Website Reconnaissance Step Sorter

About this interactive

The three steps run in order, and each answers a different question. Crawling asks what pages are there: a spider follows links from page to page to discover the site's structure, finding pages you had not seen yet. URL enumeration asks what are all the addresses: it lists every page as an addressable URL, and sitemap.xml and robots.txt hand that list over because the site published them for search engines. Scraping asks what data is on them: it pulls the values out of the HTML and reads archived or cached copies of pages since removed. Ask which question the action answers — discover, list, or extract — not which tool it names.

What you'll learn

Place each website reconnaissance action in its step: crawling to discover pages, URL enumeration to list every address, and scraping to extract the data
Explain the role of sitemap.xml, robots.txt, and archived or cached pages in revealing a site's structure

Aligned to

CompTIA PenTest+

2.1 Given a scenario, apply information gathering techniques.

CompTIA Security+

2.2 Explain common threat vectors and attack surfaces.

Key terms

Website Scraping: The automated extraction of data from web pages, typically by parsing HTML content discovered through URL enumeration.
Web Crawling: The automated, systematic exploration of websites to discover and index pages and associated resources, also known as web spidering.
URL Enumeration: The process of identifying and cataloging all uniform resource locators associated with a target website to map its available resources.
Uniform Resource Locator: A standardized address used to identify and locate specific resources on the web, consisting of components such as schema, host, directory, and file path.
robots.txt: A text file placed on a web server that instructs web crawlers which pages or directories they are permitted or prohibited from accessing.
sitemap.xml: An XML file that lists a website's URLs to help search engine crawlers discover and index all available site resources.

Topics

Interactive Categorize

About TechKnowSurge

TechKnowSurge builds IT and cybersecurity professionals through hands-on, concept-first training built around real understanding — not memorization. Free interactive tools, structured programs, and 25+ years of real-world experience, all in one place.

Explore free tools and programs →