Websites Reconnaissance

In today's age, we communicate a lot about our personal lives and our professional lives, and that's no different for companies as well. They post a lot of information on their website that's easily found if we're targeting a certain company or a certain organization. So let's take a look at website reconnaissance.

More than likely, we can find out a lot about our target on their website. In one of my other lessons, I showed how companies can have many different domains, and also many host records with those domains, and they could be pointed to many different IP addresses, which really point all to resources and can have vulnerabilities. But even on these websites themselves, we can find a lot of information.

When a company has a website, usually there's a lot of resources behind it, and these resources contain data. We can extract data from these web pages and web resources, and we don't necessarily need to do it manually. There are ways that we can extract data and get data out of these by doing some sort of scanning on them.

So here are some things that we can do for website reconnaissance. We can start out by website crawling, and from there we can go into URL enumeration, and then we can do website scraping.

Website crawling and URL enumeration

Website crawling, or web spidering, is systematically exploring and navigating the internet to gather information. So web crawling is automatically going to find those sites and those resources associated with those sites.

Part of what's happening is URL enumeration: finding all the URLs that are associated with a site. There's something called a uniform resource locator, or URL. A URL is a way to address a specific resource in all of the world. We have the host right here and the domain right here, so this specifies where it's going to find this data. It's going to actually do an nslookup, or it's going to look up this in DNS and figure out what the IP address is. From there, it's going to take a look at the directory — in this case, resources — and what resource it is, or file it is, which is picture2.jpeg. It also does tell us what protocol is going to be used here. And it gives us some additional extras here at the end, so we can do some sort of query or be able to address a specific fragment on the page. But essentially what we're seeing here is a URL, or a uniform resource locator. By identifying all the resources that are out there, we can better scrape this data or be able to look at data really in the whole world.

What a URL points to on the server

Let's take a little further look into this URL and what it looks like. Essentially, when you have a domain — in this case it's www.techknowdj.com — it's pointing to some sort of resource. In this case, it's a web server and it's hosting out web pages. This server right here has a bunch of directories on it, and one of those directories is housing the whole website. So what we see right here is the directory that's housing the website.

Within this server we see a /var/www/html/ and then whatever resources are part of this website right here. Well, not all of this shows up to the rest of the world. In fact, this first part, this /var/www/html, really doesn't show up to anybody else. No one else is going to see that part of it. But what they do see is this end part right here that shows up in the URL. So we can see that this first part right here doesn't show up in the URL at all. This www isn't the same thing as this www — those are two different things there. But what we do see is the end here does mimic the same thing. That's because this is just pointing to a resource on the server that's housing www.techknowdj.com. So that's how it accesses these different resources and how we address these different resources.

URL enumeration is just going through and finding all of the URLs that are associated with this website. So we see this www.techknowdj.com and then we're finding all of the resources that are associated with this site, and we call that URL enumeration.

Sitemaps and robots.txt

Ultimately, if we own a website, good chances are we want that data to be discovered. We want search engines to be able to discover it. And there are a couple of ways that we can make those search engines aware of all of the site resources that we have.

One of those is the sitemap.xml. The sitemap.xml allows different web crawlers out there — different ones like Google and Yahoo and Bing and all of these different search engines — to crawl our website and be able to discover resources on it. A similar file is robots.txt, which really tells those web crawlers what they can search, what they can find, and what they shouldn't automatically find and shouldn't scrape data from. Both these files allow us to access what the website looks like and shortcut this grabbing data and doing URL enumeration with it.

Website scraping

Once we understand what all of the URLs are, then we can do website scraping. So we do the URL enumeration, and then we search this and scrape it for data. This is that next step, website scraping. Website scraping is automatically extracting data from websites.

Some of the data that we're scraping is from hypertext markup language, or HTML. Some of these pages are just files on the server and they end with HTM or HTML, and that's because this is the code that they're written in. Here's an example of HTML. HTML has certain markup language — that's what the ML is. It marks up this document, but then also has some data that's involved in this document. So what we can do is we can start scraping and looking for this data on this document, and we call this website scraping: being able to scrape these pages for data. So we would take that URL enumeration of all the pages that we found on this site and then start scraping it for data that we're looking for.

Archives and caches

Sometimes a company realizes that they exposed some data that they didn't want exposed. Maybe one of these pages had some usernames and passwords on it, or had some sort of sensitive information, so they delete the page or they reconfigure the page. But the thing is that there are archives out there and there is caching out there, so this still could exist out there. So one of the things that we might want to do is look at historical records or archives of these pages and search that for data as well.

Platforms and plugins

A very different concept from website scraping is just technically how this web page is implemented, or how this website is implemented. A lot of times it's running on some sort of — maybe it's Apache, or some sort of WordPress, or maybe it's Drupal, or it's running on these other technologies — and those technologies could have weaknesses. Not only that, but there are certain plugins as well. So there's these different platforms, and then there's also plugins to those, and each one of these could have their own weaknesses.

So as we're looking through and scraping the data off of this, one of the things that we're going to be looking for is what these plugins are and what the vulnerabilities are with those plugins, because this might be a way into that company.