Monitoring & Troubleshooting Software

Troubleshooting Software

Beyond the command line tools already covered, there is other software that can help you troubleshoot, along with monitoring services you can run on your network. Some of the software that can help in the troubleshooting process would be protocol analyzers, bandwidth speed testers, port scanners, iPerf, TFTP servers, and terminal emulators, and there are several more that can help out as well. These are the ones we'll cover right now.

A protocol analyzer is something like Wireshark. So we use Wireshark, and you can see how it can capture the data and we can dig in to see what's happening with the data. We can also mirror some of these ports, so if we want to know what traffic is going between here, we can capture some of that data with it and then really drill down and see what's going on.

Then there is the bandwidth speed tester. Many times there are bandwidth issues, and so perhaps you just go to Google and type in speed test, or you go to Ookla and do a speed test from there, and be able to test out to see what the bandwidth is doing and whether you're struggling to get the bandwidth that you're paying for.

A port scanner is software that can scan other machines. So I could take this machine and scan ports on this other machine to see what ports are open on the machine. That can also help determine if there are maybe ports that shouldn't be open on the machine, or should be open. So you can get some information from there.

iPerf is some software that you can set up. It has both a client and a server side, so I can set up a client on one network and maybe a server on the other network, and then measure the bandwidth and traffic in between those two. It'll set up some streaming between these two devices to measure what kind of throughput you're getting.

Then you have the TFTP server. The TFTP server is similar to an FTP server, where you can store data on it. Perhaps you're uploading logs to the TFTP server, or configuration files. So that can be used to help troubleshoot these networks.

And then you also have the terminal emulator. Let's say you're having some issues with a slow network and you jump onto this switch using a terminal emulator. What that is is it's simulating as if you're on that device, and you can enter commands into that device and see what information is being sent back. A lot of times it will show you debug information on there. So you can start troubleshooting, or interfaces are flapping up and down, or you can view the configurations again. There are different things you can do to help troubleshoot on that piece of equipment using a terminal emulator.

Then you have the Wi-Fi analyzer. If you're here in a building, a Wi-Fi analyzer lets you analyze what kind of wireless signals you're getting there, which channels are being used, which bandwidths are being used, and which devices are sending that out. So if you've got access points around your building, what are the MAC addresses of those access points? It can help really tune your wireless network and figure out what's going wrong in your wireless network.

Types of Monitoring

There are many different types of monitoring, different ways that we can monitor this traffic. An example is ICMP, service monitoring, SNMP, event logs, NetFlow, packet capturing, and analytics.

ICMP is like ping. We talked about ping and pinging things, and there's some software where that's what it does: it will ping a device, and when it goes down it can do some sort of notification and let you know. It's one of the great ways of monitoring your network, where you have a service that's monitoring your network and it just continually pings your equipment to see if it's even up. I have network monitoring tools that I have going all the time, and it will notify me, give me a text message or give me a call if something goes down. So that's the MP.

You can leverage ICMP or some sort of service monitoring. It doesn't have to just be ping. It could be that if there's a web server and it's HTTP, you could have a service that's monitoring that HTTP service, or FTP, or whatever service it is. It just continues to ask, do you still have this web page available? Do you still have this web page available? Or maybe it's not just a simple do you have this web page available. Maybe it's also monitoring how fast it responds, or looking in the web page for certain text inside the web page. One of our service monitors that I have will display a web page that will say unhealthy if something is going awry on the back end. So my service is just monitoring it, and as long as it says healthy it's fine. As soon as it turns to say unhealthy, then it will notify me and send me a message back.

Then you have SNMP, simple network management protocol. This is a protocol that can be used to gather information from your equipment. So maybe you have an SNMP server that's monitoring your devices here, and then it goes and requests information or pulls information, or this switch will send information to this device, and then you can start seeing what's happening on that device. There are monitoring tools that will monitor this as well. In fact, one of my tools will monitor the ping, the HTTP, it'll do SNMP, and the information it collects with SNMP is like: what is the CPU doing? What are the hard drives doing? What is the RAM doing? What is the other memory on the machine doing? Are there hard drives that are filling up? So it'll monitor lots and lots of different aspects of the machine, and it does that through these SNMP messages.

There are event logs. Your servers will have logs for the services that they have, and when something goes awry you can go and check those event logs to see what is happening on those machines. It will give you some additional information in these event logs that you can go and look up. A lot of times you'll have an event log per service that you have running on that machine, or there are some system-wide event logs that will be monitoring. And you can correlate between these different event logs to see what's happening and give you a full picture of what's happening on that machine.

Then you have NetFlow. NetFlow will monitor the traffic going across the device. So for instance, maybe we have a firewall or a router here and we have NetFlow turned on. Then that will deliver information. Maybe we have a laptop that it's being delivered to, and then we can monitor what's happening going across that connection right there. Maybe we're monitoring for some sort of security issue, or for some sort of performance issues that we're having.

Packet capturing, like we mentioned before, is like Wireshark. You can use Wireshark to capture these packets and see what's going on with that.

And then you have analytics. A lot of times you'll install things like this on web pages. If you have a web server, then you will probably want to see what the analytics are on that, so you install different programs. Google Analytics is an example of this, and it will measure things like this: you put code in your web page so that when customers reach that web page, it will go into this code and give that information to Google, saying this is how long that information took to download, I am now connected to this service, I am having this kind of experience with this service. So it'll report certain things back to Google Analytics, and then you jump onto Google Analytics and see how the performance of your web browser is doing. There are several different types of analytic programs out there and they all function a little differently, but that's one example of how you can monitor what the customer is actually experiencing, because a lot of these don't necessarily emulate the customer experience. These analytic programs will actually report back on how fast the customer downloaded a page, how fast it's working, and what pages they visited, and give you all sorts of great information.

Monitoring Services

There are a lot of different types of data we could collect. We could collect ping and HTTP, and we mentioned that there are logs on different machines. The problem is that there is a lot of this different information out there that we can collect for troubleshooting, but if we've got to go to every single piece of equipment for that troubleshooting, that could be problematic. So there are some ways that we could work this to actually get better information.

We could set up a syslog server, and then we would get logs, we would get information being sent to the syslog server, so that way we could gather information. There are also log aggregation servers and services like Splunk that I've used. What that would be is I'd set up a server here, and then all of the logs of the machines would connect into it and deposit, or this machine would go reach out and grab the logs and put it all in one spot. The advantage of that is now you can start doing searches and seeing what all of your logs are doing when you have a problem or some sort of issue. You can see maybe it's a cascade of issues: this happens, and then this happens, then this happens. A log aggregator will pull that all into one spot, and so you can analyze that.

I always like to have some sort of monitoring service out in the cloud. If I've got a web server right here, that service is monitoring this server right here. Maybe it's just checking that web page once again, maybe it's checking something specific, some sort of word on there or some sort of button or something specifically on that. And so that way it will notify me if things go down, and it tests from all around the world. So I get reports from everywhere, all around the world, letting me know if my service is going down, and if it does go down, whether it's just in certain countries that it goes down or if it's in this country. Some sort of outside monitoring service can be really helpful, so you get notifications faster than they come from the customer. Or perhaps it's at night and your customers aren't online at night, and you can actually fix it before they are online. So it can be really helpful to have these services set up.

Some of the services I use do real-time monitoring, where they have something embedded into the web page that will report back to the service, similar to the Google Analytics that I explained before.

There are a lot of different types of monitoring services out there. Whether it's a server you can set up on your network that will collect this data, or a service up in the cloud, I've even seen a combination of the two, where part of the service is local and part of the service is in the cloud, and then you log into it and it will aggregate all of that information so you have it all pulled together. But if you are running enterprise networks, then you're going to need some sort of monitoring services to make sure that you are up and running and that you're not having issues.