Troubleshooting Steps

When things go wrong

One of the challenges of this career is that things do go wrong, and when things go wrong, lots of people notice. When you're dealing with networks, people rely on those networks, and when they go down, they can't do business anymore. Most businesses nowadays have some sort of component where they need to get online and work online to communicate with customers and make sales and do all sorts of other business-related items. So when the network goes down, there's a lot of people knocking on your door.

The problem is that I've seen a lot of employees — and I'm guilty of this myself — when things go down, there's a bit of a panic, and you start changing different things on the network trying to figure out what change will make it come back up again. This can be very chaotic and cause a lot of problems. So we need to know what steps we can go through so that we don't take this chaotic approach. Number one, we could possibly cause more damage to the network and cause problems on the network. Number two, how do we do it in such a way that we can get that network up and running quickly? So let's talk about some troubleshooting steps that you can use to make sure you're successful when you're troubleshooting for the labs you're doing for this curriculum, or even in the future.

The troubleshooting process

One of the first things we're going to talk about is the troubleshooting process. We'll talk about the steps that you will go through to troubleshoot issues from a very high level, and then we're going to get into each one of those issues and go a little more in depth. Those steps are to define a problem, to do some sort of information gathering, and then we will establish a theory. Once we've established a theory, we will test that theory. We will create a plan. We will implement that plan. We will verify that it's up and running, and then we will finish off by documentation.

There are times that we go through this process and we realize we actually haven't fixed the core issue yet, and we'll have to go back to the drawing board and establish a new theory. So there are times that we'll have to repeat some of these steps again. In fact, when you get in there with troubleshooting, it gets a little messy. Which is why, at the end of your troubleshooting, if you haven't done it correctly, then you have tried all these different things and you're not sure what you've tried and what you haven't tried, and it can cause a lot of problems.

Caveman or professional

The first question I would ask you is, what is your approach right now? When there is something that happens, what steps do you go through to troubleshoot that issue? You can apply this to networks and technology, or you can apply it to other things outside of your career and technology. How do you approach things? Do you approach it in — well, I've got a caveman here — do you approach it in kind of a caveman fashion where you just try a bunch of different things to get over it? Or do you take a very logical, professional approach to troubleshooting?

The problem is that when you go down this route of taking the caveman approach of just tackling the issue, you could be making a lot of changes. Make a change. Does this work? No, it doesn't. Go back and make another change. Does it work? No, it doesn't. Make another change. Does it work? No, it doesn't. And if you just keep trying this, what happens is now you've made a lot of different changes. You could have broken something else while you made all these changes. So now, not only are you dealing with the core issue that caused the problem to begin with, but you're also dealing with some other issue — you broke the network in some other way as well. So we need to take a more logical approach to make sure that we're not creating more problems, more compound issues.

And then also, when we get to the end and maybe we have fixed the issue but we're not familiar with what changes we made along the way, that can become problematic too, because now we've got a configuration out there that we don't know if it's secure or not secure, or if we've caused other problems. So that can be really problematic also.

We all tend to probably start out more towards that caveman side of the spectrum. This is a spectrum here, and we all kind of start out towards there, and as we go along and we become more educated, we work towards this more logical approach. This will allow us, really in the long run, to be able to execute things more quickly and get that network up and running more quickly, with less issues once it is back up and running. So we want to make a transition from this caveman approach to a more professional approach.

CompTIA and Cisco

So then the question is, how do we approach this logically, and what steps do we take? The answer is actually different depending on what you're studying. There's the CompTIA Network+ certification, which has one set of protocols that you go through for actually troubleshooting an issue, and then you've got the Cisco method of doing it. But there really is a correlation between these two.

With CompTIA, you identify the problem. With Cisco, you define the problem, gather information, analyze the information, and eliminate possibilities. So in both cases you're really trying to boil it down, or render it down, to find what is the real problem here.

With CompTIA you establish a theory. With Cisco you propose some sort of hypothesis. But the idea with it is you just develop: what could the issues be? Then you go into test theory, or test hypothesis, and this is to test it to see if what you think it is is the problem or not. In some cases with this, when you test the hypothesis, when you actually test the theory, when you actually implement, you make the change and it fixes the issue and the issue is resolved. Although there are times when you implement a change and it's just a band-aid. It's just to temporarily fix the issue, and it buys you enough time to go back and really fix the core issue.

Then you get into establish a plan of action, implement the solution, and verify functionality — or in the Cisco realm, it's solve the problem. And then within the Cisco realm, you'll also notice that there's a whole matrix within here that you jump around from spot to spot as you troubleshoot. It's actually got a diagram that kind of shows you how you jump around in there, and we'll talk a little bit about that within here. And then finally with the CompTIA, we see document, which I believe is a very important step as well. So whether you're troubleshooting Cisco equipment or troubleshooting other equipment, whatever the case may be, you're going to want to document what changes you have, and possibly even document changes as you go along. As you make your changes and you test all of this out, you'll probably want to do some documentation there.

So just realize that there are some different steps that you can go through, but we're going to go through this and figure out what exactly the differences are. We're going to talk about these steps from a higher level than this.

Define the problem

The first step in all of this is to define the problem. Now, this seems like pretty basic, and sometimes with networking it is. Sometimes in networking you get woken up in the middle of the night because a notification system came in and let you know there is an issue going on on your network. But other times it's a user running into your office and saying, hey, there's a problem and you need to fix this. So really, finding the problem might be really simplistic. If the network goes down, you know immediately what the problem is. But there are times when I will sit in a meeting and people are discussing what issues are happening and it's not really well defined. And so this is an important step.

Also realize that as you're doing this, what you're doing is you're identifying the symptoms that need to be solved. So you're not really identifying the root cause yet, but you're identifying some of the symptoms that you see that you want changed. That's ultimately what you are identifying: what is wrong that you want changed, to either go back to the way it was or change to have a different result. So you define that problem.

A few notes about this. Like I say, you could be looking at the tip of the iceberg. There could be a lot more that's going wrong than you realize. It could be compound issues, which means there's multiple issues that are playing into this. And I would say always question the obvious. A lot of times what I'll do is I'll get notification that maybe there's a CPU spike, or maybe there's some sort of issue with a hard drive filling up. And when I hear those things, that could be the problem, or it could actually be resulting from something else. We'll get more into that as we go through these steps. But do spend a little bit of time defining the problem if it needs that time and attention. Like I say, sometimes it's going to be obvious.

Information gathering

The next step in this process is the information gathering. You're going to gather information on what is happening on your network, and so you're going to continue to dig in. In fact, this is not just a step that you're going to do and then it's going to end. You're going to continue to do this throughout this process. You're going to continue to gather this information.

So what are some of the things that you need to know? You need to know the who, what, when, where. So start digging in. Ask users what's going on. Users might be the ones that are reporting it. You need to ask which users are reporting it, which users are affected. Is it one user? It might not be that urgent if it's one user, but it might be the CEO and it might be urgent. Or it might be all your employees, and that's going to have a certain amount of urgency. If it is your end users, your customers, that are being affected, then that could have a huge impact. So figure out who is involved in all this.

Figure out when they are being impacted. Is it all the time? Is it just some of the time? Is it going up and down? Those intermittent issues can be really problematic to try to troubleshoot, because it goes down, you have all sorts of reports and all sorts of problems, and then when you start digging in it goes back up and then you're like, okay, well, now it's up, so how do you troubleshoot? And there are ways you can still troubleshoot, but that can be problematic.

You ask yourself what devices this is occurring on and what exactly is happening. What has changed? Is there something that's changed? More often than not, I find that there is an issue on your network, or there's an issue that's happening, because somebody implemented the wrong change. So they've gone out and they've made a change, and that change brought the system down. So look back just before you made the change — maybe look back minutes, maybe look back days, maybe look back a couple of weeks. Sometimes changes will be delayed before they fully take effect.

So just realize that is going to be important too: assessing what and when, and then where. Where are you having issues, and where is it being reported from? Look for information from your different systems, from your notification systems that are out there. Look for information on all your systems to figure this out. You're going to look at logs and monitors, and we'll talk more about logs and monitors. So look at all of these sources of information to figure out, okay, now I'm gathering all of this and now I can take a look and see the big picture. In fact, there's systems out there that do exactly that: gather logs from all of these systems and put it into one system so you can more easily troubleshoot these issues.

Establish a theory

Once you've gathered that information, you're going to have to start developing a theory. Establish some sort of working theory of what's going on here. It might be obvious — like I say, a change might have happened and then three seconds later the network went down, and it might be pretty obvious, okay, this is what's going on. But like I say, sometimes it'll be a while for a change to take effect, and it could be even weeks or maybe even months that you notice this. So establish some sort of theory of what's going on. And even in this process, this overlaps with that information gathering. You're going to continue to gather information as you can and establish some sort of theory at the same time.

Test the theory

The next step is to test your theory. A lot of times testing the theory is actually creating the solution. For instance, let's say there's some sort of configuration change that happened. Maybe a user reset their password, and then right after that a service went down. Perhaps that user is tied to that service, and so now you have a theory that we need to change that service to use a different user. So you go in there and you test the theory by changing that user to have a different user for that service, and then perhaps that fixes that issue.

So you're going to test whatever theory that is, and that may be the permanent fix, or it may be something that's just leading to what the permanent fix is going to be. Whatever the case may be, make sure you document this. One of the issues I've seen is, once again, people will test a bunch of stuff out. So if you change the user on that service and then it doesn't fix the issue, you're going to want to go back and change that later on. So you're going to want to just document and make sure that you know what you did so you can go and reverse it later.

Create a plan, implement, and verify

Next is to create a plan. Create your action plan of what steps you're going to do in order to fix the issue. Once you've determined this is a working theory, we've tested it, that sure seems like what the issue is, then you create a plan. As I mentioned before, a lot of times what my team will do and what I'll do is, once we discover something, we will fix it, and then that is a temporary fix, and then we'll create a plan to put a long-term solution into place.

Once we've established that plan of action, then we have to take action. So take action by implementing the change. Put it into place and make your users happy by fixing the problem. And then finally we verify that it has in fact fixed the issue. We test it. Whatever issue we had before, we've gone through the process and done the same thing, and now it's working as designed. So we verify that the fix is in place.

I mentioned this a few times: when you go through here, taking these steps one by one will help you in making sure that you are logical in how you approach this. In reality, when you actually get rolling on this, what's going to happen is you're going to gather some information and you realize, okay, well now I've got a theory already from this information, some sort of educated guess on what's going on. You're going to jump to testing that theory out, and maybe you're fixed at that point in time. So what you do is you go back and forth through these until you get it solved.

But I think what really helps when you visualize this process is that there is usually not enough time spent in this information gathering. Way too many people jump to this theory part right here and get into testing the theory out when it really wasn't a solid theory to begin with. So that's really what this system helps you to do: not jump ahead and cause more problems, but instead take a more logical approach to it. But you will have some repeat as you go through this and find that you get to the verification stage and it didn't work. You've got to go back to the drawing board, and maybe there's more information that you have now to create a new theory. And so quite often that's the case: whatever information you gathered from that test, and when you go to verify it, then it brings you back to that information gathering — you have a new piece of information now.

Document your findings

And then an important step that you don't want to miss out on is document what you find. There are times that there will be some sort of issue and my team will be talking about it and we'll say, well, we've encountered this before, let's go back and look at our records, and then we find that we didn't record enough information to really help us out. So record what goes on and document what goes on, so that way next time you can troubleshoot and get over this much quicker and much faster. So document your findings.

Recap

So there you go. What we did is we talked about the troubleshooting process, which is define a problem, do some information gathering, establish a theory, test the theory, create a plan, implement, and verify. Once you verify it, then document. And then within here, as we mentioned, you may need to jump around a little bit with this. But the big key to this is don't make a lot of changes without the proper information, and take a logical approach to making the right changes, so that way you can troubleshoot this efficiently and quickly and not end up with more problems in the end.