Documentation

It is kind of amazing how much goes into documentation, and how many people you will find on both sides of this. There are people who are a little anti-documentation and really just think that you should be making progress, and then people on the other side who just want to document all the time, or at least think that you should be documenting all the time. The answer is somewhere in between.

Why Document

It is so critical that we do document things. There have been many times when I have had an issue on my network and I have been trying to troubleshoot it and figure out what is going on. I take a look at a piece of equipment and I see maybe there is a high CPU on it, and I think, well, that is strange, why is there such a high CPU on it? So I will assume that that is a problem. But the thing is, I do not know. Is it always at 80%, or does it just happen to be 80% right now because of the issue that we are having? I do not know that information. What that boils down to is that I need to have some sort of records beforehand of whether that is part of how this piece of equipment operates, or whether it is unique right now, different right now, so I can identify if things have changed or not. That is one reason to document.

There are other reasons as well. For instance, if a device died and you need to replace it, how are you going to replace it if you did not know what the configurations were on that device? Or let us say somebody is out someday and there is an emergency and you need to tackle something, and there is some knowledge that that person has that no one else has. Documentation can save the day. They can step in, and you can look at the documentation, decipher what is going on and be able to fix things. Or what happens when there is maybe a security incident and we do not know how far back this security incident goes? We need documentation to know when we made certain changes, to understand what the security ramifications of whatever happened are. Another one would be, let us say we fixed an issue and then we encounter the issue again. If we did not document that fix, then we would not know how to fix it again.

I can bring up time and time again why there are so many benefits to documentation. You definitely should document, and I am going to say that most people do not do enough documentation. Probably 80 to 90% of the people out there do not do enough documentation. So you drive it home: you need to be documenting. It is very critical that you do this documentation step whenever you are rolling out a project, whenever you are fixing something, whenever you make a change.

Finding The Right Balance

But now that I say you have to document, that it is so important and there is a whole host of reasons why you document, I will also say that there are some people who want to just document like crazy, who put it as the number one priority. That is not the case either. You want your network to be up and running, and documentation kind of takes a backseat to some of these other priorities that happen as well. So you can actually get overemphasized with documentation, but I rarely see that happen. In fact, even the people that overemphasize the need for documentation and put it at such a high priority, I find that a lot of times they are the same people that do not document enough. Most of the time I run into cases where there is not enough documentation.

There are also some problems with overdocumenting. If we have so much information and we cannot find the information that we need, that is problematic, and if it is not organized. So there is a problem when we have way too many documents and maybe there is too much old information that is kind of cluttering things up. We need to expire some of that. But like I say, most of the time I find people are on the not-enough-documentation side. There is finding the right balance in between.

Smaller businesses tend to go to underdocumentation. The bigger businesses tend to go for more process-oriented, and there is a reason, because there is some security that happens with that, and there is some ability to bounce back when there are issues. So there is a reason why they gravitate towards the more documentation side.

You are going to have to adjust this for the different businesses and find what the right mix is depending on the culture of the business. Are they a move-forward-fast, fail-fast business, where you come back and fix your failures so you can move on at an even faster pace? That is more of the startup, smaller businesses. Or is it a business where they want to make sure that any changes they make are not ever going to end up with issues in production? An example like that is that we want our banks to be secure, right? So we want a slow process in a bank. We do not want to change things drastically, and so when you get into that type of security that a bank has, they veer towards more documentation. Finding the right balance is going to be important on your networks.

Common Network Documentation

So what kind of documentation should you have? I am not going to be able to cover everything here, but one of them is a network map, and there actually are a couple of different kinds of network maps.

You have one that is a physical network map of how buildings are laid out. Here we have got building A, B, C. We have got an annex over here, and how the annex gets connected through this wireless connection, and they all connect into this core switch. So we have got this campus area network, and we have a network map for it. This would be called a physical topology. This is the physical topology and how it is set up.

But there is also virtually how we have things set up, which could be different. For instance, virtually we may have this network that is for administration, a network that is for operations, a network for sales, a network for servers. How that plays out is maybe sales is both in building A and building C, there are salespeople in both those locations, and for operations there are people in building B and C. So the two maps could look very different between them. This is a physical topology and this would be a logical topology, how it logically is set up.

Network maps are one of the things I do when I get into a new business, where they have all these jacks around. The problem is that if you do not know where all of that equipment is and how it is set up, when there is an issue it makes it really hard to troubleshoot. So one of the first things I do is make sure that I understand what the network is like and start developing my network map, so that way when things go down I know how to troubleshoot it. This is a very important one to make sure you have on hand.

Another one is the baseline configurations, establishing a baseline for how things are set up. Because if I am taking a look at a piece of equipment and I see some sort of pattern with it and I think, oh, that looks suspicious, I have no idea what it looked like before. So what we do is we capture a baseline configuration. The baseline configuration is going to be a capture of what the device looks like. Maybe it is the performance, in the case of the example I am giving right here. It is a capture of the performance, so we understand how the equipment has been operating, so when something goes awry we can compare it to that and see what the differences are.

The other thing I have in here is the configurations. If something goes awry and something is different and I cannot find what changes have happened recently, I may need to pull out the old configuration file of how that looked and compare it to the new one to see if something has changed recently. So that is another part of this baseline configuration: I would want to capture the configuration of my different devices. I even have a method for when I go and make changes to my different devices, so that I capture a new baseline every time I make changes to that device.

Documentation Guidelines

I have created some documentation guidelines based off of my experiences in the past, and you are not going to see this on any test or anything like that. I have just come up with some guidelines for when I am creating documentation, for how to make it more manageable.

First of all, one thing you have to do is keep it organized, because if it is disorganized and in different spots, when something goes awry, being able to find that documentation could be hard. So make sure that you are organized. One of the places I stepped in, which was struggling at first, had documentation in all different spots, and so what we did is we organized it and put it all in one spot.

That brings us to the next one: limit the location of documentation. There are lots of different ways to store documents, in lots of different formats, in lots of different locations that we can put it in, and different technologies that we can put it in. If we are using all of these different types of storage, then that can be problematic as well. How do we access this, and once again, where do we find it?

I do end up in quite a few locations, just because I want passwords in something that is secure, that is going to store it encrypted and not going to give access to everyone. And then there are some documents I want a little bit more dynamic, so I want it on something like Google Docs or Microsoft Office 365, something that is going to be more dynamic so everybody can take a look and see what is adjusting in it. And then a lot of my documentation is on a wiki spaces, Confluence type of setup, where you get on and you create documents of your workflow or documents of the different systems. So I do end up putting it in multiple locations, but I try to limit that as much as possible, as much as it will allow me to limit it.

Make sure that you have access to that documentation in the case of an outage. There are times when I have had documentation on the hardware, where maybe it is on the virtual hardware, maybe I have got some virtual hardware that is operating in this virtual machine cluster, and then I lose access to it, and then how do you troubleshoot? So however you are keeping this data, make sure it is accessible if there is a network outage, and consider maybe having two locations for it.

This is a big one for me: do not have the same information in multiple forms, multiple documents, multiple locations. Just try to keep one central place for that information. An example of this is that there are times when there are issues at work and there is a report, a template, that is filled out for those issues. Well, that is not always good for all of the people within the business. What happens is that this part of the business wants the report done in this format, and this part of the business in this format, and this part of the business in this format, and then you end up with four different copies of the same piece of documentation to meet the needs of each one of these locations. That generates a lot of extra work and it also creates confusion. And then when things get updated, they do not all get updated. They are going to get off. Documentation gets off, and that is one reason why you perform audits, because you want to make sure your documentation is accurate. But if you have got four different locations, all four of those locations is going to end up having different information. It just happens again and again and again. So for that purpose I say, well, let us just create one form, and then within that form somehow we have got to call out different sections to meet the needs of these different people. I am pretty adamant about that, about not having multiple copies of that same information.

And then I have on here back up that documentation. Make sure you have a backup of that documentation, and once again make sure that it is accessible in the event of an outage.

Steps To Documentation

The steps to documentation are pretty easy. Determine the type of data that you need to store, whether it is those baseline configurations, or backup copies of things, or maybe it is the network diagram or a system diagram. Somehow determine the types of data that you want to collect.

Identify the devices that you are going to collect that information on. What devices do you need baseline configurations for? What systems do you need all mapped out?

Then establish durations. In this case right here, for instance, if you are doing a baseline configuration for performance, there are going to be durations for that. Are you going to perform the duration over a period of a couple of hours, a couple of days, a couple of months, a couple of years? Probably not a couple of years, but there could be different durations that you are going to establish to figure out what the baseline is. And then you will have to figure out how to refresh that, when to update those baseline configurations.

Make sure you go through and actually take a conscious effort of identifying what type of documentation you are going to have, where it is going to be located at, and identify the devices, the durations, and when to update it. Think through all of these.

And then one thing that has to happen is everybody on the team has to be on the same page, because everybody has to have a common understanding of how to access this documentation and how to record the specific information. I will go through with my teams and make sure that they understand what their expectation is and where to find that documentation when they are onboarding, and then even on a yearly basis, to make sure we are all on the same page.

Do not overlook documentation. It is certainly not the top priority, but what happens because it is not the top priority is that a lot of times it gets put on the back burner, and you should definitely not do that.