Change Management

Change Is the Riskiest Thing We Do

In all my years of experience in IT, both as a technician and as a manager, I find that the riskiest thing that we can do in a company, in an organization, is make changes. When people make changes, that's when we introduce issues into our network — whether we bring a system down, or maybe we make a wrong change which creates an exposure of information, which could lead to a confidentiality breach. Something has gone wrong in our network because of a change that we made, more often than not.

So what we need to do is somehow mitigate that risk, lessen the risk in making changes, and the way we can do that is a change control process.

Changes Are Behind Most Problems

When I have an issue on my network — let's say our systems go down — one of the first things that I do is start thinking about what changes have we made recently that would have affected this. And almost always I find some sort of change that we made that led to this downtime.

Same thing with vulnerabilities. If there's some sort of vulnerability that's found on our network, like maybe it's a permission issue, one thing that I do is go back and look: when did we make this change, when did we implement this wrong? And a lot of times we implement it correctly at one point, but somewhere along the lines we changed the implementation and now it's opened up a vulnerability. So really we have problems that exist on our network, and a lot of them exist because we made the wrong change to the network.

Standard Operating Procedures

One of the things that helps with this is if we have procedures, if we have standard operating procedures. Standard operating procedures are those procedures that we follow to carry out our day-to-day functions, for us to carry out some sort of process to make something happen. We call them standard operating procedures because they're the standard that we're operating by, that we're carrying out; it's a procedure that we're carrying out. The better that we can document these procedures, the better off that we can follow these procedures and become more efficient and accurate with what we do.

Now, one of the problems is, as we are making certain changes to the network, there is no standard operation that we have. That is, when we start tackling something or doing something, it's unique. We've never done it before, we're implementing something new, it's software we've never rolled out before, it's a piece of hardware that we're installing for the first time. So there's no standard operating procedure for that.

Well, actually there kind of is. There is one that we can implement, and that is change management. If we have proper change management, we can better think through this process of rolling out that piece of equipment, of rolling out that hardware or software, of rolling that out in a way that's going to set us up for the best way for success.

Request, Ownership and Planning

Change management starts out with a request. There's a change request that's made. Perhaps it's outside of the department — somebody made a request to roll out some new software for the company. Or maybe it's within the department and we determined that we need a new piece of hardware set up, and so we start a change request form. But it all starts out with a change request.

From there, there needs to be an ownership of that change. It gets assigned to somebody to carry out the functions that are needed to make this change happen.

Then we go into the planning phase. The planning phase is this idea that we are going to plan how this change is going to be rolled out in the best, most efficient, most accurate, most secure way of rolling this out. So we think about our goals. We think about communication and what communication looks like: who do we communicate, how do we communicate, when do we communicate. We think about the process of rolling this out and what that process looks like, to eliminate things like downtime, or eliminate the effect that downtime might have on everyone. We take a look at measurements of what we're going to measure during this process. And then we even create a backout plan, that if this change is not successful, how do we reverse this change.

Evaluation and the Change Control Board

Then we get into the evaluate phase. In the evaluate phase we take a look at the plan, we take a look at the process, we take a look at the communication plan, we take a look at all the details to make sure it is aligned with our policies and that there are no errors with it.

Sometimes this might be just a self-check, a self-evaluation of all of this. But the problem with that is, if we have created this plan and we're also checking it, then that causes a problem, because what we're doing is we're checking our own work, and we have already determined that it's a good plan, otherwise we wouldn't have created it. And we might overlook things that, if we had somebody else take a look at it, they would discover.

So sometimes we go through a change control board, or a CAB. A change control board will take a look at the changes to make sure that it falls in line with the policies and the process is good, that it's not going to cause any issues, and they will then accept or reject this plan.

Implementation, Monitoring and Closure

Once we've accepted this plan, the next thing we're going to do is implement the plan. Now, this could be a really simple process, because we've already created the plan and now we just need to follow it. So really, creating this plan right here can really help with this implementation process.

Of course, once we've made that change, then we take a look and we monitor this: have we caused any adverse effects, have we achieved the goal that we wanted to achieve?

Then we wrap things up with a change closure, where we update any kind of documentation, communicate any kind of changes that we had made to those who need to be communicated, close any associated tickets, and we close out all the paperwork and everything with this.

During this whole time we're documenting and communicating, and this allows us to be really transparent in what we're doing and be able to recall everything. If we need to recall anything, we've got the documentation to go back and take a look at. We can reuse that same process if we have to redo something similar in the future, make a similar change in the future. So this all really leads to a lot of success for this rollout.