Planning Considerations

One of the great things about a change management process is that you start thinking more in depth about what it is that you're doing, how you're going to do it, and some considerations around planning — and you do that well before you actually get into implementation.

Service impacts

A lot of times when we're making changes to systems we're going to restart a service or restart a server, and we've got to start thinking about what the impact is going to be. So we're going to think about service impacts, and we start thinking about how everything is interrelated: making one change, how is it going to affect everything else?

We take a look at the downtime and service restrictions. We take a look at whether we're going to need to do any service restarts or application restarts, or any kind of dependency with other software, with other hardware, with other systems. We take a look and see if these are legacy systems, if it's going to affect any legacy systems. We take a look at maintenance windows and allow and deny lists. So we start looking at all these different aspects when it comes to planning out our implementation.

Testing and environments

We also start thinking about what we are going to do during the testing phase, so we plan out how this is going to be tested afterwards. This has a couple of advantages. Number one, we can test the results afterward to make sure that whatever change we made was made successfully, and made to what we were trying to impact, what we were trying to do. The other advantage is that it helps us think through all of those other dependencies and service restarts and everything else, as we start thinking about what the end result is going to be from the change we're implementing.

We may also have some environments that are available to us. For instance, we have a production environment. This is where our users are actually connecting — maybe their customers, maybe their employees, whatever type of system this is, whether it's customer facing or internal. We have users that are logging onto this production system, but we don't want to make changes to this and bring the system down or cause problems to this system, so we test it out first.

We might have a staging environment. A staging environment is an environment that we can test changes out in first, before we roll it out to production. And we might not just have one — we might have a stage environment, a test environment, a dev environment — and we can roll it out to each one of these.

It does two things for us. It tests the actual change that we make: maybe the change is that we're adding software, we're installing software, so it tests out the software itself, and whether the software is going to be successful running on the server. But it also tests our process for rolling out the software. Maybe the software runs just fine on this server right here, but maybe what we realize during the rollout process is that we had to restart the server and we didn't realize that before, and now when we go to production we know that we need to schedule it during a maintenance window.

Security implications

We're also going to think about security implications and impacts. That's where we're going to think about allow and deny lists. Allow and deny lists either allow maybe certain ports, maybe certain software — they allow something or deny something. Once again we need to think about legacy applications, because those legacy applications can be vulnerable to certain things, so we've got to think about that. It's also part of the change control process to always evaluate against our policies to make sure that our policies are being enforced and we're following them.

Who is impacted, and timing

I like to think of who will be impacted — not only who will be impacted during this rollout process, but also, let's say we expect it to not have any downtime, in the slight chance that there is downtime, who would that impact? If it is going to impact somebody, if there happens to be downtime, I still notify them. I let them know, I communicate to them to let them know that there could be some possible downtime.

Timing in some of these rollouts can be critical, making sure that we don't do it during busy times. I was part of a system where we'd have millions of users on it every year, and what we wanted to do is make sure that we didn't have downtime during our busy time, so we'd schedule it during a time when people were off the system.

There are times when we need to do it maybe midday, or maybe during the busy time, and it's best if we can schedule downtime to do that. That is when we say that there's going to be a specific time that we're going to bring things down. Or maybe it's even just on the weekend — if we think that there's going to be people on this at the time that we need to do the maintenance, we should schedule a downtime and do the proper notification.

When we're scheduling downtime we need to think about our service level agreements. Those service level agreements a lot of times allow for scheduled downtime, but only during certain times of the day. So if we're scheduling downtime, we just need to think about those SLAs and when it is the most appropriate time to have a downtime.

Maintenance windows and communication

This is where maintenance windows can be handy. One of the things that I like to do is have an ongoing maintenance window. For instance, one of the places I worked for was a SaaS company. We had our maintenance window every Thursday, after all of our customers were essentially off of the system, so it would be a Thursday evening that we would have as our maintenance window, and we could do maintenance on the system, we could roll out new software, we could make all the changes that we wanted to make.

I can tell you that a lot of success around these changes is going to be around communication, because there are times when things do just go down that you weren't expecting to go down during these maintenance times. If I've communicated ahead of time, there's a lot more forgiveness. People will still reach out sometimes, but they'll say, "Oh yeah, that's right, we're in maintenance, they did notify me of that." However, if you don't notify people ahead of time, communicate ahead of time that there could be some downtime, or if there is downtime, then they get to the service expecting it to be up and running and then it's not. It can be very problematic and you'll have a lot of angry users that are trying to use the system.

Backout plans

Another thing that can save you a lot of pain is a backout plan. There are times that we implement a change and we roll out a change, and then once we get it out there it causes all sorts of problems. If we thought ahead of time about a backout plan — maybe we took a snapshot, maybe we did an extra backup, maybe we saved the configurations, maybe we did something that allows us to fall back to an earlier version, an earlier state, much faster, much easier — it could save us a lot of time. I can tell you these backout plans have been a lifesaver for me in the past.

Version control

This might be a good time to talk about version control. Version control is this idea that every time we make a change, and especially around configurations, we're creating a version and we're tracking those versions as it goes. Now we can see the details of the change and go back and say, what does version five have?

We also have some accountability, because we can see what has happened along the way, and usually the person that's made these changes is tracked with these different versions. It also gives us a historical record of everything that we see, and it creates that backout plan of how we roll back. So versioning control is a great way to manage the different changes that we make, especially when it comes to versioning control around our configurations.