Detecting, Declaring, and Escalating

Detecting and declaring an incident perhaps is one of the hardest parts about an incident, or at least sometimes when you're defining what an incident is, it can get really kind of murky.

The earlier that we can detect an incident and declare an incident, it really helps move up the timeline into resolving the issue and limiting how much impact it has. But this is not an easy step.

Detection

When it comes to detection, we can detect things fairly early on. Let's give an example of maybe a system is not performing well and it's going offline. It's going to start out as functioning as designed, and then at some point in time we may start seeing some issues arise on it, so there's signs of malfunction. Maybe we even start getting some system alerts. Eventually that system goes offline, and then customers start reporting in. But where in this process do we declare this is an incident and start getting into our incident response mode?

Well, one thing that helps with that is our monitoring. So we want to monitor for things like performance trends, security, anomalies. We want to monitor over all of this stuff to understand what our systems are doing and understand when things are going awry.

Declaring too early

But declaring things too early or too often can be problematic as well. Let's say we're always looking for these signs of malfunction and we declare an incident right away. Well, there's a lot that goes into incident response and a lot of people get notified, so getting everybody on board right away could be problematic. The problem with declaring too soon is that it could trigger a bunch of events, a bunch of events that could cause unnecessary panic and unnecessary work. You can always undeclare that there was an incident, but still it causes some panic when you set off these incidents and say this is an incident, we're going into incident response. And also it could be kind of like the person who cried wolf: you could declare it too much, and that causes problems as well.

Now that doesn't mean that we don't respond to something. That is, we are going to start looking into things as soon as we can to see what's going on. But before we hit the button to say panic everyone, everyone get involved into troubleshooting, before we do that, there's some steps that we want to take before we declare this is an incident.

Verifying a true positive

One of those is that we need to verify that this actually is an issue, that something is coming up. And so we are going to want to take a look at what state this is. Is it a true positive, true negative, false positive or false negative? When it comes to alerting, if our systems are alerting off of something, it could be a positive, which means that it is saying that there's something wrong, something is going on; or a negative, saying that nothing is going on, it's not reporting anything. And true means that it's accurate, and false means it's not.

What we want to do is we want true positives: if there's an incident that's occurring, then we need to respond to it. So we received an alert because it's positive, and also that alert is valid. And so that's the first step, that what we want to do is make sure that it is a true positive. So we need to go through some sort of validation to make sure that it is a true positive.

Who can call an incident

Not only do we need to validate that it's a true positive, but that it does need to require an official incident response, that we need to go into incident response mode and start following those processes. You don't necessarily want just anybody within your organization to say there's an incident, you need to drop everything and start researching it, start looking into it. And maybe you do want that. Maybe it's a small organization and that's fine. But the problem is that I've had a lot of users that, at the slightest drop, maybe they can't get to the site, and they say, "Oh, this is an incident. This is an issue. Drop everything. Research into it. Look into it." And it happens to be their internet connection. They weren't able to get to the site because their internet was down. And so I've had issues like that before.

So you want to determine who can actually call an incident. Who in the company would say there is an incident happening, I've looked into it, and now what we need to do is we need to declare that this is an incident? Common people who are able to call an incident within the company might be the IT department, maybe it's the security department, customer support, or maybe there are some other departments. Often these other departments might contact customer support or contact the security department or contact the IT department. They do the initial research into this, making sure that it is valid and that it meets certain guidelines, and then declare the incident.

Guidelines and thresholds

And we do want to set up guidelines on what constitutes as being an incident. Is it just one user calls in and complains, well, this feature is not working correctly? Do we want to call an incident right away? That would be the extreme, where probably not, for most cases. But let's say we had 10 users call in about a feature that's being down. Or maybe it's a whole service that's being down. Maybe it's the whole website that's down. Maybe then we look into it. Is 10 enough? I don't know. Maybe you only have 20 users on the site, and then that's probably a good sign that there's something wrong. But maybe you have millions of users and only 10 called in. Well, then maybe it's not an issue. And so we need some sort of threshold. At what point in time do we say this is an incident? Let's declare it as being an incident and get everyone involved into troubleshooting this issue and getting them on board with fixing it.

Triage and getting everyone on board

We'll also want to do some sort of triage. Depending on what the incident is, we may need to call certain people in. So what a triage essentially is is an initial step, initial looking into it, to figure out where this belongs, where it's going to go, and how we're going to approach troubleshooting, and who is going to be involved with the troubleshooting process. So we're going to triage this incident to figure out who all is involved and how we're going to approach it.

Now that we've declared that there is an incident, we start following the procedures to start getting everyone on board. We make sure that customer support realizes that there's an issue so they can relay that to customers and users who are calling in, who are using our systems. Maybe we're looking at the incident response manager and making sure that they're on board so that they can start coordinating all of this. And then we also bring in system administrators, database administrators, developers. We bring everybody in that could help troubleshoot this issue.

Now that we get everybody on board, we're going to start analyzing what the issue is and doing some troubleshooting to figure out what's going on on the system.