Architecting Availability

One of the things that we're trying to achieve when we're setting up services for our end users is high availability. But what exactly is high availability, and how do we achieve that?

Many times when we think of security and implementing security on our networks, our infrastructure and within our organizations, what we think about is somebody trying to hack them and making it secure against those. Essentially, we're trying to think about confidentiality and integrity of our systems. But often we overlook this availability. Although we know that it's important, we overlook that this is part of security, part of the security of the organization and within our infrastructures, and really is needed to deliver great services to our end user.

Downtime and what it costs

When our systems are not available, we call that downtime. There is times when it goes down, our services go down to our end user. So this is a timeline right here. Let's say that there is an incident and things go down. Our services go down, our customers can't get to our site. Maybe it's an e-commerce site. Maybe it lasts a minute, 10 minutes, 30 minutes, an hour, a few hours, whatever the case may be. We at some point in time go back online.

One of the issues is that we have this loss of revenue during that time. There are customers that can't purchase our services. Some of those customers may come back and still purchase our services, but some of them we just might have lost, and they'll never come back because now they've gone to a competitor. So that's the impact, the immediate impact. We notice when there's a drop in revenue during this time, but it extends much beyond that, because maybe we would have made some lifetime customers if they bought our services during this window, and now we're losing that lifetime customer. So it's not just the revenue that we would have made during this time, but it could extend far beyond that.

Plus, there's a lot of follow-up that happens when our systems go down. We want to figure out what went wrong, what happened, how do we solve this for the future so it doesn't happen again. So there's a lot of follow-up that needs to happen past this extent of the downtime, that can really impede our work progress.

So what is the impact of downtime? This is a slide I often use with any cyber security incident, but this one in particular, downtime. We could get a loss of revenue. Perhaps we could get some of these others, like loss of assets, maybe even some judgments and fines. We probably need to send our customers some sort of notification, which takes time and energy. We lose maybe some of our competitive advantage. Definitely there could be a loss of reputation with this, and a loss of customers with this. And so with this availability, especially if it happens often or long enough, there can be some really long-term impacts to the business.

What threatens availability

But what threatens availability? We have those threat agents out there. If they're conducting some sort of attack on us, like a denial of service attack, that means our services are going down. That's problematic. There's also utility disruption, so it could be the internet or power or something along those lines that causes us to go down. There can also be failure: failure on hardware, failure on software, that can cause our services to go down. Maybe it's because of some sort of destruction like a fire, earthquake, or natural disaster. Or, more often than not, I found there's some sort of human error, like a misconfiguration or something they've done wrong which has brought down our services. When I say human, I usually mean employees that are working on the systems they're trying to maintain the uptime on, and they're the ones that are bringing it down. More often than not, I find that this is what the issue is.

Availability and high availability

So let's talk about what availability is. Availability just means you can get to the site, that when you try to get to our e-commerce site, it is up and running, that those services are available. And so what it takes is that the back end, the internet connection, the hardware, the databases, the infrastructure is all up and running to support this availability.

What we're trying to achieve is high availability, or HA for short. High availability just means that these services are usually up and running when the customer expects it to be up and running. So high availability is somewhat subjective. For instance, maybe we have a very time-sensitive site, an e-commerce site where people are on 24 hours a day. Then just a little bit of downtime can be very impactful to our system. However, let's say we're maybe part of an education system and no one's really on our systems during the summertime because all the students are off. That doesn't play into this high availability as much, right? It's not as critical. But when school is in session and those students are sitting at their desk, then we want to make sure that they're available. So it's very subjective on what high availability actually means. But essentially, when the customer wants access to those services, they have access to those services.

Faults, downtime and fault tolerance

So what disrupts high availability? A fault in the system could disrupt that. What is a fault? A fault is something that happens on the system that causes an issue. So for instance, maybe this server goes down because there is a hardware malfunction. Maybe the NIC card on there goes offline. That's a fault. That's something that has occurred that's problematic.

What this creates is downtime. Downtime is the measurement of what the customer is experiencing, what the end user is experiencing. So in this case, because this server has a fault and it goes offline, now what we have is our services are offline, and we can measure downtime in how long it is. Is it minutes? Is it hours? Is it days? So we can measure the downtime of how long our services are offline.

So what we want to do is create something that's fault tolerant. What fault tolerant means is that if there is a fault on a system, other systems take the place of it and keep it back up and running. Our example was a NIC card on this server, that it's the network interface card, and if that goes offline that's problematic, our services go down. So we duplicate that NIC card. We have two network interface cards on there, and we have them plugged into two different switches, and we have some redundancy there. Or better yet, maybe we set up more servers on here so a whole server can go offline and be fault tolerant.

Redundancy and weak points

So one way we achieve this, or the primary way that we achieve this, is through redundancy. Rather than having a NIC card that's running, we have multiple NIC cards. Rather than having one server that's running, we have multiple servers that are running. We duplicate what we have on here to make things redundant, and then it becomes fault tolerant.

So what are some ways that we can do this? To eliminate downtime, we can eliminate weak points within our network. For instance, if our internet service provider is not providing a stable network for us, maybe it's time to find another internet service provider. We're creating redundancy within our network, so here we have multiple servers that are running, we've got multiple database servers that are running here. We start monitoring and alerting on us when things go down so we can respond very quickly to them. So response time is important, and making sure we're planning ahead of time. There's lots of things that can contribute to limiting how much downtime we have on our network.

So what we do is we create things like redundancy on our network. That way, if a server goes down we are fault tolerant, where another server kicks in, making our services to the end user highly available.