High Availability

Service Levels

When it comes to our networks, or really any service that we're delivering to an end user, we want certain service levels. When we are delivering a service to someone, whether it be some sort of end user or customer, there are certain key elements that need to exist.

What I mean by that is that if I'm paying for a service, I want to make sure that service is available for me. If I am paying for an internet service, I want to make sure that it's up. If I'm paying for some sort of online software and I can't get to it, that is no use to me. So we need to have that service available.

Another thing that we would like to have is some sort of performance with that. If we can get to that software, or we can get across that network, but it's really slow and it's a really miserable experience, then obviously I'm going to get really frustrated as a customer and I'm not going to want that service anymore.

Another thing we will want is some sort of level of security, and depending on what it is, we're going to want more or less security. For instance, if I'm accessing a bank, and that's a service that I'm accessing, I want to make sure that connection is secure. I want to make sure that they're safeguarding my money, and I want to make sure that they are safeguarding my personal information. So there's a certain amount of security that I'm going to want out of my service.

And then finally there's one that an end customer doesn't necessarily want, but that we want as somebody who needs to manage these services: it needs to be manageable. When we scale up, we want our networks to be able to grow and our services to be able to grow without a lot of headaches. So manageability matters as well, although here we're going to focus on availability.

Service Level Agreements

There is also something called service level agreements, or SLAs. SLAs are between somebody who's delivering a service and somebody who's receiving that service. You can find the SLAs in possibly a contract or a terms of service. What it is is an agreement on what level of service somebody's going to get.

A lot of times they specify something like the availability: how much uptime is going to be available. For instance, if I have an internet service provider, how often are they down? They will most likely have some sort of service level agreement around availability, and if they don't meet this availability standard then you could be due some money back. They could give you a credit on your next bill, because they have not met their service level agreement.

Same thing with performance. If I have subscribed to a 200 Meg connection or a gigabit connection, then I'm guaranteed a certain amount of performance for that availability, for that uptime, and if I'm not getting that, it could be that I actually get a credit back on my bill. Some services have security as well.

So an SLA is between somebody who's delivering a service and somebody who's receiving that service, and it's an agreement or an understanding of what level of service is going to be delivered.

Uptime and the Nines

One of those SLA examples is uptime. This is a really common measurement of how much this service is available to somebody, and more importantly, it actually measures what the downtime of a server is.

What I mean by this is that if I have an internet service provider, they may guarantee me some amount of uptime, and they'll guarantee that uptime for the month, or they'll guarantee it for the quarter, or they'll guarantee it for the year. So if I bypass that during the month, the quarter or the year, then maybe I'll get some sort of money back for that month, quarter or year.

An example of this is somebody saying that I have an uptime guarantee of 99% for the year. If they have a guarantee of 99% uptime for the year, that equates to them being able to be down for 3.65 days out of that year, so that's actually quite a bit.

If I have three nines, and this is a really common measurement — a lot of AWS services have a 99.9% guarantee there — what they're saying is that during the year they could be down for a little over eight hours before they ever will credit your bill back for this uptime. So three nines is very common.

And 99.99% means that they can be down for almost an hour out of that year. Now that's going to be quite a bit more difficult, that's going to be really hard to have four nines there. Notice what I say: two nines, three nines, four nines, five nines. We commonly express this not as 99.99% uptime; we express it as four nines and five nines.

For a lot of services, five nines is considered the premium. If you can guarantee five nines, that's pretty powerful. If you maintain five nines, that's a very powerful statement right there, and what that means is that you've only been down for just a little over five minutes out of the year. That can be pretty amazing if your service is running at this level.

Redundancy on Multiple Levels

The thing is that there are going to be problems on your network. Somebody could be hacking you, or equipment could go down, or a service could go down. There are things that are going to happen on your network that will go wrong. It's just a question of time, not of whether it is going to go wrong. To compensate for this, if we want to maintain high availability, we have to create redundancies within our network, and it comes on multiple levels.

First of all, we have to have redundancy within the infrastructure. What that means is that this network right here has power going to it, and we need to make sure that if that power fails there's a backup solution for that. Many data centers have two or three different power sources from different types of power to back up their whole network. And the internet service provider — there's some sort of connectivity that needs to have redundancy with it as well. Also, as well as power and your connections, there are things like the HVAC or cooling that have to have redundancy within that network as well.

Sometimes we have redundancy from an infrastructure standpoint where we have a whole other building that's maybe in another location, or, as what I set up, a whole other region. Some of the services that I offer for the company that I work for are within the Oregon data center, but I also have redundancy over in a Virginia data center as well, so clear across the country there's this redundancy with it. That is the infrastructure redundancy within your network.

You also want equipment redundancy. Here I've got multiple routers, or multiple layer 3 switches, multiple access switches, and multiple lines between all of these switches. I want redundancy even within this equipment: not only is the equipment redundant, but within the equipment I have redundant power supplies, redundant controllers, and redundant links on those switches and that equipment. So there's a lot of redundancy that's built in on every single level.

The other thing too is that not only can all of this equipment possibly fail, but if you have a server that is running a certain service, that can fail as well. That's why things like SQL and a lot of these different databases and a lot of different services have multiple servers running that same service, so that if one were to fail it could fail over to another service. That's a whole other level.

Redundancy in Our Network

Within our network we have created some redundancy. For instance, our switches have multiple links to each of the switches, and even this configuration here has redundancy involved as well. If we wanted to create redundancy with these routers, maybe we have another line connecting router 1 and router 3. Perhaps we have a whole other internet service provider that's providing us services to the internet. Perhaps on this network right here we have two routers that are running side by side, so that if one were to fail then it would switch over to the other one. Even on our switches, the switches that I'm using have redundant power supplies on them. So we have been incorporating redundancy within our network already.

Now with this network, what I can do is I could actually plug this switch directly into this router right here. This one's already running OSPF on it and this one is running OSPF on it, and so we would create redundancy automatically. It would create and know the best route to get to these networks automatically, because that's what a routing protocol does.

However, what we don't have redundancy in here is this core switch. This core switch is integral to providing services at this level down here, so if this core switch goes down, we are not going to have the same level of connectivity on these different devices.

First Hop Redundancy Protocol

What we can do is create redundancy between two different switches. But of course, how do we implement this, because this switch is holding the IP address that is the default gateway for these other devices? How do we, when this switch goes down, have the other switch take over and take on that same IP address?

For instance, this laptop right here has a default gateway of 10.1.1.1, which is on an SVI interface of our layer 3 switch. So if this switch were to go down, how does this switch over here take on this 10.1.1.1 IP address to become the default gateway for this laptop?

Well, that's where first hop redundancy protocol comes in. It's the first hop, the first connection, the first layer 3 connection point of all of this. That is FHRP, and this is how we would create redundancy for our default gateways.