HA - Redundancy

Why Redundancy

In this industry it's really just understood that things are going to fail: you're going to have equipment failures, you're going to have software errors, you're going to have things that come up that are going to be problematic. So it's important to start planning for that, and one way that we plan to make sure that things are fault tolerant is we incorporate redundancy within our equipment and within our services.

Here we have some servers that are offering services to our customers, and what we're going to want to do is create redundancy. Not just one server, because a server could go down, but multiple servers, so that in case one of these services goes down one of the others will take its place. Same thing with the database: we want multiple versions of this database, so if one fails the other one would pick up.

Where Do We Need Redundancy

Where do we need redundancy? The answer to this is pretty much everywhere. Everywhere that you can think that the system relies on, you need redundancy in.

• Power. Your system relies on power, so you're going to need redundancy there.
• Cooling.
• Networking and the internet.
• Hardware and software.
• Your data.
• Location. You'll want redundancy in your location as well.

This is a setup that I set up in a data center, and you can see that each one of these is a piece of hardware. There are three of these; they're network addressable storage, and each one of these has two power supplies. You can see the full power supply right here for this middle one, and you can see that there's another duplicate of that right here, although some of it is cut off. So you could have a whole power supply fail and this is still going to be up and running.

Each of these then plugs into this power distribution right here — this is a power bar. One of these power supplies is plugged into this power bar, and the other one runs to the other side and is plugged into a power bar on the other side. So you could have the power bar fail and this equipment will still be up and running. Each one of these also goes to a whole other circuit, and one of those circuits could fail and you'd still have this up and running. So you can see redundancy through the whole process here.

Power Redundancy

You also want redundancy in the power supply. Most of our supply comes from the grid, from some sort of line coming into the company providing us power, but you want redundancy with that. One example of redundancy is with a UPS, or uninterrupted power supply. This is actually a different business that I worked in, and at the bottom rack here you can see two different UPSs — you can see just the front of one, and then you can see the top and front of another.

UPSs have a battery inside of them, and they also plug into the wall, and then you plug your devices into these. So what will happen is if the power from the wall goes out, then the battery will kick in and continue to supply power, and so you won't have any interruption in your service. That's what we want.

Now these power supplies do run out of power over time. For this particular business we wanted them to be up and running for3 minutes, so these were to supply enough power for 30 minutes and keep their phones, keep their switches, keep some of their networking gear up and running.

So what we need is a long-term solution, and that can come in with a backup generator. This is a backup generator, and I would go out and test this on a weekly basis to make sure that it was up and running. This will kick in and start generating power if the power goes out. So the UPS supplied temporary power until the generator could kick on and supply a long-term power source.

Clustering

Clustering is one way that we can create redundancy with software and services. In this example right here we have a web server — in fact we've got three web servers here. Any one of these could reply to this client when the client is trying to connect. What these web servers do is they're set up with clustering and are communicating with each other, so that a client can communicate with one of these servers and get responses from this server. If that server goes down, another server can pick up and start responding to that client.

There are several ways that this can be set up, but a good example is with the database server. You have a primary database server and these web servers are communicating with it, and then if something were to happen with this, clustering services would make the secondary server now responsible for replying and reply back to these web servers. That's an example of clustering, although there are multiple ways that you can set up clustering.

Load Balancing

Load balancing is another way to accomplish software and service redundancy. The purpose of a load balancer is to distribute the connections that are coming in. There are a lot of connections coming in from a lot of different users, and this load balancer will then go and choose one of these servers to send the request off to, who will then respond to that client. That's load balancing technology.

But the nice thing about this is that if one of these servers goes down, this load balancer is actually checking those services to make sure they're up and running, and when the service goes down it will take that out of the pool of available servers and then not send traffic to it, and just send traffic to the other devices. So load balancing is another way that actually creates some redundancy with the services — although this doesn't work with many things like some databases, because these databases are too dynamic. So load balancing doesn't work in all scenarios, but it can work for some scenarios.

RAID

We need to create redundancy even with our drives, and therefore what we're creating redundancy with is our data. One way to do that is with a RAID configuration.

A good example of creating redundancy with RAID is using a RAID 1. It's mirroring, so whatever data gets sent to the first disk also gets put on the second disk. You've got a complete set of data on both disks. Any one of these disks could go down and you still have the full set of data.

RAID 0 is not a good example of redundancy. That's because the first block of data gets sent to the first disk, the second to the second disk, the third to the first disk again, the fourth to the fourth disk, and so it just splits the data. If you were to lose one of these disks you would actually lose your data. The advantage of this is that it speeds up the process. One of the slow parts about storing data is the writes, and sometimes the reads as well, but writes is a very slow process, and so by striping it across multiple disks we can actually speed this process up.

So how do we get some of the advantages of speeding the process up with striping, but also have some redundancy? We can do that with a RAID 5. A RAID 5 means that we can have two or more disks that are devoted towards striping — we can stripe across all of these drives — and then we have another drive that we're going to save as a parity. What that does is it allows us to lose any one of these drives, and this parity drive actually allows us to recreate that data. So we have both the advantages of striping, but we also have some resilience, in that if one of these drives dies we can recreate that data.

RAID 6 is the same way, except it has two parity disks, and so we can actually lose two drives if we have two parity disks and still recreate the data. And remember, we could have two of the drives being mirrored, or three or four — we can actually have multiple drives that are being striped to as well.

We also have a RAID 10, also known as a RAID 1+0, and the idea behind this is it has striping and mirroring. It first stripes to one set of mirrored disks and then stripes to the second set of mirrored disks. So block one goes to the first set, block two to the second set, block three to the first set again, block four to the second set again. But each one of these drives is mirrored, so they're duplicates of each other, and we could still lose a drive and be okay, and then we get the advantages of that striping as well. You don't see this quite as much out there as RAID 5; RAID 5 is probably the most commonly used out there.

Replication and Backup

We're also going to need to have redundancy within our data, and replication is an example of one way that we can create redundancy with our data. That is, we have our data in one location and then we replicate that data to another location, and the idea behind replication is that it's always being updated at some interval or in real time.

Backup is another solution, although the idea behind backup is that it's a point in time. We're going to take a copy of possibly this database right here and create a copy of it, and that is going to be used as our backup, and we can always then fall back on that copy of the data.

Location Redundancy

You also want location redundancy, and I'm going to give AWS as an example of this. In fact, I'm going to give you an example of a business that I moved up into the AWS cloud. What we did is we set up our servers in a few different availability zones within AWS, and availability zones are essentially different data centers. Some of our services were in availability zone one, some were in availability zone two, and some were in three, being different data centers. They were clustered services, and if any one of these availability zones went down then another set of servers would automatically take over and it wouldn't be any issue — we'd see no downtime with our application.

That's how we had set it up, but the problem is they were all in the same region. So what if there's a region-wide incident? We also had a set of servers that were in another region, so if these went down we could go through a disaster recovery process to bring them back up in a whole other region.

Provider Redundancy

You may also want to consider redundancy amongst your providers. If this is up in the cloud, we call that multicloud. An example of this is maybe we have our primary servers in AWS; however, if we have an issue, maybe we can fail that over to Azure.

So we have the original that is up and running, the primary. We could have that on prem, which means that we manage all of the data center and it's all on premise. Or perhaps we have some sort of colocation, so our first copy is on a colocation — that would be rented out space in a data center that we have our equipment in. Or perhaps we put it up into AWS and have our first copy on AWS, or maybe we have it inside of Google.

The idea behind this provider redundancy is to have a different copy. So if we have one here in AWS, maybe we have a colocation with a second copy, something we have access to. Or maybe we might even have a third location that's in another cloud provider. So there's some sort of redundancy in here, in that if a provider has some sort of issues, we have backups to that as well.

Hot, Warm, and Cold Sites

This can obviously get really costly if you have all these copies in all these different areas, so we may want to consider what kind of redundancy this is going to be.

Here we have the original — this is going to be our copy that's up and running. We could have a hot site. A hot site means that all of these servers and all of the data are all live, and all we need to do is trigger something that says customers, don't go here any more, you're actually going to be sent to this hot site, and now the hot site is the live site. This requires a lot of connection and bandwidth and making sure that everything's up to date, so this is going to be the most expensive option.

The warm site means that maybe what we'd have is just some servers that are already set up ready to go, but the data is not populated. We have a backup copy over there, and that backup copy is going to be something that we can restore onto these servers and get things up and running.

Or perhaps we just have a cold site. Maybe we just have some ability and some idea of how we're going to set this up, but we don't actually have the equipment or the data over there yet. We call that a cold site.

How Much Redundancy: N+1

There's also this question of, well, how many do we need? What is the redundancy that we need? Generally speaking, there's this N+1. N+1 is just the idea of whatever number you need, add one to it. A good example would be a spare tire: you have four tires on your vehicle, you could add an extra tire, and now you've added one, so now you have five tires available to you. This is one way that you can create this.

Or if you had really critical systems, maybe you have an N+2, or N plus whatever. But N+1 is a pretty big standard that's out there that most people consider good for meeting the needs that they have. There are times that we need more than that — maybe N+2, maybe N+3. It's just going to depend on the scenario, but N+1 is pretty standard.