Clustering

Clustering is one great way to group up servers and services, so if something goes down, something else can take its place.

Clustering is a way to group up servers and services so they all act essentially as one. So here we have a bank of servers right here. We could create a cluster out of this, and then they are all essentially running those same services on each one of those servers. So we can distribute the load amongst those servers, or use them for redundancy of these different servers.

We can also do it on our databases. In fact, almost always, if I have redundancy with the databases I'm almost always using clustering services in one form or another. If it's the servers on the front end, then sometimes I'm using clustering services, sometimes I'm not.

Now, sometimes I find clustering services is like a separate service, a separate install on these servers, and then other applications use it to do failovers. While other times it's kind of baked into the actual application code. And let me give you a couple examples of this. Redis is a database. Redis we can install on servers and have this database up and running. Redis has what I'm considering baked into the code clustering features. So we would install Redis and then we would turn on clustering features to get it up and running. Whereas other technologies or other software like SQL utilizes other services. So what you would actually do is you would install Windows Server Failover Clustering onto the server and get it up and running for those failover services, and then SQL you would install on there and configure SQL to use those failover services. So it's like two different applications that are running.

So now, how these clustering services work depends on what the application is and how you configure it. But essentially what they all do is just monitor each other. So in this case right here, let's say we have a bank of servers and we create a cluster out of it. And then what they're doing is they're monitoring each other to see if they go offline. And then if something goes offline, then it just takes it temporarily out of the cluster and it no longer is functioning. And then once you can bring it back up online and address whatever issue is happening to it, then you can bring it back into that cluster and make it alive again. So that's clustering services. And same thing with the database — the databases will monitor each other to make sure that they're both up and running.

Failover

So one way that this clustering technology works is doing something called failovers. So if we're making connection to a machine and one of those machines goes offline, the connections that were going to that machine need to fail over to another machine. So we call this failing over. The same thing happens to databases: these machines are maybe connecting to this database right here, and then if this goes offline maybe it needs to fail over to another machine here. So that's called failover.

Active and passive nodes

There are active and passive nodes. So right here we have a cluster, and maybe we have all active nodes. So these are all active nodes here — active, active, active and active. Now what they can do is, if they're all active then they can share the load, so it doesn't matter which server we go to, it can help service things out. Versus, some technologies will have an active/passive. So this is active and this is passive back here. And in this case right here, everything gets just communicated to the active node, and then if the active node goes down then the passive node then becomes active.

So two different ways of doing this, and this depends on your configuration and the technology, on whether you can even do active/active or if you have to do this active/passive. And there are pros and cons to this. For instance, one of the reasons why database servers are often active/passive is because there's a lot of changes that happen quite frequently, and if you have two database servers that are making changes at the same time, you can have contention with that. So sometimes it doesn't make sense to have active/active, although a lot of times that's the ideal. Sometimes it just doesn't really play out well, so sometimes we have to go with the active/passive node.

Terminology

Now there are some terms that come with clustering, but they're not universal. So you might use one set of terms when it comes to Redis and another set of terms when it comes to SQL, and another set when it comes to VMware. And so just to be a little familiar with what these are, let's go over them.

One of them is active/passive. Now this is pretty straightforward: one of your servers, one of your nodes within the cluster, is active and the other one is passive. The passive one is still updating, ready to take over if the active one goes offline.

Now the terms primary and secondary, and master and slave, can mean similar type things as well. Just, the active one is the primary one. But it also can denote which one is usually the primary one, or which one is usually the master one. And I don't typically use this term master and slave. It's gone out of favor now, and really we're trying to transition more to this primary and secondary, but you need to be aware of it because there are still some technologies out there that use that terminology, and there's still a lot of documentation that uses that terminology.

But what we have here is this primary could just be the active one, or the primary could be the one that we usually are up and running on, and the secondary is this one back here. And one of the things that makes that important is because maybe, for instance, we have this big beefy front end here that we usually are using, but we have this secondary one that we don't want to spend as much money on. And so it is the passive one, and we have maybe less RAM on it and less CPU and less compute resources. But if something were to happen and we have to fail over to it, then what we would do is we would beef it up and make it stronger if we have to run on it for any kind of duration.

So that is some difference in the terminology that we see out there. Basically they mean the same thing, but the primary and secondary could mean that we have a database that we're primarily using and a database that we occasionally will fail over to.

Clustering and maintenance

Clustering and failovers can really help us out in maintenance situations as well. Here we've got a database server that's being accessed by all these servers, and they're providing services that we don't want to go down. So what happens is that we have two databases here, and one of them is going to be the live copy and the other one may be a passive copy. Well, what we're going to do is we're going to do all of our patching on this passive copy here. And then when we're ready, what we can do is we can fail everything over to that — what was the passive copy now becomes the active copy — to do the updates to the other database server. Then once we do that, we can either fail back or we can keep it running, depending on what our scenario is. But we can fail over to keep this high availability for our customers. We can fail over during this maintenance period without ever having downtime.

VMware

Let's look at some examples to see how these clustering services may behave and act. The first one is VMware, and what clustering services look like on VMware.

Each one of these boxes is called ESXi. ESXi is just their hypervisor to keep those servers up and running, and these are all virtual servers, and what they do is they link up to some sort of storage in the back end. So I'm representing this by a folder right here. Now this is shared storage and it's duplicated. It has its own way of some sort of distribution to make sure that there's backup and clustering and all that. So we're just looking at this as being the file storage right now. So we have the files, all the files that make up some sort of virtual machine.

And so now what happens is that that virtual machine needs to run on one of these ESXi boxes. So what happens is we turn up that VM on one of those boxes and it's up and running. So now that server is up and running. But let's say this physical server goes offline and is no longer available. Now what we can do is we can turn that machine up on another machine, because we've still got the files and folders. Now this is not necessarily using clustering — I can do this without setting up clustering on VMware. But it is more helpful, better, if we do cluster these machines, and it can actually make this much faster and quicker just to turn it up on another machine.

I can take this to the next level and use VMware's high availability feature. Their high availability feature lets us run this machine, just like I mentioned before, on this ESXi box right here. But what it will do is it will also keep a duplicate copy of its memory. Because we do have the files and folders and stuff of this machine right here, but what we don't have is the running state of this machine right here — it is running on the RAM and using the CPU of this physical server. So what we do is we keep a second copy on another server right here. And what that does is now we have a duplicate of the RAM. And then if anything changes on that virtual machine RAM on this machine, it will automatically update it on another machine. If this goes offline, then what it will do is, it already has the running state, it already has a copy of the RAM, and it can just essentially — it's not even turning on that machine, it just activates that machine. And now that machine is up and running as if nothing ever happened. And it's made possible because of VMware's clustering services: the ability to cluster these machines and be able to communicate and have a duplicate set of RAM, all the memory for that virtual machine, on a whole other physical ESXi box.

Redis

Let's look at a different service, a clustering of Redis, and that behaves very differently. Redis is a database server. So we have databases right here, but really they're just servers as well — we have a server that's running these databases. So I'm going to represent this by three servers that we have right here. And this is clustered services, and this clustering helps share the load amongst these servers. And we only have one active copy of the database, and so the data could be held on machine one, on machine two or machine three.

Now these machines need to access the database. So what they'll do is they'll reach out to the node that they think has the information that they're looking for. And so node three might be one that they reach out to and say, "Hey, I'm looking for this information." And node three is going to respond back and say, "Oh, that's great, but I don't have that information. You actually need to look at node two." And then it will reference node two, and node two will then respond and give it the proper information.

Now the way this works is it'll keep an actual duplicate, a passive duplicate copy of this. So if one of these fails, then machine one or two will just pick up the slack and then be able to host out that same information. So now somebody reaches out to what they think is node three, but it's no longer live. So it starts reaching out to node two. Node two says, "Oh, that information is on node one now." And then it reaches out to node one. So this is one of the clustering technologies and how it works.

SQL

Another example is SQL. SQL uses clustering technologies. It doesn't have to, but we can set it up to use it, and it gives us that duplicate information here.

So let's say we have what's called an Always On availability group. So what's happening is that information is being actively replicated to two copies of the database. We only have one copy that's actually active, and the other server that's running here. So let's say we have a two node server here. So what happens is that all the machines are actually just communicating to one server, the one that is on. And now let's say something happens to that server — then that goes offline, and then everything starts communicating to the other server.

And this can happen in many different ways. Number one is, there could be just an update that these machines know which machines to reach out to, and they check one and if it's down they check on the other one. But that's not as common of a setup as the other two I'm going to tell you about.

One mechanism that we can use is if they're on the same network, so they have IP addresses in the same range. Each server will have their own IP address, and then they'll have a shared combined IP address. These machines reach out to this shared IP address, and this shared IP address is associated with just one of the machines, the primary database. Now if this machine were to go offline, the other machine then would take over this virtual IP, and now all the machines are sending it to the other machine. So that is one way that we can handle this.

If they're on different networks, they can't really have a shared IP address, because how do you route to an IP address that changes networks? You can't really do it that way. So another way that we can do this is that they have a DNS record, and that DNS record points to some host machine. We're just going to call this SQL, so it's a DNS record that points to this machine SQL, and it's associated with one of the IP addresses here. And so if this server goes offline, then what happens is the DNS record for this cluster right here gets changed and points to the secondary IP address, and that secondary machine takes over.

So you can see there's different setups, different ways that we can set up this clustering to happen.