A buffer overflow attack occurs when data written to a memory buffer exceeds its allocated size, spilling into adjacent memory and causing system instability, crashes, or unauthorized code execution. Understanding this vulnerability is foundational to recognizing how denial-of-service attacks and arbitrary code execution exploits are carried out.
Buffer Overflow Attack
A buffer overflow is one of those attacks that can be used as a denial of service attack.
The quick answer to this is that a buffer overflow is just like the name sounds: there's a buffer and it overflows. What that looks like is, inside of memory you have different buffers, and when there's information that's written to those buffers that's longer than the buffer, it overflows into other memory, causing problems. Really, this is a problem with the application and how the application is handling memory.
When a buffer overflows, it can cause unexpected behavior. Some of that behavior could be that the system crashes, or there's some sort of access control loss, or perhaps there is arbitrary code execution, meaning there's some sort of malware that can be executed leveraging the fact that the buffer has overflowed.
We're actually going to take a more in-depth look into what buffer overflow is so we can better understand what's happening here. First of all, let's understand that a computer is made up of several components: a CPU, a GPU, a network interface card. You have the BIOS, you have several inputs and outputs into the system. Essentially, before things get executed by the CPU, everything goes into the RAM. A big part of how your computer operates is how it manages the RAM, and what goes into the RAM and what comes out of the RAM.
RAM is also called memory. The way your computer uses memory is that it stores data in there right before it gets access to the CPU, the central processing unit, or as data is coming out of the CPU then it gets put into the RAM as well, or the memory as well.
Now, the way that memory works is that there's some sort of allocation to it, and addressing. What I mean by that is that memory is broken up into smaller chunks that can be addressed. So that way, when you're trying to access a certain piece of information, then you have an address to access that information. Now you have a way to call that information, or place that information, based off of some sort of addressing. So we've got a lower level addressing up to a higher level addressing, and this is all broken down into these different allocations.
I'm on a laptop here and I actually have this suite of programs called Sysinternals. I'm going to scroll down to the RAM, and I'm going to open up this RAMMap64. What we're going to do is take a look at the RAM allocation and see what the RAM allocation looks like.
This first page just shows us overall how this RAM is allocated. So we've got process, private, mapped, sharable, page table. We're not going to get into any kind of granularity of understanding what all this means. You can see how we break up RAM, we break up the memory into these different allocations so we can use it for these different purposes.
We're going to actually take a look at this file details tab. In the file details tab, this is showing us all of the different files that are running, or opened, or being stored right now in memory, in RAM, and it shows you the size of each of these files. We're just going to choose a random one. I'm going to open up this. So this is the system32 newdev.dll. I'm going to open that up, and what this is showing us is all of the physical addressing on that RAM, and it's mapped to this specific file. So we can see that this file is taking up quite a few allocation spots inside of memory. So this just gives you a glimpse into these file allocations within RAM and what those look like.
Programs running on your machine are going to need to store some information. Here we've got an example of a couple of strings that are right next to each other inside of memory. So your program is going to need to store certain information. In this case right here, we've got a couple of stored values that we're storing in memory. We have a string one right here and a string two. And let's just use the case where it's the same. In this case, we're storing this bit of information, which is "this fits". So "this fits" is then stored in each one of these memory allocations. If we were to break that apart, here's the hex that's actually being stored. So we see these values that are being stored within this. They're the same because "this fits" is the same string that we're saving in each one of these.
Now, as part of this, to make this efficient, maybe we determined what size this string is going to be. So this is going to be eight bytes long is what we're going to store "this fits" in for string one. And really it doesn't matter for this example for string two, but let's say it's the same amount.
So let's now get into buffer overflow. Essentially what a buffer overflow is is now we try to save another piece of data, but it's longer than eight bytes in this example. So instead of "this fits", now we say "more digits". So "more digits" is longer and it runs into this second string, and the t happened to be the same here, but the s is different. So there is a different value now inside of string 2, because we saved a longer piece of information than what string one could really handle, and it overwritten some of string two. Now, as I mentioned, this really is an application problem, that the application wasn't written in a way to properly handle the situation.
There are two major types of buffer overflow. There's a heap overflow and there's a stack overflow. I'm not going to get in depth into the difference between heap and stack, but we're going to address this visually so we at least have some concept of what the difference is between heap and stack.
First of all, just know that the data structure within RAM is stored kind of like this, where we have some kernel information, the stack information on one side, we've got text data and heap on the other side. For the purpose of this we don't need to get real in depth into what each one of those are, just know that this is a common architecture of how things are structured when it comes to memory.
Now, both the heap and the stack are going to grow, and they're both going to grow inwards. The stack will grow this way and the heap will grow the other way.
I zoomed in on the stack right here, and what we see here is several different data points within the stack. One of them is the return address. That's what RE is, the return address here. It's not exactly important for this lesson to really understand what the return address is, but it is something that we can actually leverage to attack a system. So once again, we're not going to go in depth into that in this lesson, but maybe another lesson.
So what we have here then at the end is the buffer, and this is going to be where we store some data points in that buffer. So what is the heap and stack overflows? Well, that's this buffer. We have this buffer here, and if we fill up this buffer and it runs into the EBP and the return address and some of these other data points, that's when we start running into issues and problems. And so that would be a stack overflow. Well, the same type of thing can happen in the heap, and then that's called a heap overflow.
So now, when we start overwriting some of these other fields, then it could cause one of many different issues on our systems. Our application can perform maybe really randomly and cause some really strange issues, or maybe it crashes the system. This is where we get the denial of service attack, because if that system crashes, now that system is down and is not offering services to others. It could also leverage some sort of access control. Maybe overrunning this with certain code allows us to access systems that we wouldn't normally have access to. Or there's a way to actually store some lines of code into here that are deceptive, that are nefarious, that are malware, or some sort of arbitrary code execution could happen with this that allows us to access the system in a way that's not intended to. So there's quite a few problems that can exist from this.
So let's take a look at what this actually looks like. I'm on a machine that has a program called Overflow 32. So this Overflow 32 is going to allow us to do a buffer overflow, but we're going to run it inside of another program, this GDB, which is a debugging program, which will allow us to see into the memory and see what that's doing.
So the first thing we'll do is just run the program and see what this looks like. It's a very simple program. All it's doing is asking us for a password. And that's it, that's pretty much all the program does. And so we'll just enter some data in here. Let's do 4 A's, and password is entered. And that's it, it ran the whole program.
Now let's do an overrun. Now, this is only supposed to accept 10 characters, but it allows us to input more than just 10 characters. So we're going to run this again and see what happens when we put too many characters in here. So I got a ton of characters. We're going to copy it and paste it in here. We'll hit enter. And what happens is that the program stops running and we get a segmentation fault. And this is because we've overran the buffer.
So what does that look like inside of memory? Well, let's take a look. I'm going to use break 9. You don't need to be concerned with what that is, but it's going to essentially break it at a part of the program where we can take a look at the memory and see what's happening inside of memory. So we'll run this again. We will clear out what we had before, and then we'll do our series of A's again and hit enter. And now we're going to take a look - see, the program did a break on line 9 - and we're going to take a look at the memory.
So what's happened here is this is the buffer. Essentially these right here allow us to log this password and store this password in here. 41. It keeps repeating 41 because those are the A's. A gets represented as a 41. Remember, this is all being stored in as AS asy, or as in bits in the back end. It's not being stored as an actual A. So we've got these bits.
Now, like I say, this essentially is our buffer right here, and our buffer ends, and then we've got some other data here. One of them is this return address. And we've overwritten our return address. So when it tries to return it, it will try to return to a different address, and then the system crashes, or in this case it's just the program that crashes. And so now we've got a denial of service attack, because we've shut down this program, because we've overran the memory. It's only allotted a certain amount and we've surpassed that amount.
So here's that buffer overflow attack, where we just have, essentially like the word says, buffer overflows into these other fields causing problems. The two major types that we have is a heap and a stack overflow. And the consequences of these can be varied depending on what has overflowed and how it's overflowed.
TechKnowSurge builds IT and cybersecurity professionals through hands-on, concept-first training built around real understanding — not memorization. Free interactive tools, structured programs, and 25+ years of real-world experience, all in one place.
Explore free tools and programs →