TechKnowSurge
CompTIA CySA+ 3.2 NIST CSF RS.MA-01 NIST 800-53 IR-4 ISC2 CISSP 7.6 CompTIA CySA+ 3.3 NIST CSF RS.MA-04 NIST 800-53 IR-8 NIST CSF RS.CO-02
TechKnowSurge Catalog Videos Incident Response Example
VideoSecurityFree

Incident Response Example

A real-world incident response walkthrough traces a high-traffic application outage from initial anomaly detection through containment, root cause analysis, and post-incident review. The scenario illustrates how escalation procedures, cross-team coordination, and lessons learned drive continuous improvement to an organization's response plan.

⚑ Complete this video to capture a CTF flag worth 1 point.

About this video

Incident response plans vary significantly across organizations and incident types, and this scenario demonstrates one realistic example of how a mature, tested plan plays out under pressure. The environment involves a large-scale application with a backend database serving hundreds of thousands of users. When the system begins showing early signs of malfunction, initial alerts go unnoticed before escalating notifications finally prompt a system administrator to begin investigating whether the situation represents a true positive requiring escalation. Once the system goes fully offline, an incident is formally declared, triggering a coordinated response that brings in customer support, an incident response manager, database administrators, and developers. Customer-facing communication is handled promptly, with support staff keeping users informed while the technical team works through analysis, containment, and eradication. A database server restart resolves the immediate outage, and continued log analysis identifies the root cause: application code incorrectly calling a stored procedure that was overloading the database CPU. The recovery phase extends beyond restoring service to include transparent communication with customers and stakeholders about both the fix and the ongoing investigation. A postmortem meeting produces an after-action report documenting the full incident timeline, process gaps, and corrective actions. Key lessons include the need for more sensitive early-stage alerting and earlier involvement of the full technical team, alongside a permanent code-level fix slated to move through the development pipeline as the long-term resolution.

What you'll learn

What's covered

Incident Response Scenario

Aligned to

CompTIA CySA+
3.2 Given a scenario, perform incident response activities.
3.3 Explain the preparation and post-incident activity phases of the incident management life cycle.
NIST CSF
RS.MA-01 The incident response plan is executed in coordination with relevant third parties once an incident is declared.
RS.MA-04 Incidents are escalated or elevated as needed.
RS.CO-02 Internal and external stakeholders are notified of incidents.
NIST 800-53
IR-4 Incident Handling
IR-8 Incident Response Plan
ISC2 CISSP
7.6 Conduct incident management

Key terms

Incident Response
IR
A structured process for identifying, containing, eradicating, and recovering from security incidents.
Escalation
The process of elevating a security issue — such as a critical vulnerability or active incident — to a higher priority or authority level when the standard response process is insufficient or too slow to address the threat.
Containment
The phase of incident response focused on limiting the spread and impact of a security incident to prevent further damage.
Eradication
The phase of incident response in which the root cause and components of an incident, such as malware, are completely removed from affected systems.
Recovery
The phase of incident response in which affected systems and services are restored to normal operation after an incident.
Post-Incident Review
A follow-up phase after an incident is resolved in which teams analyze root causes, evaluate response effectiveness, and implement improvements to reduce future recurrence.

Topics

Incident Response Cybersecurity Escalation Procedures Root Cause Analysis Post Incident Review Cross Team Coordination

Transcript

In reality our incident response plan could look very different from company to company. Not only that, but there are a lot of scenarios within the company, and so this is just one example of how we could react and how we could set things up.

The Scenario

Our scenario is that we have an application, an application that we have hundreds of thousands of users getting on, and it has a database on the back end. Something is malfunctioning with this, and we are going to see how we go through this incident response process.

In this scenario too, we already have a well-developed incident response plan and we have tested it out many times. We have had other incidents that we have tested this flow out on and refined it a bit, so we are already pretty far along in this process. Now what we are going to do is apply what we have learned and how we approach things to this scenario.

Normal Operations

We start out the scenario where things are functioning as designed. There are no issues, and since there are no issues there is nothing really to report. However, we do have some dashboards that show the health of the system. Here is a picture of one of the offices that I set up, where we have monitors that I had set up and programmed so we can see what is going on within our systems. My team sat in this room right here and we could visually see what was going on at any given time. We could look up and see nothing is really happening.

Now, this is not a solution for recognizing that things are going wrong if something were to go wrong. But what we can do is see that things are functioning well every time we look up, and also see what some of these standard stats are, some of the things that we are used to seeing on a day-to-day basis.

The First Signs Of Trouble

But then this system starts experiencing some errors and we start seeing some signs of malfunction. It is not significant yet, but it is just some signs that things are not going in the right direction. Maybe at this point in time our systems do show that there are some signs that something is a little off, but maybe no one looks up, or maybe it is not significant enough to really trigger anybody to take action.

Maybe even the system gets to a point where it starts triggering some alerts and sends out an email to the team. Email is not something that we check consistently, or a lot of times we do not check consistently, so maybe it is that no one on the team actually saw this email come in.

At some point in time maybe the system starts experiencing issues that trigger a more significant alert. At this point in time it is not a full incident, but we still trigger SMS and text and a call to make sure that somebody is keeping an eye on the situation.

The Technician Investigates

At this point in time our technician starts investigating to see what is going on. Maybe first he just looks up at the dashboard to see if there are any kind of inconsistencies with the data that we are receiving, and then starts researching things by getting into systems and seeing what is going on. So at this point in time we just have a single system administrator that is investigating the issue.

The technician might also send a quick email off to customer support just to let them know that he is investigating some sort of issue, and to let him know if anything comes up, if any customers complain, so he can get a pulse on the situation to figure out how much it is affecting the end user.

One thing this technician is trying to do is to evaluate: is this a valid report? Is it a true positive or is it a false positive? And if it is a true positive, is it to the point where it needs to be escalated to a higher tier technician, or to bring other people on, or to get an incident response manager going? What is the level of urgency with this issue?

Declaring The Incident

Then something unfortunate happens: the system goes offline. So now the system is offline and it truly is an issue. At this point in time the system administrator that is investigating this issue declares that this is an incident. By declaring an incident, this suddenly triggers a series of events and alerts that are going to notify quite a few more people to get involved in this investigation.

Some of the people that we bring on is customer support, letting them know that this is an incident, and we start communicating with them at a much greater level. Also we bring on an incident response manager to make sure that we are responding to the incident in a proper way. They are going to coordinate things like communication and the plan, and making sure that the right people are on board. Then we have the system administrator, and maybe we bring on database administrators and some developers to figure out what is going on on the system.

Shortly after that we start getting customers reporting that there is an issue. They are calling in and letting us know that they cannot get to our services. Customer support has already been communicated to, so they already know how to handle the situation. They let the customers know that we know that they are experiencing an issue right now and we are investigating it, that we do not have an estimated time of when we are going to fix it, but as soon as we do understand what the problem is and fix the issue we will let you know.

Analysis, Containment, Eradication And Recovery

Now that we detected and declared that this is an incident, it is all hands on deck and we go through the process of analyzing, containing, eradicating and recovery.

During this troubleshooting process, what they really know is that the users are having problems connecting to the application. From the application they look at that and they realize that it is calling something on the database server, and there is something wrong on the database server. So they look at the database server and see that it is a CPU issue on the database server. They are not sure exactly what it is and they want to do more investigation, but right now they have a lot of angry customers that are not happy. So what they will do is restart the database server. The restart on the database server corrects the issue and now everything is functioning fine.

So now they have contained the issue. They are not 100% sure if they fixed this issue or not, but to keep good communication open they have sent out messages to the company and to customer support and let everybody know that for now they feel like they fixed the issue, but they are not 100% sure if it is going to come back or not, and they are continuing to investigate what is going on.

After a half hour of making sure that the systems are running fine and looking at all the data, they figure that at this point in time they have contained and eradicated the issue by restarting the database. They are still a little unsure what is happening, but they are still pouring over the logs and still pouring over the data to figure out what the root cause of this is.

From a recovery standpoint, customer support is already contacting the customers and letting them know that they feel like they fixed the issue, but that they are not 100% sure what the issue was and are still investigating to make sure that there is a long-term fix in place.

The Root Cause

After further investigation they realized that the database CPU issue was due to a stored procedure. What it was is that the application was calling that stored procedure, and it was calling it in an incorrect way, and they traced it down to some sort of code that was on the application as being the root cause of all of this. The reason why the restart worked on this database server is because it terminated that process, and when it came back up then everything was fine. So fixing the code is the long-term solution here.

Lessons Learned

There were quite a few lessons learned here. They scheduled a postmortem meeting, a meeting after this incident, and they discussed all the details of this incident.

One of the changes they decided to make is that they could be called and alerted a little sooner, so they decided to make this trigger a little more sensitive and move it a little bit earlier in the process so they could start investigating earlier.

What they realized is that it took a team effort to fix this particular issue. The system administrator needed to work on the web server, the database administrator needed to work on the database server, and the developers needed to work on the code. So it took a team to really figure out what the issue was. What they decided to do was change a little bit of their processes to include the whole team earlier on in this process, so they could start analyzing and troubleshooting sooner.

This also prompted a technology change, a change to the code on the application server, and so they are going to put that in as a long-term fix that is going to go through development to fix that issue on the application server, on that code.

All these changes went into an after action report. The after action report summarizes everything that happened during this exchange, to include things that we are going to fix, both the processes and the technology. We also included the timeline of events and everything that happened.

About TechKnowSurge

TechKnowSurge builds IT and cybersecurity professionals through hands-on, concept-first training built around real understanding — not memorization. Free interactive tools, structured programs, and 25+ years of real-world experience, all in one place.

Explore free tools and programs →