TechKnowSurge

NIST 800-53 AU-11 NIST 800-53 AU-4 ISC2 CISSP 2.4 ISC2 CISSP 2.5

VideoSecurityFree

Log Management and Archiving

Managing log data at scale requires balancing system performance, storage capacity, compliance retention requirements, and the secure disposal of personally identifiable information. Log rotation, archiving, and structured disposal policies are core practices for keeping logging infrastructure reliable and audit-ready.

⚑ Complete this video to capture a CTF flag worth 1 point.

About this video

When multiple systems generate logs simultaneously, the sheer volume of data can become a serious operational challenge. Storage capacity fills quickly, system performance degrades, and navigating large log datasets becomes increasingly difficult. Because different systems handle retention in different ways—some storing logs indefinitely, others capping storage by record count or time period—organizations cannot rely on default behaviors alone to manage log data responsibly. Log rotation is one technique used to address this problem, cycling out older log entries to make room for new ones and prevent any single system from becoming overwhelmed. However, rotation alone does not satisfy compliance obligations. Many regulatory frameworks require organizations to retain logs for a defined period, often a year or more, which means deleted logs still need to exist somewhere. Archiving addresses this by moving older logs out of active systems like an ELK stack and into separate storage, preserving them for security investigations, audits, or incident response without taxing production infrastructure. Disposal is the third dimension of log lifecycle management and carries its own compliance considerations. Logs frequently contain personally identifiable information such as IP addresses, and many regulatory and privacy frameworks prohibit retaining PII longer than it is operationally necessary. Organizations must define clear disposal schedules to ensure that sensitive log data is removed in a timely, documented manner. A complete log management strategy accounts for all three stages—rotation, archiving, and disposal—to keep systems performant, audits defensible, and data handling practices legally sound.

What you'll learn

Explain the concepts of log rotation, log archiving, and log disposal, including their roles in managing storage capacity, meeting compliance retention requirements, and protecting personally identifiable information.

What's covered

Log Management & Cleanup

Log Storage Challenges
- Too Many Logs
  - System Performance Impact
  - Storage Concerns
- Storage Retention Variations
  - Indefinite Storage
  - Record Count Limits
  - Time-Based Retention
  - Storage Size Limits
Log Rotation
- Cycling Out Old Logs
  - Makes Room for New Logs
- ELK System Example
  - Limited to Few Months Data
Log Archiving
- Compliance Retention Requirements
  - One Year Minimum Example
- Move Logs to Alternate Location
  - Backup Copy for Incidents
Log Disposal
- PII in Logs
  - IP Addresses as PII
- Dispose After Use Period

Aligned to

NIST 800-53

AU-11 Audit Record Retention

AU-4 Audit Log Storage Capacity

ISC2 CISSP

2.4 Manage data lifecycle

2.5 Ensure appropriate asset retention

Key terms

Log Management: The process of collecting, storing, analyzing, and monitoring log data generated by systems and applications.
Log Rotation: The process of cycling out old log files to make room for new ones, preventing systems from becoming overloaded with log data.
Log Archiving: The practice of moving older log files out of active systems into separate storage to meet retention requirements while freeing operational capacity.
Log Disposal: The deliberate deletion or destruction of log data after its required retention period has ended, particularly to remove sensitive information such as PII.
Personally Identifiable Information: Personally Identifiable Information is any data that can be used alone or in combination to identify, contact, or locate an individual, requiring protection under privacy laws and organizational security policies.
Confidentiality: The principle that information is accessible only to those authorized to access it.

Topics

Log Management Log Rotation Log Archiving Data Retention Compliance Pii Protection Cybersecurity

Transcript

Now since we've got all of these different systems that are logging, one issue that we can have is we can have so many logs it becomes very cumbersome to work with them, to deal with them. It could bog down our systems, it can cause a lot of problems as far as storage is concerned. So we might need to do some cleanup of these logs.

How Long Systems Store Logs

Systems store different logs for a certain period of time. Some of them will store indefinitely, while others will store, once they get a certain number of records, they'll start erasing the oldest records. While others keep it for a certain period of time, while others have a certain storage limit. So they each have a different level of logging that it can store, and a level of how much will be stored in these different systems.

Log Rotation and Archiving

Now one thing we need to maybe do is some sort of log rotation. That is, some of these systems can get bogged down quite a bit. For instance, in my monitoring, the log aggregation that we had really bogged down if we got too many logs in it. We would use ELK, and the system that we had could only hold a few months worth of ELK data, because there was tons of logs going in it. So what we had to do is we had to cycle out some of the old logs, and so that way it'd make room for the new logs.

But we also had certain retention requirements, so from a compliance standpoint we had to keep a year's worth of logs. So one thing that we did is we archived those logs. We took them out of the ELK system and we stored them in another location, and now we have a backup copy of all of the logs. So if we need to do a restore and research some sort of security incident, we could do that, and we had to do that for compliance reasons.

Disposal of Logs

So we've got to think about log archiving and how we can archive things to store it for a certain period of time. But we also have to think about disposal, and that is, there are times when there is certain information in those logs. For instance, a lot of entities will consider IP addresses as being personally identifiable information, and you shouldn't keep PII information for any longer than you need to have it, for any longer than you need to use it. So at that point in time we need to make sure we dispose of those logs.

Read the full transcript Show less

About TechKnowSurge

TechKnowSurge builds IT and cybersecurity professionals through hands-on, concept-first training built around real understanding — not memorization. Free interactive tools, structured programs, and 25+ years of real-world experience, all in one place.

Explore free tools and programs →