The Silent Log Rotation Failure Most Admins Miss (Northpoint’s Fix)

Log rotation is a cornerstone of system administration, yet one subtle misconfiguration causes logs to vanish silently, leaving teams blind to critical events. This article reveals the exact failure mode—where logrotate deletes logs before services can reopen file handles—and provides Northpoint’s proven fix. We explain why traditional approaches fail, compare three rotation strategies, and give a step-by-step guide to implementing copytruncate with proper signaling. This overview reflects widely shared professional practices as of May 2026; verify critical details against current official guidance where applicable.

Understanding the Silent Log Rotation Failure

Many administrators assume that once logrotate is configured, logs are safely managed. However, a common oversight occurs when a service writes to a log file that logrotate compresses and removes without the service reopening its file handle. The service continues writing to the old inode, which now points to a deleted file, so new log entries are lost. This is the silent failure: logs appear to rotate normally, but critical data disappears. The root cause is a mismatch between logrotate’s default behavior and how the service handles file descriptors. Most services, like Apache or Nginx, open log files at startup and keep them open; they do not automatically detect file rotation. Without a post-rotation signal (e.g., SIGHUP) or the copytruncate directive, the service writes to a ghost file. In one composite scenario, a team noticed that their application’s logs stopped growing after rotation, but they only discovered the data loss during an audit weeks later. The failure is silent because logrotate’s exit status is often ignored, and no alert is triggered. Understanding this mechanism is the first step to preventing it.

How File Descriptors Cause Data Loss

When logrotate renames or deletes a log file, the inode changes. The service, holding the original file descriptor, continues writing to the old inode. If that inode is unlinked, the data goes into a void. The copytruncate directive avoids this by copying the file and truncating the original, keeping the same inode. This is the fix Northpoint recommends for services that cannot be easily signaled.

Composite Scenario: The Ghost Log

A development team observed that their error logs contained entries only up to midnight, then nothing. They assumed rotation worked. In reality, logrotate had compressed the file, but the Java application never received a signal. The team lost three days of error data. This scenario illustrates the critical need for proper configuration.

Why Default Configurations Are Risky

Many default logrotate configurations use 'rotate' and 'compress' without 'copytruncate' or 'postrotate'. This works only for services that reopen logs on SIGHUP. For services that do not, logs vanish. Administrators must audit each service’s behavior.

Why Traditional Rotation Methods Fail

Traditional log rotation methods, such as the default 'create' directive, rely on renaming the old log and creating a new file with the same name. This works well for services that close and reopen log files on each write, like syslog. However, most modern services, especially those using long-lived processes (e.g., web servers, application servers, databases), keep file descriptors open indefinitely. When logrotate performs a rename, the service continues writing to the old file descriptor, which now points to the renamed (and soon-to-be-compressed) file. The new log file remains empty. The 'create' directive then creates a new empty file, but the service never writes to it. Over time, the old file is compressed and deleted, and all log entries written after rotation are lost. This failure is exacerbated by delayed compression: logrotate may compress the old log only after the next rotation, further confusing administrators. In a typical project, a team configured logrotate with 'rotate 7' and 'compress' for their Node.js application. After a week, they noticed the current log file was empty, while the compressed archives contained only a few hours of data each. They had lost nearly six days of logs. This scenario highlights the need for a method that preserves the original file descriptor.

The 'create' Directive Pitfall

The 'create' directive tells logrotate to create a new log file after renaming the old one. But if the service never opens the new file, it remains empty. This is a common mistake: administrators assume the service will automatically detect the new file. Rarely does this happen without explicit signaling.

Signaling Issues with SIGHUP

Some services require a SIGHUP signal to reopen log files. However, not all services handle SIGHUP correctly, and the signal may be ignored or cause unintended side effects. For example, a misconfigured SIGHUP handler could restart the service, causing downtime. This makes signaling an unreliable solution.

Copytruncate as a Reliable Alternative

The 'copytruncate' directive works by copying the current log file to the archive name and then truncating the original file to zero length. Because the file’s inode never changes, the service continues writing seamlessly. This method avoids the need for signaling and ensures no data loss. Northpoint’s fix uses copytruncate as the default for all services that do not explicitly require a different method.

Northpoint’s Fix: The Copytruncate Strategy

Northpoint’s approach to preventing silent log rotation failure centers on the 'copytruncate' directive, combined with careful consideration of timing and compression. The core idea is to never change the inode of the active log file. Instead, logrotate copies the current contents to a rotated archive and then truncates the original file. The service, still holding the original file descriptor, writes to the same inode, which now has zero length. New log entries fill the file as usual. This method works with any service that writes to a file descriptor, regardless of whether it supports signals. Northpoint recommends this as the default for most services, except those that explicitly require the 'create' method (e.g., syslog). However, copytruncate has a small race condition: between the copy and truncation, a few log lines may be lost. For most applications, this is acceptable because the loss is limited to a few bytes. To minimize this, Northpoint suggests using the 'delaycompress' directive to postpone compression until the next rotation, reducing the window of data loss. Additionally, setting 'rotate' to a higher value keeps more archives available. In practice, Northpoint’s fix has been adopted by several teams who previously suffered from silent data loss. One team reported that after switching to copytruncate, they never lost a single log line again. The fix is simple to implement and requires no changes to the service configuration.

Step-by-Step Configuration

To implement Northpoint’s fix, edit the logrotate configuration file for your service (e.g., /etc/logrotate.d/nginx). Add the 'copytruncate' directive, along with 'delaycompress' and a reasonable 'rotate' count. Example: /var/log/nginx/*.log { daily copytruncate delaycompress rotate 30 compress missingok notifempty }.

When to Avoid Copytruncate

Copytruncate is not suitable for services that require atomic file operations, such as those that read the log file while it is being written. In such rare cases, the 'create' directive with proper signaling may be necessary. However, for the vast majority of services, copytruncate is safe and effective.

Testing Your Configuration

After applying the fix, test by running logrotate in debug mode: logrotate -d /etc/logrotate.d/yourconfig. Check that the 'copytruncate' directive is listed. Then force a rotation with 'logrotate -f' and verify that the service continues writing to the same file without interruption.

Comparing Three Rotation Approaches

Administrators often face a choice between three primary log rotation methods: the default 'create' method, the 'copytruncate' method, and manual rotation via scripts. Each has distinct trade-offs in terms of data loss risk, complexity, and service compatibility. The default 'create' method is simple to configure and works well with services that reopen logs on each write (e.g., syslog, rsyslog). However, it fails with services that hold file descriptors open, leading to data loss. The 'copytruncate' method avoids data loss by keeping the same inode, but it introduces a small race condition that can lose a few bytes during the copy-truncate operation. It also consumes more I/O because it copies the entire file. Manual rotation scripts offer full control but require custom code and careful handling of signals. They are often used in legacy systems or when specific business logic is needed. Below is a comparison table summarizing key factors:

For most modern infrastructures, copytruncate offers the best balance of safety and simplicity. Manual scripts are reserved for special cases where atomic operations are required. The default 'create' method should only be used when you are certain the service handles log reopening.

When to Choose Each Method

Choose 'create' for syslog, rsyslog, and other services that flush logs frequently. Choose copytruncate for web servers, application servers, and databases. Choose manual scripts when you need to integrate with custom monitoring or enforce specific retention policies.

Real-World Performance Impact

In a composite scenario, a team running a high-traffic Nginx server measured the I/O impact of copytruncate. With a 100MB log file rotating daily, the copy operation took about 2 seconds and used moderate disk I/O. The team considered this negligible compared to the cost of lost log data. They also noted that delaycompress spread the compression load, reducing peak I/O.

Step-by-Step Implementation Guide

Implementing Northpoint’s fix requires careful attention to configuration details. Follow these steps to ensure your logs rotate safely:

1. Identify all log files managed by logrotate. Run 'logrotate -d /etc/logrotate.conf' to list configurations. Note which services use which files.
2. For each service that uses long-lived processes (e.g., nginx, apache, tomcat, java apps), edit the corresponding file in /etc/logrotate.d/. Add 'copytruncate' after the rotation frequency line. Also add 'delaycompress' to postpone compression until the next rotation.
3. Set an appropriate 'rotate' count. For example, 'rotate 30' keeps 30 days of logs. Adjust based on disk space and retention policy.
4. Ensure 'compress' is present if you need compression. Use 'delaycompress' to avoid compressing the just-rotated file immediately.
5. Remove any 'postrotate' lines that send signals unless the service explicitly requires them. Copytruncate makes signaling unnecessary.
6. Test the configuration with 'logrotate -d /etc/logrotate.d/yourconfig'. Verify that 'copytruncate' appears in the output.
7. Force a rotation with 'logrotate -f /etc/logrotate.d/yourconfig'. Monitor the log file to ensure the service continues writing to it. Check that the old log is archived correctly.
8. Monitor for a few days to confirm no data loss. Compare the size of the current log file with the archived ones. If the current log is empty, the configuration is wrong.
9. Document the change in your runbook so that future administrators understand why copytruncate is used.

By following these steps, you eliminate the silent failure mode and ensure continuous log collection. This guide assumes you have root access and are comfortable editing configuration files.

Common Pitfalls During Implementation

One common mistake is forgetting to add 'delaycompress'. Without it, logrotate compresses the copied file immediately, which can cause a race condition if the service writes during compression. Another pitfall is using 'copytruncate' with services that require the file to be rotated atomically, such as some database systems. In such cases, test thoroughly before deploying.

Verification Checklist

After implementation, run this checklist: (1) Current log file is non-zero and growing. (2) Archived logs contain data up to the rotation time. (3) No errors in /var/log/messages or logrotate’s own log. (4) Service logs show no gaps. (5) Disk usage remains within bounds.

Real-World Examples and Lessons Learned

Composite scenarios from multiple organizations illustrate the impact of silent log rotation failure. In one case, a financial services company discovered that their transaction logs had been silently truncated for three months due to a misconfigured logrotate. The default 'create' method was used for their Java application, which never reopened the log file. The team only noticed when an audit revealed missing records. After switching to copytruncate, they recovered full log continuity. Another scenario involved a SaaS provider whose Nginx logs stopped rotating properly after an upgrade. The upgrade changed the service’s behavior, and the existing logrotate configuration no longer worked. They lost two days of access logs, which delayed a security investigation. By adopting Northpoint’s fix, they prevented future incidents. A third example comes from a DevOps team managing a Kubernetes cluster. They used sidecar containers for logging, and the logrotate configuration on the host was not aligned with the container lifecycle. After implementing copytruncate with appropriate delays, they eliminated gaps in container logs. These examples underscore that the silent failure can affect any organization, regardless of scale. The common thread is that the failure is often discovered only after data loss has occurred, making prevention critical.

Composite Scenario: The Three-Month Gap

A mid-sized e-commerce company used Tomcat for their application. Logrotate was configured with the default 'create' method. After three months, a compliance audit required log review. The team discovered that logs from the last three months were incomplete: only the first few hours after each rotation were recorded. They traced the issue to logrotate, which had been rotating correctly, but Tomcat never reopened the file. The gap cost them significant effort to reconstruct events. After switching to copytruncate, they never faced the issue again.

Composite Scenario: The Upgrade Surprise

A web hosting provider upgraded their Apache server from version 2.2 to 2.4. The upgrade changed how Apache handles log file descriptors. Post-upgrade, logrotate’s 'create' method caused logs to be written to a deleted file. The provider lost two days of logs for hundreds of customers before they identified the cause. Implementing copytruncate resolved the problem.

Composite Scenario: Container Logging Challenges

A DevOps team used Fluentd to collect logs from containers. The host’s logrotate configuration rotated the container log files, but Fluentd held file descriptors open. Data loss occurred regularly. By applying copytruncate on the host and aligning rotation intervals, they achieved seamless log collection.

Common Questions and Answers About Log Rotation

Administrators frequently ask about log rotation best practices. Here are answers to the most common questions:

Q: Does copytruncate cause any data loss? A: Yes, a very small amount—typically a few bytes—may be lost between the copy and truncation. For most applications, this is acceptable. If zero data loss is required, consider using a logging library that rotates internally (e.g., log4j’s rolling file appender).

Q: Can I use copytruncate with syslog? A: It is not recommended because syslog writes many small messages and the race window could lose more data. Syslog typically reopens logs on each write, so the 'create' method is preferred.

Q: How often should I rotate logs? A: It depends on log volume and disk space. Common frequencies are daily, weekly, or when logs reach a certain size (size-based rotation). Northpoint recommends daily rotation with copytruncate for most services.

Q: What is the impact of delaycompress? A: Delaycompress postpones compression until the next rotation. This reduces I/O spikes and ensures the just-rotated file is not compressed while the service might still be writing to it (though copytruncate already prevents that). It also keeps the most recent archive uncompressed for quick access.

Q: Should I remove postrotate scripts entirely? A: If you use copytruncate, postrotate scripts that send signals are usually unnecessary. However, if you have custom scripts that perform other actions (e.g., notify monitoring), keep them but ensure they do not interfere with the file handle.

Q: How do I test log rotation without losing existing logs? A: Use 'logrotate -d' for dry-run. Create a test log file and run logrotate manually with a test configuration. Monitor the file descriptor with 'lsof' to ensure the service writes to the same inode.

Q: What if my service requires atomic moves? A: In rare cases, such as with some database systems, atomic moves are required for consistency. In those cases, use the 'create' method with a proper signal, or use a dedicated logging solution.

Conclusion

Silent log rotation failure is a pervasive problem that can undermine system observability and security. By understanding the mechanism—services writing to deleted file descriptors—administrators can take proactive steps to prevent data loss. Northpoint’s fix, centered on the copytruncate directive, offers a simple, reliable solution that works with virtually any service. Combined with delaycompress and careful testing, it ensures that no log line is ever lost. We encourage every administrator to audit their logrotate configurations today and adopt this fix. Remember, logs are your first line of defense in troubleshooting and forensics; don’t let them disappear silently. This guide reflects widely shared professional practices as of May 2026; verify critical details against current official guidance where applicable.