Keep logs from filling the disk

Logs are the most common cause of a full disk, and the fix is boring configuration nobody applies until after the first incident. Apply it before.

1. Find out what the journal is currently using

journalctl --disk-usage
ls -lh /var/log/journal/*/ | head -n 5

By default the journal will grow to 10% of the filesystem and only stops at 15% free. On a 20GB root volume that is 2GB of logs before anything intervenes — usually fine, occasionally the whole problem.

2. Cap it explicitly

sudo mkdir -p /etc/systemd/journald.conf.d
sudo tee /etc/systemd/journald.conf.d/99-limits.conf >/dev/null <<'CONF'
[Journal]
Storage=persistent
SystemMaxUse=200M
SystemKeepFree=500M
SystemMaxFileSize=20M
MaxRetentionSec=2week
CONF
sudo systemctl restart systemd-journald
journalctl --disk-usage

Four numbers with different jobs. SystemMaxUse is the ceiling. SystemKeepFree is the floor the journal leaves for everyone else, and it wins when the two conflict. SystemMaxFileSize bounds an individual file so rotation happens in reasonable chunks. MaxRetentionSec discards by age regardless of size — worth setting for a retention policy you have to defend.

3. Force a vacuum and watch it work

for i in $(seq 1 5000); do
  logger -t volumetest "filler line $i padding padding"
done
journalctl --disk-usage
sudo journalctl --vacuum-size=50M
journalctl --disk-usage
sudo journalctl --vacuum-time=1h
journalctl --disk-usage

--vacuum-size and --vacuum-time are the manual versions of the config, and they are what you run during an incident when the disk is full right now and you cannot wait for rotation.

4. Rotate an application log with logrotate

The journal only covers services logging to it. An app writing its own file needs logrotate.

sudo mkdir -p /var/log/labapp
sudo sh -c 'for i in $(seq 1 20000); do
  echo "$(date -Is) labapp request id=$i" >> /var/log/labapp/app.log
done'
ls -lh /var/log/labapp/

sudo tee /etc/logrotate.d/labapp >/dev/null <<'CONF'
/var/log/labapp/*.log {
    daily
    rotate 7
    maxsize 1M
    missingok
    notifempty
    compress
    delaycompress
    copytruncate
    create 0640 root adm
}
CONF

Two directives deserve attention. maxsize 1M rotates on size even before the daily schedule, which is what protects you from a burst. copytruncate copies the file and truncates the original in place, so a process holding the descriptor keeps writing to the same inode — without it, an app that does not reopen its log writes into a deleted file forever, and its space is never reclaimed.

5. Prove the rotation happens

sudo logrotate --force /etc/logrotate.d/labapp
ls -lh /var/log/labapp/
sudo cat /var/lib/logrotate/status 2>/dev/null | grep labapp || \
  sudo cat /var/lib/logrotate.status | grep labapp

You now have app.log (fresh) plus a rotated copy. --force ignores the schedule; --debug shows what would happen and changes nothing. The status file is where logrotate remembers when it last ran — if rotation "stopped working", that file being unwritable is a common cause.

Clean up

sudo rm -f /etc/logrotate.d/labapp
sudo rm -rf /var/log/labapp
sudo rm -f /etc/systemd/journald.conf.d/99-limits.conf
sudo systemctl restart systemd-journald

Where this goes next

The disk is safe. Next: the request that works from one host and times out from another.