Challenge: the box that filled up at 3am

This is the incident the intermediate labs were preparation for. It combines the disk lab, the logging lab, and the service lab, and it contains the one disk-full symptom that defeats people the first time they meet it.

Set up the scenario

Run this exactly as written. It builds a small volume, starts a service on it, and creates the fault.

sudo mkdir -p /srv/incident
sudo fallocate -l 512M /srv/incident/disk.img
LOOP=$(sudo losetup --find --show /srv/incident/disk.img)
sudo pvcreate -y "$LOOP" >/dev/null
sudo vgcreate incvg "$LOOP" >/dev/null
sudo lvcreate -y -n incdata -l 100%FREE incvg >/dev/null
sudo mkfs.ext4 -q /dev/incvg/incdata
sudo mkdir -p /var/lib/incident
sudo mount /dev/incvg/incdata /var/lib/incident
echo "LOOP=$LOOP  — write this down"

sudo tee /opt/chatty.sh >/dev/null <<'SCRIPT'
#!/bin/bash
LOG=/var/lib/incident/chatty.log
while true; do
  head -c 2000000 /dev/urandom | base64 >> "$LOG"
  sleep 1
done
SCRIPT
sudo chmod 755 /opt/chatty.sh
sudo tee /etc/systemd/system/chatty.service >/dev/null <<'UNIT'
[Unit]
Description=Chatty service (fills the disk on purpose)
[Service]
ExecStart=/opt/chatty.sh
Restart=always
[Install]
WantedBy=multi-user.target
UNIT
sudo systemctl daemon-reload
sudo systemctl enable --now chatty >/dev/null
echo "waiting for the disk to fill..."
sleep 90
sudo rm -f /var/lib/incident/chatty.log
df -h /var/lib/incident

The situation

The volume mounted at /var/lib/incident is full or nearly full. The obvious offending file has already been deleted — by a well-meaning colleague at 3am — and yet:

df -h /var/lib/incident
sudo du -sh /var/lib/incident

df says the space is used. du says almost nothing is there. Both are telling the truth.

Your goal

1. Explain why df and du disagree. One sentence.
2. Reclaim the space without rebooting and without unmounting the filesystem.
3. Prove the space came back with df.
4. Stop the service from doing it again, by capping how much it can write rather than by deleting the service. It must still run.
5. Grow the volume to 1GB anyway, online, because the original sizing was wrong.

Constraints

• No reboot. A reboot fixes step 2 by accident and teaches you nothing.
• Do not unmount at any point.
• For step 4, the cap must be enforced by the system, not by editing /opt/chatty.sh. The script is a stand-in for a vendor application you cannot change.
• Step 5 must happen with the filesystem mounted and the service running.

Hints, in increasing order of spoiler

• Step 1: a file has a name and it has an inode. Deleting the name does not free the blocks if something still holds the inode open. lsof has a flag for exactly this — it appeared in the disk lab.
• Step 2: you can either make the holder let go, or truncate what it is holding. systemctl restart is the blunt version; truncate -s 0 /proc/<pid>/fd/<n> is the surgical one. Both are correct answers, and knowing why you would choose each is the point.
• Step 4: the logging lab used logrotate with copytruncate for an app that will not reopen its log. The service lab used cgroup limits. Only one of those caps disk writes for a process writing a single ever-growing file — and think about which layer MemoryMax operates at before you reach for it.
• Step 5: pvcreate, vgextend, lvextend, resize2fs. In that order. The volume group has no free extents, so you need a second loop device first.

What success looks like

Then leave it running for five minutes and check df again. If usage climbs back to 100%, step 4 is not actually enforcing anything.

Clean up

sudo systemctl disable --now chatty
sudo rm -f /etc/systemd/system/chatty.service /opt/chatty.sh
sudo rm -f /etc/logrotate.d/chatty
sudo systemctl daemon-reload
sudo umount /var/lib/incident
sudo lvremove -y /dev/incvg/incdata >/dev/null
sudo vgremove -y incvg >/dev/null
for l in $(losetup -a | grep incident | cut -d: -f1); do
  sudo pvremove -y "$l" >/dev/null 2>&1; sudo losetup -d "$l"
done
sudo rm -rf /srv/incident /var/lib/incident
losetup -a | grep -c incident || echo "0 loop devices left"

Where this goes next

You can hold a Linux box together under pressure. The Cloud Engineering track picks this up from the CLI, and the Docker track shows you what a container is actually doing to the process table you just learned to read.