Grow a filesystem that filled up

Every disk-full incident ends one of two ways: you delete something, or you grow the volume. This lab is the second, practised safely — everything happens on loopback files, so there is no real disk to lose.

1. Make two fake disks

sudo mkdir -p /srv/loopdisks
sudo fallocate -l 1G /srv/loopdisks/disk1.img
sudo fallocate -l 1G /srv/loopdisks/disk2.img
LOOP1=$(sudo losetup --find --show /srv/loopdisks/disk1.img)
LOOP2=$(sudo losetup --find --show /srv/loopdisks/disk2.img)
echo "LOOP1=$LOOP1 LOOP2=$LOOP2"
lsblk "$LOOP1" "$LOOP2"

A loop device presents a file as a block device. The kernel treats these exactly like disks, so every command below is the one you would run on real hardware or an EBS volume.

Keep $LOOP1 and $LOOP2 — you need them later. If your shell resets, losetup -a lists them again.

2. Build the LVM stack

sudo pvcreate "$LOOP1"
sudo vgcreate labvg "$LOOP1"
sudo lvcreate -n labdata -l 100%FREE labvg
sudo mkfs.ext4 /dev/labvg/labdata
sudo mkdir -p /mnt/labdata
sudo mount /dev/labvg/labdata /mnt/labdata
df -h /mnt/labdata

Three layers, and the names matter when you are reading someone else's incident notes. PV is the physical volume (a disk). VG is the volume group (a pool of PVs). LV is the logical volume (what you format and mount). Growing a filesystem means adding capacity at whichever layer has run out.

3. Fill it until writes fail

sudo dd if=/dev/zero of=/mnt/labdata/ballast \
  bs=1M count=2000 status=none
echo "dd exit=$?"
df -h /mnt/labdata
sudo sh -c 'echo hello > /mnt/labdata/newfile'

dd stops short and the write fails with "No space left on device". This is the state you are paged for.

Before growing anything, check the thing people forget — inodes:

df -i /mnt/labdata

A filesystem can be 100% full on inodes with free bytes, from millions of tiny files. df -h looks fine and every write fails. Growing the volume does not help; you need -i to see it.

4. Extend it, online, with nothing unmounted

sudo pvcreate "$LOOP2"
sudo vgextend labvg "$LOOP2"
sudo vgs labvg
sudo lvextend -l +100%FREE /dev/labvg/labdata
sudo lvs labvg
sudo resize2fs /dev/labvg/labdata
df -h /mnt/labdata

The order is the whole lesson, and it is bottom-up: add the PV, extend the VG, extend the LV, then grow the filesystem. Miss the last step and lvs shows more space while df shows none — the classic "I extended it and nothing happened".

resize2fs grows a mounted ext4 filesystem in place. For XFS the command is xfs_growfs /mnt/labdata, and XFS cannot shrink at all, ever.

5. Find what actually filled it

The real incident continues here: something is still writing.

sudo du -x -h --max-depth=1 /mnt/labdata | sort -h | tail -n 5
sudo find /mnt/labdata -type f -size +100M -exec ls -lh {} +
sudo lsof +L1 2>/dev/null | head -n 5

du -x stays on one filesystem, so it does not wander into a mount you did not mean to measure. The last command is the one nobody remembers: +L1 lists open files with a link count below one — deleted files still held open by a process. Their space is not returned until the holder exits, which is why du and df disagree and why deleting the log did not help.

Clean up

Reverse order, or the removals fail.

sudo umount /mnt/labdata
sudo lvremove -y /dev/labvg/labdata
sudo vgremove -y labvg
sudo pvremove -y "$LOOP1" "$LOOP2"
sudo losetup -d "$LOOP1" "$LOOP2"
sudo rm -rf /srv/loopdisks /mnt/labdata
lsblk | grep -c loop || echo "0 loop devices left"

Where this goes next

You can grow a volume under pressure. Next: stopping the logs that filled it from doing it again.