Day 06: Processes and signals

kill -9 is where most people start and it should be where you end. Understanding the difference between the signals is the difference between a clean shutdown and a corrupt file.

1. Start something to look at

mkdir -p ~/labs/proc && cd ~/labs/proc
sleep 3000 &
sleep 3000 &
jobs
ps -o pid,ppid,stat,etime,cmd -C sleep

& backgrounds the job. ps -C selects by command name, which beats ps aux | grep sleep because it never matches the grep itself.

2. Read a process from /proc

PID=$(pgrep -n sleep)
echo "$PID"
ls /proc/$PID
cat /proc/$PID/cmdline | tr '\0' ' '; echo
ls -l /proc/$PID/cwd
cat /proc/$PID/status | head -n 12

/proc/<pid> is the kernel exposing a live process as a directory. cmdline is the exact argv (null-separated, hence the tr), cwd is a symlink to its working directory, status gives state, memory and the uid it runs as. Every process-inspection tool you have ever used is reading these files.

3. Send signals in increasing severity

PID=$(pgrep -n sleep)
kill -TERM $PID
sleep 1
pgrep -c sleep

kill with no flag sends TERM (15), which asks the process to shut down and lets it clean up. A well-written program flushes buffers and removes its pidfile here. Now the rude one:

PID=$(pgrep -n sleep)
kill -KILL $PID
sleep 1
pgrep -c sleep

KILL (9) is handled by the kernel, not the process — it never gets a chance to run cleanup code. Use it only after TERM has visibly failed.

4. Find the process holding something open

sleep 3000 > ~/labs/proc/held.log &
sudo lsof ~/labs/proc/held.log
sudo fuser -v ~/labs/proc/held.log

This is the answer to "device or resource busy" and to "I deleted the file but the disk is still full". A deleted file whose descriptor is still open keeps its blocks until the holder exits — lsof | grep deleted finds those.

5. See load and priority

uptime
nice -n 19 sleep 300 &
ps -o pid,ni,cmd -C sleep

The three numbers from uptime are load over 1, 5 and 15 minutes: the average count of runnable-or-waiting processes. On a 4-core box a load of 4.0 means fully busy; 20.0 means badly oversubscribed. Compare against nproc, never against a fixed number.

ni is the nice value: −20 is greedy, 19 is generous, 0 is default. Only root can go negative.

Clean up

pkill sleep
cd ~ && rm -rf ~/labs/proc

Where this goes next

You can find and stop a process by hand. Tomorrow: the thing that does it for you on every boot, restarts it when it dies, and is the reason nobody writes init scripts any more.