Skip to content
Linux Administration
Lab 25 of 27·45mIntermediate

Triage a machine that is up and too slow

Decide whether you are short of CPU, memory, or disk in under five minutes, using load, pressure stall information, and iostat.

You need

  • A Linux VM with sudo access
  • sysstat and stress-ng (apt-get install -y sysstat stress-ng)

Do first

"It is slow" needs to become "it is waiting on X" before you can fix anything. There are three candidates. This lab creates each bottleneck deliberately so you can see what each one looks like.

1. Establish a baseline

nproc
uptime
free -h
vmstat 1 3

Read load against nproc, always. A load of 8 on 16 cores is half-busy; on 2 cores it is a queue four deep. In vmstat, the columns that matter are r (runnable, waiting for CPU), b (blocked on I/O), si/so (swap in/out), and wa (CPU time spent waiting for I/O).

Verify

nproc; uptime | grep -o "load average.*" # core count, then three load numbers to read against it

2. Use pressure stall information first

cat /proc/pressure/cpu
cat /proc/pressure/memory
cat /proc/pressure/io

PSI is the fastest triage on any modern kernel and it is criminally underused. Each line gives the percentage of time _something_ was stalled waiting on that resource, averaged over 10s, 60s and 300s. some means at least one task stalled; full means everything did.

Read all three, take the largest avg10, and you have your bottleneck in one screen — no interpretation of load averages required.

Verify

cat /proc/pressure/io | head -n 1 # some avg10=... avg60=... avg300=... total=...

3. Create CPU starvation and recognise it

stress-ng --cpu "$(nproc)" --timeout 30s &
sleep 10
uptime
cat /proc/pressure/cpu
vmstat 1 3
top -bn1 | head -n 12
wait

The signature: load climbs toward and past nproc, /proc/pressure/cpu avg10 rises, vmstat's r column exceeds the core count, and wa stays near zero. Nothing is waiting on disk — there simply is not enough CPU.

Verify

cat /proc/pressure/cpu | grep -o "avg10=[0-9.]*" # avg10 near 0 once stress-ng has finished

4. Create memory pressure and recognise it

free -h
stress-ng --vm 2 --vm-bytes 40% --timeout 30s &
sleep 10
free -h
cat /proc/pressure/memory
vmstat 1 3
wait

A different signature: free shows available collapsing, /proc/pressure/memory rises, and if swap exists si/so become non-zero. Sustained swap traffic is the worst of the three — everything becomes disk-slow at once.

Read the available column, not free. Linux uses spare memory as page cache by design, so low free with high available is a healthy machine, and "we are out of memory" read off the wrong column is the most common false alarm in this job.

Verify

free -h --si | awk 'NR==2{print "available:", $NF}' # a sizeable figure once stress-ng has finished

5. Create I/O pressure and recognise it

iostat -xz 1 3
stress-ng --hdd 2 --hdd-bytes 512M --timeout 30s &
sleep 10
iostat -xz 1 3
cat /proc/pressure/io
vmstat 1 3
wait

The columns worth reading in iostat -xz: %util (how busy the device is), aqu-sz (average queue depth), and r_await/w_await (milliseconds a request waited). High await with high %util is a saturated device. High await with _low_ %util on cloud storage usually means you have exhausted a provisioned IOPS or throughput allowance — the device is idle because the allowance is spent.

The matching signature elsewhere: vmstat's wa climbs and b is non-zero, while r stays low. Plenty of CPU, all of it waiting.

Verify

iostat -xz 1 2 | tail -n 5 # device rows with %util and await columns

6. Find the process responsible

sudo pidstat -u -r -d 1 3 2>/dev/null | tail -n 20
ps -eo pid,ppid,pcpu,pmem,rss,etime,cmd --sort=-pcpu | head -n 8
ps -eo pid,pcpu,pmem,rss,cmd --sort=-rss | head -n 8
sudo iotop -b -n 2 2>/dev/null | head -n 12 || echo "iotop not installed"

pidstat -u -r -d gives CPU, memory and disk per process in one table, which is the fastest way from "the machine is I/O bound" to "this PID is doing it".

Verify

ps -eo pcpu,cmd --sort=-pcpu | head -n 2 # a header and the current top CPU consumer

The five-minute routine

  1. cat /proc/pressure/{cpu,memory,io} — which resource, in one screen
  2. uptime against nproc — how deep the queue is
  3. free -h, reading available — memory headroom
  4. iostat -xz 1 3 — %util, aqu-sz, await for storage
  5. pidstat -u -r -d 1 3 — which process
  6. dmesg -T | tail — OOM kills and device errors leave evidence here
sudo dmesg -T | grep -i -E "oom|killed process|error" | tail -n 5

Where this goes next

Six guided labs. The last one is a challenge: a machine with a real problem and no indication of which of the three it is.