Day 07: systemd units

Every "why did the app not come back after the reboot" question ends here. A unit file is about fifteen lines and replaces a hundred lines of init script.

1. Read the state of something already running

systemctl status ssh || systemctl status sshd
systemctl is-enabled ssh || systemctl is-enabled sshd
systemctl is-active ssh || systemctl is-active sshd

Two words that are constantly confused: active means running right now, enabled means it will start at boot. A service can be active and disabled — running now, gone after a reboot — which is the single most common cause of that question.

2. Write a service

sudo mkdir -p /opt/heartbeat
sudo tee /opt/heartbeat/run.sh >/dev/null <<'SCRIPT'
#!/bin/bash
while true; do
  echo "heartbeat $(date -Is)"
  sleep 5
done
SCRIPT
sudo chmod 755 /opt/heartbeat/run.sh

Now the unit:

sudo tee /etc/systemd/system/heartbeat.service >/dev/null <<'UNIT'
[Unit]
Description=Heartbeat lab service
After=network.target

[Service]
Type=simple
ExecStart=/opt/heartbeat/run.sh
Restart=on-failure
RestartSec=2
User=nobody
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target
UNIT

Three lines carry most of the weight. Restart=on-failure is the supervision you would otherwise write yourself. User=nobody means a compromise of this script is not a compromise of root. WantedBy=multi-user.target is what enable hooks into — without an [Install] section, systemctl enable has nothing to do.

3. Load and start it

sudo systemctl daemon-reload
sudo systemctl enable --now heartbeat
systemctl status heartbeat --no-pager

daemon-reload is required after any unit file change — systemd caches them, and forgetting this is why your edit "did nothing".

4. Prove the supervision works

systemctl show heartbeat -p MainPID
sudo kill -KILL "$(systemctl show heartbeat -p MainPID --value)"
sleep 4
systemctl show heartbeat -p MainPID
systemctl status heartbeat --no-pager | head -n 5

The PID changed. You killed it and systemd started it again, because Restart=on-failure and a KILL counts as failure. Compare with a deliberate stop:

sudo systemctl stop heartbeat
sleep 4
systemctl is-active heartbeat

It stays stopped. An administrative stop is not a failure, so on-failure correctly does nothing.

5. Override a unit without editing it

Never edit a file in /lib/systemd/system — a package update overwrites it. Use a drop-in:

sudo systemctl start heartbeat
sudo systemctl edit heartbeat --force --full=false 2>/dev/null || true
sudo mkdir -p /etc/systemd/system/heartbeat.service.d
sudo tee /etc/systemd/system/heartbeat.service.d/override.conf >/dev/null <<'DROPIN'
[Service]
RestartSec=10
DROPIN
sudo systemctl daemon-reload
systemctl show heartbeat -p RestartSec

Drop-ins in /etc win over the shipped unit and survive upgrades. systemctl cat heartbeat shows the merged result, which is what you want when a setting is not taking effect.

Clean up

sudo systemctl disable --now heartbeat
sudo rm -rf /etc/systemd/system/heartbeat.service \
  /etc/systemd/system/heartbeat.service.d /opt/heartbeat
sudo systemctl daemon-reload

Where this goes next

Your service wrote to the journal and you have not read it yet. Tomorrow is entirely about that.