Why ralli-roadmap returned 502 — supervising a bare node process

ralli-roadmap.oaoisme.top (the live rebuild-progress dashboard) started returning 502 Bad Gateway. This is the story of why, and the fix that stops it recurring.

What a 502 actually means here

Nginx terminates TLS and reverse-proxies the host to a local upstream:

ralli-roadmap.oaoisme.top  ──►  proxy_pass http://127.0.0.1:3500;

A 502 is nginx saying "I'm fine, but the thing I proxy to didn't answer." So the fault is never nginx — it's the upstream on :3500. Confirmed in seconds:

ss -ltnp | grep :3500     # → nothing listening
ps aux | grep server.js   # → no roadmap process

The app was simply not running.

Root cause: a bare, unsupervised process

The dashboard is a tiny Express app (/root/work/ralli-roadmap/server.js). It had been started by hand — no systemd unit, no Docker, no pm2. The box runs earlyoom, and during a long, memory-heavy job elsewhere it logged mem avail: 24%. When memory gets tight something gets killed; a hand-started process has nothing to restart it, so it stayed dead and every request became a 502.

The app itself was healthy the whole time — booting it by hand came straight back up (200, and it re-scanned the repo fine). The bug was never in the code; it was the absence of supervision. Contrast its neighbour ralli-backend.service, which has survived the same pressure because systemd keeps restarting it.

The lesson: a service reachable from the internet must be supervised. "I started it in a shell once" is not a deployment — it's a 502 waiting for the next OOM, crash, or reboot.

The fix: a systemd unit (mirroring ralli-backend)

/etc/systemd/system/ralli-roadmap.service:

[Unit]
Description=ralli-roadmap (live rebuild-progress dashboard) — 127.0.0.1:3500
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
WorkingDirectory=/root/work/ralli-roadmap
ExecStart=/usr/bin/node server.js
Environment=NODE_ENV=production
Restart=always
RestartSec=3
OOMScoreAdjust=-400          # earlyoom should pick the dev/test hogs, not this 50MB app
StandardOutput=append:/var/log/ralli-roadmap.log
StandardError=append:/var/log/ralli-roadmap.log

[Install]
WantedBy=multi-user.target

systemctl daemon-reload
systemctl enable --now ralli-roadmap.service

Two properties make this durable:

• Restart=always + RestartSec=3 — a crash, an OOM kill, or any exit is followed by an automatic respawn three seconds later.
• OOMScoreAdjust=-400 — earlyoom/the kernel deprioritise this tiny dashboard as an OOM victim, so the real memory hogs (dev/test runs) get reaped first. Belt and braces: even if it is killed, systemd brings it back.
• WantedBy=multi-user.target + enable — it also survives a reboot.

Proving it actually recovers

Don't trust Restart=always — test it. Kill the managed process and watch systemd respawn:

OLD=$(systemctl show -p MainPID --value ralli-roadmap.service)
kill -9 "$OLD"
sleep 4
systemctl show -p MainPID --value ralli-roadmap.service   # → a NEW pid
curl -s -o /dev/null -w '%{http_code}\n' https://ralli-roadmap.oaoisme.top/   # → 200

New PID, public URL back to 200. The 502 class of outage is closed for this site.

Generalise it

If you ever find a public host 502-ing, the checklist is always the same: it's the upstream, not nginx. grep proxy_pass the vhost to get the port, ss -ltnp | grep :PORT to see if anything's home, and if the answer is "a hand-started process died" — give it a systemd unit with Restart=always rather than starting it by hand again. The second fix is the only one that's actually a fix. See also Why tmux kept dying — OOM resilience for the memory-pressure side of the same coin.