Why tmux kept dying — OOM resilience

The symptom

tmux would, every so often, vanish entirely — not one session, all of them at once. You'd come back to no server running on /tmp/tmux-0/default and every window/pane gone. Reattaching was impossible because the server process itself was dead.

That last detail is the tell. A tmux server hosts every session in one process. If a single session crashed you'd lose that session; losing all of them means the server died. So the question was never "why does my session close" — it was "what keeps killing the server process."

The root cause (two layers)

Layer 1 — the box runs out of memory. journalctl showed the kernel OOM (out-of-memory) killer firing 33 times in 14 days. The machine has 7.8 GiB RAM but only a 512 MiB swapfile that was ~100 % full — effectively zero breathing room. When a heavy node process (a build/test) spiked to ~1.4 GB resident, the kernel had nowhere to page memory out to, so it invoked the OOM killer to free RAM by killing a process.

Layer 2 — systemd amplified one kill into a total wipeout. This is the part that turned "a build got killed" into "every session died." Modern Linux runs processes inside systemd cgroup units/scopes. The tmux session tree lived inside a transient scope, and the system-wide default was:

DefaultOOMPolicy=stop

OOMPolicy=stop means: if any process in this unit is killed by the OOM killer, stop the entire unit. So when the kernel reaped the runaway node, systemd reacted by tearing down the whole scope — SIGTERM then SIGKILL to everything in it, including the tmux server. The journal showed it verbatim:

tmux-spawn-…scope: A process of this unit has been killed by the OOM killer.
tmux-spawn-…scope: Stopping timed out. Killing.
tmux-spawn-…scope: Killing process … (bash) with signal SIGKILL.
tmux-spawn-…scope: Failed with result 'oom-kill'.

The runaway process was the intended victim. The tmux server was collateral of the stop policy.

The fix — four parts

1. Real memory headroom: an 8 GB swapfile

The single biggest lever. Swap is the kernel's overflow space; with only a tiny, full 512 MB swap, any spike went straight to OOM. We replaced it with a fully-allocated 8 GB swapfile and made it persistent:

dd if=/dev/zero of=/swapfile bs=1M count=8192   # NOT fallocate — see gotcha
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile
# /etc/fstab:
/swapfile none swap sw 0 0

Gotcha — XFS + swapfiles: the root filesystem here is XFS. A fallocated file on XFS can have non-contiguous extents that swapon rejects ("swapfile has holes"). Using dd writes a real, fully-allocated file that swapon always accepts. The old swap also wasn't in /etc/fstab at all — it was only runtime-tracked from a manual swapon, so it wouldn't have survived a reboot. The new one is in fstab, which systemd-fstab-generator turns into a swapfile.swap unit at boot.

2. An OOM-resilient tmux server (the real bug fix)

Layer 1 makes OOM rare; this makes OOM non-catastrophic for tmux. We give the server its own systemd unit with OOMPolicy=continue:

# /etc/systemd/system/tmux.service
[Service]
Type=forking
Environment=HOME=/root
ExecStart=/usr/bin/tmux new-session -d -s main
ExecStop=/usr/bin/tmux kill-server
Restart=always
OOMPolicy=continue     # <-- the fix
KillMode=process

OOMPolicy=continue is the opposite of the default: if a process in this unit is OOM-killed, do nothing to the unit. So the kernel reaps the runaway pane process, and the tmux server plus every other session keep running.

Because tmux uses one server per socket and this one owns the default socket (/tmp/tmux-0/default), your normal commands are unchanged — tmux new -s dev, tmux attach -t dev connect to this protected server automatically. Bonus: since the server is owned by a system service rather than a login session, it also survives SSH logouts.

Why we deliberately did NOT set OOMScoreAdjust. The tempting move is to give tmux a strongly negative OOMScoreAdjust so the kernel never picks it. But oom_score_adj is inherited by child processes at fork — and tmux forks every pane as a child. Protecting the server would protect every runaway build running inside it, so the kernel would go kill something else (mongod, docker, the GUI) instead of the actual glutton. We want the runaway to be the victim. The tmux server is tiny (~1 MB RSS), so the kernel's heuristic never picks it anyway. The correct fix is the policy (continue), not the score.

Why the keep-alive session. A tmux server with zero sessions exits immediately. ExecStart=… new-session -d -s main keeps one session alive so the server (and thus Type=forking's main PID) persists.

3. Trim the idle desktop

This is a headless server, yet it booted to graphical.target and ran an idle lightdm + Xorg + pulseaudio + gnome-keyring + gvfs stack (~292 MB) — which was itself repeatedly OOM-killed and respawned, generating much of the OOM noise. RDP (:3389) is UFW-blocked externally anyway, so nothing reached it. We reclaimed it:

systemctl set-default multi-user.target
systemctl stop lightdm && systemctl disable lightdm
systemctl stop user@112.service   # the lightdm user's lingering session

The xrdp package is left installed (reversible: systemctl set-default graphical.target).

4. A graceful early-OOM guard: earlyoom

Belt-and-suspenders. earlyoom watches free memory and, at a low-memory threshold (default 10 % RAM / 10 % swap free), cleanly SIGTERMs the single biggest consumer before the kernel's hard OOM killer fires. That avoids the kernel's blunter, freeze-prone OOM path entirely.

apt-get install -y earlyoom && systemctl enable --now earlyoom

How to verify / operate

swapon --show                                  # 8G swapfile present
systemctl show tmux.service -p OOMPolicy        # OOMPolicy=continue
tmux ls                                         # server alive on default socket
systemctl is-active earlyoom                    # active
systemctl get-default                           # multi-user.target
journalctl --since "1 day ago" | grep -c "invoked oom-killer"   # should trend to ~0

How the fix was proven

Rather than wait for a real OOM, the mechanism was tested in a contained, cgroup-local way: two transient units, each with MemoryMax=64M, MemorySwapMax=0, running a memory bomb (gets OOM-killed) plus a long-lived survivor process — with no global memory impact.

Unit policy	Outcome when the bomb was OOM-killed
OOMPolicy=stop (old default)	active=failed, result=oom-kill — whole unit torn down, survivor killed (the tmux bug)
OOMPolicy=continue (new)	active=active, result=success — survivor lived (sessions survive)

That is the difference between losing every session and losing only the runaway build.

The one-paragraph takeaway

tmux wasn't buggy. The box was memory-starved (tiny full swap + a useless idle desktop), and systemd's default OOMPolicy=stop turned every OOM kill into a full tmux-scope teardown. Give the kernel headroom (8 GB swap), run tmux under OOMPolicy=continue so a reaped runaway doesn't take the server with it, remove the idle desktop, and add earlyoom as a graceful guard.