/etc/environment is not read by systemd, and I believed it was for a month

UV_USE_IO_URING=0 was supposed to be the fix. In late August the Flux Cumulus fleet running on Ubuntu 22.04 picked up a nasty failure mode on Node 1: npm install entering uninterruptible sleep, refusing to die, and taking the node from CONFIRMED to expired inside a few hours. The kernel journal named the io_uring submission-queue polling thread:

[Fri Aug 21 08:22:44 2026] INFO: task iou-sqp-1334:1346 blocked for more than 120 seconds.
[Fri Aug 21 08:22:44 2026]       Not tainted 5.15.0-187-generic #197-Ubuntu
[Fri Aug 21 08:22:44 2026] task:iou-sqp-1334    state:D stack:    0 pid: 1346 ppid: 1333 flags:0x00004000
[Fri Aug 21 08:24:45 2026] INFO: task iou-sqp-1334:1346 blocked for more than 241 seconds.

Same task, same PID, still blocked two minutes later. state:D — uninterruptible sleep. SIGKILL does nothing to a task blocked inside the kernel. ppid: 1333 — the parent is the npm install shell.

Node 20's libuv routes async filesystem work through io_uring, and npm install is almost entirely async filesystem work. On this kernel that combination produced the deadlock captured above. UV_USE_IO_URING=0 tells libuv not to use io_uring — the variable is documented in Node's CLI reference, implemented by a dependency, with no stability guarantees attached. Set it before Node starts and the path that hangs never runs. I did not chase the kernel-side root cause: a reproducible trigger and a workaround that eliminated it was enough at the time.

I set it in /etc/environment on every node and verified it was in place.

For most of the next month, it was never in effect at boot on any node.

// the verification that proved the wrong thing

I verified the fix in August by reading the pm2 God Daemon's environment directly out of /proc. pm2 kill && pm2 list to force the daemon to restart, then sudo cat /proc/$(pgrep -f "God Daemon" | head -1)/environ | tr '\0' '\n' | grep UV_USE. The variable was there. The daemon had it.

The check was correct. What it proved was not what I thought.

/etc/environment is read by PAM for login sessions. When I ran pm2 kill && pm2 list in my SSH session, PAM had already given the shell the variable. The daemon pm2 list spawned was a child of that shell — it inherited the environment from its parent. My /proc check then read the child's environment and correctly reported the variable was present.

The daemon inherited from the shell, not from any durable configuration. It was a fact about a process that would not exist after the next reboot. The verification passed honestly, and it proved the wrong thing, because the object under test had been created by the act of testing.

That's the lead beat of this piece: a verification that passes honestly and proves the wrong thing, because the object under test was created by the act of testing. Everything downstream is either mechanism or consequence.

// what actually pulled the trigger

The other thing worth knowing is what triggered Node 1's August incident. The deadlock needed something to cause npm install to run. That something was in the standard FluxNode crontab from Step 7 of the deployment guide: @reboot sleep 120 && pm2 restart flux. A fixed 120-second timer, set to fire into the middle of npm install on every single boot.

Not bad luck. A scheduled collision on a loop, with an unkillable process at the end of it.

The timer wasn't Node 1 specific. Every node in the fleet ran the same line. Why only Node 1 hit the deadlock is plausible inference — it was the only node with a genuine long-running npm install for the timer to land in the middle of during that window — but that's inference, not finding. The cron entry itself was universal.

That cron entry is now removed; a */10 DOS-state monitor covers restart recovery instead.

// why /etc/environment never reached the daemon

At the next reboot after that August verification, systemd resurrected the pm2 daemon. The generated unit's environment block set exactly two variables:

Environment=PATH=/home/flux/.nvm/versions/node/v20.9.0/bin:...
Environment=PM2_HOME=/home/flux/.pm2
ExecStart=/home/flux/.nvm/.../pm2/bin/pm2 resurrect

PATH and PM2_HOME. No UV_USE_IO_URING.

The mechanism: systemd's system manager doesn't read /etc/environment. PAM does, for login sessions. systemd does not. A daemon resurrected by systemd at boot inherits exactly what the unit file declares — which in this case was two variables that did not include the one I was relying on.

That's the kind of thing you either know or you don't. I didn't.

// the fix

The fix is a systemd drop-in. On every node:

sudo mkdir -p /etc/systemd/system/pm2-flux.service.d/
sudo tee /etc/systemd/system/pm2-flux.service.d/override.conf > /dev/null <<'EOF'
[Service]
Environment=UV_USE_IO_URING=0
EOF
sudo systemctl daemon-reload

Drop-ins under /etc/systemd/system/ survive unit regeneration and application updates. The variable is now part of the unit's declared environment, so the daemon inherits it whether it was started by hand or resurrected at boot.

No restart is needed. The drop-in applies at the next boot — which is exactly when the workaround is needed. Applied to all seven nodes on 10–11 September, no node leaving CONFIRMED during the change.

Worth noting the fleet runs Node v20.9.0. From 20.11.1 onward io_uring is opt-in rather than opt-out, so a reader on a current Node likely isn't exposed to this at all — the drop-in is protection for the version we're actually on.

// exposed, never bitten

With the fix in place, I swept the kernel journals across every retained boot on all seven nodes, counting iou-sqp and hung-task events:

for b in $(sudo journalctl -k --list-boots | awk '{print $1}'); do
  printf '%3s  %s\n' "$b" \
    "$(sudo journalctl -k -b "$b" --no-pager 2>/dev/null | \
       grep -c 'iou-sqp\|blocked for more than')"
done

Node 1 showed 67 hung-task reports across five consecutive boots in a three-hour window on 21 August, then zero — including the boot that spanned the entire unprotected month. The rest of the fleet: zero across every retained boot.

The gap cost nothing. But that isn't the framing. The @reboot timer was live on every node for the whole month. What kept the fleet from a repeat wasn't the missing workaround — it was that nothing landed in the middle of an npm install window on any other node in that time. Exposed, never bitten.

The rack power migration piece landed on a rhyming lesson about a different safety measure — an autostart flag disabled for a reason, then quietly missing for weeks after that reason no longer applied. The pattern is the same: nominal protection, silent absence, no cost until an unrelated event exposes both.

// the truncation trap

Immediately after writing the drop-in on the first node, I ran the verification I should have run in August:

systemctl show pm2-flux.service -p Environment

It returned a single line, truncated at terminal width, with a trailing > and the pager sitting at (END). All I could see was PATH. The drop-in had worked; the output couldn't show it.

Environment is emitted as one space-separated line, and any pager that folds long lines will truncate it. The fix is trivial: systemctl show pm2-flux.service -p Environment --no-pager | tr ' ' '\n' | grep UV_USE. Cost of the trap: about a minute of thinking I had to rewrite a drop-in that was already correct.

The instrument lying twice in one investigation: once at a cost of a month, once at a cost of a minute. Both times the tool did exactly what it was asked. Both times the framing of the question was where the mistake sat.

// one more thing from the same three commands

The same systemctl show output also revealed the unit's ExecStart: pm2 resurrect, restoring the process list from ~/.pm2/dump.pm2. Anyone applying the older /etc/environment fix pattern — pm2 kill to force the daemon to inherit the variable — walks into a related trap: kill without saving first, and the resurrect at boot restores an empty process list. The node boots with no application.

pm2 save --force after any kill. One command, one sentence.