# /etc/environment is not read by systemd, and I believed it was for a month

`UV_USE_IO_URING=0` was supposed to be the fix. In late August the Flux Cumulus fleet running on Ubuntu 22.04 picked up a nasty failure mode on Node 1: `npm install` entering uninterruptible sleep, refusing to die, and taking the node from CONFIRMED to expired inside a few hours. The kernel journal named the io\_uring submission-queue polling thread:

```plaintext
[Fri Aug 21 08:22:44 2026] INFO: task iou-sqp-1334:1346 blocked for more than 120 seconds.
[Fri Aug 21 08:22:44 2026]       Not tainted 5.15.0-187-generic #197-Ubuntu
[Fri Aug 21 08:22:44 2026] task:iou-sqp-1334    state:D stack:    0 pid: 1346 ppid: 1333 flags:0x00004000
[Fri Aug 21 08:24:45 2026] INFO: task iou-sqp-1334:1346 blocked for more than 241 seconds.
```

Same task, same PID, still blocked two minutes later. `state:D` — uninterruptible sleep. `SIGKILL` does nothing to a task blocked inside the kernel. `ppid: 1333` — the parent is the `npm install` shell.

Node 20's libuv routes async filesystem work through io\_uring, and `npm install` is almost entirely async filesystem work. On this kernel that combination produced the deadlock captured above. `UV_USE_IO_URING=0` tells libuv not to use io\_uring — the variable is documented in Node's CLI reference, implemented by a dependency, with no stability guarantees attached. Set it before Node starts and the path that hangs never runs. I did not chase the kernel-side root cause: a reproducible trigger and a workaround that eliminated it was enough at the time.

I set it in `/etc/environment` on every node and verified it was in place.

For most of the next month, it was never in effect at boot on any node.

## // the verification that proved the wrong thing

I verified the fix in August by reading the pm2 God Daemon's environment directly out of `/proc`. `pm2 kill && pm2 list` to force the daemon to restart, then `sudo cat /proc/$(pgrep -f "God Daemon" | head -1)/environ | tr '\0' '\n' | grep UV_USE`. The variable was there. The daemon had it.

The check was correct. What it proved was not what I thought.

`/etc/environment` is read by PAM for login sessions. When I ran `pm2 kill && pm2 list` in my SSH session, PAM had already given the shell the variable. The daemon `pm2 list` spawned was a child of that shell — it inherited the environment from its parent. My `/proc` check then read the child's environment and correctly reported the variable was present.

The daemon inherited from the shell, not from any durable configuration. It was a fact about a process that would not exist after the next reboot. The verification passed honestly, and it proved the wrong thing, because the object under test had been created by the act of testing.

That's the lead beat of this piece: a verification that passes honestly and proves the wrong thing, because the object under test was created by the act of testing. Everything downstream is either mechanism or consequence.

## // what actually pulled the trigger

The other thing worth knowing is what triggered Node 1's August incident. The deadlock needed something to cause `npm install` to run. That something was in the standard FluxNode crontab from Step 7 of the deployment guide: `@reboot sleep 120 && pm2 restart flux`. A fixed 120-second timer, set to fire into the middle of `npm install` on every single boot.

Not bad luck. A scheduled collision on a loop, with an unkillable process at the end of it.

The timer wasn't Node 1 specific. Every node in the fleet ran the same line. Why only Node 1 hit the deadlock is plausible inference — it was the only node with a genuine long-running `npm install` for the timer to land in the middle of during that window — but that's inference, not finding. The cron entry itself was universal.

That cron entry is now removed; a `*/10` DOS-state monitor covers restart recovery instead.

## // why /etc/environment never reached the daemon

At the next reboot after that August verification, systemd resurrected the pm2 daemon. The generated unit's environment block set exactly two variables:

```plaintext
Environment=PATH=/home/flux/.nvm/versions/node/v20.9.0/bin:...
Environment=PM2_HOME=/home/flux/.pm2
ExecStart=/home/flux/.nvm/.../pm2/bin/pm2 resurrect
```

PATH and PM2\_HOME. No `UV_USE_IO_URING`.

The mechanism: systemd's system manager doesn't read `/etc/environment`. PAM does, for login sessions. systemd does not. A daemon resurrected by systemd at boot inherits exactly what the unit file declares — which in this case was two variables that did not include the one I was relying on.

That's the kind of thing you either know or you don't. I didn't.

## // the fix

The fix is a systemd drop-in. On every node:

```bash
sudo mkdir -p /etc/systemd/system/pm2-flux.service.d/
sudo tee /etc/systemd/system/pm2-flux.service.d/override.conf > /dev/null <<'EOF'
[Service]
Environment=UV_USE_IO_URING=0
EOF
sudo systemctl daemon-reload
```

Drop-ins under `/etc/systemd/system/` survive unit regeneration and application updates. The variable is now part of the unit's declared environment, so the daemon inherits it whether it was started by hand or resurrected at boot.

No restart is needed. The drop-in applies at the next boot — which is exactly when the workaround is needed. Applied to all seven nodes on 10–11 September, no node leaving CONFIRMED during the change.

Worth noting the fleet runs Node v20.9.0. From 20.11.1 onward io\_uring is opt-in rather than opt-out, so a reader on a current Node likely isn't exposed to this at all — the drop-in is protection for the version we're actually on.

## // exposed, never bitten

With the fix in place, I swept the kernel journals across every retained boot on all seven nodes, counting `iou-sqp` and hung-task events:

```bash
for b in $(sudo journalctl -k --list-boots | awk '{print $1}'); do
  printf '%3s  %s\n' "$b" \
    "$(sudo journalctl -k -b "$b" --no-pager 2>/dev/null | \
       grep -c 'iou-sqp\|blocked for more than')"
done
```

Node 1 showed 67 hung-task reports across five consecutive boots in a three-hour window on 21 August, then zero — including the boot that spanned the entire unprotected month. The rest of the fleet: zero across every retained boot.

The gap cost nothing. But that isn't the framing. The `@reboot` timer was live on every node for the whole month. What kept the fleet from a repeat wasn't the missing workaround — it was that nothing landed in the middle of an `npm install` window on any other node in that time. Exposed, never bitten.

The [rack power migration piece](https://jjozzietech.com.au/rack-power-migration/) landed on a rhyming lesson about a different safety measure — an autostart flag disabled for a reason, then quietly missing for weeks after that reason no longer applied. The pattern is the same: nominal protection, silent absence, no cost until an unrelated event exposes both.

## // the truncation trap

Immediately after writing the drop-in on the first node, I ran the verification I should have run in August:

```plaintext
systemctl show pm2-flux.service -p Environment
```

It returned a single line, truncated at terminal width, with a trailing `>` and the pager sitting at `(END)`. All I could see was `PATH`. The drop-in had worked; the output couldn't show it.

`Environment` is emitted as one space-separated line, and any pager that folds long lines will truncate it. The fix is trivial: `systemctl show pm2-flux.service -p Environment --no-pager | tr ' ' '\n' | grep UV_USE`. Cost of the trap: about a minute of thinking I had to rewrite a drop-in that was already correct.

The instrument lying twice in one investigation: once at a cost of a month, once at a cost of a minute. Both times the tool did exactly what it was asked. Both times the framing of the question was where the mistake sat.

## // one more thing from the same three commands

The same `systemctl show` output also revealed the unit's `ExecStart`: `pm2 resurrect`, restoring the process list from `~/.pm2/dump.pm2`. Anyone applying the older `/etc/environment` fix pattern — `pm2 kill` to force the daemon to inherit the variable — walks into a related trap: kill without saving first, and the resurrect at boot restores an empty process list. The node boots with no application.

`pm2 save --force` after any kill. One command, one sentence.