Gotchas (hard-won)

1. Never readdir/ls a FUSE mount to test liveness — on a stale FUSE mount (server died) it blocks waiting for the dead server. Staleness is detected by the recorded daemon PID's liveness (/proc/<pid>/comm == bubblebox-fuse) in mount.rs::ensure_mounted. is_mountpoint uses the mountpoint command (stat-based, never blocks, detects stale mounts).

2. mount() must unmount before create_dir_all — create_dir_all on a stale FUSE mount errors EEXIST. mount() best-effort fusermount3 -u (stderr silenced — it prints "entry not found" when nothing's mounted) then waits for clear, before any dir creation.

3. Composefs object names ≠ sha256(content) (empirically a different digest, likely fs-verity-style). So per-blob content verification on pull is not done yet — pull verifies the descriptor via composefs-info measure-file and relies on HTTPS + authenticated blob names for the rest. Settling the digest algorithm is a deferred task.

4. overlay lowerdir order = priority, leftmost wins. run_composed builds lowerdir=primary:dep1:dep2 so the primary package's files shadow deps on conflict (main-wins). Read-only overlay (no upperdir) — the sandbox rootfs is read-only; writes go to bind-mounted /home, /tmp, /persist.

5. zsh does not word-split unquoted $VAR — when calling bubblebox export $TOOLS from a zsh recipe/script, use a real array (tools=(a b c); … $tools) or ${=TOOLS}, else it's passed as one giant arg.

6. run/compose-inner take trailing args (trailing_var_arg + allow_hyphen_values), so bubblebox run rg --version works without -- (the wrapper form). -- --foo also works. Don't switch back to last = true.

7. The release binary is large (~22 MB) because of aws-sdk-s3 + reqwest + tokio. That's the accepted cost of a self-contained tool (no rclone/aws-cli host dep). A lighter S3 crate could shrink it later.

8. Descriptor digests are not reproducible across extractions — re-building or re-importing the SAME image yields a different descriptor digest (podman export carries non-normalized metadata; mkcomposefs folds it in). Store blobs still dedupe by content, so the cost is one orphaned descriptor (+ any unshared blobs) per rebuild until gc --apply. Don't rely on digest-equality to detect "same image".

9. NixOS setuid helpers live in /run/wrappers/bin, which non-login environments omit from PATH — a bare fusermount3 there resolves to the unprivileged store copy and every FUSE mount dies with fusermount3: mount failed: Operation not permitted. This silently ran the bee nightly CI red for two weeks (2026-09-05→19). The engine now prepends the wrappers dir to the FUSE server's PATH and tries the wrapper binary first on unmount (8d59009); tools/verify.sh/nightly.sh harden their own PATH the same way. Same PATH class as NixOS sudo (see homelab-ops).

10. Non-login/agent shells differ from login shells in more than PATH — --userns=keep-id breaks in them (crun readlink ''; use rootless podman's default mapping, it already maps container-root to you), ssh needs -F /dev/null on this host (bad-perms systemd ssh_config include), and DSH agent sandboxes give /run/user read-only plus a private /tmp. bubblebox doctor first, then suspect the environment, not the code.

11. bwrap --bind-try tolerates a missing SOURCE but not a missing DEST — an entrypoint that top-level-binds a host dir (e.g. /sys for btop's sensors) still needs that directory pre-created in the image rootfs, or the run dies "Can't mkdir /sys: Read-only file system" (composefs roots are read-only; bwrap only tolerates EEXIST). Same class as #9/#10: the standard-dir set in archetype-C finals exists for exactly this reason — extend it whenever an entrypoint binds a new top-level path (btop added /sys, 2026-09-23).

12. Bind destinations cannot be created at run time — the engine now fails fast and tells you — a sandbox bind (or bind_try, or even --dir) whose destination is missing from the image hits bwrap's dst mkdir on the read-only rootfs and dies "Can't mkdir : Read-only file system" LATE and cryptic (--dir is no escape: same mkdir, same EROFS, bwrap 0.11.2). run now validates every bind before exec (single-package AND compose-inner) and errors early, naming the bind and the fix: image-side RUN mkdir -p <dst> (#11's tmux/nvim /dirs/ pattern). dst existence is judged against the sandbox VIEW — host-overlaid dirs (/run /var /etc /home /mnt /media /opt /srv) stat host-side; /tmp /dev /proc are engine-mounted writable pseudo-fs. Two adjacent fixes in the same change: an UNSET $VAR in an entrypoint expands to an empty string, and bwrap does NOT treat an empty bind_try source as missing (it falls into the dst mkdir → EROFS) — push_bind now skips empty-source try binds with a warning (this is what makes the openssh-clients agent-socket design safe on hosts with no runtime dir), and required binds with empty/missing sources error loudly. Engine PR #1, 2026-09-24.