This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

bubblebox GC + Generations — Design

Status: Phases 1–3 implemented. The substrate for the deferred GC / generations / remove / upgrade / list / lockfile work (see the wiki Status page). This page is the design record; the wiki tracks what's built. See the Progress log at the end for what's done vs pending.

1. Goals & non-goals

Goals. Reclaim disk from orphaned store objects; enable version pinning + rollback; deliver reproducible environments (lockfiles); land the deferred remove / upgrade / list / generations features as one coherent layer.

Non-goals (this design). Nix-style lib-object decomposition (images resolve dependency conflicts at build time — multi-version is only for rollback/pinning, narrower than nix's reason). Dotfiles. System config.

Key asymmetry vs nix. bubblebox packages are self-contained OCI images, so dependency conflicts are resolved at image-build time, not in the store. The store therefore only needs multi-version for keeping old tool versions around — not for nix's hash-coexistence-of-shared-libs reason. This bounds the project: the store stays descriptor-per-image.

2. Decision stack

#	Decision
1	GC roots = generations (realized manifests), not bare installed/ records. The compose closure is baked into each generation.
2	Binary-generations (digest-pinned). Required for "version" to mean anything on rollback (config-generations can't roll back a binary).
3	Whole-tools only. No lib-object decomposition; images handle conflicts.
4	Live override + frozen binaries. run reads the live effective entrypoint for sandbox config (binds/env/writable_*); binaries come from the generation's pinned digests. Structural change = new generation; cosmetic change = live.
5	Single profile now; design doesn't preclude multi-profile / transient devshell profiles later.
6	Minimal v1 GC. Dry-run default + --apply. No flock in v1 (document the gc+apply race; add when background GC or a daemon appears). No --check-remote / --keep-since.
7	Pruning ≠ GC. Generation-pruning (drop manifests) and store-GC (reclaim bytes) are separate ops.
8	Phase 1 mechanism: digest-addressed descriptors via pkg→digest symlink (readers untouched); migration is manual.
9	Declarative model: profile.toml (inputs) + profile.lock (realization) — flake.nix / flake.lock analog. Pure-declarative now (apply); install/remove/upgrade sugar deferred. Delivers the deferred lockfile (locked decision #10).
10	Generation IDs: unordered, content-addressed (hash of manifest); ordering by created_at + optional name. profile.lock IS the current generation; generations/ is the archive.

3. Architecture

Store layout (final)

~/.config/bubblebox/                     (user-authored / committable)
  profile.toml                           INPUTS — desired set (flake.nix analog)
  profile.lock                           CURRENT realization (flake.lock analog; generated)
  entrypoints/<pkg>.toml                 user overrides (LIVE — read by run)
  sources.toml, caches.toml              (OPTIONAL now — superseded when profile.toml declares registries; kept as a fallback for ad-hoc `source add`/`cache add`)

~/.local/share/bubblebox/                (regenerable state)
  store/<XX>/<rest>                      content blobs — composefs-digest, deduped (UNCHANGED)
  descriptors/<digest>.cfs               digest-addressed descriptors (Phase 1)
  descriptors/<pkg>.cfs                  symlinks → <digest>.cfs (Phase 1; materialized from active gen in Phase 2)
  generations/<hash>.toml                realized manifests — the archive (Phase 2)
  remote-defaults/<pkg>.toml             downloaded default entrypoints (cache for `config cat`)
  sources/, logs/                        (unchanged)
  installed/                             RETIRED in Phase 2 (subsumed by generations)

Object graph (what GC walks)

generations/<hash>.toml ─┐
profile.lock ────────────┼─→ [[package]] digest → descriptors/<digest>.cfs
live FUSE mounts ────────┘                         ↓
                                    composefs-info objects(store, <digest>.cfs)
                                                  ↓
                                          store/<XX>/<rest>  (blobs)

4. Schemas

profile.toml — inputs (you author this). Self-contained: declares the registries it resolves from (profile-authoritative — no separate source add/ cache add step) plus the package set as a concise array. Note TOML ordering: top-level keys (packages) must precede the [[source]]/[[cache]] headers.

# The set. Floating (resolved from the registries, pinned-in-lock between
# applies). Forward-compatible with Phase 4 tags ("nvim:stable").
packages = ["nvim", "rg", "fd"]

# Registries — when present, apply resolves using ONLY these.
[[source]]
name = "personal"
kind = "local"             # or: kind = "git", url = "https://..."
path = "~/Code/cn/.../bubblebox"

[[cache]]                  # optional
name = "public"
url = "https://cache.example.com"
priority = 10              # optional

Pinning a specific digest ([pins]) and per-package from = "<registry>" are deferred (the lock covers normal reproducibility).

profile.lock = generations/<hash>.toml (same format; lock is a copy of current gen):

created_at = 1717900000
name = "pre-experiment"        # optional human label (archive use)
roots = ["nvim", "rg"]         # profile.toml entries → get ~/.local/bin wrappers
[[package]]                    # full closure: roots + compose deps, all digests concrete
name = "nvim"
digest = "def456..."
[package.origin]               # reused from installed::Origin (installed.rs:17)
kind = "cache"
url = "https://..."
[[package]]
name = "fd"                    # nvim's compose dep: in closure, NOT in roots (no wrapper)
digest = "789..."
[package.origin]
kind = "cache"
url = "https://..."

5. Phased plan

Phase 1 — Foundation: digest-addressed descriptors

Goal. Descriptors move from descriptors/<pkg>.cfs (one per package, overwritten) to digest-addressed (descriptors/<digest>.cfs) with descriptors/<pkg>.cfs as a symlink → <digest>.cfs. No user-visible behavior change; enables version coexistence.

Mechanism. Readers that open(2) the pkg path transparently follow the symlink → every reader stays untouched. Only writers change.

Touch points:

site	now	Phase 1
build.rs:78,112	mkcomposefs writes descriptors/<pkg>.cfs	write temp → measure → rename to <digest>.cfs → symlink <pkg>
cache.rs:91,102	pull writes bytes to <pkg>.cfs	pull already has the digest (resolve_index, cache.rs:94) → write <digest>.cfs → symlink <pkg>

Readers untouched (symlink-transparent): mount.rs:27,192, composefs.rs:14 (measure_file), cache.rs:98,104,226,489. build.rs:64 and cache.rs:488 already compute + store the digest in installed/.

Migration (manual — single user). Nuke descriptors/, re-build / pull. Blobs in store/ are content-addressed and reused by mkcomposefs --digest-store (build.rs:111); only the regenerable descriptors get rewritten.

Acceptance. build rg twice (different inputs) → two descriptors/<digest>.cfs, <pkg>.cfs symlinked to latest; run rg works; old digest becomes a reclaimable orphan.

Phase 2 — Generations: declarative profile + lockfile

Goal. profile.toml (inputs) + profile.lock (realized) + generations/ (archive). One reconcile command (apply) produces a generation. Retire installed/.

apply semantics:

bubblebox apply              # realize profile.toml from profile.lock (no re-resolution)
                             #   → pull missing pinned digests → write gen + lock if changed → materialize
bubblebox apply --refresh    # also re-resolve FLOATING inputs against caches; update lock
bubblebox plan [--refresh] [--json]   # dry-run apply: print exactly what would change —
                             #   packages/fetch/wrappers/files/units + the would-be generation id
                             #   (read-only; hash-only ingest; exits 1 when apply would fail)

plan mirrors apply's resolution branch-for-branch and reuses the same functions (resolve_floating, descriptor_digest, export_diff, and the plan_* twins of home.rs's reconcile fns), so the diff can't drift from what apply does — the --json report is a stable contract for LLM/CI consumers (same convention as info/list --json). Caveat noted in its output: an unlocked float with a local build may be superseded by the cache version at apply time, shifting digests/the predicted id.

Idempotent: if the realized set matches current, no new generation. The closure comes from the existing config::load_effective(...).compose_list() (cache.rs:464); each member's digest from descriptor_digest (cache.rs:488); Origin reused from installed.rs.

Reproducibility. Clone profile.toml + profile.lock → apply pulls the exact pinned closure from recorded origins, zero resolution — bit-for-bit reproducible. This is the deferred lockfile (decision #10).

Profile location (v0.3.0). The profile can live anywhere, e.g. a host repo: bubblebox --profile <file-or-dir> or BUBBLEBOX_PROFILE, else ~/.config/bubblebox/profile.toml. The resolved path is canonicalized, so a symlinked ~/.config/bubblebox/profile.toml resolves files//units/ next to its target. This is the recommended setup, because wrappers and devshells can't pass flags. profile.lock stays host state in ~/.config/bubblebox/; when the profile lives elsewhere, apply / generations switch / no-change apply also mirror the lock beside profile.toml (commit that copy). The thinkpad (cn hosts/thinkpad/bubblebox) works this way.

Gotcha: a profile [[cache]] overrides channel caches entirely, including their public_key. If the profile declares a cache without public_key, signature verification is silently OFF for it. Repeat the key from the source's channel.toml.

materialize(generation) (shared primitive): (1) repoint each descriptors/<pkg>.cfs → active gen's digest; (2) re-sync ~/.local/bin to roots (via export.rs). Runs last, only after all pulls succeed. Live mounts are safe — the FUSE daemon holds its fd open; repointing the symlink doesn't affect it, and the next ensure_mounted (mount.rs:204, mtime compare) remounts on the changed binding.

Retire installed/. Seed generation #1 from existing installed/<pkg>.toml on first run, or nuke + reinstall. remote-defaults/ stays as a download cache for config cat.

Edge (decision #4). A live override that composes a package not in the active generation → run fails clean ("<pkg> not installed; add it to profile.toml and apply").

Phase 3 — GC

Algorithm:

roots = ⋃ package.digest over:
   ├─ all generations/<hash>.toml + profile.lock
   └─ live FUSE mounts (resolve mounted pkg → symlink target digest; mount.rs:174 liveness)

live_blobs = ⋃ composefs-info objects(store, <digest>.cfs) over roots   # composefs.rs:29

sweep:
   descriptors/<digest>.cfs ∉ live         → delete
   descriptors/<pkg>.cfs target gone        → delete (regenerable from profile.lock)
   store/<XX>/<rest>       ∉ live_blobs     → delete

Safety (no lock v1). The race: apply fetches blob X (pull computes missing_objects, cache.rs:114-style) before writing the manifest referencing X → GC sees X orphan → sweeps → apply writes a manifest pointing at a missing blob. Seconds-window, low-probability, silent failure. Operational rule: don't run gc --apply while apply runs. Dry-run default limits the blast radius. Add a flock when background GC or a devshell daemon appears.

Report (dry-run):

$ bubblebox gc
roots: 3 generations + profile.lock + 1 live mount
live:        47 descriptors, 12,481 blobs (4.2 GB)
reclaimable:  6 descriptors,    312 blobs (148 MB)
hint: 48 MB in 5 old generations — `bubblebox generations --older-than 30d`, then re-run gc.
run `bubblebox gc --apply` to reclaim the 148 MB.

Code. New cli/src/gc.rs + Cmd::Gc in main.rs. Reuses composefs::referenced_objects (composefs.rs:29) and mount.rs liveness (expose a live_mounted_digests helper). Root-collection, blob-set union, and sweep-decisions are pure logic over temp-dir fixtures — testable like the existing expand_env / deep_merge tests.

6. CLI reference (final)

# Phase 2 — declarative:
apply [--refresh]                                    realize + materialize
generations                                          list (by created_at; hash + label)
generations --delete <hash> | --keep-last N | --older-than Nd   prune manifests

# Phase 3 — GC:
gc [--apply]                                          dry-run default; sweep
                                                       (pass global -v to list each orphan path)

# unchanged surface:
run build pull push source cache config mount unmount export init

7. Deferred (with triggers)

Item	Trigger
install / remove / upgrade sugar over profile.toml	if the editor-round-trip proves annoying
store flock	background/scheduled GC, or a devshell daemon
gc --check-remote (annotate permanent-loss vs re-fetchable)	once reclaim volume justifies it
gc --keep-since grace	if "just-upgraded oops" becomes common
multi-profile / named profiles	when devshell or work/personal splits land (devshells landed in Phase 4 — see Devshells; named/multiple persistent profiles still deferred)
devshell (transient profile, shared binding machinery)	Phase 4 v1 landed — reusable base shells + project bubblebox.toml + base inheritance + init hook; see Devshells (transient pull-on-entry + multi-base still deferred there)
versioning: free-form name:tag (Phase 4) — packages = ["nvim:stable"]; cache index/<pkg>/<tag>; push --tag; local tag cmd; lock records {name,tag,digest,origin}. Container-style (no semver ordering). Note: tags give selection+repro, NOT simultaneous two-version-of-same-name coexistence (that's devshell/multi-profile)	when you need friendly version names / named streams beyond what the lock already pins
[pins] (force-pin a digest in profile.toml) + per-package from = "<registry>"	if the lock's repro isn't enough, or registry name-collisions appear
per-blob content verify	when the composefs digest algorithm settles (gotcha #3; orthogonal to GC)

8. Risks & open questions

• Migration of existing pkg-named descriptors — manual nuke + rebuild (single user).
• apply + gc race — documented; no lock v1.
• Live override ∉ active generation — run fails clean (decision #4 edge).
• composefs digest algorithm "unsettled" (gotcha #3) — doesn't block GC (we trust the descriptor via composefs-info objects); only blocks orthogonal per-blob verify.
• Devshell north-star not yet mapped in detail — but the declarative + binding-materialization design is shaped to accommodate it as a transient profile reusing the same machinery.

Progress log

Phase 1 — digest-addressed descriptors — ✅ implemented

Descriptors now publish at descriptors/<digest>.cfs with descriptors/<pkg>.cfs as a relative symlink → <digest>.cfs. Readers (mount, measure_file, missing_objects, push's fs::copy) are untouched — they open the pkg path and follow the symlink transparently. Only the two writers changed:

• composefs.rs — new pub(crate) fn publish_descriptor(desc_dir, pkg, staging, digest): atomic publication (rename staging → <digest>.cfs; build a temp symlink and rename it over <pkg>.cfs). Handles the gradual migration case (overwrites an old regular file with the symlink) and is idempotent on identical re-publish. 4 unit tests cover the digest-addressing, pre-Phase-1 regular-file replacement, version coexistence on rebuild, and idempotent same-digest re-publish.
• build.rs — build_descriptor writes mkcomposefs output to a staging path under the build tmp dir, then measures + publishes; now returns the digest so build records provenance without re-measuring.
• cache.rs — pull_package fetches the descriptor into a staging path (.{pkg}.cfs.staging), verifies measure_file, then publishes. local_matches is unchanged and symlink-transparent (exists() + measure_file follow the link).

Verified: cargo build, cargo clippy --all-targets (clean against the workspace's clippy::all = deny), cargo test — 32/32 pass (28 existing + 4 new).

Migration (manual, single user): existing descriptors/<pkg>.cfs regular files keep working (readers open them regardless). They flip to the new digest-addressed + symlink layout on the next build/pull of each package, or in one shot: rm -rf "$BUBBLEBOX_DATA_DIR/descriptors" then re-build/ pull each package (store blobs are content-addressed and reused; only the regenerable descriptors are rewritten).

User acceptance step (not yet run): bubblebox build rg twice against different inputs should leave two descriptors/<digest>.cfs with descriptors/rg.cfs symlinked to the latest; bubblebox run rg -- --version should still work following the symlink; the older digest becomes a reclaimable orphan (Phase 3 GC reclaims it; until then it accumulates harmlessly).

Phase 2 — declarative profile + generations — ✅ implemented

The declarative reconcile works end-to-end: apply realizes profile.toml → pulls the closure → writes a content-addressed generation + profile.lock → materializes the descriptor bindings. run/mount are unchanged (they follow the symlinks apply points).

New module cli/src/profile.rs:

• Profile / ProfilePkg (profile.toml: name + optional pinned digest).
• Generation / GenPkg (the realized manifest = profile.lock = each generations/<id>.toml); reuses installed::Origin (nested [package.origin]).
• generation_id — sha256 over the canonical (sorted) closure, excluding origin/time so identical realizations dedup (verified: re-apply is a no-op, identical inputs in shuffled order produce the same id).
• materialize_bindings — repoints descriptors/<pkg>.cfs for every package in the generation (via the extracted composefs::point_binding).

New CLI: apply [--refresh], generations list.

Wiring: sha2 dep added; Common gained profile_path / profile_lock_path / generations_dir; cache::{http_client, pull_packages, resolve_index, descriptor_digest} exposed pub(crate) and reused by apply. The two-pass pull (inputs first so their default entrypoints land, then the closure's extra deps) mirrors the proven install path.

Verified: cargo build, cargo clippy --all-targets (clean), cargo test 39/39 (7 new). e2e (offline, isolated env, real rg/fd/nvim descriptors): single-input apply; pinned-digest resolve; idempotent re-apply (no new generation); 2-input profile; entrypoint compose-closure expansion (profile.toml = nvim → generation captures [nvim,rg,fd], roots=["nvim"]); generation history preserved across applies (old gens stay as rollback targets); generations list marks the current one; the no-profile.toml error path.

e2e (networked, full cache round-trip): push rg --stage-dir (with a compose=[rg,fd] default entrypoint) + push fd → served over HTTP via python3 -m http.server → isolated env → cache add → floating profile.toml (name = "rg") → apply --refresh correctly: resolved rg's digest from index/rg, pulled its descriptor + 1274 blobs + default entrypoint, expanded the closure to [rg,fd], pulled fd (1 additional blob — cross-package dedup working), wrote the 2-package generation (roots=["rg"]), and materialized both descriptor bindings. The networked path reuses pull_packages / resolve_index unchanged; both offline and online apply paths are validated.

Implemented Phase 2 tails:

• Wrapper auto-sync. apply/switch reconcile ~/.local/bin to the generation's roots via export_diff + export::export (add new roots, remove gone roots). Safe — only wrappers a previous generation's roots claimed are ever removed; manual export / imperative install wrappers are untouched. apply --no-export opts out.
• generations switch <id> (rollback) — reactivate an archived generation (copy → profile.lock, materialize, re-sync wrappers); no new generation created (the "move pointer" model). Ids match by unique prefix.
• Pruning: generations delete <id> (refuses the current), keep-last N, older-than Nd. These drop manifests only; reclaiming store bytes is the separate Phase 3 GC pass.
• Retired installed/. build/install no longer write installed/<pkg>.toml (the generation manifest is now the single manifest); installed.rs is trimmed to just the Origin type (reused by the manifest). Existing on-disk installed/ files are inert (nothing reads them).

Verified: e2e (isolated env incl. HOME): auto-export creates wrappers on apply; --no-export is a no-op; dropping a root removes its wrapper; switch to an older generation re-creates the removed wrapper; delete refuses the current and allows a non-current generation; keep-last prunes. Plus 41 unit tests (5 new: match_id prefix/ambiguous/not-found, export_diff).

Still deferred from Phase 2: install/remove/upgrade sugar over profile.toml (the editor-round-trip has been fine so far).

Phase 3 — GC — ✅ implemented

New cli/src/gc.rs + Cmd::Gc { --apply }. Root set = every generation manifest + profile.lock + every pkg binding (descriptors/<pkg>.cfs, symlink or regular file) + live FUSE mounts. Including pkg bindings is what makes GC safe alongside imperative build/install packages that aren't (yet) in any generation — their descriptor + blobs are rooted by the binding, so only truly orphaned objects are reclaimed.

Algorithm: live_digests from the roots; live_blobs = ⋃ composefs-info objects over the live descriptor files (composefs::referenced_objects); sweep digest-named descriptor files whose digest ∉ live_digests and store blobs whose <XX>/<rest> ∉ live_blobs. Pkg bindings are never candidates. Dry-run by default (reports count + human-readable bytes); --apply deletes and prunes newly-empty shard dirs.

Reclaiming old package versions is the documented two-step: prune old generations (generations --delete/--keep-last/--older-than), then gc. No store lock in v1 (the apply+gc race is the documented, low-probability residual — don't gc --apply while apply runs).

Verified: cargo build + cargo clippy (clean) + 45 tests (4 new: digest- name classification, orphan-digest selection skips live + bindings, orphan-blob selection by relpath, human-readable sizes). e2e (isolated env, real rg/fd/bat descriptors + 1277 blobs): dry-run reported exactly the planted orphans (a fake digest file + a dummy blob); the imperative bat (regular-file descriptor, in no generation) and the generation-pinned rg/fd with all 1277 live blobs were protected; --apply reclaimed exactly the 2 orphans.

Profile restructure — self-contained, concise profile.toml — ✅ implemented

profile.toml is now a concise packages = [...] array (was one [[package]] block per tool) AND self-contained: it declares the registries it resolves from ([[source]] / [[cache]]). Profile-authoritative resolution moved into Common::load: when the profile declares any registry, every command (build/run/config/apply/pull) resolves using ONLY those (a source_override + cache_override, honored at the sources::load / caches::load / substituters chokepoints), so sources.toml/caches.toml are fully redundant when a profile declares registries (and the bubblebox-source-add recipe is retired). Empty profile-registry section → falls back to those files. Git sources declared in a profile are cloned on first use (sources::ensure_ready, called once in main).

[pins] and per-package from are deferred (the lock covers repro); Phase 4 versioning (free-form name:tag) is designed-deferred (see §7).

Verified: cargo build + cargo clippy (clean) + 48 tests; on the host, with sources.toml/installed//*.profile removed, apply / run nvim (compose closure) / config cat / source list all resolve via the profile-declared source.