chad/cn
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

thinkpad image: build once, promote on merge; fix image-rebuild hang #140

Open
glen wants to merge 1 commit from thinkpad-image-promote into main
pull from: thinkpad-image-promote
merge into: chad:main
Conversation 0 Commits 1 Files changed 7 +220 -51
glen commented 2026-11-06 18:41:54 -05:00
Collaborator
Copy link

Problem

Merging #139 built the thinkpad image three times: the PR job, the push-to-main job, then cjust image-rebuild. CI deleted both of its builds right after verifying them. Then cjust image-rebuild hung forever: it followed the log with journalctl -f, which doesn't exit when the service finishes, so its chained image-upgrade never ran.

Change

Build once, promote on merge (ci.yml thinkpad-image job → new hosts/bees/thinkpad-image-ci.sh):

  • PR: build, then the shared verify gates, then push the verified image to zot as cn/thinkpad-host:ctx-<git tree hash of hosts/thinkpad/host-image>. A ctx-* tag only exists if the gates passed.
  • Merge: copy that exact image's manifest to its version tag and to :stable. This is a registry-side retag: seconds, no rebuild, and it ships the bytes that passed review. After a merge, once the job is green, cjust image-upgrade on think is all you need.
  • Fallback: if there's no candidate, or a nightly has published something newer since the PR built (staleness guard compares build stamps), it runs the nightly unit with systemctl start --wait.
  • Keyed on the build-context tree hash, not the commit. The PR's merge-ref build stays promotable when unrelated commits land on main in between. Any change to the image directory means a new key and a fresh build.

zot: keep the last 5 ctx-* candidates. Adds restartTriggers on zot.service so retention edits apply when bees deploys, not at the next reboot (nothing did this before).

cjust image-rebuild: waits with systemctl start --wait while streaming the log, then stops the tail, so the image-upgrade step actually runs. Docs now point to "after a merge: cjust image-upgrade"; image-rebuild is for pulling in today's Fedora updates.

The nightly is unchanged (it's still the daily base-image and dnf refresh).

Verification

  • shellcheck 0.11 is clean on the new script, the build script, and the extracted image-rebuild recipe body.
  • Tested the promote logic against a fake OCI registry (same manifest and blob API calls the script makes against zot):
    • A: fresh candidate. Retagged; stable, the version tag and ctx-* end up with the same config digest.
    • B: no candidate. Falls back to systemctl start --wait thinkpad-image-build.service.
    • C: candidate older than :stable. Falls back to a fresh build.
    • D: empty registry. Retagged.
    • E: PR mode. Order confirmed as build, then verify, then push; nothing is pushed if verify fails (set -e).
  • On bees: the runner user (crussell) has passwordless sudo for systemctl, and zot's tag listing is reachable over Nebula. zot currently holds stable plus four 44-* tags; ctx-* doesn't collide with the existing ^[0-9]+- retention pattern.

Self-test

This PR changes thinkpad-image-ci.sh, which is now in the path filter, so the thinkpad-image job runs on this PR in pr mode and pushes a ctx-* candidate. Merging it exercises promote for real.

Accepted risk

Retagging uses the plain registry API (curl PUT of the manifest), since the runner has no skopeo or jq. zot is Nebula-only and unauthenticated, the same trust model the nightly's podman push already relies on.

## Problem Merging #139 built the thinkpad image three times: the PR job, the push-to-main job, then `cjust image-rebuild`. CI deleted both of its builds right after verifying them. Then `cjust image-rebuild` hung forever: it followed the log with `journalctl -f`, which doesn't exit when the service finishes, so its chained `image-upgrade` never ran. ## Change **Build once, promote on merge** (`ci.yml` thinkpad-image job → new `hosts/bees/thinkpad-image-ci.sh`): - **PR:** build, then the shared verify gates, then push the verified image to zot as `cn/thinkpad-host:ctx-<git tree hash of hosts/thinkpad/host-image>`. A `ctx-*` tag only exists if the gates passed. - **Merge:** copy that exact image's manifest to its version tag and to `:stable`. This is a registry-side retag: seconds, no rebuild, and it ships the bytes that passed review. After a merge, once the job is green, `cjust image-upgrade` on think is all you need. - **Fallback:** if there's no candidate, or a nightly has published something newer since the PR built (staleness guard compares build stamps), it runs the nightly unit with `systemctl start --wait`. - **Keyed on the build-context tree hash, not the commit.** The PR's merge-ref build stays promotable when unrelated commits land on main in between. Any change to the image directory means a new key and a fresh build. **zot:** keep the last 5 `ctx-*` candidates. Adds `restartTriggers` on `zot.service` so retention edits apply when bees deploys, not at the next reboot (nothing did this before). **`cjust image-rebuild`:** waits with `systemctl start --wait` while streaming the log, then stops the tail, so the `image-upgrade` step actually runs. Docs now point to "after a merge: `cjust image-upgrade`"; `image-rebuild` is for pulling in today's Fedora updates. The nightly is unchanged (it's still the daily base-image and dnf refresh). ## Verification - `shellcheck` 0.11 is clean on the new script, the build script, and the extracted `image-rebuild` recipe body. - Tested the promote logic against a fake OCI registry (same manifest and blob API calls the script makes against zot): - **A: fresh candidate.** Retagged; `stable`, the version tag and `ctx-*` end up with the same config digest. - **B: no candidate.** Falls back to `systemctl start --wait thinkpad-image-build.service`. - **C: candidate older than `:stable`.** Falls back to a fresh build. - **D: empty registry.** Retagged. - **E: PR mode.** Order confirmed as build, then verify, then push; nothing is pushed if verify fails (`set -e`). - On bees: the runner user (crussell) has passwordless sudo for `systemctl`, and zot's tag listing is reachable over Nebula. zot currently holds `stable` plus four `44-*` tags; `ctx-*` doesn't collide with the existing `^[0-9]+-` retention pattern. ## Self-test This PR changes `thinkpad-image-ci.sh`, which is now in the path filter, so the thinkpad-image job runs on this PR in `pr` mode and pushes a `ctx-*` candidate. Merging it exercises `promote` for real. ## Accepted risk Retagging uses the plain registry API (`curl` PUT of the manifest), since the runner has no skopeo or jq. zot is Nebula-only and unauthenticated, the same trust model the nightly's `podman push` already relies on.
thinkpad image: build once, promote on merge; fix image-rebuild hang
All checks were successful
ci / changes (pull_request) Successful in 1s
Details
ci / check (pull_request) Successful in 17s
Details
ci / thinkpad-image (pull_request) Successful in 6m4s
Details
d745cd30a2 
Before: merging an image change built it three times (PR job, push-to-main
job, then the nightly / cjust image-rebuild). CI deleted both of its builds
right after verifying them, so a merge only reached the laptop after another
full build. cjust image-rebuild also never finished: it followed the build
log with journalctl -f, which doesn't exit when the service does, so its
chained image-upgrade never ran (Ctrl-C was the only way out).

Now:
- PR (ci.yml thinkpad-image): build, verify, then push the verified image to
  zot as cn/thinkpad-host:ctx-<git tree hash of hosts/thinkpad/host-image>.
- Merge: retag that exact image to :stable + its version tag. Registry-side
  manifest copy, seconds, no rebuild, and it ships the bytes that passed
  review. Falls back to the nightly unit (systemctl start --wait) if there is
  no candidate or a nightly published something newer since the PR built
  (staleness guard on the build stamps).
- Keyed on the build-context tree hash, so the PR's merge-ref build stays
  promotable when unrelated commits land on main in between.
- Logic in hosts/bees/thinkpad-image-ci.sh (shellcheck-clean; tested against
  a fake OCI registry: retag, no-candidate fallback, stale fallback, empty
  registry, PR build/verify/push order).
- zot: keep the last 5 ctx-* candidates; restartTriggers on zot.service so
  retention edits apply on deploy instead of the next reboot.
- cjust image-rebuild: wait with systemctl start --wait (log streamed
  alongside, then stopped), so the upgrade step actually runs. Docs now say
  'after a merge: cjust image-upgrade'.

The nightly is unchanged: still the daily Fedora base/dnf refresh.
All checks were successful
ci / changes (pull_request) Successful in 1s
Details
ci / check (pull_request) Successful in 17s
Required
Details
ci / thinkpad-image (pull_request) Successful in 6m4s
Details
This pull request can be merged automatically.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin thinkpad-image-promote:thinkpad-image-promote
git switch thinkpad-image-promote
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
chad/cn!140
No description provided.