dpkg calls the OLD package's prerm on an UPGRADE as well as on a removal (rpm passes a
remaining-instance count), and preremove.sh ran `systemctl disable --now monky-deployd.timer`
unconditionally. Upgrading env-dev-01 and env-dev-08 to 0.1.8 today stopped and disabled both
agents.
The failure is silent, which is the dangerous part: the box stays reachable, the containers keep
running, and nothing reports that check-ins have ceased — the backend just stops converging. A
fleet upgrade would have taken every agent offline at once and looked like a success.
preremove.sh now returns early for every upgrade shape (upgrade, failed-upgrade, deconfigure,
rpm's 1) and only disables on a real removal. postinstall.sh try-restarts the long-lived proxy
unit so it picks up the new code; the timer needs nothing, since each tick is a fresh process.
Tests drive the script with a fake systemctl on PATH and assert an upgrade touches no units.
OPERATIONS.md warns that a box coming FROM 0.1.8 or earlier still needs its timer re-enabled by
hand, because the old prerm has already run by then.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KLB7jieMNRkTsJ2epr4Ds1
Three faults from one agent-managed onboarding (env-dev-08, 2026-09-09), each of which pointed
the diagnosis away from the actual fault.
1. install.sh granted the agent's read on the ziti identity with a POSIX ACL. ziti-edge-tunnel
rewrites that file on a controller config update and the rewrite drops the ACL: the agent
applied cleanly at 01:21 and was failing every tick by 01:32. Group membership survives the
rewrite (the file stays ziti:ziti 0640), so install.sh and the package postinstall now add
monky-deployd to the `ziti` group, and a default ACL on the identity directory carries the
grant onto a newly created file. The explicit ACLs stay for the boxes that need them.
2. openziti.load() accepts an unreadable or malformed identity: the C SDK logs "configuration is
invalid" and returns a context that only fails at dial, as a bare TypeError, which the
transport reported as a missing intercept or a policy gap. The SDK transport now reads and
parses the identity itself and names the real fault first.
3. The disk pre-flight ran only when the bundle declared disk_need_bytes, so a bundle without one
died mid-pull with containerd's "no space left on device" — which reads as a registry fault.
A bundle that declares no size now has to clear the headroom floor, and the pre-flight measures
containerd's root as well as the docker data-root: docker 29 keeps image layers in the
containerd image store, and on env-dev-08 those sat on different filesystems (93 GiB free where
the agent looked, 2.8 GiB where the pull wrote).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KLB7jieMNRkTsJ2epr4Ds1
Design merged first: monky-design-docs !225 (doc 24 §4a). Pairs with monky-tenancy!40, which
copies the estate-wide read-only Harbor robot into each environment's own prefix and marks the
manifest entry `use: registry-auth`.
- That entry is not an env var (it would otherwise land in .env and therefore in every container's
environment). The agent parses it — JSON, or `username:password` with the new `registry_host` —
and writes `<state_dir>/docker/config.json` 0600 in a directory it owns, with an explicit
DOCKER_CONFIG pointing the docker CLI at it. The unit runs as monky-deployd, so a `docker login`
by a human or by root is invisible to the agent: that is what made env-dev-01 look like it had no
credential at all after the operator had just logged in.
- `compose pull` failures now carry the registry's own message ("no basic auth credentials",
"manifest unknown", DNS) into the journal and the report instead of `rc=1`.
- Tests: both credential shapes, the refusals, 0600/0700 modes, idempotent rewrite, and that the
runner never silently falls back to a human's $HOME.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KLB7jieMNRkTsJ2epr4Ds1
Every apply requested a new lease, so a failing deploy retried by the 60 s
timer burned tenancy's 5-leases-per-hour budget and then failed on
LEASE_RATE_LIMITED forever (env-qa-02 pilot). Now: reuse a lease-derived
token while lookup-self says it is valid; swap the bootstrap token for a
lease once; if tenancy rate-limits the lease while a working token exists,
apply with it and defer the swap.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KLB7jieMNRkTsJ2epr4Ds1
install.sh wrote tenancy.port 8081 (the in-pod port) but the
monky.tenancy.deploy intercept is 443, so the SDK found no intercept and the
check-in failed with 'service not available' followed by the SDK's bare
TypeError. Also wrap non-OSError SDK failures in a TransportError that names
the host:port and what to check.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KLB7jieMNRkTsJ2epr4Ds1
On env-qa-02 (ziti-edge-tunnel 1.18.7) the package's ExecStartPre script is
not executable by user ziti (203/EXEC), so the unit never started, and under
set -e `systemctl enable --now` aborted install.sh before config.yaml, ACLs,
the bootstrap grant and the timer were written — a second kit was needed.
- run-host drop-in clears ExecStartPre= (run-host does not auto-enrol JWTs)
- a tunneler start failure is a WARNING until everything else is installed,
then a hard error at the end (re-run needs no new kit)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KLB7jieMNRkTsJ2epr4Ds1
The first real kit run (env-qa-02) downloaded
monky-deployd_26.04 LTS (Resolute Raccoon)_amd64.deb: sourcing /etc/os-release
inline overwrote VERSION. Script vars are now DEPLOYD_VERSION and os-release is
read in a subshell.
Per the operator's rule (end users run ONE script), install.sh now installs
Docker Engine when absent — Docker's apt suite for the host codename, falling
back to Ubuntu's docker.io — with --docker-data-root <dir> to place the
data-root before first start. Existing Docker is left untouched.
bash -n + shellcheck -S warning clean.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KLB7jieMNRkTsJ2epr4Ds1
The GitLab project is private (its parent groups are private, so it cannot be
made public): the v0.1.1 one-liner answered 401 anonymously. install.sh gains
--token / MONKY_DEPLOYD_TOKEN and sends `DEPLOY-TOKEN: <token>` (a GitLab deploy
token, scope read_package_registry only, revocable) on every registry download,
the script itself included; the token goes through a 0600 curl -K file (never the
command line, the log or an xtrace). The grant is taken via --bootstrap-file when
the script is piped (stdin IS the script). Ansible: monky_deployd_download_token
(vaulted) -> DEPLOY-TOKEN header, no_log. Docs explain why, the token's scope and
the --source gitea alternative (split-horizon Gitea, cbs/iac#102).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KLB7jieMNRkTsJ2epr4Ds1
Inside the estate gitea.cbs.tikali.net resolves to jump1's RED EIP (10.10.0.175),
which has no HTTP ingress, so backend boxes could not download the install
artefacts from the Gitea mirror (cbs/iac#102). scm.tikali.ai is reachable from
those boxes and the project is now public, so the GitLab generic package
registry becomes the PRIMARY source:
- packaging/install.sh: default source = scm.tikali.ai generic package registry
(projects/69/packages/generic/monky-deployd/<ver>/...); `--source gitea` /
MONKY_DEPLOYD_SOURCE=gitea keeps the Gitea release as the off-estate
alternative; --base-url / MONKY_DEPLOYD_BASE_URL still override the base.
- ansible role defaults: monky_deployd_base_url/_deb_url point at the registry,
Gitea layout kept as a commented alternative.
- README / docs/OPERATIONS.md / CLAUDE.md / CI comments + release description:
both locations keep being published (release + release:gitea).
- Version 0.1.1 (the tag gate refuses v* tags whose version != __version__);
tests compare against __version__ instead of a literal. No agent change.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KLB7jieMNRkTsJ2epr4Ds1
mdella