deployd#3 (DD-0620): every kit for a backend registered since 2026-09-08 died at `--site`. env-dev-08 (2026-09-11..13): two days of "identity is not readable" ticks — ziti-edge-tunnel re-creates the file with mode 0600, the ACL mask goes to ---, group membership stops helping. identity-acl.sh + monky-deployd-identity-acl.path re-apply the grant on every directory change. Doc-Drift: DD-0620 fixed Closes #3 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ASnneBmT7rfaJLE8NGNw7S
11 KiB
Changelog
0.1.10 — sites fmt|pdx|roam, and a read grant that outlives the tunneller (2026-09-13)
- Install kits for every backend registered since 2026-09-08 died at
--site(deployd#3, DD-0620): tenancy 0.7.x issuesfmt | pdx | roam, deployd still enforcedcbs | pdx.SITES,install.shandconfig.example.yamlnow acceptfmt | cbs | pdx | roam(cbs= the deprecated alias offmton rows registered before the rename) and a test pins the set to tenancy's. - The agent's read on the ziti identity now survives a tunneller rewrite for real. 0.1.8 made
monky-deployda member of thezitigroup; that is not enough:ziti-edge-tunnelre-creates the file with mode0600, which sets the ACL mask to---and defeats the group read and the named-user entry alike (env-dev-08, 2026-09-11 → 09-13: two days of "identity is not readable" ticks under a healthy tunnel; the backend wentoffline). Newidentity-acl.shre-applies the entries and the mask;monky-deployd-identity-acl.pathre-runs it whenever the identity directory changes;install.shand the package postinstall arm it.
0.1.9 — an upgrade no longer stops the agent (2026-09-09)
dpkg -iover a running agent disabled it. dpkg calls the OLD package'sprermon an upgrade as well as on a removal (rpm passes a remaining-instance count), andpreremove.shransystemctl disable --now monky-deployd.timerunconditionally. Upgrading env-dev-01 and env-dev-08 from 0.1.6/0.1.7 to 0.1.8 stopped and disabled both agents. It is silent: the box stays up, the containers keep running, and nothing reports that check-ins have ceased — the backend simply stops converging.preremove.shnow returns early for every upgrade shape (upgrade,failed-upgrade,deconfigure, rpm's1), andpostinstall.shtry-restarts the long-lived proxy unit so it picks up the new code. A fleet upgrade would have taken every agent offline at once.
0.1.8 — onboarding: keep the identity readable, refuse a full disk (2026-09-09)
Three faults from one onboarding (env-dev-08, agent-managed, 2026-09-09), each of which sent the diagnosis somewhere other than the fault.
- The agent's read on the ziti identity now survives a tunneller rewrite.
install.shgranted it with a POSIX ACL.ziti-edge-tunnelrewrites the identity file whenever the controller sends a config update, and the rewrite drops the ACL — the agent had applied cleanly and was failing every tick six minutes later. The durable grant is group membership:install.shand the package's postinstall now addmonky-deploydto the identity file's group (ziti) when it exists, and a default ACL on the identity directory carries the grant onto a freshly created file. The explicit ACLs stay. - An unusable identity says so.
openziti.load()accepts an unreadable or malformed identity, logs "configuration is invalid" at the C layer and hands back a context that fails LATER, at dial, as a bareTypeError— which the transport reported as a missing intercept or a policy gap. The SDK transport now reads and parses the identity itself first, and names the real fault (absent, not readable by this uid, or not JSON) before anything touches the mesh. - A full disk is refused before the pull, and the right disk is measured. The pre-flight ran only
when the bundle declared
disk_need_bytes; a bundle without one sailed past it and died mid-pull with containerd'sno space left on device, which reads as a registry fault. A bundle that declares no size now has to clear the headroom floor. The pre-flight also measures containerd's root as well as the docker data-root: docker 29 keeps image layers in the containerd image store, and on env-dev-08 those were on different filesystems — 93 GiB free where the agent looked, 2.8 GiB where the pull wrote.
0.1.7 — registry credential + a pull error you can read (2026-09-08)
- No hand
docker loginon a new box (monky-design-docs !225, doc 24 §4a). A manifest entry markeduse: registry-authis the Harbor pull credential, not an env var: it is written to<state_dir>/docker/config.json(0600, in a directory the agent owns) and the docker CLI is pointed at it with an explicitDOCKER_CONFIG. That last part is the trap — the unit runs asmonky-deployd, so adocker loginby a human or by root is invisible to the agent and looks exactly like no credential at all (env-dev-01, 2026-09-08). Both seeded shapes are accepted: a JSON object, orusername:passwordwith the registry from the newregistry_hostconfig. compose pullfailures carry the registry's own message instead ofrc=1. "no basic auth credentials", "manifest unknown" and DNS failures now reach the journal and the report to tenancy.
0.1.6 — 2026-09-07
- A valid lease token is reused across applies. Every apply requested a new lease, so a failing deploy
retried by the 60 s timer burned tenancy's 5-leases-per-hour budget and then failed on
LEASE_RATE_LIMITEDforever (env-qa-02 pilot). Now: reuse a lease-derived token whilelookup-selfsays it is valid; swap the bootstrap token for a lease once; and if tenancy rate-limits the lease while a working token exists, apply with it and defer the swap instead of failing the deploy.
0.1.5 — 2026-09-07
-
tenancy.portdefaults to 443 everywhere — the ansible role default and theTenancyCfgdefault now matchinstall.sh/config.example.yaml(0.1.5 only changed those two); PROTOCOL.md states the intercept port separately from the in-pod 8081, that the broker does not add#monky-deploy-agentyet, that the jwt mount keeps nojtistate, and thatrelease:giteahas not run (doc-drift DD-0523/0525/0527/0528). -
${VAR}inside comment lines is not a reference. The renderer's.env.templateheader literally says "substitutes every ${VAR}", which the refusal check counted as an unresolved variable (ENV_INCOMPLETE: unresolved: VAR) — the first bundle on env-qa-02 was refused for it. -
config:
tenancy.portis the service's intercept port (443), not the in-pod 8081. With 8081 the SDK found no intercept and the dial failed (service not available, then a bareTypeErrorfrom the SDK's fallback).install.shnow writes 443;config.example.yamlupdated. -
transport sdk: clear error instead of a TypeError when an address has no intercept or the identity has no dial policy for the service (
TransportErrornames the host:port and what to check).
0.1.4 — 2026-09-07
- install.sh:
ziti-edge-tunnel.servicefailed to start after enrolment (203/EXEC: the package'sExecStartPrescript is not executable by userzition ziti-edge-tunnel 1.18.x) and, because the unit was enabled with--nowunderset -e, the script aborted BEFORE writing config/ACLs/grant/timer — a second kit was needed. The run-host drop-in now clearsExecStartPre=; a tunneler start failure is a WARNING until everything else is in place, then a hard error at the end.
0.1.3 — 2026-09-06
- install.sh:
--versionwas clobbered by/etc/os-release(it definesVERSION), so the kit tried to downloadmonky-deployd_26.04 LTS (Resolute Raccoon)_amd64.deb(env-qa-02 pilot, first real kit run). Script variables are nowDEPLOYD_VERSION; os-release is read in a subshell. - install.sh installs Docker Engine when absent (Docker's apt suite for the codename → fallback
docker.io), with--docker-data-root <dir>to place the data-root before first start. One script for the end user, per the operator's rule.
v0.1.2 — 2026-09-05
- The GitLab project is private (its parent groups are private, so it cannot be made public;
found when the v0.1.1 one-liner returned 401 anonymously).
install.shgains--token <deploy-token>/MONKY_DEPLOYD_TOKEN: every download from the generic package registry — the script itself included — sendsDEPLOY-TOKEN: <token>, a read-only GitLab deploy token (scoperead_package_registryonly, revocable), seeded in OpenBao atmonky/monky-tenancy/deployd-download(keytoken) and handed to the box by the monky-tenancy install kit. The token never reaches the command line, the log or an xtrace (curl-Kconfig file, 0600, deleted after the download;set +xforced). The one-liner now fetchesinstall.shfrom the registry (…/monky-deployd/<ver>/install.sh) instead of-/raw/main, and takes the bootstrap grant via--bootstrap-file— withcurl … | bash -s --stdin IS the script, so it is no longer read for the grant in that mode.--source gitea(no token) stays the off-estate alternative. - Ansible role:
monky_deployd_download_token(vaulted) →DEPLOY-TOKENheader on both fetches,no_log: true. - README / OPERATIONS: why (private project; split-horizon Gitea, cbs/iac#102), token scope, revocation. No agent behaviour change.
v0.1.1 — 2026-09-05
install.sh/ ansible role / README: the primary download is the public GitLab project's generic package registry on scm.tikali.ai (/api/v4/projects/69/packages/generic/monky-deployd/<ver>/…); the one-liner fetches the script fromscm.tikali.ai/…/-/raw/main/packaging/install.sh. Inside the estategitea.cbs.tikali.netis split-horizon to jump1's RED EIP (no HTTP ingress), so backend boxes could not fetch the artefacts (cbs/iac#102). The Gitea release stays the off-estate alternative (--source gitea/MONKY_DEPLOYD_SOURCE=gitea). No agent behaviour change.
v0.1.0 — 2026-09-05
First release (MONKY-ADR-0028 §D, Reconciliation v2, Gate 1 v2).
- Stdlib-only Python 3.12 agent:
run --once|loop,status,bootstrap,--prune. - Protocol:
POST /v1/agent/checkin(action apply|none|down),GET /v1/agent/bundle/{env}/{sha}(sha256 verified),POST /v1/agent/lease→ deploy grant (login_jwt),POST /v1/agent/report(result applied|failed|down, redactedlog_tail). An AppRole-shaped lease is refused (LEASE_SHAPE). - OpenBao:
POST /v1/auth/jwt-tenancy/login {"role":"see-env","jwt":…}; KV-v2 reads pinned to the manifest's versions, paths pinned tomonky/data/<env>/see/; renew-self / re-lease before max TTL. - Refusals:
ENV_INCOMPLETE(names only),PRIVILEGED_REFUSED,ROLLBACK_REFUSED,DISK_INSUFFICIENT(need × 1.5 + 2 GiBvs docker data-root),BUNDLE_SHA_MISMATCH,BUNDLE_ENV_MISMATCH. - Transports
sdk(openziti SDK),proxy(ziti tunnel proxy18443/18200),system. - Exit codes 0 / 75 / 78 (
AGENT_ENV_MISMATCH, no retry storm) / 1; laptop mode (offline exits 0). - Packaging: hardened
monky-deployd.serviceoneshot + 60 s timer,monky-deployd-proxy.service,nfpm.debwith/opt/monky-deployd/venv,packaging/install.sh(Ubuntu 26.04), ansible role skeleton. - Known divergence: monky-tenancy
main(MR !15) still ships the AppRole lease/kit; the JWT-grant follow-up is the tenancy side of this release (docs/PROTOCOL.md "Divergences").