Merge branch 'fix/upgrade-keeps-the-timer' into 'main'

fix(packaging): an upgrade must not stop and disable the agent — 0.1.9

See merge request tikali/applications/monky/monky-deployd!14

CHANGELOG.md

@@ -1,6 +1,53 @@
 <!-- xlate:verbatim-fences -->
 # Changelog
 
+## 0.1.9 — an upgrade no longer stops the agent (2026-09-09)
+
+- **`dpkg -i` over a running agent disabled it.** dpkg calls the OLD package's `prerm` on an
+  **upgrade** as well as on a removal (rpm passes a remaining-instance count), and `preremove.sh`
+  ran `systemctl disable --now monky-deployd.timer` unconditionally. Upgrading env-dev-01 and
+  env-dev-08 from 0.1.6/0.1.7 to 0.1.8 stopped and **disabled** both agents. It is silent: the box
+  stays up, the containers keep running, and nothing reports that check-ins have ceased — the
+  backend simply stops converging. `preremove.sh` now returns early for every upgrade shape
+  (`upgrade`, `failed-upgrade`, `deconfigure`, rpm's `1`), and `postinstall.sh` `try-restart`s the
+  long-lived proxy unit so it picks up the new code. A fleet upgrade would have taken every agent
+  offline at once.
+
+## 0.1.8 — onboarding: keep the identity readable, refuse a full disk (2026-09-09)
+
+Three faults from one onboarding (env-dev-08, agent-managed, 2026-09-09), each of which sent the
+diagnosis somewhere other than the fault.
+
+- **The agent's read on the ziti identity now survives a tunneller rewrite.** `install.sh` granted it
+  with a POSIX ACL. `ziti-edge-tunnel` rewrites the identity file whenever the controller sends a config
+  update, and the rewrite drops the ACL — the agent had applied cleanly and was failing every tick six
+  minutes later. The durable grant is **group membership**: `install.sh` and the package's postinstall
+  now add `monky-deployd` to the identity file's group (`ziti`) when it exists, and a **default ACL** on
+  the identity directory carries the grant onto a freshly created file. The explicit ACLs stay.
+- **An unusable identity says so.** `openziti.load()` accepts an unreadable or malformed identity, logs
+  "configuration is invalid" at the C layer and hands back a context that fails LATER, at dial, as a
+  bare `TypeError` — which the transport reported as a missing intercept or a policy gap. The SDK
+  transport now reads and parses the identity itself first, and names the real fault (absent, not
+  readable by this uid, or not JSON) before anything touches the mesh.
+- **A full disk is refused before the pull, and the right disk is measured.** The pre-flight ran only
+  when the bundle declared `disk_need_bytes`; a bundle without one sailed past it and died mid-pull with
+  containerd's `no space left on device`, which reads as a registry fault. A bundle that declares no size
+  now has to clear the headroom floor. The pre-flight also measures **containerd's root as well as the
+  docker data-root**: docker 29 keeps image layers in the containerd image store, and on env-dev-08 those
+  were on different filesystems — 93 GiB free where the agent looked, 2.8 GiB where the pull wrote.
+
+## 0.1.7 — registry credential + a pull error you can read (2026-09-08)
+
+- **No hand `docker login` on a new box** (monky-design-docs !225, doc 24 §4a). A manifest entry marked
+  `use: registry-auth` is the Harbor pull credential, not an env var: it is written to
+  `<state_dir>/docker/config.json` (0600, in a directory the agent owns) and the docker CLI is pointed at it
+  with an explicit **`DOCKER_CONFIG`**. That last part is the trap — the unit runs as `monky-deployd`, so a
+  `docker login` by a human or by root is invisible to the agent and looks exactly like no credential at all
+  (env-dev-01, 2026-09-08). Both seeded shapes are accepted: a JSON object, or `username:password` with the
+  registry from the new `registry_host` config.
+- **`compose pull` failures carry the registry's own message** instead of `rc=1`. "no basic auth
+  credentials", "manifest unknown" and DNS failures now reach the journal and the report to tenancy.
+
 ## 0.1.6 — 2026-09-07
 
 - **A valid lease token is reused across applies.** Every apply requested a new lease, so a failing deploy

config.example.yaml

@@ -44,3 +44,7 @@ laptop_mode: false                   # true: offline exits 0 quietly; run withou
 # compose_project: monky-env-qa-02   # docker compose project name
 # docker_bin: docker
 # log_level: INFO
+
+# The registry the bundle pulls from. Used only when the seeded pull credential is a bare
+# `username:password` (a JSON credential names its own registry). doc 24 §4a.
+registry_host: harbor.tikali.net

docs/OPERATIONS.md

@@ -22,7 +22,17 @@ monky-deployd status --json | jq .
 Files: `/etc/monky-deployd/config.yaml` (0640 root:monky-deployd), `/etc/monky-deployd/openbao-ca.pem`,
 `/etc/monky-deployd/bootstrap.jwt` (only until the first login), `/var/lib/monky-deployd/{bao.token,state.json,lock}`,
 `/var/lib/monky-deployd/<env>/releases/<sha>/` + `current` (the compose project dir, `.env` 0600),
-`/opt/openziti/etc/identities/monky-host.<env>.json` (ziti:ziti 0600 + ACL `u:monky-deployd:r`).
+`/opt/openziti/etc/identities/monky-host.<env>.json` (ziti:ziti 0640; the agent reads it as a member of
+the `ziti` **group**, plus an explicit ACL `u:monky-deployd:r` and a default ACL on the directory).
+
+> The group is the grant that matters. `ziti-edge-tunnel` rewrites the identity file whenever the
+> controller sends a config update, and the rewrite drops POSIX ACLs — an ACL-only box works until the
+> first refresh and then fails every tick (env-dev-08, 2026-09-09). On a host where the tunneller was
+> installed independently of the kit, check `id monky-deployd` for `ziti` before anything else.
+
+> **Upgrading the agent.** `dpkg -i` keeps the timer enabled from 0.1.9 on. On a box upgraded from
+> 0.1.8 or earlier the old package's `prerm` already ran and **disabled** it — check
+> `systemctl is-enabled monky-deployd.timer` after the upgrade and `systemctl enable --now` it if needed.
 
 ## Reading the journal
 
@@ -31,9 +41,10 @@ Files: `/etc/monky-deployd/config.yaml` (0640 root:monky-deployd), `/etc/monky-d
 | `checkin: action=none desired=… applied=…` then `healthy; heartbeat reported` | converged |
 | `checkin: action=apply …` → `read 3 secret(s): GEMINI_API_KEY, …` → `promoted release …` → `applied …` | a deploy |
 | `refused: ENV_INCOMPLETE: unresolved: X` | the bundle needs a variable no manifest entry supplies — fix the descriptor / set the secret in the console; nothing was started |
-| `refused: DISK_INSUFFICIENT: docker data-root has N MiB free, bundle needs M MiB` | free space (the env-dev-09 lesson): grow the data-root disk or prune |
+| `refused: DISK_INSUFFICIENT: image storage (…) has N MiB free, …` | free space: grow the disk or prune. The paths in the message are every filesystem a pull can fill — docker's data-root AND containerd's root, which docker 29 fills with image layers and which is often a different disk (env-dev-08). A bundle that declares no size must still clear `disk.headroom_bytes` |
 | `refused: PRIVILEGED_REFUSED` / `ROLLBACK_REFUSED` | the bundle needs `allow_privileged` / `allow_rollback` in its `agent` profile |
 | `temporary network failure` (exit 75) | mesh/tenancy unreachable — check `ziti-edge-tunnel`, the identity's terminators, `monky.tenancy.deploy` health |
+| `transport sdk: identity … is not readable by this user` | the agent lost its read on the identity file (usually a tunneller rewrite dropping the ACL) — `usermod -a -G ziti monky-deployd` |
 | `AGENT_UNAUTHENTICATED: bearer refused` (exit 1) | the grant was superseded (kit re-revealed / retire) or the token revoked → re-run the install kit |
 | `AGENT_ENV_MISMATCH` (exit 78) | the token belongs to another env than `config.yaml` — fix the config or re-issue the identity; the timer keeps firing but every tick exits 78 immediately (no storm) |
 | `failed: docker compose pull failed (rc=1)` | registry/pull problem; compose output is in the report's tail and in the journal |

monky_deployd/__init__.py

@@ -4,4 +4,4 @@ Dials monky-tenancy over the mesh with the box's host identity, fetches the rend
 leases a deploy grant, logs in to OpenBao, reads its own secrets, runs `docker compose`,
 reports. Stdlib only; the optional `openziti` SDK is the `sdk` transport."""
 
-__version__ = "0.1.6"
+__version__ = "0.1.9"

monky_deployd/agent.py

@@ -24,6 +24,7 @@ from pathlib import Path
 
 from monky_deployd import __version__
 from monky_deployd import bundle as bundlemod
+from monky_deployd import registry as registrymod
 from monky_deployd import state as statemod
 from monky_deployd.bao import BaoClient, BaoError, BaoToken, ManifestPathError, kv_data_path
 from monky_deployd.bundle import Bundle, BundleError
@@ -86,7 +87,7 @@ class Agent:
     def __init__(self, cfg: Config, *, prune: bool = False, docker: Docker | None = None):
         self.cfg = cfg
         self.prune = prune
-        self.docker = docker or Docker(cfg.docker_bin)
+        self.docker = docker or Docker(cfg.docker_bin, docker_config=str(cfg.docker_config_dir))
         self.transport = build(cfg)
         self.tenancy_http = HttpClient(
             self.transport, cfg.tenancy.scheme, cfg.tenancy.host, cfg.tenancy.port, timeout=cfg.tenancy.timeout_s
@@ -274,13 +275,29 @@ class Agent:
         if hdr_sha and hdr_sha != b.sha:
             log.warning("bundle header sha %s disagrees with content %s", _short(hdr_sha), _short(b.sha))
         self._refusal_checks(b)
-        # secrets: lease -> login -> reads (values never logged; names only)
+        # secrets: lease -> login -> reads (values never logged; names only). An entry marked
+        # `use: registry-auth` is NOT an env var — it is the pull credential, written to the
+        # agent's own Docker config instead of `.env` (doc 24 §4a).
         entries = b.manifest.get("entries", [])
+        registry_entries = [e for e in entries if e.get("use") == registrymod.USE]
+        env_entries = [e for e in entries if e.get("use") != registrymod.USE]
         if entries:
             token = self._token_for_apply()
-            for e in entries:
+            for e in env_entries:
                 self._values[e["var"]] = self.bao.kv_read(token, e["path"], cfg.env_id, e.get("version"))
-            log.info("read %d secret(s): %s", len(entries), ", ".join(sorted(self._values)))
+            for e in registry_entries:
+                raw = self.bao.kv_read(token, e["path"], cfg.env_id, e.get("version"))
+                try:
+                    auth = registrymod.parse(raw, default_registry=cfg.registry_host)
+                    registrymod.write_docker_config(cfg.docker_config_dir, auth)
+                except registrymod.RegistryAuthError as exc:
+                    log.warning("registry credential unusable (%s) — private images will not pull", exc)
+            log.info(
+                "read %d secret(s): %s%s",
+                len(entries),
+                ", ".join(sorted(self._values)),
+                " (+ registry credential)" if registry_entries else "",
+            )
         env_text = bundlemod.render_env(b.env_template, self._values)
         leftover = bundlemod.referenced_vars(env_text)
         if leftover:
@@ -288,7 +305,17 @@ class Agent:
         release = self._promote(self._stage(b, env_text))
         compose = self._compose()
         log.info("compose pull")
-        compose.pull()
+        try:
+            compose.pull()
+        except ComposeError as exc:
+            # the registry's own message is the diagnosis ("no basic auth credentials", "manifest
+            # unknown", a DNS failure). Swallowing it behind `rc=1` cost an SSH hunt on env-dev-01.
+            detail = " | ".join(line.strip() for line in (exc.output or "").splitlines() if line.strip())[-600:]
+            self.state.last_result = "failed"
+            self.state.last_error = f"compose pull failed: {detail or exc}"
+            log.error("compose pull failed (rc=%s): %s", exc.rc, detail or "(no output)")
+            self._report("failed", b.sha, detail=f"compose pull: {detail or exc}"[:900])
+            return EX_FAIL
         log.info("compose up -d --remove-orphans")
         compose.up()
         ok, containers = compose.wait_healthy(cfg.healthy_timeout_s)
@@ -326,16 +353,24 @@ class Agent:
             raise Refusal("PRIVILEGED_REFUSED", ", ".join(findings) + " (bundle.json allow_privileged is not set)")
         if self.state.is_rollback(b.sha) and not b.flag("allow_rollback"):
             raise Refusal("ROLLBACK_REFUSED", f"{b.sha[:12]} was applied before; allow_rollback is not set")
+        # A bundle that declares its size gets the full sum; one that does not still has to clear
+        # the headroom floor. Without the floor a nearly-full box sails past this check and dies
+        # mid-pull with containerd's "no space left on device", which reads as a registry fault
+        # and costs an SSH hunt (env-dev-08, 2026-09-09).
         need = b.disk_need_bytes
-        if need:
+        free = self.docker.free_bytes()
-            free = self.docker.free_bytes()
+        required = int(need * cfg.disk.factor + cfg.disk.headroom_bytes) if need else cfg.disk.headroom_bytes
-            required = int(need * cfg.disk.factor + cfg.disk.headroom_bytes)
+        if free is not None and free < required:
-            if free is not None and free < required:
+            sized = (
-                raise Refusal(
+                f"bundle needs {required // 2**20} MiB "
-                    "DISK_INSUFFICIENT",
+                f"({need // 2**20} MiB x {cfg.disk.factor} + {cfg.disk.headroom_bytes // 2**20} MiB headroom)"
-                    f"docker data-root has {free // 2**20} MiB free, bundle needs {required // 2**20} MiB "
+                if need
-                    f"({need // 2**20} MiB x {cfg.disk.factor} + {cfg.disk.headroom_bytes // 2**20} MiB headroom)",
+                else f"a pull needs at least {required // 2**20} MiB headroom (bundle declares no size)"
-                )
+            )
+            raise Refusal(
+                "DISK_INSUFFICIENT",
+                f"image storage ({', '.join(self.docker.storage_paths())}) has {free // 2**20} MiB free, {sized}",
+            )
 
     # -- credentials -------------------------------------------------------------------------------------
     def _ensure_token(self) -> str:

monky_deployd/compose.py

@@ -12,6 +12,10 @@ import time
 from dataclasses import dataclass
 from pathlib import Path
 
+# containerd's default root: docker 29's image store lives here, often on another filesystem
+# than DockerRootDir. Both are checked before a pull (see Docker.storage_paths).
+CONTAINERD_ROOTS = ("/var/lib/containerd",)
+
 log = logging.getLogger("monky-deployd.compose")
 
 
@@ -41,9 +45,13 @@ class Container:
 
 
 class Docker:
-    def __init__(self, docker_bin: str = "docker", timeout_s: int = 600):
+    def __init__(self, docker_bin: str = "docker", timeout_s: int = 600, docker_config: str | None = None):
         self.bin = docker_bin
         self.timeout_s = timeout_s
+        # doc 24 §4a: registry credentials live in a directory the AGENT owns, named explicitly
+        # rather than inherited from $HOME. The unit runs as `monky-deployd`, so a `docker login`
+        # by a human or by root is invisible here — which is exactly what cost env-dev-01 an hour.
+        self.docker_config = docker_config
 
     def available(self) -> bool:
         return shutil.which(self.bin) is not None
@@ -59,7 +67,7 @@ class Docker:
                 capture_output=True,
                 text=True,
                 timeout=timeout or self.timeout_s,
-                env={**os.environ, "COMPOSE_INTERACTIVE_NO_CLI": "1"},
+                env=self._env(),
             )
         except FileNotFoundError as exc:
             raise ComposeError(what, 127, f"{self.bin} not found") from exc
@@ -70,6 +78,12 @@ class Docker:
             raise ComposeError(what, p.returncode, out.strip())
         return out
 
+    def _env(self) -> dict:
+        env = {**os.environ, "COMPOSE_INTERACTIVE_NO_CLI": "1"}
+        if self.docker_config:
+            env["DOCKER_CONFIG"] = self.docker_config
+        return env
+
     # -- facts ------------------------------------------------------------------------------------
     def version(self) -> str | None:
         try:
@@ -90,8 +104,23 @@ class Docker:
             root = ""
         return root or "/var/lib/docker"
 
-    def free_bytes(self, path: str | None = None) -> int | None:
+    def storage_paths(self) -> list[str]:
-        p = path or self.data_root()
+        """Every filesystem a `compose pull` can fill.
+
+        docker 29 keeps IMAGE layers in the containerd image store (containerd's own root,
+        /var/lib/containerd by default), NOT under DockerRootDir. On a box where those two sit
+        on different filesystems, measuring only the data-root reports plenty of room while the
+        pull dies with "no space left on device" (env-dev-08, 2026-09-09: 93 GiB free on the
+        data-root, 2.8 GiB on the root filesystem that held containerd).
+        """
+        paths = [self.data_root()]
+        for extra in CONTAINERD_ROOTS:
+            if os.path.isdir(extra):
+                paths.append(extra)
+        return paths
+
+    def _free_at(self, path: str) -> int | None:
+        p = path
         while p and not os.path.exists(p):
             p = os.path.dirname(p)
         try:
@@ -100,6 +129,12 @@ class Docker:
             return None
         return st.f_bavail * st.f_frsize
 
+    def free_bytes(self, path: str | None = None) -> int | None:
+        """Free bytes on `path`, or the TIGHTEST of the image-storage filesystems."""
+        paths = [path] if path else self.storage_paths()
+        seen = [v for v in (self._free_at(p) for p in paths) if v is not None]
+        return min(seen) if seen else None
+
     def image_prune(self) -> None:
         try:
             self.run(["image", "prune", "-f"], timeout=300)

monky_deployd/config.py

@@ -175,6 +175,9 @@ class Config:
     healthy_timeout_s: int = 300
     compose_project: str = ""
     docker_bin: str = "docker"
+    # doc 24 §4a: the registry the bundle pulls from, used when the seeded credential is a bare
+    # `username:password` (a JSON credential names its own registry).
+    registry_host: str = "harbor.tikali.net"
     log_level: str = "INFO"
     path: str = DEFAULT_CONFIG_PATH
 
@@ -191,6 +194,13 @@ class Config:
     def lock_path(self) -> Path:
         return Path(self.state_dir) / "lock"
 
+    @property
+    def docker_config_dir(self) -> Path:
+        """Where the agent keeps its OWN registry credentials (`DOCKER_CONFIG`). Not `$HOME`: the
+        unit runs as `monky-deployd`, and a human's or root's `docker login` must not be what the
+        agent depends on (doc 24 §4a)."""
+        return Path(self.state_dir) / "docker"
+
     @property
     def is_prod(self) -> bool:
         if self.prod is not None:

monky_deployd/registry.py

@@ -0,0 +1,80 @@
+"""The registry credential (monky-design-docs doc 24 §4a).
+
+The bundle's manifest carries one entry marked `use: registry-auth` — the estate-wide read-only
+Harbor robot, copied by monky-tenancy into this environment's own prefix so the agent can read it
+with the OpenBao policy it already has. It is NOT an env var (an entry in `.env` would put the
+registry password into every container's environment), so it never reaches the compose file: it is
+written to a Docker config **the agent owns**, and `DOCKER_CONFIG` points the docker CLI at it.
+
+That last part is the whole point. The unit runs as `monky-deployd`, whose home is the state dir,
+so a `docker login` performed by a human or by root is invisible to it — the failure looks exactly
+like "no credentials at all" (env-dev-01, 2026-09-08).
+
+Accepted shapes for the secret's value, because the seeded robot has been written both ways:
+  * a JSON object: `{"registry": …, "username": …, "password": …}`
+  * a `username:password` string, with the registry taken from `registry_host` config
+"""
+
+from __future__ import annotations
+
+import base64
+import json
+import logging
+from dataclasses import dataclass
+from pathlib import Path
+
+log = logging.getLogger("monky-deployd.registry")
+
+USE = "registry-auth"
+
+
+class RegistryAuthError(Exception):
+    pass
+
+
+@dataclass
+class RegistryAuth:
+    registry: str
+    username: str
+    password: str
+
+    def docker_config(self) -> dict:
+        token = base64.b64encode(f"{self.username}:{self.password}".encode()).decode()
+        return {"auths": {self.registry: {"auth": token}}}
+
+
+def parse(value: str, *, default_registry: str) -> RegistryAuth:
+    """`value` is whatever the KV entry held; never logged, never echoed."""
+    text = (value or "").strip()
+    if not text:
+        raise RegistryAuthError("empty registry credential")
+    if text.startswith("{"):
+        try:
+            data = json.loads(text)
+        except json.JSONDecodeError as exc:
+            raise RegistryAuthError("registry credential is not valid JSON") from exc
+        user, pw = data.get("username"), data.get("password")
+        registry = data.get("registry") or default_registry
+        if not user or not pw:
+            raise RegistryAuthError("registry credential JSON needs username + password")
+        return RegistryAuth(registry=str(registry), username=str(user), password=str(pw))
+    if ":" not in text:
+        raise RegistryAuthError("registry credential is neither JSON nor username:password")
+    user, _, pw = text.partition(":")
+    if not default_registry:
+        raise RegistryAuthError("username:password credential needs a configured registry host")
+    return RegistryAuth(registry=default_registry, username=user, password=pw)
+
+
+def write_docker_config(dir_path: Path, auth: RegistryAuth) -> Path:
+    """0600 `config.json` in a directory the agent owns; DOCKER_CONFIG points the CLI at it."""
+    dir_path = Path(dir_path)
+    dir_path.mkdir(parents=True, exist_ok=True)
+    dir_path.chmod(0o700)
+    target = dir_path / "config.json"
+    tmp = dir_path / "config.json.tmp"
+    tmp.write_text(json.dumps(auth.docker_config(), indent=2) + "\n")
+    tmp.chmod(0o600)
+    tmp.replace(target)
+    log.info("registry credential in place for %s (%s)", auth.registry, auth.username)
+    return target

monky_deployd/transport.py

@@ -16,6 +16,7 @@ from __future__ import annotations
 import http.client
 import json
 import logging
+import os
 import socket
 import ssl
 from dataclasses import dataclass
@@ -35,6 +36,27 @@ class TransportError(Exception):
 class Transport:
     name = "base"
 
+    def _check_identity_readable(self) -> None:
+        path = self.identity_path
+        try:
+            with open(path, "rb") as fh:
+                raw = fh.read()
+        except FileNotFoundError as exc:
+            raise TransportError(f"transport sdk: identity {path} does not exist") from exc
+        except PermissionError as exc:
+            raise TransportError(
+                f"transport sdk: identity {path} is not readable by this user "
+                f"(uid {os.geteuid()}) — the file is owned by the tunneller and rewritten on every "
+                "config refresh, which drops POSIX ACLs; add the agent's user to the file's group "
+                "(usually `ziti`) so the grant survives"
+            ) from exc
+        except OSError as exc:
+            raise TransportError(f"transport sdk: identity {path} is unreadable: {exc}") from exc
+        try:
+            json.loads(raw)
+        except ValueError as exc:
+            raise TransportError(f"transport sdk: identity {path} is not valid JSON: {exc}") from exc
+
     def connect(self, host: str, port: int, timeout: float) -> socket.socket:  # pragma: no cover
         raise NotImplementedError
 
@@ -87,6 +109,13 @@ class SdkTransport(Transport):
                 "transport sdk: the openziti module is not installed in this venv; "
                 "use transport: proxy (monky-deployd-proxy.service) or system"
             ) from exc
+        # openziti.load() does NOT raise on an unreadable or malformed identity: the C SDK logs
+        # "Failed to load Ziti Identity ...: configuration is invalid" and hands back a context
+        # that fails LATER, at dial, as a bare TypeError — which reads as a missing intercept or
+        # a policy gap and sends you hunting the mesh instead of the file (env-dev-08,
+        # 2026-09-09: ziti-edge-tunnel rewrote the identity and dropped the agent's ACL).
+        # So check the file ourselves first and name the real fault.
+        self._check_identity_readable()
         try:
             self._ctx = openziti.load(self.identity_path)
         except Exception as exc:

packaging/install.sh

@@ -213,8 +213,15 @@ fi
 # --- 4. config, ACLs, bootstrap grant -----------------------------------------------------------------
 install -d -m 0750 -o root -g monky-deployd "$ETC"
 install -d -m 0700 -o monky-deployd -g monky-deployd /var/lib/monky-deployd
+# The identity file is the agent's only credential for the mesh. ziti-edge-tunnel rewrites it on
+# every controller config update and the rewrite DROPS these ACLs, so the durable grant is group
+# membership (the file stays <ziti>:<ziti> 0640 across rewrites); the ACLs below still cover the
+# window before the agent's next login and hosts where the file has another group.
+# A default ACL on the directory carries the grant onto a freshly created identity file.
+if getent group ziti >/dev/null; then usermod -a -G ziti monky-deployd || true; fi
 setfacl -m u:monky-deployd:r "$IDENTITY"
 setfacl -m u:monky-deployd:rx "$IDENTITY_DIR"
+setfacl -d -m u:monky-deployd:r "$IDENTITY_DIR" 2>/dev/null || true
 setfacl -m u:monky-deployd:x /opt/openziti/etc 2>/dev/null || true
 if [ -n "$BAO_CA" ]; then
   install -m 0644 "$BAO_CA" "$ETC/openbao-ca.pem"

packaging/scripts/postinstall.sh

@@ -13,10 +13,18 @@ install -d -m 0700 -o monky-deployd -g monky-deployd /var/lib/monky-deployd
 install -d -m 0750 -o root -g monky-deployd /etc/monky-deployd
 # the agent drives docker compose: docker group membership (no root)
 if getent group docker >/dev/null; then usermod -a -G docker monky-deployd || true; fi
+# the agent reads the host's ziti identity (owned by the tunneller, mode 0640). ziti-edge-tunnel
+# REWRITES that file whenever the controller sends a config update, and the rewrite drops any
+# POSIX ACL granting the agent read — group membership is the grant that survives it.
+# (env-dev-08, 2026-09-09: the agent went from applied to "no intercept" 6 minutes after a refresh.)
+if getent group ziti >/dev/null; then usermod -a -G ziti monky-deployd || true; fi
 # the venv is relocatable only to the path it was built at; refuse a broken interpreter early
 /opt/monky-deployd/venv/bin/python -c 'import monky_deployd' || { echo "monky-deployd: venv unusable (python3 mismatch?)" >&2; exit 1; }
 if [ -d /run/systemd/system ]; then
   systemctl daemon-reload || true
-  # do NOT enable the timer here: install.sh / the ansible role do it after the config exists
+  # do NOT enable the timer here: install.sh / the ansible role do it after the config exists.
+  # An UPGRADE is different — the timer is already enabled and must keep running, so restart the
+  # long-lived proxy unit onto the new code. `try-restart` is a no-op when it is not running.
+  systemctl try-restart monky-deployd-proxy.service 2>/dev/null || true
 fi
 exit 0

packaging/scripts/preremove.sh

@@ -1,5 +1,14 @@
 #!/bin/sh
 set -e
+# dpkg calls the OLD package's prerm on an UPGRADE as well as on a removal, and rpm calls it with
+# an install count. Disabling the timer unconditionally therefore stopped the agent on every
+# upgrade and left it disabled — silently, because the box stays reachable and nothing else
+# notices that check-ins have ceased (env-dev-01 and env-dev-08, 2026-09-09).
+#   dpkg: "$1" is `remove`, `upgrade <new-version>`, `deconfigure …` or `failed-upgrade`
+#   rpm : "$1" is the number of instances that will remain — 1 on upgrade, 0 on uninstall
+case "${1:-}" in
+  upgrade | failed-upgrade | deconfigure | 1) exit 0 ;;
+esac
 if [ -d /run/systemd/system ]; then
   systemctl disable --now monky-deployd.timer 2>/dev/null || true
   systemctl disable --now monky-deployd-proxy.service 2>/dev/null || true

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "monky-deployd"
-version = "0.1.6"
+version = "0.1.9"
 description = "Monky backend pull agent: checkin -> bundle -> lease -> OpenBao -> docker compose -> report, over the ziti mesh (MONKY-ADR-0028)"
 readme = "README.md"
 requires-python = ">=3.12"

tests/conftest.py

@@ -52,6 +52,9 @@ def fake_docker(tmp_path, monkeypatch):
             }
         )
     )
+    # the real agent also measures containerd's root; on a CI runner that path belongs to the
+    # runner's own docker and would make every disk assertion depend on the runner's free space.
+    monkeypatch.setattr("monky_deployd.compose.CONTAINERD_ROOTS", ())
     monkeypatch.setenv("PATH", f"{FAKEBIN}:{os.environ['PATH']}")
     monkeypatch.setenv("FAKE_DOCKER_LOG", str(log))
     monkeypatch.setenv("FAKE_DOCKER_STATE", str(state))

tests/test_agent.py

@@ -237,6 +237,30 @@ def test_disk_refusal(bootstrapped, tenancy, fake_docker):
     assert tenancy.leases == []
 
 
+def test_disk_refusal_without_a_declared_size(bootstrapped, tenancy, fake_docker, monkeypatch):
+    """A bundle that declares no size still has to clear the headroom floor.
+
+    Without the floor a nearly-full box passes the pre-flight and dies mid-pull with
+    containerd's "no space left on device", which reads as a registry fault (env-dev-08).
+    """
+    cfg = bootstrapped
+    cfg.disk.headroom_bytes = 2 * 1024**3
+    monkeypatch.setattr("monky_deployd.compose.Docker.free_bytes", lambda self, path=None: 32 * 2**20)
+    tenancy.set_files(make_files())
+    assert tick(cfg) == EX_FAIL
+    detail = tenancy.reports[-1]["detail"]
+    assert "DISK_INSUFFICIENT" in detail and "declares no size" in detail
+    assert tenancy.leases == []
+
+
+def test_ample_disk_without_a_declared_size_applies(bootstrapped, tenancy, fake_docker, monkeypatch):
+    cfg = bootstrapped
+    monkeypatch.setattr("monky_deployd.compose.Docker.free_bytes", lambda self, path=None: 500 * 2**30)
+    tenancy.set_files(make_files())
+    assert tick(cfg) == EX_OK
+    assert tenancy.reports[-1]["result"] == "applied"
+
+
 def test_legacy_approle_lease_is_refused_loudly(bootstrapped, tenancy, fake_docker):
     cfg = bootstrapped
     tenancy.lease_shape = "approle"

tests/test_cli.py

@@ -79,6 +79,8 @@ def test_sdk_transport_uses_openziti_monkeypatch(monkeypatch, tmp_path):
     srv = socket.socket()
     srv.bind(("127.0.0.1", 0))
     srv.listen(1)
+    # the transport pre-flights the identity file before handing it to the SDK
+    (tmp_path / "id.json").write_text('{"ztAPI": "https://example.invalid"}')
     t = SdkTransport(str(tmp_path / "id.json"))
     s = t.connect("127.0.0.1", srv.getsockname()[1], 2)
     s.close()

tests/test_packaging_scripts.py

@@ -0,0 +1,46 @@
+"""The maintainer scripts must not stop the agent on an upgrade.
+
+dpkg calls the OLD package's prerm on an upgrade as well as on a removal, and rpm calls it with
+an install count. `systemctl disable --now` there stopped the agent on every upgrade and left it
+disabled — silently, because the box stays reachable and nothing else notices that check-ins have
+ceased (env-dev-01 and env-dev-08, 2026-09-09).
+"""
+
+from __future__ import annotations
+
+import shutil
+import subprocess
+from pathlib import Path
+
+import pytest
+
+PREREMOVE = Path(__file__).resolve().parents[1] / "packaging" / "scripts" / "preremove.sh"
+
+
+def _run(arg: str, tmp_path: Path) -> list[str]:
+    """Run preremove with a fake `systemctl` on PATH and report the calls it made."""
+    calls = tmp_path / "systemctl.log"
+    fake = tmp_path / "bin"
+    fake.mkdir(exist_ok=True)
+    (fake / "systemctl").write_text(f'#!/bin/sh\necho "$@" >> {calls}\nexit 0\n')
+    (fake / "systemctl").chmod(0o755)
+    env = {"PATH": f"{fake}:/usr/bin:/bin"}
+    r = subprocess.run(["sh", str(PREREMOVE), arg], env=env, capture_output=True, text=True)
+    assert r.returncode == 0, r.stderr
+    return calls.read_text().splitlines() if calls.exists() else []
+
+
+@pytest.mark.skipif(not Path("/run/systemd/system").is_dir(), reason="needs a systemd host to reach the disable branch")
+@pytest.mark.parametrize("arg", ["remove", "0"])
+def test_a_real_removal_disables_the_units(arg, tmp_path):
+    assert any("disable" in c for c in _run(arg, tmp_path))
+
+
+@pytest.mark.parametrize("arg", ["upgrade", "failed-upgrade", "deconfigure", "1"])
+def test_an_upgrade_leaves_the_units_alone(arg, tmp_path):
+    assert _run(arg, tmp_path) == []
+
+
+def test_the_script_is_shell_clean():
+    assert shutil.which("sh")
+    subprocess.run(["sh", "-n", str(PREREMOVE)], check=True)

tests/test_registry_auth.py

@@ -0,0 +1,53 @@
+"""The registry credential (doc 24 §4a): parsed from either seeded shape, written to a Docker
+config the AGENT owns, never an env var, never logged."""
+
+from __future__ import annotations
+
+import base64
+import json
+
+import pytest
+
+from monky_deployd import registry as registrymod
+
+
+def test_parses_json_and_user_colon_password():
+    a = registrymod.parse(
+        json.dumps({"registry": "harbor.tikali.net", "username": "robot$pull", "password": "p4ss"}),
+        default_registry="ignored.example",
+    )
+    assert (a.registry, a.username, a.password) == ("harbor.tikali.net", "robot$pull", "p4ss")
+    b = registrymod.parse("robot$pull:p4ss", default_registry="harbor.tikali.net")
+    assert (b.registry, b.username, b.password) == ("harbor.tikali.net", "robot$pull", "p4ss")
+    # a password containing a colon survives (partition on the FIRST one)
+    c = registrymod.parse("robot$pull:p4:ss", default_registry="h")
+    assert c.password == "p4:ss"
+
+
+@pytest.mark.parametrize("bad", ["", "   ", "no-colon-here", '{"username": "u"}', "{not json"])
+def test_refuses_what_it_cannot_use(bad):
+    with pytest.raises(registrymod.RegistryAuthError):
+        registrymod.parse(bad, default_registry="harbor.tikali.net")
+
+
+def test_docker_config_is_written_0600_in_a_directory_the_agent_owns(tmp_path):
+    auth = registrymod.parse("robot$pull:p4ss", default_registry="harbor.tikali.net")
+    target = registrymod.write_docker_config(tmp_path / "docker", auth)
+    assert target.exists()
+    assert oct(target.stat().st_mode)[-3:] == "600"
+    assert oct(target.parent.stat().st_mode)[-3:] == "700"
+    cfg = json.loads(target.read_text())
+    token = cfg["auths"]["harbor.tikali.net"]["auth"]
+    assert base64.b64decode(token).decode() == "robot$pull:p4ss"
+    # rewriting is idempotent (the agent does it every tick)
+    registrymod.write_docker_config(tmp_path / "docker", auth)
+    assert json.loads(target.read_text()) == cfg
+
+
+def test_docker_runner_points_the_cli_at_that_directory():
+    from monky_deployd.compose import Docker
+
+    d = Docker("docker", docker_config="/var/lib/monky-deployd/docker")
+    assert d._env()["DOCKER_CONFIG"] == "/var/lib/monky-deployd/docker"
+    # unset when no directory is configured — never silently fall back to a human's $HOME
+    assert "DOCKER_CONFIG" not in Docker("docker")._env()

tests/test_storage_space.py

@@ -0,0 +1,59 @@
+"""Free-space accounting across the filesystems a pull can fill.
+
+docker 29 keeps image layers in containerd's store, which is frequently on a different
+filesystem than DockerRootDir. env-dev-08 (2026-09-09) had 93 GiB free on the data-root and
+2.8 GiB on the filesystem containerd was writing to; the pull died with "no space left on
+device" after the pre-flight had reported plenty of room.
+"""
+
+from __future__ import annotations
+
+import monky_deployd.compose as compose_mod
+from monky_deployd.compose import Docker
+
+
+class _FixedDocker(Docker):
+    def __init__(self, root: str):
+        super().__init__()
+        self._root = root
+
+    def data_root(self) -> str:
+        return self._root
+
+
+def test_storage_paths_include_containerd_root_when_present(tmp_path, monkeypatch):
+    root = tmp_path / "docker"
+    containerd = tmp_path / "containerd"
+    root.mkdir()
+    containerd.mkdir()
+    monkeypatch.setattr(compose_mod, "CONTAINERD_ROOTS", (str(containerd),))
+    assert _FixedDocker(str(root)).storage_paths() == [str(root), str(containerd)]
+
+
+def test_storage_paths_skip_a_containerd_root_that_does_not_exist(tmp_path, monkeypatch):
+    root = tmp_path / "docker"
+    root.mkdir()
+    monkeypatch.setattr(compose_mod, "CONTAINERD_ROOTS", (str(tmp_path / "absent"),))
+    assert _FixedDocker(str(root)).storage_paths() == [str(root)]
+
+
+def test_free_bytes_reports_the_tightest_filesystem(tmp_path, monkeypatch):
+    root = tmp_path / "docker"
+    containerd = tmp_path / "containerd"
+    root.mkdir()
+    containerd.mkdir()
+    monkeypatch.setattr(compose_mod, "CONTAINERD_ROOTS", (str(containerd),))
+    d = _FixedDocker(str(root))
+    monkeypatch.setattr(d, "_free_at", lambda p: 900 if p == str(root) else 5)
+    assert d.free_bytes() == 5
+    # an explicit path is still measured on its own
+    assert d.free_bytes(str(root)) == 900
+
+
+def test_free_bytes_is_none_when_nothing_can_be_measured(tmp_path, monkeypatch):
+    root = tmp_path / "docker"
+    root.mkdir()
+    monkeypatch.setattr(compose_mod, "CONTAINERD_ROOTS", ())
+    d = _FixedDocker(str(root))
+    monkeypatch.setattr(d, "_free_at", lambda p: None)
+    assert d.free_bytes() is None

tests/test_transport_identity.py

@@ -0,0 +1,54 @@
+"""The SDK transport names an unusable identity file instead of failing at dial.
+
+openziti.load() accepts an unreadable or malformed identity, logs "configuration is invalid"
+at the C layer and returns a context that only fails when something dials through it — as a
+bare TypeError that reads like a missing intercept. env-dev-08 (2026-09-09) lost its identity
+ACL to a tunneller rewrite and spent every tick reporting a mesh fault it did not have.
+"""
+
+from __future__ import annotations
+
+import json
+
+import pytest
+
+from monky_deployd.transport import SdkTransport, TransportError
+
+
+def _sdk(path):
+    t = SdkTransport(str(path))
+    return t
+
+
+def test_missing_identity_is_named(tmp_path):
+    with pytest.raises(TransportError) as e:
+        _sdk(tmp_path / "absent.json")._check_identity_readable()
+    assert "does not exist" in str(e.value)
+
+
+def test_malformed_identity_is_named(tmp_path):
+    p = tmp_path / "id.json"
+    p.write_text("{not json")
+    with pytest.raises(TransportError) as e:
+        _sdk(p)._check_identity_readable()
+    assert "not valid JSON" in str(e.value)
+
+
+def test_unreadable_identity_points_at_the_group_grant(tmp_path, monkeypatch):
+    p = tmp_path / "id.json"
+    p.write_text(json.dumps({"ztAPI": "https://example.invalid"}))
+
+    def _denied(*a, **k):
+        raise PermissionError(13, "Permission denied")
+
+    monkeypatch.setattr("builtins.open", _denied)
+    with pytest.raises(TransportError) as e:
+        _sdk(p)._check_identity_readable()
+    msg = str(e.value)
+    assert "not readable" in msg and "group" in msg
+
+
+def test_a_good_identity_passes(tmp_path):
+    p = tmp_path / "id.json"
+    p.write_text(json.dumps({"ztAPI": "https://example.invalid"}))
+    _sdk(p)._check_identity_readable()  # no raise