monky-deployd/README.md

monky-deployd

The on-box pull agent for Monky backends (MONKY-ADR-0028, design doc 24). A docker VM or a laptop that hosts a backend environment runs this agent; every minute it dials monky-tenancy over the OpenZiti mesh with the box's own host identity, asks what the backend should be running, and converges: fetch the rendered bundle, lease a deploy grant, log in to OpenBao, read its own secrets, docker compose up, report. Nothing pushes into the box, no SSH, no credentials for the box are held anywhere else. The same check-in is the liveness heartbeat.

box / laptop  ◄── mesh ──►  monky-deployd: checkin → bundle → lease → OpenBao → compose up → report

Python 3.12+, stdlib only (the openziti SDK is optional and vendored into the .deb). Verified on Ubuntu 26.04.

Install (one-liner, from the enrolment kit)

An admin reveals the kit once in the console (GET /v1/backends/{id}/agent/install); it hands you the enrolment JWT, a one-time bootstrap deploy grant and the read-only download token (the kit runs the line below for you). By hand, on the box:

T=<deploy token>   # read-only GitLab deploy token (read_package_registry); the kit carries it
curl -sSf -H "DEPLOY-TOKEN: $T" https://scm.tikali.ai/api/v4/projects/69/packages/generic/monky-deployd/0.1.6/install.sh \
  | sudo bash -s -- --env env-qa-02 --site cbs --token "$T" --bootstrap-file bootstrap.jwt --enrol-jwt ./monky-host.env-qa-02.jwt
#   [--transport sdk|proxy|system] [--version 0.1.6] [--docker-data-root /home/docker-data] [--laptop] [--bao-ca openbao-ca.pem] [--source gitlab|gitea]

install.sh installs ziti-edge-tunnel (OpenZiti jammy suite) and docker-compose-plugin if absent, downloads the pinned .deb + .sha256 from the scm.tikali.ai generic package registry (https://scm.tikali.ai/api/v4/projects/69/packages/generic/monky-deployd/<ver>/monky-deployd_<ver>_amd64.deb), enrols monky-host.<env> if the identity is missing, switches the tunneler to run-host, writes /etc/monky-deployd/config.yaml, grants the agent read access to the identity (ACL), stages the bootstrap grant (0600), enables monky-deployd.timer, runs one tick and deletes the JWT.

Why a token. The GitLab project is private and cannot be made public (its parent groups are private), so anonymous downloads answer 401. Every registry fetch — the script itself included — therefore sends DEPLOY-TOKEN: <token>: a GitLab deploy token with the single scope read_package_registry (it can download packages and nothing else: no code, no API, no write; revoke it in the project's Settings → Repository → Deploy tokens and issue a new one). The operator seeds it in OpenBao at monky/monky-tenancy/deployd-download (key token); the monky-tenancy install kit reads it from there and passes --token (MONKY_DEPLOYD_TOKEN also works). The script never prints it (curl -K config file, 0600, deleted after the download; set -x is switched off). Why not the Gitea mirror: inside the estate gitea.cbs.tikali.net is split-horizon to jump1's RED EIP (10.10.0.175), which has no HTTP ingress, so backend boxes cannot reach it (cbs/iac#102); --source gitea (or MONKY_DEPLOYD_SOURCE=gitea, no token) keeps the Gitea release as the off-estate alternative.

Transports

transport	how	when
sdk (default)	the OpenZiti Python SDK dials monky.tenancy.deploy / openbao by service name with the same host identity ziti-edge-tunnel run-host uses; no tun, no root beyond the docker group	VMs and laptops with the vendored wheel
proxy	monky-deployd-proxy.service runs ziti tunnel proxy -i <identity> monky.tenancy.deploy:18443 openbao:18200 as user ziti; the agent talks to 127.0.0.1:18443/18200 (TLS SNI + cert check still bao.cbs.tikali.net)	the wheel is unavailable; ziti CLI present
system	plain DNS/TCP	laptops whose tunneler runs in run mode (tun + DNS)

Commands and exit codes

monky-deployd run --once [--prune]    # one tick (what the timer runs); --prune drops old releases + docker image prune
monky-deployd run                     # loop (laptop mode: no timer); SIGTERM stops it
monky-deployd status [--json]         # token present? applied vs desired sha, last checkin/report, compose ps
monky-deployd bootstrap [--force]     # log in to OpenBao with /etc/monky-deployd/bootstrap.jwt, store the token
monky-deployd version

exit	meaning
0	converged / heartbeat sent — or offline in laptop_mode
75	temporary network failure (mesh down, tenancy unreachable, 429); the next tick retries — SuccessExitStatus=75 in the unit
78	AGENT_ENV_MISMATCH: the token is pinned to another environment than config.yaml says; not retried (loop mode exits)
1	refusal or failure (ENV_INCOMPLETE, PRIVILEGED_REFUSED, DISK_INSUFFICIENT, ROLLBACK_REFUSED, BUNDLE_SHA_MISMATCH, compose failure, AGENT_UNAUTHENTICATED, no credentials) — reported to tenancy as failed with the code and secret names only

What a tick does

1. flock (a second concurrent tick exits 0) → load state.json.
2. Bearer = the OpenBao token at /var/lib/monky-deployd/bao.token (0600). Absent → log in to jwt-tenancy with the kit's bootstrap grant (/etc/monky-deployd/bootstrap.jwt), then delete it.
3. POST /v1/agent/checkin {env_id, agent_version, applied_sha, host, containers} → action:
   • down → docker compose down --remove-orphans (-v only if purge_volumes was requested or volumes_on_absent: purge, never on prod) → report down.
   • none → compose ps; healthy → report applied (heartbeat); unhealthy → report failed.
   • apply → GET /v1/agent/bundle/{env}/{sha}, verify sha256 over the files, then refuse on: unresolved ${VAR} (names only), manifest paths outside monky/data/<env>/see/, privileged/host network/SYS_ADMIN unless bundle.json allow_privileged, a rollback unless allow_rollback, docker data-root free space < need × 1.5 + 2 GiB → POST /v1/agent/lease → OpenBao POST /v1/auth/jwt-tenancy/login {"role":"see-env","jwt":<grant>} → KV reads per secrets.manifest.json (pinned versions) → .env written atomically 0600 into a staging dir → promoted to releases/<sha> + current → compose pull → up -d --remove-orphans → wait healthy → report applied (or failed with the compose log tail).
4. finally: renew-self when the TTL runs low, re-lease before max TTL, scrub secrets from memory, remove staging dirs, save state.

Security model

• Identity = the box's ziti host identity. Only identities with #monky-deploy-agent can dial tenancy's agent entrypoint; #openbao-client reaches OpenBao — tenancy sends both to the broker as host_extra_attrs at register (monky-tenancy !29 / monky-ziti ac258352, 2026-09-07), so a backend registered since then dials without hand-added attrs (see PROTOCOL.md §Where and how). The agent reads the identity through an ACL (setfacl -m u:monky-deployd:r), never owns it.
• The bearer to tenancy is the agent's own OpenBao token, minted by OpenBao from a tenancy-signed ES256 deploy grant (aud openbao-see-env, kind deploy-grant, 1 h; its jti becomes the token's meta.grant_jti — the mount keeps no replay state, tenancy does). Tenancy verifies it with auth/token/lookup, pins meta.env_id, and refuses a token whose meta.grant_jti was superseded (kit re-reveal, retire) → 401 AGENT_UNAUTHENTICATED. No AppRole, nothing to unwrap (Gate 1 result, 2026-09-05).
• The agent never receives a secret from tenancy. Bundles carry placeholders; the agent reads monky/data/<env_id>/see/* itself, and the OpenBao policy is templated on the token's entity (one entity per env), so another env's path is a 403 — and refused locally before any read.
• Journald never carries a value. A Redactor filter masks token shapes (hvs.*, JWTs, Authorization: Bearer) and every value the agent has read; the report's log_tail goes through the same scrubber. Secret names are logged.
• Hardened oneshot: user monky-deployd (+ docker group), NoNewPrivileges, ProtectSystem=strict, ReadWritePaths only the state dir, /etc/monky-deployd (to consume the grant) and the docker socket, UMask=0077, no capabilities.
• Prod is special: -v is never passed on a prod env, and tenancy never auto-changes prod state.

Ubuntu 26.04 verified checklist (from design doc 24 §2.1a / plan §D 2.8)

python3 --version                                  # >= 3.12 (26.04 ships 3.14)
docker compose version                             # compose plugin
systemctl is-active ziti-edge-tunnel               # run-host (drop-in run-host.conf)
ziti tunnel proxy --help                           # fallback transport present (transport proxy)
/opt/monky-deployd/venv/bin/python -c 'import openziti'   # SDK import (transport sdk)
monky-deployd status                               # token present; applied == desired
journalctl -u monky-deployd -n 50                  # "checkin: action=..." then "applied ..."
df -h $(docker info -f '{{.DockerRootDir}}')       # free >= bundle need x 1.5 + headroom
docker compose -p monky-<env> ps                   # healthy
curl monky.percept.<env>:47283/health              # from another mesh member -> 200

Repository layout

monky_deployd/   cli.py config.py transport.py tenancy.py bao.py bundle.py compose.py state.py redact.py agent.py
packaging/       install.sh  systemd/{monky-deployd.service,.timer,monky-deployd-proxy.service}  nfpm.yaml  scripts/
ansible/roles/monky_deployd/   role skeleton for osg1-07 (env-dev-06..09 rollout)
docs/            PROTOCOL.md (the four calls + the grant flow)  OPERATIONS.md (systemd, logs, retire, laptops)
tests/           hermetic: fake tenancy + fake OpenBao HTTP servers, a stub `docker` on PATH

Development

ruff check . && ruff format --check .
pytest                                  # no network, no docker: fakes only
bash -n packaging/install.sh
systemd-analyze verify packaging/systemd/*.service   # where systemd is available

CI notes

lint and test run on every MR/branch. wheel builds the openziti wheel on ubuntu:26.04 (PyPI ships an sdist that fetches ziti-sdk-c from github.com at build time — the runner needs egress to github.com and pypi.org) and package builds the .deb with nfpm (binary from GitHub releases, goreleaser apt repo as fallback). Both were proven on the v0.1.0 tag pipeline and are blocking on main/tags (manual on MRs); the .deb always ships the SDK wheel. On a v* tag release uploads to the GitLab generic package registry + release (the primary download; the project is private, so the installer sends the read-only deploy token), and release:gitea publishes the same assets on the Gitea mirror (the --source gitea alternative; automatic when GITEA_TOKEN is set, manual otherwise — see docs/OPERATIONS.md for the by-hand recipe). The GitLab registry is published by every tag pipeline; the Gitea mirror only when release:gitea runs — automatically once GITEA_TOKEN is set in CI, by hand otherwise — so check the Gitea release page before pointing an installer at it.

See also

• docs/PROTOCOL.md, docs/OPERATIONS.md, CHANGELOG.md
• monky-tenancy docs/usage.md (agent protocol), app/api/agent.py, app/schemas_backends.py
• monky-deploy (the renderer whose render_files produces the bundle)

Docker Engine (since 0.1.3)

install.sh installs Docker Engine when it is absent (Docker's apt suite for the host's Ubuntu codename, falling back to Ubuntu's docker.io), so an end user runs exactly one script. --docker-data-root <dir> writes /etc/docker/daemon.json before the daemon first starts (pilot VMs keep the data-root on a dedicated disk, e.g. /home/docker-data). A pre-existing Docker is left untouched. macOS/Windows laptops still need Docker Desktop from the user.