monky-deployd/README.md

monky-deployd

The on-box pull agent for Monky backends (MONKY-ADR-0028, design doc 24). A docker VM or a laptop that hosts a backend environment runs this agent; every minute it dials monky-tenancy over the OpenZiti mesh with the box's own host identity, asks what the backend should be running, and converges: fetch the rendered bundle, lease a deploy grant, log in to OpenBao, read its own secrets, docker compose up, report. Nothing pushes into the box, no SSH, no credentials for the box are held anywhere else. The same check-in is the liveness heartbeat.

box / laptop  ◄── mesh ──►  monky-deployd: checkin → bundle → lease → OpenBao → compose up → report

Python 3.12+, stdlib only (the openziti SDK is optional and vendored into the .deb). Verified on Ubuntu 26.04.

Install (one-liner, from the enrolment kit)

An admin reveals the kit once in the console (GET /v1/backends/{id}/agent/install); it hands you the enrolment JWT and a one-time bootstrap deploy grant. On the box:

curl -fsSL https://gitea.cbs.tikali.net/mdella/monky-deployd/raw/branch/main/packaging/install.sh \
  | sudo bash -s -- --env env-qa-02 --site cbs --enrol-jwt ./monky-host.env-qa-02.jwt < bootstrap.jwt
#   [--transport sdk|proxy|system] [--version 0.1.0] [--laptop] [--bao-ca openbao-ca.pem]

install.sh installs ziti-edge-tunnel (OpenZiti jammy suite) and docker-compose-plugin if absent, downloads the pinned .deb + .sha256 from the Gitea release, enrols monky-host.<env> if the identity is missing, switches the tunneler to run-host, writes /etc/monky-deployd/config.yaml, grants the agent read access to the identity (ACL), stages the bootstrap grant (0600), enables monky-deployd.timer, runs one tick and deletes the JWT. The GitLab project is private, so the public download is the Gitea mirror.

Transports

transport	how	when
sdk (default)	the OpenZiti Python SDK dials monky.tenancy.deploy / openbao by service name with the same host identity ziti-edge-tunnel run-host uses; no tun, no root beyond the docker group	VMs and laptops with the vendored wheel
proxy	monky-deployd-proxy.service runs ziti tunnel proxy -i <identity> monky.tenancy.deploy:18443 openbao:18200 as user ziti; the agent talks to 127.0.0.1:18443/18200 (TLS SNI + cert check still bao.cbs.tikali.net)	the wheel is unavailable; ziti CLI present
system	plain DNS/TCP	laptops whose tunneler runs in run mode (tun + DNS)

Commands and exit codes

monky-deployd run --once [--prune]    # one tick (what the timer runs); --prune drops old releases + docker image prune
monky-deployd run                     # loop (laptop mode: no timer); SIGTERM stops it
monky-deployd status [--json]         # token present? applied vs desired sha, last checkin/report, compose ps
monky-deployd bootstrap [--force]     # log in to OpenBao with /etc/monky-deployd/bootstrap.jwt, store the token
monky-deployd version

exit	meaning
0	converged / heartbeat sent — or offline in laptop_mode
75	temporary network failure (mesh down, tenancy unreachable, 429); the next tick retries — SuccessExitStatus=75 in the unit
78	AGENT_ENV_MISMATCH: the token is pinned to another environment than config.yaml says; not retried (loop mode exits)
1	refusal or failure (ENV_INCOMPLETE, PRIVILEGED_REFUSED, DISK_INSUFFICIENT, ROLLBACK_REFUSED, BUNDLE_SHA_MISMATCH, compose failure, AGENT_UNAUTHENTICATED, no credentials) — reported to tenancy as failed with the code and secret names only

What a tick does

1. flock (a second concurrent tick exits 0) → load state.json.
2. Bearer = the OpenBao token at /var/lib/monky-deployd/bao.token (0600). Absent → log in to jwt-tenancy with the kit's bootstrap grant (/etc/monky-deployd/bootstrap.jwt), then delete it.
3. POST /v1/agent/checkin {env_id, agent_version, applied_sha, host, containers} → action:
   • down → docker compose down --remove-orphans (-v only if purge_volumes was requested or volumes_on_absent: purge, never on prod) → report down.
   • none → compose ps; healthy → report applied (heartbeat); unhealthy → report failed.
   • apply → GET /v1/agent/bundle/{env}/{sha}, verify sha256 over the files, then refuse on: unresolved ${VAR} (names only), manifest paths outside monky/data/<env>/see/, privileged/host network/SYS_ADMIN unless bundle.json allow_privileged, a rollback unless allow_rollback, docker data-root free space < need × 1.5 + 2 GiB → POST /v1/agent/lease → OpenBao POST /v1/auth/jwt-tenancy/login {"role":"see-env","jwt":<grant>} → KV reads per secrets.manifest.json (pinned versions) → .env written atomically 0600 into a staging dir → promoted to releases/<sha> + current → compose pull → up -d --remove-orphans → wait healthy → report applied (or failed with the compose log tail).
4. finally: renew-self when the TTL runs low, re-lease before max TTL, scrub secrets from memory, remove staging dirs, save state.

Security model

• Identity = the box's ziti host identity. Only identities with #monky-deploy-agent can dial tenancy's agent entrypoint; #openbao-client reaches OpenBao. The agent reads the identity through an ACL (setfacl -m u:monky-deployd:r), never owns it.
• The bearer to tenancy is the agent's own OpenBao token, minted by OpenBao from a tenancy-signed ES256 deploy grant (aud openbao-see-env, kind deploy-grant, 1 h, single-use jti). Tenancy verifies it with auth/token/lookup, pins meta.env_id, and refuses a token whose meta.grant_jti was superseded (kit re-reveal, retire) → 401 AGENT_UNAUTHENTICATED. No AppRole, nothing to unwrap (Gate 1 result, 2026-09-05).
• The agent never receives a secret from tenancy. Bundles carry placeholders; the agent reads monky/data/<env_id>/see/* itself, and the OpenBao policy is templated on the token's entity (one entity per env), so another env's path is a 403 — and refused locally before any read.
• Journald never carries a value. A Redactor filter masks token shapes (hvs.*, JWTs, Authorization: Bearer) and every value the agent has read; the report's log_tail goes through the same scrubber. Secret names are logged.
• Hardened oneshot: user monky-deployd (+ docker group), NoNewPrivileges, ProtectSystem=strict, ReadWritePaths only the state dir, /etc/monky-deployd (to consume the grant) and the docker socket, UMask=0077, no capabilities.
• Prod is special: -v is never passed on a prod env, and tenancy never auto-changes prod state.

Ubuntu 26.04 verified checklist (from design doc 24 §2.1a / plan §D 2.8)

python3 --version                                  # >= 3.12 (26.04 ships 3.14)
docker compose version                             # compose plugin
systemctl is-active ziti-edge-tunnel               # run-host (drop-in run-host.conf)
ziti tunnel proxy --help                           # fallback transport present (transport proxy)
/opt/monky-deployd/venv/bin/python -c 'import openziti'   # SDK import (transport sdk)
monky-deployd status                               # token present; applied == desired
journalctl -u monky-deployd -n 50                  # "checkin: action=..." then "applied ..."
df -h $(docker info -f '{{.DockerRootDir}}')       # free >= bundle need x 1.5 + headroom
docker compose -p monky-<env> ps                   # healthy
curl monky.percept.<env>:47283/health              # from another mesh member -> 200

Repository layout

monky_deployd/   cli.py config.py transport.py tenancy.py bao.py bundle.py compose.py state.py redact.py agent.py
packaging/       install.sh  systemd/{monky-deployd.service,.timer,monky-deployd-proxy.service}  nfpm.yaml  scripts/
ansible/roles/monky_deployd/   role skeleton for osg1-07 (env-dev-06..09 rollout)
docs/            PROTOCOL.md (the four calls + the grant flow)  OPERATIONS.md (systemd, logs, retire, laptops)
tests/           hermetic: fake tenancy + fake OpenBao HTTP servers, a stub `docker` on PATH

Development

ruff check . && ruff format --check .
pytest                                  # no network, no docker: fakes only
bash -n packaging/install.sh
systemd-analyze verify packaging/systemd/*.service   # where systemd is available

CI notes

lint and test run on every MR/branch. wheel builds the openziti wheel on ubuntu:26.04 (PyPI ships an sdist that fetches ziti-sdk-c from github.com at build time — the runner needs egress to github.com and pypi.org) and package builds the .deb with nfpm (binary from GitHub releases, goreleaser apt repo as fallback). Both were proven on the v0.1.0 tag pipeline and are blocking on main/tags (manual on MRs); the .deb always ships the SDK wheel. On a v* tag release uploads to the GitLab generic package registry + release, and release:gitea publishes the same assets on the public Gitea mirror (automatic when GITEA_TOKEN is set, manual otherwise — see docs/OPERATIONS.md for the by-hand recipe).

See also

• docs/PROTOCOL.md, docs/OPERATIONS.md, CHANGELOG.md
• monky-tenancy docs/usage.md (agent protocol), app/api/agent.py, app/schemas_backends.py
• monky-deploy (the renderer whose render_files produces the bundle)