monky-deployd/README.md

<!-- xlate:verbatim-fences -->
# monky-deployd

**The on-box pull agent for Monky backends** ([MONKY-ADR-0028](https://scm.tikali.ai/tikali/applications/monky/monky-design-docs/-/blob/develop/adr/0028-backend-lifecycle-ownership-and-execution.md), [design doc 24](https://scm.tikali.ai/tikali/applications/monky/monky-design-docs/-/blob/develop/docs/24-backend-lifecycle.md)).
A docker VM or a laptop that hosts a backend environment runs this agent; every minute it dials
**monky-tenancy over the OpenZiti mesh with the box's own host identity**, asks what the backend
should be running, and converges: fetch the rendered bundle, lease a deploy grant, log in to
OpenBao, read its own secrets, `docker compose up`, report. Nothing pushes into the box, no SSH,
no credentials for the box are held anywhere else. The same check-in is the liveness heartbeat.

```
box / laptop  ◄── mesh ──►  monky-deployd: checkin → bundle → lease → OpenBao → compose up → report
```

Python 3.12+, **stdlib only** (the `openziti` SDK is optional and vendored into the `.deb`).
Verified on **Ubuntu 26.04**.

## Install (one-liner, from the enrolment kit)

An admin reveals the kit once in the console (`GET /v1/backends/{id}/agent/install`); it hands
you the enrolment JWT, a one-time **bootstrap deploy grant** and the read-only **download token**
(the kit runs the line below for you). By hand, on the box:

```sh
T=<deploy token>   # read-only GitLab deploy token (read_package_registry); the kit carries it
curl -sSf -H "DEPLOY-TOKEN: $T" https://scm.tikali.ai/api/v4/projects/69/packages/generic/monky-deployd/0.1.2/install.sh \
  | sudo bash -s -- --env env-qa-02 --site cbs --token "$T" --bootstrap-file bootstrap.jwt --enrol-jwt ./monky-host.env-qa-02.jwt
#   [--transport sdk|proxy|system] [--version 0.1.2] [--laptop] [--bao-ca openbao-ca.pem] [--source gitlab|gitea]
```

`install.sh` installs `ziti-edge-tunnel` (OpenZiti `jammy` suite) and `docker-compose-plugin` if
absent, downloads the pinned `.deb` + `.sha256` from the [scm.tikali.ai generic package registry](https://scm.tikali.ai/tikali/applications/monky/monky-deployd/-/packages)
(`https://scm.tikali.ai/api/v4/projects/69/packages/generic/monky-deployd/<ver>/monky-deployd_<ver>_amd64.deb`),
enrols `monky-host.<env>` if the identity is missing, switches the tunneler to `run-host`,
writes `/etc/monky-deployd/config.yaml`, grants the agent read access to the identity (ACL),
stages the bootstrap grant (0600), enables `monky-deployd.timer`, runs one tick and deletes the
JWT.

**Why a token.** The GitLab project is **private** and cannot be made public (its parent groups
are private), so anonymous downloads answer 401. Every registry fetch — the script itself
included — therefore sends `DEPLOY-TOKEN: <token>`: a GitLab **deploy token** with the single
scope `read_package_registry` (it can download packages and nothing else: no code, no API, no
write; revoke it in the project's *Settings → Repository → Deploy tokens* and issue a new one).
The operator seeds it in OpenBao at `monky/monky-tenancy/deployd-download` (key `token`); the
monky-tenancy install kit reads it from there and passes `--token` (`MONKY_DEPLOYD_TOKEN` also
works). The script never prints it (curl `-K` config file, 0600, deleted after the download;
`set -x` is switched off). Why not the Gitea mirror: inside the estate `gitea.cbs.tikali.net` is
split-horizon to jump1's RED EIP (`10.10.0.175`), which has no HTTP ingress, so backend boxes
cannot reach it (cbs/iac#102); `--source gitea` (or `MONKY_DEPLOYD_SOURCE=gitea`, no token) keeps
the [Gitea release](https://gitea.cbs.tikali.net/mdella/monky-deployd/releases) as the
off-estate alternative.

## Transports

| `transport` | how | when |
|---|---|---|
| `sdk` (default) | the OpenZiti Python SDK dials `monky.tenancy.deploy` / `openbao` by service name with the same host identity `ziti-edge-tunnel run-host` uses; no tun, no root beyond the docker group | VMs and laptops with the vendored wheel |
| `proxy` | `monky-deployd-proxy.service` runs `ziti tunnel proxy -i <identity> monky.tenancy.deploy:18443 openbao:18200` as user `ziti`; the agent talks to `127.0.0.1:18443/18200` (TLS SNI + cert check still `bao.cbs.tikali.net`) | the wheel is unavailable; `ziti` CLI present |
| `system` | plain DNS/TCP | laptops whose tunneler runs in `run` mode (tun + DNS) |

## Commands and exit codes

```
monky-deployd run --once [--prune]    # one tick (what the timer runs); --prune drops old releases + docker image prune
monky-deployd run                     # loop (laptop mode: no timer); SIGTERM stops it
monky-deployd status [--json]         # token present? applied vs desired sha, last checkin/report, compose ps
monky-deployd bootstrap [--force]     # log in to OpenBao with /etc/monky-deployd/bootstrap.jwt, store the token
monky-deployd version
```

| exit | meaning |
|---|---|
| `0` | converged / heartbeat sent — or offline in `laptop_mode` |
| `75` | temporary network failure (mesh down, tenancy unreachable, `429`); the next tick retries — `SuccessExitStatus=75` in the unit |
| `78` | `AGENT_ENV_MISMATCH`: the token is pinned to another environment than `config.yaml` says; **not retried** (loop mode exits) |
| `1` | refusal or failure (`ENV_INCOMPLETE`, `PRIVILEGED_REFUSED`, `DISK_INSUFFICIENT`, `ROLLBACK_REFUSED`, `BUNDLE_SHA_MISMATCH`, compose failure, `AGENT_UNAUTHENTICATED`, no credentials) — reported to tenancy as `failed` with the code and secret *names* only |

## What a tick does

1. `flock` (a second concurrent tick exits 0) → load `state.json`.
2. Bearer = the OpenBao token at `/var/lib/monky-deployd/bao.token` (0600). Absent → log in to
   `jwt-tenancy` with the kit's bootstrap grant (`/etc/monky-deployd/bootstrap.jwt`), then delete it.
3. `POST /v1/agent/checkin` `{env_id, agent_version, applied_sha, host, containers}` → `action`:
   - **`down`** → `docker compose down --remove-orphans` (`-v` only if `purge_volumes` was
     requested or `volumes_on_absent: purge`, **never on prod**) → report `down`.
   - **`none`** → `compose ps`; healthy → report `applied` (heartbeat); unhealthy → report `failed`.
   - **`apply`** → `GET /v1/agent/bundle/{env}/{sha}`, verify **sha256** over the files, then refuse on:
     unresolved `${VAR}` (names only), manifest paths outside `monky/data/<env>/see/`, `privileged`/host
     network/`SYS_ADMIN` unless `bundle.json` `allow_privileged`, a rollback unless `allow_rollback`,
     docker data-root free space `< need × 1.5 + 2 GiB` → `POST /v1/agent/lease` → OpenBao
     `POST /v1/auth/jwt-tenancy/login {"role":"see-env","jwt":<grant>}` → KV reads per
     `secrets.manifest.json` (pinned versions) → `.env` written atomically 0600 into a staging dir →
     promoted to `releases/<sha>` + `current` → `compose pull` → `up -d --remove-orphans` → wait
     healthy → report `applied` (or `failed` with the compose log tail).
4. `finally`: renew-self when the TTL runs low, re-lease before max TTL, scrub secrets from memory,
   remove staging dirs, save state.

## Security model

- **Identity = the box's ziti host identity.** Only identities with `#monky-deploy-agent` can dial
  tenancy's agent entrypoint; `#openbao-client` reaches OpenBao. The agent reads the identity
  through an ACL (`setfacl -m u:monky-deployd:r`), never owns it.
- **The bearer to tenancy is the agent's own OpenBao token**, minted by OpenBao from a
  tenancy-signed ES256 deploy grant (`aud openbao-see-env`, `kind deploy-grant`, 1 h, single-use
  `jti`). Tenancy verifies it with `auth/token/lookup`, pins `meta.env_id`, and refuses a token
  whose `meta.grant_jti` was superseded (kit re-reveal, retire) → `401 AGENT_UNAUTHENTICATED`.
  **No AppRole, nothing to unwrap** (Gate 1 result, 2026-09-05).
- **The agent never receives a secret from tenancy.** Bundles carry placeholders; the agent reads
  `monky/data/<env_id>/see/*` itself, and the OpenBao policy is templated on the token's entity
  (one entity per env), so another env's path is a 403 — and refused locally before any read.
- **Journald never carries a value.** A `Redactor` filter masks token shapes (`hvs.*`, JWTs,
  `Authorization: Bearer`) and every value the agent has read; the report's `log_tail` goes through
  the same scrubber. Secret *names* are logged.
- **Hardened oneshot**: user `monky-deployd` (+ `docker` group), `NoNewPrivileges`,
  `ProtectSystem=strict`, `ReadWritePaths` only the state dir, `/etc/monky-deployd` (to consume the
  grant) and the docker socket, `UMask=0077`, no capabilities.
- **Prod is special**: `-v` is never passed on a prod env, and tenancy never auto-changes prod state.

## Ubuntu 26.04 verified checklist (from design doc 24 §2.1a / plan §D 2.8)

```
python3 --version                                  # >= 3.12 (26.04 ships 3.14)
docker compose version                             # compose plugin
systemctl is-active ziti-edge-tunnel               # run-host (drop-in run-host.conf)
ziti tunnel proxy --help                           # fallback transport present (transport proxy)
/opt/monky-deployd/venv/bin/python -c 'import openziti'   # SDK import (transport sdk)
monky-deployd status                               # token present; applied == desired
journalctl -u monky-deployd -n 50                  # "checkin: action=..." then "applied ..."
df -h $(docker info -f '{{.DockerRootDir}}')       # free >= bundle need x 1.5 + headroom
docker compose -p monky-<env> ps                   # healthy
curl monky.percept.<env>:47283/health              # from another mesh member -> 200
```

## Repository layout

```
monky_deployd/   cli.py config.py transport.py tenancy.py bao.py bundle.py compose.py state.py redact.py agent.py
packaging/       install.sh  systemd/{monky-deployd.service,.timer,monky-deployd-proxy.service}  nfpm.yaml  scripts/
ansible/roles/monky_deployd/   role skeleton for osg1-07 (env-dev-06..09 rollout)
docs/            PROTOCOL.md (the four calls + the grant flow)  OPERATIONS.md (systemd, logs, retire, laptops)
tests/           hermetic: fake tenancy + fake OpenBao HTTP servers, a stub `docker` on PATH
```

## Development

```sh
ruff check . && ruff format --check .
pytest                                  # no network, no docker: fakes only
bash -n packaging/install.sh
systemd-analyze verify packaging/systemd/*.service   # where systemd is available
```

### CI notes

`lint` and `test` run on every MR/branch. `wheel` builds the `openziti` wheel on `ubuntu:26.04`
(PyPI ships an sdist that fetches **ziti-sdk-c from github.com** at build time — the runner needs
egress to github.com and pypi.org) and `package` builds the `.deb` with `nfpm` (binary from GitHub
releases, goreleaser apt repo as fallback). Both were proven on the v0.1.0 tag pipeline and are
blocking on `main`/tags (manual on MRs); the `.deb` always ships the SDK wheel. On a `v*` tag
`release` uploads to the GitLab generic package registry + release (**the primary download**; the
project is private, so the installer sends the read-only deploy token), and `release:gitea` publishes the same assets on the Gitea mirror (the `--source gitea`
alternative; automatic when `GITEA_TOKEN` is set, manual otherwise — see `docs/OPERATIONS.md` for
the by-hand recipe). Both locations keep being published.

## See also

- `docs/PROTOCOL.md`, `docs/OPERATIONS.md`, `CHANGELOG.md`
- monky-tenancy `docs/usage.md` (agent protocol), `app/api/agent.py`, `app/schemas_backends.py`
- monky-deploy (the renderer whose `render_files` produces the bundle)