monky-deployd/docs/PROTOCOL.md

<!-- xlate:verbatim-fences -->
# monky-deployd protocol

The agent side of the backend lifecycle agent protocol — [MONKY-ADR-0028](https://scm.tikali.ai/tikali/applications/monky/monky-design-docs/-/blob/develop/adr/0028-backend-lifecycle-ownership-and-execution.md)
§2/§3 (amendment 2026-09-05) and [design doc 24](https://scm.tikali.ai/tikali/applications/monky/monky-design-docs/-/blob/develop/docs/24-backend-lifecycle.md)
§2.1a, §3.3, §4. The tenancy side is monky-tenancy `app/agent_main.py` / `app/api/agent.py` /
`app/schemas_backends.py`; its shapes are the contract, this page is how the agent uses them.

## Where and how

- **Where:** tenancy's **agent entrypoint** — a second container (`app.agent_main`,
  `127.0.0.1:8081`) bound to the ziti service **`monky.tenancy.deploy`** by a `ziti-edge-tunnel
  run-host` sidecar. Unreachable from the public ingress. **The agent dials the service's
  INTERCEPT port, `tcp/443`** (`monky.tenancy.deploy.intercept.v1` in openziti
  `state/overlay/configs.json` — the authority for the port); the sidecar's `host.v1` forwards to
  `127.0.0.1:8081` inside the pod. `tenancy.port` in the config, `install.sh` and the ansible role all
  default to 443 (0.1.5). Only host identities carrying `#monky-deploy-agent` may dial
  (`monky.tenancy.deploy.dial`, openziti). Tenancy sends that attr and `#openbao-client` to the
  broker as `host_extra_attrs` when it registers an agent-backed backend (monky-tenancy !29,
  monky-ziti `ac258352` — both 2026-09-07), so `monky-host.<env>` carries them from creation;
  identities created before that (env-qa-02's) had them added on the controller by hand.
- **Transport:** plain HTTP inside the mesh (`transport: sdk` dials by service name;
  `proxy` → `127.0.0.1:18443`). OpenBao is the existing `openbao` ziti service (`#openbao-client`),
  dialled as `https://bao.cbs.tikali.net:8200` (an intercept name), TLS validated against the
  `openbao-ca` certificate (`bao.ca_bundle`).
- **Auth:** `Authorization: Bearer <the agent's OpenBao token>`. Tenancy verifies it with
  `auth/token/lookup` (cached 60 s), pins the request's `env_id` to the token's `meta.env_id`
  (`403 AGENT_ENV_MISMATCH` → exit 78, never retried), and compares `meta.grant_jti` against
  `backend_leases` (`401 AGENT_UNAUTHENTICATED` when the grant was superseded or revoked).
- **Rate buckets per env:** checkin 6/min, bundle 10/min, report 30/min, lease **5/h**
  (`429 LEASE_RATE_LIMITED`, `Retry-After`) → the agent treats 429 as temporary (exit 75).
- **Errors** are `{code, detail}` JSON; the agent maps them to `TenancyError` subclasses.

## Bootstrap → lease → token (the grant flow)

```
kit reveal (admin, once)     tenancy signs a bootstrap DEPLOY GRANT: ES256 JWT
                              iss <tenancy issuer>, aud openbao-see-env, sub agent:<env_id>,
                              env_id, kind deploy-grant, jti (→ token meta grant_jti; tenancy records it), exp now+1h
install.sh                   stages it at /etc/monky-deployd/bootstrap.jwt (0600)
first tick                   POST https://bao…/v1/auth/jwt-tenancy/login {"role":"see-env","jwt":<grant>}
                              → auth.client_token (ttl 24 h, max 30 d, policy see-env,
                                 metadata {env_id, grant_jti} from the mount's claim_mappings)
                             token → /var/lib/monky-deployd/bao.token (0600); grant deleted
every tick                   Bearer <token> → checkin / bundle / lease / report
apply                        POST /v1/agent/lease → {login_jwt,…} → jwt-tenancy login → NEW token
                              (becomes the bearer; the old one is revoke-self'd) → KV reads
upkeep (finally)             lookup-self; ttl < 12 h → renew-self; age > 30 d − 2 d → re-lease
```

OpenBao side (Terraform, `tikali/services/security/openbao`): mount `jwt-tenancy`
(`jwks_url` = tenancy's `GET /.well-known/agent-jwks.json`, in-cluster), role `see-env`
(`role_type=jwt`, `user_claim=env_id` ⇒ one alias/entity **per env**, `bound_audiences=["openbao-see-env"]`,
`bound_claims={"kind":"deploy-grant"}`, `claim_mappings={"env_id":"env_id","jti":"grant_jti"}`,
`token_ttl=86400`, `token_max_ttl=2592000`, `token_no_default_policy=true`), policy `see-env` =
`monky/data/{{identity.entity.aliases.<accessor>.metadata.env_id}}/see/*` read + metadata list +
`auth/token/{renew-self,lookup-self}`. **Gate 1 v2 passed 2026-09-05 07:40Z** (distinct entity per env).

**Second-reveal semantics.** A kit re-reveal (or a retire) supersedes every earlier bootstrap grant
of the backend (tenancy marks their `backend_leases` rows revoked) and revokes the token accessors
tenancy knows. The `jwt-tenancy` mount keeps **no** replay state — it validates signature, `aud`,
`bound_claims` and `exp` only (openbao `terraform/jwt-tenancy.tf`, the `see_env` role) — so a
superseded grant still logs in until its 1 h `exp`; the token it yields, like a token already minted
from it, is refused by tenancy at the first bearer call with `401 AGENT_UNAUTHENTICATED` (its
`meta.grant_jti` is superseded) — **not** `AGENT_ENV_MISMATCH`.
The agent then deletes its token; if a fresh `bootstrap.jwt` is on disk it bootstraps again in the
same tick, otherwise it exits 1 and says "re-run the install kit".

## The four calls

### `POST /v1/agent/checkin`

```json
{"env_id": "env-qa-02", "agent_version": "0.1.0", "applied_sha": "3f9c…",
 "host": {"hostname": "env-qa-02", "os": "ubuntu-26.04", "kernel": "7.0.0-30-generic",
          "docker": "28.3.0", "compose": "2.32.0", "free_bytes": 61234567890,
          "agent_version": "0.1.0", "transport": "sdk", "site": "cbs", "laptop_mode": false},
 "containers": [{"name": "see-backend", "state": "running", "health": "healthy"}]}
```
```json
{"env_id": "env-qa-02", "desired_sha": "7a10…", "action": "apply", "purge_volumes": false,
 "bundle_url": "/v1/agent/bundle/env-qa-02/7a10…", "checkin_interval_s": 60,
 "vault": {"addr": "https://bao.cbs.tikali.net:8200", "mount": "monky", "prefix": "env-qa-02/see",
           "auth_mount": "jwt-tenancy", "auth_role": "see-env"}}
```
`action`: `apply` (desired ≠ applied), `none` (converged → heartbeat), `down` (retire; `purge_volumes`
is only meaningful here). `vault.mount` is the **KV** mount; the agent adopts it if it differs from
config. `vault.addr` is informational — the transport decides how OpenBao is reached.

### `GET /v1/agent/bundle/{env_id}/{sha}`

`application/x-tar`, `Cache-Control: no-store`, `X-Bundle-Sha` (tenancy) / `X-Bundle-Sha256` (doc 24;
both accepted). Members: `docker-compose.yml` (or `compose.yaml`), `.env.template`,
`secrets.manifest.json`, `bundle.json`, optionally `.env.example`. **No secret material** — the agent
refuses a manifest entry carrying a `value`. The agent recomputes

```
sha256( for name in sorted(files): name + "\0" + content + "\0" )
```

and refuses (`BUNDLE_SHA_MISMATCH`) unless it equals `desired_sha`. `404 BUNDLE_NOT_FOUND` when
the sha is not published.

`secrets.manifest.json` (paths + versions, never values):
```json
{"vault_mode": "openbao",
 "entries": [{"var": "GEMINI_API_KEY",    "path": "monky/env-qa-02/see/gemini_api_key", "kind": "supplied",  "version": 2},
             {"var": "POSTGRES_PASSWORD", "path": "monky/env-qa-02/see/pg_password",    "kind": "generated", "version": 1},
             {"var": "SEE_ADMIN_TOKEN",   "path": "monky/env-qa-02/see/admin_token",    "kind": "generated", "version": 1}]}
```
Path forms `monky/<env>/see/<name>` (tenancy) and `monky/data/<env>/see/<name>` (doc 24) both map
to `GET /v1/monky/data/<env>/see/<name>?version=N`; the `<env>` segment **must** equal the agent's
`env_id`. `version: null` = latest. The secret document is `{"value": "…"}`.

`bundle.json` (monky-deploy `render_bundle_json`): `env_id, tier, site, bundle, target, target_class,
address, pull, renderer, images_policy, secrets_provider, agent{…}, files[]`. The agent honours
`allow_privileged`, `allow_rollback`, `disk_need_bytes` (top level or under `agent`), refuses a
`renderer` other than `compose`, and treats `tier == "prod"` as prod.

### `POST /v1/agent/lease`

```json
{"env_id": "env-qa-02"}
```
(The agent also sends `"reason": "apply" | "renew"` for its own logs; tenancy's `AgentLease` is
`{env_id}` and ignores unknown fields.)
```json
{"env_id": "env-qa-02", "login_jwt": "eyJ…", "ttl_s": 3600, "mount": "jwt-tenancy", "role": "see-env",
 "addr": "https://bao.cbs.tikali.net:8200"}
```
`addr` is the OpenBao address for the login below, top-level (tenancy `AgentLeaseOut`); the KV
mount/prefix come from `checkin`'s `vault`, not from the lease.
Then `POST /v1/auth/{mount}/login {"role": "{role}", "jwt": "{login_jwt}"}` on OpenBao. A body with
`wrapping_token` / `role_id` (the pre-Gate-1 AppRole lease) is refused with `LEASE_SHAPE` → report
`failed`. `429 LEASE_RATE_LIMITED` → exit 75.

### `POST /v1/agent/report`

```json
{"env_id": "env-qa-02", "sha": "7a10…", "result": "applied",
 "log_tail": "INFO checkin: action=apply …\nINFO applied 7a10c0ffee11 (2 container(s) healthy)",
 "containers": [{"name": "see-backend", "state": "running", "health": "healthy"}],
 "detail": "ENV_INCOMPLETE: unresolved: SEE_ADMIN_TOKEN"}         // detail only on failed
```
`result`: `applied` (sets `applied_sha`, health ok — also the heartbeat on `none`), `failed`
(health degraded; `detail` = code + names), `down` (clears `applied_sha`). `log_tail` ≤ 16 KiB, the
first 2 KiB land in the audit log — it has been through the redactor.

## Divergences (2026-09-05)

- **Resolved 2026-09-05 (before v0.1.0 was tagged):** monky-tenancy !17 (`288df791`) landed the
  tenancy side of the Gate 1 RESULT / ADR-0028 amendment — `AgentLeaseOut{env_id, login_jwt, ttl_s,
  mount, role, addr}`, the ES256 deploy-grant signer (`app/agent_keys.py`), `GET
  /.well-known/agent-jwks.json`, no AppRole and no response wrapping anywhere in tenancy. The
  `LEASE_SHAPE` refusal of a `wrapping_token`/`role_id` body stays in this agent as a guard against a
  stale tenancy, not as a description of `main`.
- **Resolved 2026-09-05:** the kit no longer generates a config file. Since monky-tenancy !22
  (`61bd0281`) the one-time install script stages the bootstrap grant and runs `install.sh --env …
  --site … --version … --bootstrap-file …`, and `install.sh` writes `/etc/monky-deployd/config.yaml`
  (tenancy service name, OpenBao address and `jwt-tenancy`/`see-env`). `tenancy.base_url:` remains an
  accepted alias for `tenancy.{scheme,host,port}` (`config.py`) for hand-written configs.
- `report` gains an optional `detail` (doc 24 §3.3); tenancy's `AgentReport` ignores unknown fields
  today — if `strict` bodies land, `detail` folds into `log_tail`.
- Bundle sha header: tenancy sends `X-Bundle-Sha`, doc 24 says `X-Bundle-Sha256`; the agent reads
  either and trusts only its own computation.