mirror of
https://scm.tikali.ai/tikali/applications/monky/monky-deployd.git
synced 2026-09-18 04:36:15 +00:00
feat: monky-deployd v0.1.0 — pull agent over the mesh (ADR-0028)
Stdlib-only Python 3.12 agent for docker VMs and laptops: flock → checkin
(bearer = the agent's OpenBao token, bootstrapped from the install kit's
jwt-tenancy deploy grant) → action apply|none|down → bundle (sha256
verified) → refusal checks (unresolved ${VAR} names only, manifest paths
pinned to monky/data/<env>/see/, privileged/host-network, rollback, disk
need×1.5+headroom) → lease → POST /v1/auth/jwt-tenancy/login → KV reads →
.env 0600 → promote → compose pull/up → wait healthy → report; finally
renew-self / re-lease before max TTL, scrub. Exit 0/75/78/1. Redactor log
filter. Transports sdk (openziti) / proxy (ziti tunnel proxy 18443/18200) /
system. Laptop mode.
Packaging: hardened oneshot + 60 s timer + proxy unit, nfpm .deb with
/opt/monky-deployd/venv, install.sh for Ubuntu 26.04 (Gitea release
download, enrol, ACLs, bootstrap from stdin), ansible role skeleton for
osg1-07. CI: lint/test on every change; wheel (openziti on ubuntu:26.04) and
package (nfpm) allow_failure until runner egress is proven; GitLab release +
release:gitea on v* tags. Docs: README, PROTOCOL, OPERATIONS, CHANGELOG,
CLAUDE/AGENTS.
Divergence noted: monky-tenancy main (MR !15) still ships the AppRole lease
and kit; this agent implements the plan's Gate 1 RESULT (login_jwt, no
unwrap) and refuses an AppRole lease loudly (LEASE_SHAPE).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KLB7jieMNRkTsJ2epr4Ds1
This commit is contained in:
@@ -0,0 +1,109 @@
|
||||
<!-- xlate:verbatim-fences -->
|
||||
# Operating monky-deployd
|
||||
|
||||
## Units
|
||||
|
||||
| unit | what |
|
||||
|---|---|
|
||||
| `monky-deployd.timer` | fires `monky-deployd.service` 60 s after the previous tick finished (+ ≤10 s jitter); `OnBootSec=90s` |
|
||||
| `monky-deployd.service` | `Type=oneshot`, `monky-deployd run --once` as user `monky-deployd` (+ `docker` group); `SuccessExitStatus=75`; hardened (`NoNewPrivileges`, `ProtectSystem=strict`, `ReadWritePaths=/var/lib/monky-deployd /etc/monky-deployd /run/docker.sock`, `UMask=0077`, no capabilities) |
|
||||
| `monky-deployd-proxy.service` | only with `transport: proxy`: `ziti tunnel proxy -i <identity> monky.tenancy.deploy:18443 openbao:18200` as user `ziti`; `EnvironmentFile=/etc/monky-deployd/proxy.env` |
|
||||
| `ziti-edge-tunnel.service` | the host identity's tunneler in **`run-host`** mode (drop-in `run-host.conf` written by `install.sh`) |
|
||||
|
||||
```sh
|
||||
systemctl status monky-deployd.timer monky-deployd.service
|
||||
systemctl list-timers monky-deployd.timer
|
||||
systemctl start monky-deployd.service # tick now (blocks until done)
|
||||
journalctl -u monky-deployd -f # priorities are real (INFO/WARNING/ERROR)
|
||||
monky-deployd status # exit 0 = token present and in sync
|
||||
monky-deployd status --json | jq .
|
||||
```
|
||||
|
||||
Files: `/etc/monky-deployd/config.yaml` (0640 root:monky-deployd), `/etc/monky-deployd/openbao-ca.pem`,
|
||||
`/etc/monky-deployd/bootstrap.jwt` (only until the first login), `/var/lib/monky-deployd/{bao.token,state.json,lock}`,
|
||||
`/var/lib/monky-deployd/<env>/releases/<sha>/` + `current` (the compose project dir, `.env` 0600),
|
||||
`/opt/openziti/etc/identities/monky-host.<env>.json` (ziti:ziti 0600 + ACL `u:monky-deployd:r`).
|
||||
|
||||
## Reading the journal
|
||||
|
||||
| line | meaning |
|
||||
|---|---|
|
||||
| `checkin: action=none desired=… applied=…` then `healthy; heartbeat reported` | converged |
|
||||
| `checkin: action=apply …` → `read 3 secret(s): GEMINI_API_KEY, …` → `promoted release …` → `applied …` | a deploy |
|
||||
| `refused: ENV_INCOMPLETE: unresolved: X` | the bundle needs a variable no manifest entry supplies — fix the descriptor / set the secret in the console; nothing was started |
|
||||
| `refused: DISK_INSUFFICIENT: docker data-root has N MiB free, bundle needs M MiB` | free space (the env-dev-09 lesson): grow the data-root disk or prune |
|
||||
| `refused: PRIVILEGED_REFUSED` / `ROLLBACK_REFUSED` | the bundle needs `allow_privileged` / `allow_rollback` in its `agent` profile |
|
||||
| `temporary network failure` (exit 75) | mesh/tenancy unreachable — check `ziti-edge-tunnel`, the identity's terminators, `monky.tenancy.deploy` health |
|
||||
| `AGENT_UNAUTHENTICATED: bearer refused` (exit 1) | the grant was superseded (kit re-revealed / retire) or the token revoked → re-run the install kit |
|
||||
| `AGENT_ENV_MISMATCH` (exit 78) | the token belongs to another env than `config.yaml` — fix the config or re-issue the identity; the timer keeps firing but every tick exits 78 immediately (no storm) |
|
||||
| `failed: docker compose pull failed (rc=1)` | registry/pull problem; compose output is in the report's tail and in the journal |
|
||||
|
||||
Values never appear in the journal (`[REDACTED]` for token shapes and every value the agent has read).
|
||||
|
||||
## Secrets on the box
|
||||
|
||||
The only credential is `/var/lib/monky-deployd/bao.token` (0600, user `monky-deployd`). `.env`
|
||||
files under `releases/<sha>/` hold the rendered values (0600, same user, `docker compose` reads
|
||||
them). Rotating a secret in the console changes the bundle sha → the next tick re-reads and
|
||||
re-applies. To force a re-read now: `systemctl start monky-deployd.service`.
|
||||
|
||||
Lost or revoked token: `monky-deployd status` shows `token ABSENT`; reveal the kit again (admin,
|
||||
`GET /v1/backends/{id}/agent/install`), paste its bootstrap grant to `/etc/monky-deployd/bootstrap.jwt`
|
||||
(0600 monky-deployd) or re-run `install.sh` (enrolment is skipped when the identity exists), then
|
||||
`monky-deployd bootstrap` or wait a tick.
|
||||
|
||||
## Retire and volumes policy
|
||||
|
||||
Retire is driven by tenancy (`POST /v1/backends/{id}/retire {confirm, force, purge_volumes}`):
|
||||
the next check-in returns `action: down` and the agent runs `docker compose down --remove-orphans`.
|
||||
Data volumes:
|
||||
|
||||
- `-v` (remove volumes) only when tenancy sent `purge_volumes: true` **or** the box's
|
||||
`volumes_on_absent: purge` — and **never on a prod env** (the request is logged and ignored).
|
||||
- default `volumes_on_absent: keep`: volumes stay for a manual `docker volume rm` later.
|
||||
|
||||
After `down` the agent reports `down`, clears `applied_sha`, removes `current` and keeps
|
||||
`releases/` (no secrets outside `.env`, which you can `shred`). Tenancy then destroys the Bao
|
||||
paths, revokes the token accessor and deletes the host identity → the following ticks fail with
|
||||
`AGENT_UNAUTHENTICATED`; `systemctl disable --now monky-deployd.timer` and `apt remove monky-deployd`
|
||||
(`apt purge` also removes `/var/lib/monky-deployd` and `/etc/monky-deployd`).
|
||||
|
||||
## Laptop mode
|
||||
|
||||
`laptop_mode: true` (`install.sh --laptop`): being offline is normal — a tick that cannot reach the
|
||||
mesh logs one INFO line and exits 0. Run without the system timer:
|
||||
|
||||
```sh
|
||||
monky-deployd run # loop, interval_s + jitter, SIGTERM stops; or a user timer with `run --once`
|
||||
```
|
||||
|
||||
Liveness grace for laptops is 24 h on the tenancy side (P0); `offline` only hides the backend from
|
||||
the picker, nothing is retired automatically.
|
||||
|
||||
## Housekeeping
|
||||
|
||||
`monky-deployd run --once --prune` removes every `releases/<sha>` except `current` and runs
|
||||
`docker image prune -f` after a successful apply. The state dir and the token survive `apt remove`;
|
||||
`apt purge` deletes them.
|
||||
|
||||
## Publishing a release to Gitea by hand
|
||||
|
||||
When `release:gitea` is manual (no `GITEA_TOKEN` in CI) — from jump1 with the Gitea API token:
|
||||
|
||||
```sh
|
||||
T=$(tr -d '\n' < ~/.gitlab_tokens/gitea-token); G=http://172.16.8.1:3000/api/v1/repos/mdella/monky-deployd
|
||||
curl -s -X POST -H "Authorization: token $T" $G/mirror-sync # pull the tag
|
||||
RID=$(curl -s -X POST -H "Authorization: token $T" -H 'Content-Type: application/json' $G/releases \
|
||||
-d '{"tag_name":"v0.1.0","name":"monky-deployd v0.1.0","body":"See CHANGELOG.md"}' | jq -r .id)
|
||||
for f in monky-deployd_0.1.0_amd64.deb monky-deployd_0.1.0_amd64.deb.sha256 install.sh; do
|
||||
curl -s -X POST -H "Authorization: token $T" -F "attachment=@$f" "$G/releases/$RID/assets?name=$f"; done
|
||||
```
|
||||
|
||||
The assets are then at `https://gitea.cbs.tikali.net/mdella/monky-deployd/releases/download/v0.1.0/<file>`,
|
||||
which is what `install.sh` fetches (GitLab artifacts from the tag pipeline's `package` job).
|
||||
|
||||
## Ansible (osg1-07)
|
||||
|
||||
`ansible/roles/monky_deployd/` is the role skeleton to copy into osg1-07: `.deb` from the Gitea
|
||||
release (sha256-verified), config template, bootstrap grant from a vault var, ACL on the identity,
|
||||
timer, one tick. Rolled to env-dev-06..09 after the env-qa-02 pilot; env-dev-01 last.
|
||||
@@ -0,0 +1,148 @@
|
||||
<!-- xlate:verbatim-fences -->
|
||||
# monky-deployd protocol
|
||||
|
||||
The agent side of the backend lifecycle agent protocol — [MONKY-ADR-0028](https://scm.tikali.ai/tikali/applications/monky/monky-design-docs/-/blob/develop/adr/0028-backend-lifecycle-ownership-and-execution.md)
|
||||
§2/§3 (amendment 2026-09-05) and [design doc 24](https://scm.tikali.ai/tikali/applications/monky/monky-design-docs/-/blob/develop/docs/24-backend-lifecycle.md)
|
||||
§2.1a, §3.3, §4. The tenancy side is monky-tenancy `app/agent_main.py` / `app/api/agent.py` /
|
||||
`app/schemas_backends.py`; its shapes are the contract, this page is how the agent uses them.
|
||||
|
||||
## Where and how
|
||||
|
||||
- **Where:** tenancy's **agent entrypoint** — a second container (`app.agent_main`,
|
||||
`127.0.0.1:8081`) bound to the ziti service **`monky.tenancy.deploy`** by a `ziti-edge-tunnel
|
||||
run-host` sidecar. Unreachable from the public ingress. Only host identities carrying
|
||||
`#monky-deploy-agent` may dial (the broker adds the attr when the identity is created at kit reveal).
|
||||
- **Transport:** plain HTTP inside the mesh (`transport: sdk` dials by service name;
|
||||
`proxy` → `127.0.0.1:18443`). OpenBao is the existing `openbao` ziti service (`#openbao-client`),
|
||||
dialled as `https://bao.cbs.tikali.net:8200` (an intercept name), TLS validated against the
|
||||
`openbao-ca` certificate (`bao.ca_bundle`).
|
||||
- **Auth:** `Authorization: Bearer <the agent's OpenBao token>`. Tenancy verifies it with
|
||||
`auth/token/lookup` (cached 60 s), pins the request's `env_id` to the token's `meta.env_id`
|
||||
(`403 AGENT_ENV_MISMATCH` → exit 78, never retried), and compares `meta.grant_jti` against
|
||||
`backend_leases` (`401 AGENT_UNAUTHENTICATED` when the grant was superseded or revoked).
|
||||
- **Rate buckets per env:** checkin 6/min, bundle 10/min, report 30/min, lease **5/h**
|
||||
(`429 LEASE_RATE_LIMITED`, `Retry-After`) → the agent treats 429 as temporary (exit 75).
|
||||
- **Errors** are `{code, detail}` JSON; the agent maps them to `TenancyError` subclasses.
|
||||
|
||||
## Bootstrap → lease → token (the grant flow)
|
||||
|
||||
```
|
||||
kit reveal (admin, once) tenancy signs a bootstrap DEPLOY GRANT: ES256 JWT
|
||||
iss <tenancy issuer>, aud openbao-see-env, sub agent:<env_id>,
|
||||
env_id, kind deploy-grant, jti (single-use, recorded), exp now+1h
|
||||
install.sh stages it at /etc/monky-deployd/bootstrap.jwt (0600)
|
||||
first tick POST https://bao…/v1/auth/jwt-tenancy/login {"role":"see-env","jwt":<grant>}
|
||||
→ auth.client_token (ttl 24 h, max 30 d, policy see-env,
|
||||
metadata {env_id, grant_jti} from the mount's claim_mappings)
|
||||
token → /var/lib/monky-deployd/bao.token (0600); grant deleted
|
||||
every tick Bearer <token> → checkin / bundle / lease / report
|
||||
apply POST /v1/agent/lease → {login_jwt,…} → jwt-tenancy login → NEW token
|
||||
(becomes the bearer; the old one is revoke-self'd) → KV reads
|
||||
upkeep (finally) lookup-self; ttl < 12 h → renew-self; age > 30 d − 2 d → re-lease
|
||||
```
|
||||
|
||||
OpenBao side (Terraform, `tikali/services/security/openbao`): mount `jwt-tenancy`
|
||||
(`jwks_url` = tenancy's `GET /.well-known/agent-jwks.json`, in-cluster), role `see-env`
|
||||
(`role_type=jwt`, `user_claim=env_id` ⇒ one alias/entity **per env**, `bound_audiences=["openbao-see-env"]`,
|
||||
`bound_claims={"kind":"deploy-grant"}`, `claim_mappings={"env_id":"env_id","jti":"grant_jti"}`,
|
||||
`token_ttl=86400`, `token_max_ttl=2592000`, `token_no_default_policy=true`), policy `see-env` =
|
||||
`monky/data/{{identity.entity.aliases.<accessor>.metadata.env_id}}/see/*` read + metadata list +
|
||||
`auth/token/{renew-self,lookup-self}`. **Gate 1 v2 passed 2026-09-05 07:40Z** (distinct entity per env).
|
||||
|
||||
**Second-reveal semantics.** A kit re-reveal (or a retire) supersedes every earlier grant of the
|
||||
backend and revokes the token accessors tenancy knows. The old kit's grant fails at login
|
||||
(unknown/used `jti`), and a token already minted from it is refused at the next call with
|
||||
`401 AGENT_UNAUTHENTICATED` (its `meta.grant_jti` is superseded) — **not** `AGENT_ENV_MISMATCH`.
|
||||
The agent then deletes its token; if a fresh `bootstrap.jwt` is on disk it bootstraps again in the
|
||||
same tick, otherwise it exits 1 and says "re-run the install kit".
|
||||
|
||||
## The four calls
|
||||
|
||||
### `POST /v1/agent/checkin`
|
||||
|
||||
```json
|
||||
{"env_id": "env-qa-02", "agent_version": "0.1.0", "applied_sha": "3f9c…",
|
||||
"host": {"hostname": "env-qa-02", "os": "ubuntu-26.04", "kernel": "7.0.0-30-generic",
|
||||
"docker": "28.3.0", "compose": "2.32.0", "free_bytes": 61234567890,
|
||||
"agent_version": "0.1.0", "transport": "sdk", "site": "cbs", "laptop_mode": false},
|
||||
"containers": [{"name": "see-backend", "state": "running", "health": "healthy"}]}
|
||||
```
|
||||
```json
|
||||
{"env_id": "env-qa-02", "desired_sha": "7a10…", "action": "apply", "purge_volumes": false,
|
||||
"bundle_url": "/v1/agent/bundle/env-qa-02/7a10…", "checkin_interval_s": 60,
|
||||
"vault": {"addr": "https://bao.cbs.tikali.net:8200", "mount": "monky", "prefix": "env-qa-02/see"}}
|
||||
```
|
||||
`action`: `apply` (desired ≠ applied), `none` (converged → heartbeat), `down` (retire; `purge_volumes`
|
||||
is only meaningful here). `vault.mount` is the **KV** mount; the agent adopts it if it differs from
|
||||
config. `vault.addr` is informational — the transport decides how OpenBao is reached.
|
||||
|
||||
### `GET /v1/agent/bundle/{env_id}/{sha}`
|
||||
|
||||
`application/x-tar`, `Cache-Control: no-store`, `X-Bundle-Sha` (tenancy) / `X-Bundle-Sha256` (doc 24;
|
||||
both accepted). Members: `docker-compose.yml` (or `compose.yaml`), `.env.template`,
|
||||
`secrets.manifest.json`, `bundle.json`, optionally `.env.example`. **No secret material** — the agent
|
||||
refuses a manifest entry carrying a `value`. The agent recomputes
|
||||
|
||||
```
|
||||
sha256( for name in sorted(files): name + "\0" + content + "\0" )
|
||||
```
|
||||
|
||||
and refuses (`BUNDLE_SHA_MISMATCH`) unless it equals `desired_sha`. `404 BUNDLE_NOT_FOUND` when
|
||||
the sha is not published.
|
||||
|
||||
`secrets.manifest.json` (paths + versions, never values):
|
||||
```json
|
||||
{"vault_mode": "openbao",
|
||||
"entries": [{"var": "GEMINI_API_KEY", "path": "monky/env-qa-02/see/gemini_api_key", "kind": "supplied", "version": 2},
|
||||
{"var": "POSTGRES_PASSWORD", "path": "monky/env-qa-02/see/pg_password", "kind": "generated", "version": 1},
|
||||
{"var": "SEE_ADMIN_TOKEN", "path": "monky/env-qa-02/see/admin_token", "kind": "generated", "version": 1}]}
|
||||
```
|
||||
Path forms `monky/<env>/see/<name>` (tenancy) and `monky/data/<env>/see/<name>` (doc 24) both map
|
||||
to `GET /v1/monky/data/<env>/see/<name>?version=N`; the `<env>` segment **must** equal the agent's
|
||||
`env_id`. `version: null` = latest. The secret document is `{"value": "…"}`.
|
||||
|
||||
`bundle.json` (monky-deploy `render_bundle_json`): `env_id, tier, site, bundle, target, target_class,
|
||||
address, pull, renderer, images_policy, secrets_provider, agent{…}, files[]`. The agent honours
|
||||
`allow_privileged`, `allow_rollback`, `disk_need_bytes` (top level or under `agent`), refuses a
|
||||
`renderer` other than `compose`, and treats `tier == "prod"` as prod.
|
||||
|
||||
### `POST /v1/agent/lease`
|
||||
|
||||
```json
|
||||
{"env_id": "env-qa-02", "reason": "apply"} // reason: apply | renew
|
||||
```
|
||||
```json
|
||||
{"env_id": "env-qa-02", "login_jwt": "eyJ…", "ttl_s": 3600, "mount": "jwt-tenancy", "role": "see-env",
|
||||
"vault": {"addr": "https://bao.cbs.tikali.net:8200", "mount": "monky"}}
|
||||
```
|
||||
Then `POST /v1/auth/{mount}/login {"role": "{role}", "jwt": "{login_jwt}"}` on OpenBao. A body with
|
||||
`wrapping_token` / `role_id` (the pre-Gate-1 AppRole lease) is refused with `LEASE_SHAPE` → report
|
||||
`failed`. `429 LEASE_RATE_LIMITED` → exit 75.
|
||||
|
||||
### `POST /v1/agent/report`
|
||||
|
||||
```json
|
||||
{"env_id": "env-qa-02", "sha": "7a10…", "result": "applied",
|
||||
"log_tail": "INFO checkin: action=apply …\nINFO applied 7a10c0ffee11 (2 container(s) healthy)",
|
||||
"containers": [{"name": "see-backend", "state": "running", "health": "healthy"}],
|
||||
"detail": "ENV_INCOMPLETE: unresolved: SEE_ADMIN_TOKEN"} // detail only on failed
|
||||
```
|
||||
`result`: `applied` (sets `applied_sha`, health ok — also the heartbeat on `none`), `failed`
|
||||
(health degraded; `detail` = code + names), `down` (clears `applied_sha`). `log_tail` ≤ 16 KiB, the
|
||||
first 2 KiB land in the audit log — it has been through the redactor.
|
||||
|
||||
## Divergences (2026-09-05)
|
||||
|
||||
- **monky-tenancy `main` (MR !15) still implements the AppRole lease and install kit**
|
||||
(`AgentLeaseOut{wrapping_token, role_id}`, `bootstrap.wrap`, `bao.approle` in the kit's config). The
|
||||
binding design is the plan's Gate 1 RESULT / ADR-0028 amendment: `{login_jwt, ttl_s, mount, role}`
|
||||
and `POST /v1/auth/jwt-tenancy/login`. This agent implements the latter; against an un-migrated
|
||||
tenancy it reports `failed` with `LEASE_SHAPE` and refuses `bao.approle` in its config. The tenancy
|
||||
follow-up (deploy-grant signer, JWKS, `lease` shape, kit → `bootstrap.jwt`) is tracked on
|
||||
monky-tenancy.
|
||||
- The kit's generated config uses `tenancy.base_url: http://monky.tenancy.deploy:8081` — accepted as
|
||||
an alias for `tenancy.{scheme,host,port}`.
|
||||
- `report` gains an optional `detail` (doc 24 §3.3); tenancy's `AgentReport` ignores unknown fields
|
||||
today — if `strict` bodies land, `detail` folds into `log_tail`.
|
||||
- Bundle sha header: tenancy sends `X-Bundle-Sha`, doc 24 says `X-Bundle-Sha256`; the agent reads
|
||||
either and trusts only its own computation.
|
||||
Reference in New Issue
Block a user