mirror of
https://scm.tikali.ai/tikali/applications/monky/monky-deployd.git
synced 2026-09-18 05:36:15 +00:00
mdella
1c42e913a8
Stdlib-only Python 3.12 agent for docker VMs and laptops: flock → checkin
(bearer = the agent's OpenBao token, bootstrapped from the install kit's
jwt-tenancy deploy grant) → action apply|none|down → bundle (sha256
verified) → refusal checks (unresolved ${VAR} names only, manifest paths
pinned to monky/data/<env>/see/, privileged/host-network, rollback, disk
need×1.5+headroom) → lease → POST /v1/auth/jwt-tenancy/login → KV reads →
.env 0600 → promote → compose pull/up → wait healthy → report; finally
renew-self / re-lease before max TTL, scrub. Exit 0/75/78/1. Redactor log
filter. Transports sdk (openziti) / proxy (ziti tunnel proxy 18443/18200) /
system. Laptop mode.
Packaging: hardened oneshot + 60 s timer + proxy unit, nfpm .deb with
/opt/monky-deployd/venv, install.sh for Ubuntu 26.04 (Gitea release
download, enrol, ACLs, bootstrap from stdin), ansible role skeleton for
osg1-07. CI: lint/test on every change; wheel (openziti on ubuntu:26.04) and
package (nfpm) allow_failure until runner egress is proven; GitLab release +
release:gitea on v* tags. Docs: README, PROTOCOL, OPERATIONS, CHANGELOG,
CLAUDE/AGENTS.
Divergence noted: monky-tenancy main (MR !15) still ships the AppRole lease
and kit; this agent implements the plan's Gate 1 RESULT (login_jwt, no
unwrap) and refuses an AppRole lease loudly (LEASE_SHAPE).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KLB7jieMNRkTsJ2epr4Ds1
152 lines
9.7 KiB
Markdown
152 lines
9.7 KiB
Markdown
<!-- xlate:verbatim-fences -->
|
||
# monky-deployd
|
||
|
||
**The on-box pull agent for Monky backends** ([MONKY-ADR-0028](https://scm.tikali.ai/tikali/applications/monky/monky-design-docs/-/blob/develop/adr/0028-backend-lifecycle-ownership-and-execution.md), [design doc 24](https://scm.tikali.ai/tikali/applications/monky/monky-design-docs/-/blob/develop/docs/24-backend-lifecycle.md)).
|
||
A docker VM or a laptop that hosts a backend environment runs this agent; every minute it dials
|
||
**monky-tenancy over the OpenZiti mesh with the box's own host identity**, asks what the backend
|
||
should be running, and converges: fetch the rendered bundle, lease a deploy grant, log in to
|
||
OpenBao, read its own secrets, `docker compose up`, report. Nothing pushes into the box, no SSH,
|
||
no credentials for the box are held anywhere else. The same check-in is the liveness heartbeat.
|
||
|
||
```
|
||
box / laptop ◄── mesh ──► monky-deployd: checkin → bundle → lease → OpenBao → compose up → report
|
||
```
|
||
|
||
Python 3.12+, **stdlib only** (the `openziti` SDK is optional and vendored into the `.deb`).
|
||
Verified on **Ubuntu 26.04**.
|
||
|
||
## Install (one-liner, from the enrolment kit)
|
||
|
||
An admin reveals the kit once in the console (`GET /v1/backends/{id}/agent/install`); it hands
|
||
you the enrolment JWT and a one-time **bootstrap deploy grant**. On the box:
|
||
|
||
```sh
|
||
curl -fsSL https://gitea.cbs.tikali.net/mdella/monky-deployd/raw/branch/main/packaging/install.sh \
|
||
| sudo bash -s -- --env env-qa-02 --site cbs --enrol-jwt ./monky-host.env-qa-02.jwt < bootstrap.jwt
|
||
# [--transport sdk|proxy|system] [--version 0.1.0] [--laptop] [--bao-ca openbao-ca.pem]
|
||
```
|
||
|
||
`install.sh` installs `ziti-edge-tunnel` (OpenZiti `jammy` suite) and `docker-compose-plugin` if
|
||
absent, downloads the pinned `.deb` + `.sha256` from the [Gitea release](https://gitea.cbs.tikali.net/mdella/monky-deployd/releases),
|
||
enrols `monky-host.<env>` if the identity is missing, switches the tunneler to `run-host`,
|
||
writes `/etc/monky-deployd/config.yaml`, grants the agent read access to the identity (ACL),
|
||
stages the bootstrap grant (0600), enables `monky-deployd.timer`, runs one tick and deletes the
|
||
JWT. The GitLab project is private, so **the public download is the Gitea mirror**.
|
||
|
||
## Transports
|
||
|
||
| `transport` | how | when |
|
||
|---|---|---|
|
||
| `sdk` (default) | the OpenZiti Python SDK dials `monky.tenancy.deploy` / `openbao` by service name with the same host identity `ziti-edge-tunnel run-host` uses; no tun, no root beyond the docker group | VMs and laptops with the vendored wheel |
|
||
| `proxy` | `monky-deployd-proxy.service` runs `ziti tunnel proxy -i <identity> monky.tenancy.deploy:18443 openbao:18200` as user `ziti`; the agent talks to `127.0.0.1:18443/18200` (TLS SNI + cert check still `bao.cbs.tikali.net`) | the wheel is unavailable; `ziti` CLI present |
|
||
| `system` | plain DNS/TCP | laptops whose tunneler runs in `run` mode (tun + DNS) |
|
||
|
||
## Commands and exit codes
|
||
|
||
```
|
||
monky-deployd run --once [--prune] # one tick (what the timer runs); --prune drops old releases + docker image prune
|
||
monky-deployd run # loop (laptop mode: no timer); SIGTERM stops it
|
||
monky-deployd status [--json] # token present? applied vs desired sha, last checkin/report, compose ps
|
||
monky-deployd bootstrap [--force] # log in to OpenBao with /etc/monky-deployd/bootstrap.jwt, store the token
|
||
monky-deployd version
|
||
```
|
||
|
||
| exit | meaning |
|
||
|---|---|
|
||
| `0` | converged / heartbeat sent — or offline in `laptop_mode` |
|
||
| `75` | temporary network failure (mesh down, tenancy unreachable, `429`); the next tick retries — `SuccessExitStatus=75` in the unit |
|
||
| `78` | `AGENT_ENV_MISMATCH`: the token is pinned to another environment than `config.yaml` says; **not retried** (loop mode exits) |
|
||
| `1` | refusal or failure (`ENV_INCOMPLETE`, `PRIVILEGED_REFUSED`, `DISK_INSUFFICIENT`, `ROLLBACK_REFUSED`, `BUNDLE_SHA_MISMATCH`, compose failure, `AGENT_UNAUTHENTICATED`, no credentials) — reported to tenancy as `failed` with the code and secret *names* only |
|
||
|
||
## What a tick does
|
||
|
||
1. `flock` (a second concurrent tick exits 0) → load `state.json`.
|
||
2. Bearer = the OpenBao token at `/var/lib/monky-deployd/bao.token` (0600). Absent → log in to
|
||
`jwt-tenancy` with the kit's bootstrap grant (`/etc/monky-deployd/bootstrap.jwt`), then delete it.
|
||
3. `POST /v1/agent/checkin` `{env_id, agent_version, applied_sha, host, containers}` → `action`:
|
||
- **`down`** → `docker compose down --remove-orphans` (`-v` only if `purge_volumes` was
|
||
requested or `volumes_on_absent: purge`, **never on prod**) → report `down`.
|
||
- **`none`** → `compose ps`; healthy → report `applied` (heartbeat); unhealthy → report `failed`.
|
||
- **`apply`** → `GET /v1/agent/bundle/{env}/{sha}`, verify **sha256** over the files, then refuse on:
|
||
unresolved `${VAR}` (names only), manifest paths outside `monky/data/<env>/see/`, `privileged`/host
|
||
network/`SYS_ADMIN` unless `bundle.json` `allow_privileged`, a rollback unless `allow_rollback`,
|
||
docker data-root free space `< need × 1.5 + 2 GiB` → `POST /v1/agent/lease` → OpenBao
|
||
`POST /v1/auth/jwt-tenancy/login {"role":"see-env","jwt":<grant>}` → KV reads per
|
||
`secrets.manifest.json` (pinned versions) → `.env` written atomically 0600 into a staging dir →
|
||
promoted to `releases/<sha>` + `current` → `compose pull` → `up -d --remove-orphans` → wait
|
||
healthy → report `applied` (or `failed` with the compose log tail).
|
||
4. `finally`: renew-self when the TTL runs low, re-lease before max TTL, scrub secrets from memory,
|
||
remove staging dirs, save state.
|
||
|
||
## Security model
|
||
|
||
- **Identity = the box's ziti host identity.** Only identities with `#monky-deploy-agent` can dial
|
||
tenancy's agent entrypoint; `#openbao-client` reaches OpenBao. The agent reads the identity
|
||
through an ACL (`setfacl -m u:monky-deployd:r`), never owns it.
|
||
- **The bearer to tenancy is the agent's own OpenBao token**, minted by OpenBao from a
|
||
tenancy-signed ES256 deploy grant (`aud openbao-see-env`, `kind deploy-grant`, 1 h, single-use
|
||
`jti`). Tenancy verifies it with `auth/token/lookup`, pins `meta.env_id`, and refuses a token
|
||
whose `meta.grant_jti` was superseded (kit re-reveal, retire) → `401 AGENT_UNAUTHENTICATED`.
|
||
**No AppRole, nothing to unwrap** (Gate 1 result, 2026-09-05).
|
||
- **The agent never receives a secret from tenancy.** Bundles carry placeholders; the agent reads
|
||
`monky/data/<env_id>/see/*` itself, and the OpenBao policy is templated on the token's entity
|
||
(one entity per env), so another env's path is a 403 — and refused locally before any read.
|
||
- **Journald never carries a value.** A `Redactor` filter masks token shapes (`hvs.*`, JWTs,
|
||
`Authorization: Bearer`) and every value the agent has read; the report's `log_tail` goes through
|
||
the same scrubber. Secret *names* are logged.
|
||
- **Hardened oneshot**: user `monky-deployd` (+ `docker` group), `NoNewPrivileges`,
|
||
`ProtectSystem=strict`, `ReadWritePaths` only the state dir, `/etc/monky-deployd` (to consume the
|
||
grant) and the docker socket, `UMask=0077`, no capabilities.
|
||
- **Prod is special**: `-v` is never passed on a prod env, and tenancy never auto-changes prod state.
|
||
|
||
## Ubuntu 26.04 verified checklist (from design doc 24 §2.1a / plan §D 2.8)
|
||
|
||
```
|
||
python3 --version # >= 3.12 (26.04 ships 3.14)
|
||
docker compose version # compose plugin
|
||
systemctl is-active ziti-edge-tunnel # run-host (drop-in run-host.conf)
|
||
ziti tunnel proxy --help # fallback transport present (transport proxy)
|
||
/opt/monky-deployd/venv/bin/python -c 'import openziti' # SDK import (transport sdk)
|
||
monky-deployd status # token present; applied == desired
|
||
journalctl -u monky-deployd -n 50 # "checkin: action=..." then "applied ..."
|
||
df -h $(docker info -f '{{.DockerRootDir}}') # free >= bundle need x 1.5 + headroom
|
||
docker compose -p monky-<env> ps # healthy
|
||
curl monky.percept.<env>:47283/health # from another mesh member -> 200
|
||
```
|
||
|
||
## Repository layout
|
||
|
||
```
|
||
monky_deployd/ cli.py config.py transport.py tenancy.py bao.py bundle.py compose.py state.py redact.py agent.py
|
||
packaging/ install.sh systemd/{monky-deployd.service,.timer,monky-deployd-proxy.service} nfpm.yaml scripts/
|
||
ansible/roles/monky_deployd/ role skeleton for osg1-07 (env-dev-06..09 rollout)
|
||
docs/ PROTOCOL.md (the four calls + the grant flow) OPERATIONS.md (systemd, logs, retire, laptops)
|
||
tests/ hermetic: fake tenancy + fake OpenBao HTTP servers, a stub `docker` on PATH
|
||
```
|
||
|
||
## Development
|
||
|
||
```sh
|
||
ruff check . && ruff format --check .
|
||
pytest # no network, no docker: fakes only
|
||
bash -n packaging/install.sh
|
||
systemd-analyze verify packaging/systemd/*.service # where systemd is available
|
||
```
|
||
|
||
### CI notes
|
||
|
||
`lint` and `test` run on every MR/branch. `wheel` builds the `openziti` wheel on `ubuntu:26.04`
|
||
(PyPI ships an sdist that fetches **ziti-sdk-c from github.com** at build time — the runner needs
|
||
egress to github.com and pypi.org) and `package` builds the `.deb` with `nfpm` (binary from GitHub
|
||
releases, goreleaser apt repo as fallback). Both are `allow_failure: true` until proven on this
|
||
runner; without the wheel the `.deb` still works with `transport: proxy|system`. On a `v*` tag
|
||
`release` uploads to the GitLab generic package registry + release, and `release:gitea` publishes
|
||
the same assets on the public Gitea mirror (automatic when `GITEA_TOKEN` is set, manual otherwise
|
||
— see `docs/OPERATIONS.md` for the by-hand recipe).
|
||
|
||
## See also
|
||
|
||
- `docs/PROTOCOL.md`, `docs/OPERATIONS.md`, `CHANGELOG.md`
|
||
- monky-tenancy `docs/usage.md` (agent protocol), `app/api/agent.py`, `app/schemas_backends.py`
|
||
- monky-deploy (the renderer whose `render_files` produces the bundle)
|