# Operating monky-deployd ## Units | unit | what | |---|---| | `monky-deployd.timer` | fires `monky-deployd.service` 60 s after the previous tick finished (+ ≤10 s jitter); `OnBootSec=90s` | | `monky-deployd.service` | `Type=oneshot`, `monky-deployd run --once` as user `monky-deployd` (+ `docker` group); `SuccessExitStatus=75`; hardened (`NoNewPrivileges`, `ProtectSystem=strict`, `ReadWritePaths=/var/lib/monky-deployd /etc/monky-deployd /run/docker.sock`, `UMask=0077`, no capabilities) | | `monky-deployd-proxy.service` | only with `transport: proxy`: `ziti tunnel proxy -i monky.tenancy.deploy:18443 openbao:18200` as user `ziti`; `EnvironmentFile=/etc/monky-deployd/proxy.env` | | `ziti-edge-tunnel.service` | the host identity's tunneler in **`run-host`** mode (drop-in `run-host.conf` written by `install.sh`) | ```sh systemctl status monky-deployd.timer monky-deployd.service systemctl list-timers monky-deployd.timer systemctl start monky-deployd.service # tick now (blocks until done) journalctl -u monky-deployd -f # priorities are real (INFO/WARNING/ERROR) monky-deployd status # exit 0 = token present and in sync monky-deployd status --json | jq . ``` Files: `/etc/monky-deployd/config.yaml` (0640 root:monky-deployd), `/etc/monky-deployd/openbao-ca.pem`, `/etc/monky-deployd/bootstrap.jwt` (only until the first login), `/var/lib/monky-deployd/{bao.token,state.json,lock}`, `/var/lib/monky-deployd//releases//` + `current` (the compose project dir, `.env` 0600), `/opt/openziti/etc/identities/monky-host..json` (ziti:ziti 0600 + ACL `u:monky-deployd:r`). ## Reading the journal | line | meaning | |---|---| | `checkin: action=none desired=… applied=…` then `healthy; heartbeat reported` | converged | | `checkin: action=apply …` → `read 3 secret(s): GEMINI_API_KEY, …` → `promoted release …` → `applied …` | a deploy | | `refused: ENV_INCOMPLETE: unresolved: X` | the bundle needs a variable no manifest entry supplies — fix the descriptor / set the secret in the console; nothing was started | | `refused: DISK_INSUFFICIENT: docker data-root has N MiB free, bundle needs M MiB` | free space (the env-dev-09 lesson): grow the data-root disk or prune | | `refused: PRIVILEGED_REFUSED` / `ROLLBACK_REFUSED` | the bundle needs `allow_privileged` / `allow_rollback` in its `agent` profile | | `temporary network failure` (exit 75) | mesh/tenancy unreachable — check `ziti-edge-tunnel`, the identity's terminators, `monky.tenancy.deploy` health | | `AGENT_UNAUTHENTICATED: bearer refused` (exit 1) | the grant was superseded (kit re-revealed / retire) or the token revoked → re-run the install kit | | `AGENT_ENV_MISMATCH` (exit 78) | the token belongs to another env than `config.yaml` — fix the config or re-issue the identity; the timer keeps firing but every tick exits 78 immediately (no storm) | | `failed: docker compose pull failed (rc=1)` | registry/pull problem; compose output is in the report's tail and in the journal | Values never appear in the journal (`[REDACTED]` for token shapes and every value the agent has read). ## Secrets on the box The only credential is `/var/lib/monky-deployd/bao.token` (0600, user `monky-deployd`). `.env` files under `releases//` hold the rendered values (0600, same user, `docker compose` reads them). Rotating a secret in the console changes the bundle sha → the next tick re-reads and re-applies. To force a re-read now: `systemctl start monky-deployd.service`. Lost or revoked token: `monky-deployd status` shows `token ABSENT`; reveal the kit again (admin, `GET /v1/backends/{id}/agent/install`), paste its bootstrap grant to `/etc/monky-deployd/bootstrap.jwt` (0600 monky-deployd) or re-run `install.sh` (enrolment is skipped when the identity exists), then `monky-deployd bootstrap` or wait a tick. ## Retire and volumes policy Retire is driven by tenancy (`POST /v1/backends/{id}/retire {confirm, force, purge_volumes}`): the next check-in returns `action: down` and the agent runs `docker compose down --remove-orphans`. Data volumes: - `-v` (remove volumes) only when tenancy sent `purge_volumes: true` **or** the box's `volumes_on_absent: purge` — and **never on a prod env** (the request is logged and ignored). - default `volumes_on_absent: keep`: volumes stay for a manual `docker volume rm` later. After `down` the agent reports `down`, clears `applied_sha`, removes `current` and keeps `releases/` (no secrets outside `.env`, which you can `shred`). Tenancy then destroys the Bao paths, revokes the token accessor and deletes the host identity → the following ticks fail with `AGENT_UNAUTHENTICATED`; `systemctl disable --now monky-deployd.timer` and `apt remove monky-deployd` (`apt purge` also removes `/var/lib/monky-deployd` and `/etc/monky-deployd`). ## Laptop mode `laptop_mode: true` (`install.sh --laptop`): being offline is normal — a tick that cannot reach the mesh logs one INFO line and exits 0. Run without the system timer: ```sh monky-deployd run # loop, interval_s + jitter, SIGTERM stops; or a user timer with `run --once` ``` Liveness grace for laptops is 24 h on the tenancy side (P0); `offline` only hides the backend from the picker, nothing is retired automatically. ## Housekeeping `monky-deployd run --once --prune` removes every `releases/` except `current` and runs `docker image prune -f` after a successful apply. The state dir and the token survive `apt remove`; `apt purge` deletes them. ## Where the installer downloads from The primary source is the **scm.tikali.ai generic package registry** of this (public) project: `https://scm.tikali.ai/api/v4/projects/69/packages/generic/monky-deployd//` for `monky-deployd__amd64.deb`, `.sha256` and `install.sh`; the one-liner fetches the script from `https://scm.tikali.ai/tikali/applications/monky/monky-deployd/-/raw/main/packaging/install.sh`. Inside the estate `gitea.cbs.tikali.net` is split-horizon to jump1's RED EIP (`10.10.0.175`), which has no HTTP ingress, so backend boxes cannot fetch from the Gitea mirror (cbs/iac#102). Off-estate, `install.sh --source gitea` (ansible: `monky_deployd_base_url`/`_deb_url`) uses the Gitea release instead. The tag pipeline publishes to both (`release`, `release:gitea`). ## Publishing a release to Gitea by hand When `release:gitea` is manual (no `GITEA_TOKEN` in CI) — from jump1 with the Gitea API token: ```sh T=$(tr -d '\n' < ~/.gitlab_tokens/gitea-token); G=http://172.16.8.1:3000/api/v1/repos/mdella/monky-deployd curl -s -X POST -H "Authorization: token $T" $G/mirror-sync # pull the tag RID=$(curl -s -X POST -H "Authorization: token $T" -H 'Content-Type: application/json' $G/releases \ -d '{"tag_name":"v0.1.0","name":"monky-deployd v0.1.0","body":"See CHANGELOG.md"}' | jq -r .id) for f in monky-deployd_0.1.0_amd64.deb monky-deployd_0.1.0_amd64.deb.sha256 install.sh; do curl -s -X POST -H "Authorization: token $T" -F "attachment=@$f" "$G/releases/$RID/assets?name=$f"; done ``` The assets are then at `https://gitea.cbs.tikali.net/mdella/monky-deployd/releases/download/v0.1.0/`, which is what `install.sh --source gitea` fetches (GitLab artifacts from the tag pipeline's `package` job). ## Ansible (osg1-07) `ansible/roles/monky_deployd/` is the role skeleton to copy into osg1-07: `.deb` from the scm.tikali.ai package registry (sha256-verified), config template, bootstrap grant from a vault var, ACL on the identity, timer, one tick. Rolled to env-dev-06..09 after the env-qa-02 pilot; env-dev-01 last.