monky-deployd/docs/OPERATIONS.md

Operating monky-deployd

Units

unit	what
monky-deployd.timer	fires monky-deployd.service 60 s after the previous tick finished (+ ≤10 s jitter); OnBootSec=90s
monky-deployd.service	Type=oneshot, monky-deployd run --once as user monky-deployd (+ docker group); SuccessExitStatus=75; hardened (NoNewPrivileges, ProtectSystem=strict, ReadWritePaths=/var/lib/monky-deployd /etc/monky-deployd /run/docker.sock, UMask=0077, no capabilities)
monky-deployd-proxy.service	only with transport: proxy: ziti tunnel proxy -i <identity> monky.tenancy.deploy:18443 openbao:18200 as user ziti; EnvironmentFile=/etc/monky-deployd/proxy.env
ziti-edge-tunnel.service	the host identity's tunneler in run-host mode (drop-in run-host.conf written by install.sh)

systemctl status monky-deployd.timer monky-deployd.service
systemctl list-timers monky-deployd.timer
systemctl start monky-deployd.service          # tick now (blocks until done)
journalctl -u monky-deployd -f                 # priorities are real (INFO/WARNING/ERROR)
monky-deployd status                           # exit 0 = token present and in sync
monky-deployd status --json | jq .

Files: /etc/monky-deployd/config.yaml (0640 root:monky-deployd), /etc/monky-deployd/openbao-ca.pem, /etc/monky-deployd/bootstrap.jwt (only until the first login), /var/lib/monky-deployd/{bao.token,state.json,lock}, /var/lib/monky-deployd/<env>/releases/<sha>/ + current (the compose project dir, .env 0600), /opt/openziti/etc/identities/monky-host.<env>.json (ziti:ziti 0640; the agent reads it as a member of the ziti group, plus an explicit ACL u:monky-deployd:r and a default ACL on the directory).

The group is the grant that matters. ziti-edge-tunnel rewrites the identity file whenever the controller sends a config update, and the rewrite drops POSIX ACLs — an ACL-only box works until the first refresh and then fails every tick (env-dev-08, 2026-09-09). On a host where the tunneller was installed independently of the kit, check id monky-deployd for ziti before anything else.

Upgrading the agent. dpkg -i keeps the timer enabled from 0.1.9 on. On a box upgraded from 0.1.8 or earlier the old package's prerm already ran and disabled it — check systemctl is-enabled monky-deployd.timer after the upgrade and systemctl enable --now it if needed.

Reading the journal

line	meaning
checkin: action=none desired=… applied=… then healthy; heartbeat reported	converged
checkin: action=apply … → read 3 secret(s): GEMINI_API_KEY, … → promoted release … → applied …	a deploy
refused: ENV_INCOMPLETE: unresolved: X	the bundle needs a variable no manifest entry supplies — fix the descriptor / set the secret in the console; nothing was started
refused: DISK_INSUFFICIENT: image storage (…) has N MiB free, …	free space: grow the disk or prune. The paths in the message are every filesystem a pull can fill — docker's data-root AND containerd's root, which docker 29 fills with image layers and which is often a different disk (env-dev-08). A bundle that declares no size must still clear disk.headroom_bytes
refused: PRIVILEGED_REFUSED / ROLLBACK_REFUSED	the bundle needs allow_privileged / allow_rollback in its agent profile
temporary network failure (exit 75)	mesh/tenancy unreachable — check ziti-edge-tunnel, the identity's terminators, monky.tenancy.deploy health
transport sdk: identity … is not readable by this user	the agent lost its read on the identity file (usually a tunneller rewrite dropping the ACL) — usermod -a -G ziti monky-deployd
AGENT_UNAUTHENTICATED: bearer refused (exit 1)	the grant was superseded (kit re-revealed / retire) or the token revoked → re-run the install kit
AGENT_ENV_MISMATCH (exit 78)	the token belongs to another env than config.yaml — fix the config or re-issue the identity; the timer keeps firing but every tick exits 78 immediately (no storm)
failed: docker compose pull failed (rc=1)	registry/pull problem; compose output is in the report's tail and in the journal

Values never appear in the journal ([REDACTED] for token shapes and every value the agent has read).

Secrets on the box

The only credential is /var/lib/monky-deployd/bao.token (0600, user monky-deployd). .env files under releases/<sha>/ hold the rendered values (0600, same user, docker compose reads them). Rotating a secret in the console changes the bundle sha → the next tick re-reads and re-applies. To force a re-read now: systemctl start monky-deployd.service.

Lost or revoked token: monky-deployd status shows token ABSENT; reveal the kit again (admin, GET /v1/backends/{id}/agent/install), paste its bootstrap grant to /etc/monky-deployd/bootstrap.jwt (0600 monky-deployd) or re-run install.sh (enrolment is skipped when the identity exists), then monky-deployd bootstrap or wait a tick.

Retire and volumes policy

Retire is driven by tenancy (POST /v1/backends/{id}/retire {confirm, force, purge_volumes}): the next check-in returns action: down and the agent runs docker compose down --remove-orphans. Data volumes:

• -v (remove volumes) only when tenancy sent purge_volumes: true or the box's volumes_on_absent: purge — and never on a prod env (the request is logged and ignored).
• default volumes_on_absent: keep: volumes stay for a manual docker volume rm later.

After down the agent reports down, clears applied_sha, removes current and keeps releases/ (no secrets outside .env, which you can shred). Tenancy then destroys the Bao paths, revokes the token accessor and deletes the host identity → the following ticks fail with AGENT_UNAUTHENTICATED; systemctl disable --now monky-deployd.timer and apt remove monky-deployd (apt purge also removes /var/lib/monky-deployd and /etc/monky-deployd).

Laptop mode

laptop_mode: true (install.sh --laptop): being offline is normal — a tick that cannot reach the mesh logs one INFO line and exits 0. Run without the system timer:

monky-deployd run             # loop, interval_s + jitter, SIGTERM stops; or a user timer with `run --once`

Liveness grace for laptops is 24 h on the tenancy side (P0); offline only hides the backend from the picker, nothing is retired automatically.

Housekeeping

monky-deployd run --once --prune removes every releases/<sha> except current and runs docker image prune -f after a successful apply. The state dir and the token survive apt remove; apt purge deletes them.

Where the installer downloads from

The primary source is the scm.tikali.ai generic package registry of this project: https://scm.tikali.ai/api/v4/projects/69/packages/generic/monky-deployd/<ver>/<file> for monky-deployd_<ver>_amd64.deb, .sha256 and install.sh. The project is private (its parent groups are private, so it cannot be made public): every fetch, the script itself included, sends the DEPLOY-TOKEN header with a read-only GitLab deploy token — scope read_package_registry only, nothing else (no repository, no API, no write); revocable at any time in the project's Settings → Repository → Deploy tokens. It lives in OpenBao at monky/monky-tenancy/deployd-download (key token); the monky-tenancy install kit carries it and passes --token, the ansible role sends it from the vaulted monky_deployd_download_token. The one-liner:

curl -sSf -H "DEPLOY-TOKEN: $T" https://scm.tikali.ai/api/v4/projects/69/packages/generic/monky-deployd/<ver>/install.sh \
  | sudo bash -s -- --env <id> --site <site> --token "$T" --bootstrap-file bootstrap.jwt

install.sh never prints the token (it goes through a 0600 curl -K file that is deleted after the download; xtrace is switched off). A 401 on the download means the token is missing, revoked or lacks the scope. Inside the estate gitea.cbs.tikali.net is split-horizon to jump1's RED EIP (10.10.0.175), which has no HTTP ingress, so backend boxes cannot fetch from the Gitea mirror (cbs/iac#102). Off-estate, install.sh --source gitea (ansible: monky_deployd_base_url/_deb_url, no token) uses the Gitea release instead. The tag pipeline always publishes to GitLab (release); release:gitea is automatic only with GITEA_TOKEN in CI and a manual job otherwise — it has not run on a tag pipeline yet.

Publishing a release to Gitea by hand

When release:gitea is manual (no GITEA_TOKEN in CI) — from jump1 with the Gitea API token:

T=$(tr -d '\n' < ~/.gitlab_tokens/gitea-token); G=http://172.16.8.1:3000/api/v1/repos/mdella/monky-deployd
curl -s -X POST -H "Authorization: token $T" $G/mirror-sync                      # pull the tag
RID=$(curl -s -X POST -H "Authorization: token $T" -H 'Content-Type: application/json' $G/releases \
      -d '{"tag_name":"v0.1.0","name":"monky-deployd v0.1.0","body":"See CHANGELOG.md"}' | jq -r .id)
for f in monky-deployd_0.1.0_amd64.deb monky-deployd_0.1.0_amd64.deb.sha256 install.sh; do
  curl -s -X POST -H "Authorization: token $T" -F "attachment=@$f" "$G/releases/$RID/assets?name=$f"; done

The assets are then at https://gitea.cbs.tikali.net/mdella/monky-deployd/releases/download/v0.1.0/<file>, which is what install.sh --source gitea fetches (GitLab artifacts from the tag pipeline's package job).

Ansible (osg1-07)

ansible/roles/monky_deployd/ is the role skeleton to copy into osg1-07: .deb from the scm.tikali.ai package registry (sha256-verified), config template, bootstrap grant from a vault var, ACL on the identity, timer, one tick. Rolled to env-dev-06..09 after the env-qa-02 pilot; env-dev-01 last.