mirror of
https://scm.tikali.ai/tikali/applications/monky/monky-deployd.git
synced 2026-09-18 05:36:15 +00:00
mdella
43615a6fda
Three faults from one agent-managed onboarding (env-dev-08, 2026-09-09), each of which pointed the diagnosis away from the actual fault. 1. install.sh granted the agent's read on the ziti identity with a POSIX ACL. ziti-edge-tunnel rewrites that file on a controller config update and the rewrite drops the ACL: the agent applied cleanly at 01:21 and was failing every tick by 01:32. Group membership survives the rewrite (the file stays ziti:ziti 0640), so install.sh and the package postinstall now add monky-deployd to the `ziti` group, and a default ACL on the identity directory carries the grant onto a newly created file. The explicit ACLs stay for the boxes that need them. 2. openziti.load() accepts an unreadable or malformed identity: the C SDK logs "configuration is invalid" and returns a context that only fails at dial, as a bare TypeError, which the transport reported as a missing intercept or a policy gap. The SDK transport now reads and parses the identity itself and names the real fault first. 3. The disk pre-flight ran only when the bundle declared disk_need_bytes, so a bundle without one died mid-pull with containerd's "no space left on device" — which reads as a registry fault. A bundle that declares no size now has to clear the headroom floor, and the pre-flight measures containerd's root as well as the docker data-root: docker 29 keeps image layers in the containerd image store, and on env-dev-08 those sat on different filesystems (93 GiB free where the agent looked, 2.8 GiB where the pull wrote). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KLB7jieMNRkTsJ2epr4Ds1
142 lines
9.4 KiB
Markdown
142 lines
9.4 KiB
Markdown
<!-- xlate:verbatim-fences -->
|
|
# Operating monky-deployd
|
|
|
|
## Units
|
|
|
|
| unit | what |
|
|
|---|---|
|
|
| `monky-deployd.timer` | fires `monky-deployd.service` 60 s after the previous tick finished (+ ≤10 s jitter); `OnBootSec=90s` |
|
|
| `monky-deployd.service` | `Type=oneshot`, `monky-deployd run --once` as user `monky-deployd` (+ `docker` group); `SuccessExitStatus=75`; hardened (`NoNewPrivileges`, `ProtectSystem=strict`, `ReadWritePaths=/var/lib/monky-deployd /etc/monky-deployd /run/docker.sock`, `UMask=0077`, no capabilities) |
|
|
| `monky-deployd-proxy.service` | only with `transport: proxy`: `ziti tunnel proxy -i <identity> monky.tenancy.deploy:18443 openbao:18200` as user `ziti`; `EnvironmentFile=/etc/monky-deployd/proxy.env` |
|
|
| `ziti-edge-tunnel.service` | the host identity's tunneler in **`run-host`** mode (drop-in `run-host.conf` written by `install.sh`) |
|
|
|
|
```sh
|
|
systemctl status monky-deployd.timer monky-deployd.service
|
|
systemctl list-timers monky-deployd.timer
|
|
systemctl start monky-deployd.service # tick now (blocks until done)
|
|
journalctl -u monky-deployd -f # priorities are real (INFO/WARNING/ERROR)
|
|
monky-deployd status # exit 0 = token present and in sync
|
|
monky-deployd status --json | jq .
|
|
```
|
|
|
|
Files: `/etc/monky-deployd/config.yaml` (0640 root:monky-deployd), `/etc/monky-deployd/openbao-ca.pem`,
|
|
`/etc/monky-deployd/bootstrap.jwt` (only until the first login), `/var/lib/monky-deployd/{bao.token,state.json,lock}`,
|
|
`/var/lib/monky-deployd/<env>/releases/<sha>/` + `current` (the compose project dir, `.env` 0600),
|
|
`/opt/openziti/etc/identities/monky-host.<env>.json` (ziti:ziti 0640; the agent reads it as a member of
|
|
the `ziti` **group**, plus an explicit ACL `u:monky-deployd:r` and a default ACL on the directory).
|
|
|
|
> The group is the grant that matters. `ziti-edge-tunnel` rewrites the identity file whenever the
|
|
> controller sends a config update, and the rewrite drops POSIX ACLs — an ACL-only box works until the
|
|
> first refresh and then fails every tick (env-dev-08, 2026-09-09). On a host where the tunneller was
|
|
> installed independently of the kit, check `id monky-deployd` for `ziti` before anything else.
|
|
|
|
## Reading the journal
|
|
|
|
| line | meaning |
|
|
|---|---|
|
|
| `checkin: action=none desired=… applied=…` then `healthy; heartbeat reported` | converged |
|
|
| `checkin: action=apply …` → `read 3 secret(s): GEMINI_API_KEY, …` → `promoted release …` → `applied …` | a deploy |
|
|
| `refused: ENV_INCOMPLETE: unresolved: X` | the bundle needs a variable no manifest entry supplies — fix the descriptor / set the secret in the console; nothing was started |
|
|
| `refused: DISK_INSUFFICIENT: image storage (…) has N MiB free, …` | free space: grow the disk or prune. The paths in the message are every filesystem a pull can fill — docker's data-root AND containerd's root, which docker 29 fills with image layers and which is often a different disk (env-dev-08). A bundle that declares no size must still clear `disk.headroom_bytes` |
|
|
| `refused: PRIVILEGED_REFUSED` / `ROLLBACK_REFUSED` | the bundle needs `allow_privileged` / `allow_rollback` in its `agent` profile |
|
|
| `temporary network failure` (exit 75) | mesh/tenancy unreachable — check `ziti-edge-tunnel`, the identity's terminators, `monky.tenancy.deploy` health |
|
|
| `transport sdk: identity … is not readable by this user` | the agent lost its read on the identity file (usually a tunneller rewrite dropping the ACL) — `usermod -a -G ziti monky-deployd` |
|
|
| `AGENT_UNAUTHENTICATED: bearer refused` (exit 1) | the grant was superseded (kit re-revealed / retire) or the token revoked → re-run the install kit |
|
|
| `AGENT_ENV_MISMATCH` (exit 78) | the token belongs to another env than `config.yaml` — fix the config or re-issue the identity; the timer keeps firing but every tick exits 78 immediately (no storm) |
|
|
| `failed: docker compose pull failed (rc=1)` | registry/pull problem; compose output is in the report's tail and in the journal |
|
|
|
|
Values never appear in the journal (`[REDACTED]` for token shapes and every value the agent has read).
|
|
|
|
## Secrets on the box
|
|
|
|
The only credential is `/var/lib/monky-deployd/bao.token` (0600, user `monky-deployd`). `.env`
|
|
files under `releases/<sha>/` hold the rendered values (0600, same user, `docker compose` reads
|
|
them). Rotating a secret in the console changes the bundle sha → the next tick re-reads and
|
|
re-applies. To force a re-read now: `systemctl start monky-deployd.service`.
|
|
|
|
Lost or revoked token: `monky-deployd status` shows `token ABSENT`; reveal the kit again (admin,
|
|
`GET /v1/backends/{id}/agent/install`), paste its bootstrap grant to `/etc/monky-deployd/bootstrap.jwt`
|
|
(0600 monky-deployd) or re-run `install.sh` (enrolment is skipped when the identity exists), then
|
|
`monky-deployd bootstrap` or wait a tick.
|
|
|
|
## Retire and volumes policy
|
|
|
|
Retire is driven by tenancy (`POST /v1/backends/{id}/retire {confirm, force, purge_volumes}`):
|
|
the next check-in returns `action: down` and the agent runs `docker compose down --remove-orphans`.
|
|
Data volumes:
|
|
|
|
- `-v` (remove volumes) only when tenancy sent `purge_volumes: true` **or** the box's
|
|
`volumes_on_absent: purge` — and **never on a prod env** (the request is logged and ignored).
|
|
- default `volumes_on_absent: keep`: volumes stay for a manual `docker volume rm` later.
|
|
|
|
After `down` the agent reports `down`, clears `applied_sha`, removes `current` and keeps
|
|
`releases/` (no secrets outside `.env`, which you can `shred`). Tenancy then destroys the Bao
|
|
paths, revokes the token accessor and deletes the host identity → the following ticks fail with
|
|
`AGENT_UNAUTHENTICATED`; `systemctl disable --now monky-deployd.timer` and `apt remove monky-deployd`
|
|
(`apt purge` also removes `/var/lib/monky-deployd` and `/etc/monky-deployd`).
|
|
|
|
## Laptop mode
|
|
|
|
`laptop_mode: true` (`install.sh --laptop`): being offline is normal — a tick that cannot reach the
|
|
mesh logs one INFO line and exits 0. Run without the system timer:
|
|
|
|
```sh
|
|
monky-deployd run # loop, interval_s + jitter, SIGTERM stops; or a user timer with `run --once`
|
|
```
|
|
|
|
Liveness grace for laptops is 24 h on the tenancy side (P0); `offline` only hides the backend from
|
|
the picker, nothing is retired automatically.
|
|
|
|
## Housekeeping
|
|
|
|
`monky-deployd run --once --prune` removes every `releases/<sha>` except `current` and runs
|
|
`docker image prune -f` after a successful apply. The state dir and the token survive `apt remove`;
|
|
`apt purge` deletes them.
|
|
|
|
## Where the installer downloads from
|
|
|
|
The primary source is the **scm.tikali.ai generic package registry** of this project:
|
|
`https://scm.tikali.ai/api/v4/projects/69/packages/generic/monky-deployd/<ver>/<file>` for
|
|
`monky-deployd_<ver>_amd64.deb`, `.sha256` and `install.sh`. The project is **private** (its parent
|
|
groups are private, so it cannot be made public): every fetch, the script itself included, sends the
|
|
`DEPLOY-TOKEN` header with a read-only GitLab **deploy token** — scope `read_package_registry` only,
|
|
nothing else (no repository, no API, no write); revocable at any time in the project's *Settings →
|
|
Repository → Deploy tokens*. It lives in OpenBao at `monky/monky-tenancy/deployd-download` (key
|
|
`token`); the monky-tenancy install kit carries it and passes `--token`, the ansible role sends it
|
|
from the vaulted `monky_deployd_download_token`. The one-liner:
|
|
|
|
```sh
|
|
curl -sSf -H "DEPLOY-TOKEN: $T" https://scm.tikali.ai/api/v4/projects/69/packages/generic/monky-deployd/<ver>/install.sh \
|
|
| sudo bash -s -- --env <id> --site <site> --token "$T" --bootstrap-file bootstrap.jwt
|
|
```
|
|
|
|
`install.sh` never prints the token (it goes through a 0600 curl `-K` file that is deleted after the
|
|
download; xtrace is switched off). A 401 on the download means the token is missing, revoked or
|
|
lacks the scope. Inside the estate `gitea.cbs.tikali.net` is split-horizon to jump1's RED EIP
|
|
(`10.10.0.175`), which has no HTTP ingress, so backend boxes cannot fetch from the Gitea mirror
|
|
(cbs/iac#102). Off-estate, `install.sh --source gitea` (ansible: `monky_deployd_base_url`/`_deb_url`,
|
|
no token) uses the Gitea release instead. The tag pipeline always publishes to GitLab (`release`); `release:gitea` is automatic only with
|
|
`GITEA_TOKEN` in CI and a manual job otherwise — it has not run on a tag pipeline yet.
|
|
|
|
## Publishing a release to Gitea by hand
|
|
|
|
When `release:gitea` is manual (no `GITEA_TOKEN` in CI) — from jump1 with the Gitea API token:
|
|
|
|
```sh
|
|
T=$(tr -d '\n' < ~/.gitlab_tokens/gitea-token); G=http://172.16.8.1:3000/api/v1/repos/mdella/monky-deployd
|
|
curl -s -X POST -H "Authorization: token $T" $G/mirror-sync # pull the tag
|
|
RID=$(curl -s -X POST -H "Authorization: token $T" -H 'Content-Type: application/json' $G/releases \
|
|
-d '{"tag_name":"v0.1.0","name":"monky-deployd v0.1.0","body":"See CHANGELOG.md"}' | jq -r .id)
|
|
for f in monky-deployd_0.1.0_amd64.deb monky-deployd_0.1.0_amd64.deb.sha256 install.sh; do
|
|
curl -s -X POST -H "Authorization: token $T" -F "attachment=@$f" "$G/releases/$RID/assets?name=$f"; done
|
|
```
|
|
|
|
The assets are then at `https://gitea.cbs.tikali.net/mdella/monky-deployd/releases/download/v0.1.0/<file>`,
|
|
which is what `install.sh --source gitea` fetches (GitLab artifacts from the tag pipeline's `package` job).
|
|
|
|
## Ansible (osg1-07)
|
|
|
|
`ansible/roles/monky_deployd/` is the role skeleton to copy into osg1-07: `.deb` from the
|
|
scm.tikali.ai package registry (sha256-verified), config template, bootstrap grant from a vault var, ACL on the identity,
|
|
timer, one tick. Rolled to env-dev-06..09 after the env-qa-02 pilot; env-dev-01 last.
|