21 Commits

Author SHA1 Message Date
mdella 17a64843f1 Merge branch 'fix/onboarding-identity-and-disk' into 'main'
fix(onboarding): identity read that survives a rewrite, disk refused before the pull — 0.1.8

See merge request tikali/applications/monky/monky-deployd!13
2026-09-08 18:58:55 -07:00
mdella 43615a6fda fix(onboarding): identity read that survives a rewrite, disk refused before the pull — 0.1.8
Three faults from one agent-managed onboarding (env-dev-08, 2026-09-09), each of which pointed
the diagnosis away from the actual fault.

1. install.sh granted the agent's read on the ziti identity with a POSIX ACL. ziti-edge-tunnel
   rewrites that file on a controller config update and the rewrite drops the ACL: the agent
   applied cleanly at 01:21 and was failing every tick by 01:32. Group membership survives the
   rewrite (the file stays ziti:ziti 0640), so install.sh and the package postinstall now add
   monky-deployd to the `ziti` group, and a default ACL on the identity directory carries the
   grant onto a newly created file. The explicit ACLs stay for the boxes that need them.

2. openziti.load() accepts an unreadable or malformed identity: the C SDK logs "configuration is
   invalid" and returns a context that only fails at dial, as a bare TypeError, which the
   transport reported as a missing intercept or a policy gap. The SDK transport now reads and
   parses the identity itself and names the real fault first.

3. The disk pre-flight ran only when the bundle declared disk_need_bytes, so a bundle without one
   died mid-pull with containerd's "no space left on device" — which reads as a registry fault.
   A bundle that declares no size now has to clear the headroom floor, and the pre-flight measures
   containerd's root as well as the docker data-root: docker 29 keeps image layers in the
   containerd image store, and on env-dev-08 those sat on different filesystems (93 GiB free where
   the agent looked, 2.8 GiB where the pull wrote).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KLB7jieMNRkTsJ2epr4Ds1
2026-09-09 01:48:49 +00:00
mdella cc9dcebd7e Merge branch 'feat/registry-auth' into 'main'
feat: registry credential from the manifest (no hand docker login) + readable pull errors

See merge request tikali/applications/monky/monky-deployd!12
2026-09-08 09:54:48 -07:00
mdella b25c6b3b8b feat: pull private images without a hand docker login; surface the pull error
Design merged first: monky-design-docs !225 (doc 24 §4a). Pairs with monky-tenancy!40, which
copies the estate-wide read-only Harbor robot into each environment's own prefix and marks the
manifest entry `use: registry-auth`.

- That entry is not an env var (it would otherwise land in .env and therefore in every container's
  environment). The agent parses it — JSON, or `username:password` with the new `registry_host` —
  and writes `<state_dir>/docker/config.json` 0600 in a directory it owns, with an explicit
  DOCKER_CONFIG pointing the docker CLI at it. The unit runs as monky-deployd, so a `docker login`
  by a human or by root is invisible to the agent: that is what made env-dev-01 look like it had no
  credential at all after the operator had just logged in.
- `compose pull` failures now carry the registry's own message ("no basic auth credentials",
  "manifest unknown", DNS) into the journal and the report instead of `rc=1`.
- Tests: both credential shapes, the refusals, 0600/0700 modes, idempotent rewrite, and that the
  runner never silently falls back to a human's $HOME.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KLB7jieMNRkTsJ2epr4Ds1
2026-09-08 15:45:22 +00:00
monky-deployd-merger 35949a9246 Merge branch 'fix/reuse-lease-token' into 'main'
fix(agent): reuse a valid lease token; rate-limited lease no longer blocks a deploy; 0.1.6

See merge request tikali/applications/monky/monky-deployd!11
2026-09-07 20:27:03 +00:00
mdella a30014a096 fix(agent): reuse a valid lease token across applies; a rate-limited lease no longer blocks a deploy; 0.1.6
Every apply requested a new lease, so a failing deploy retried by the 60 s
timer burned tenancy's 5-leases-per-hour budget and then failed on
LEASE_RATE_LIMITED forever (env-qa-02 pilot). Now: reuse a lease-derived
token while lookup-self says it is valid; swap the bootstrap token for a
lease once; if tenancy rate-limits the lease while a working token exists,
apply with it and defer the swap.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KLB7jieMNRkTsJ2epr4Ds1
2026-09-07 20:25:38 +00:00
mdella 47c3840c95 Merge branch 'docs/attrs-now-added-by-broker' into 'main'
docs: the broker adds the agent attrs at register since 2026-09-07 — !9's 'not implemented yet' was stale on arrival

See merge request tikali/applications/monky/monky-deployd!10
2026-09-07 12:07:48 -07:00
Claude-Docs-Manager d0153dd71c docs: the broker DOES add the agent attrs at register since 2026-09-07 — !9's "not implemented yet" was stale on arrival
!9 (d34189c6) rewrote PROTOCOL.md §Where and how and README §Security model
to say monky-ziti does not add #monky-deploy-agent / #openbao-client yet and an
operator adds them by hand. That was verified against monky-ziti at the sweep
HEAD b44c50a4 (2026-09-05), not its live HEAD: monky-ziti ac258352 (2026-09-07
06:12Z, "feat(environments): host_extra_attrs — host-template-only attrs for
the agent's dial right", merged a82359dd) and monky-tenancy !29 9b84f9ee
(06:23Z, app/jobs_backends.py sends host_extra_attrs for non-k8s backends)
landed hours before !9 was opened. Both sentences now state what happens:
tenancy sends the attrs at register, the broker puts them on the host identity,
and only pre-!29 identities (env-qa-02's) were hand-added.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AW3QqEpwLV69KHn24Re45Q
2026-09-07 09:01:28 -07:00
mdella 6e8b2bf41d Merge branch 'docs/dd-0523-0525-0527-0528' into 'main'
docs+defaults: 443 everywhere the agent dials; broker attr not added yet; no jti state on the mount; release:gitea has not run (DD-0523/0525/0527/0528)

See merge request tikali/applications/monky/monky-deployd!9
2026-09-07 08:45:28 -07:00
Claude-Docs-Manager d34189c625 docs+defaults: 443 everywhere the agent dials; the attr the broker does not add yet; no jti state on the mount; release:gitea has not run
DD-0523 — !7 (31586c30, 0.1.5) moved install.sh and config.example.yaml to the
443 intercept after env-qa-02 hit "service not available". The ansible role
default (monky_deployd_tenancy_port) and TenancyCfg.port still said 8081, so
an ansible-installed box or a config that omits `port` still dialled the wrong
port; both now default to 443, the proxy-mapping and config tests follow, and
PROTOCOL.md §Where and how states the intercept port separately from the
in-pod 8081 and names openziti state/overlay/configs.json as the authority.

DD-0525 — PROTOCOL.md and README said "the broker adds the attr when the
identity is created at kit reveal". monky-ziti at b44c50a4 has no such code
(app/fabric.py host_identity_attrs carries the env template only) and openziti
docs/services.md says "Nothing carries the attr yet". Both now state the
dependency: an operator adds #monky-deploy-agent/#openbao-client on the
controller until the ADR-0028 addendum lands in monky-ziti.

DD-0527 — "the old kit's grant fails at login (unknown/used jti)". The
jwt-tenancy mount keeps no replay state (openbao terraform/jwt-tenancy.tf
see_env role: signature, aud, bound_claims, exp); a superseded grant logs in
until exp and the refusal is tenancy's 401 on the first bearer call. The
second-reveal paragraph, the grant-flow diagram and README §Security model say
so; FakeBao no longer pops a grant at login (the suite's superseded-token test
already goes through FakeTenancy.superseded_jtis, which is the real model).

DD-0528 — "Both locations keep being published": release:gitea has been a
never-run manual job on every tag pipeline (6999, 7044, 7066); README and
OPERATIONS.md now say when the Gitea mirror is published and that it has not
been yet.

Gates (local, py3.12): ruff format, ruff check, pytest 50 passed,
bash -n packaging/install.sh. `git grep 8081` afterwards hits only the in-pod
listener statements.

Doc-Drift: DD-0523 fixed
Doc-Drift: DD-0525 fixed
Doc-Drift: DD-0527 fixed
Doc-Drift: DD-0528 fixed
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AW3QqEpwLV69KHn24Re45Q
2026-09-07 08:22:02 -07:00
mdella b749a6d3b4 Merge branch 'docs/dd-0524-0526-protocol-shapes' into 'main'
docs(protocol): AppRole divergence is resolved history; lease example + fake follow tenancy's AgentLeaseOut (DD-0524/DD-0526)

See merge request tikali/applications/monky/monky-deployd!8
2026-09-06 23:23:32 -07:00
Claude-Docs-Manager 05724edde2 docs(protocol): the AppRole divergence is resolved history; the lease example and fake follow tenancy's AgentLeaseOut
DD-0524 — PROTOCOL.md §Divergences bullets 1-2 said monky-tenancy `main`
"still implements the AppRole lease and install kit" and that "the kit's
generated config uses tenancy.base_url". Both were false when the page was
published: tenancy !17 (288df791, merged 07:48Z) shipped AgentLeaseOut{env_id,
login_jwt, ttl_s, mount, role, addr}, the ES256 grant signer and the JWKS
sixteen minutes before v0.1.0 was tagged, and !22 (61bd0281) made the kit run
install.sh with flags instead of writing a config. The two bullets are now
dated "Resolved" notes; bullets 3-4 (report `detail`, X-Bundle-Sha) stand.

DD-0526 — the lease example sent `reason` and received a nested `vault{}`;
tenancy's AgentLease is `{env_id}` and AgentLeaseOut carries `addr` at the top
level (no vault object). The example now shows tenancy's shapes (with a note
that the agent still sends `reason` and tenancy ignores unknown fields), the
checkin example gains `auth_mount`/`auth_role` so it is the full AgentVaultOut,
tests/fakes.py emits `addr` the way tenancy does, and tenancy.py's docstring
and lease() read `addr` first (the `vault{}` fallback is kept so an older fake
or tenancy still leases). Verified against monky-tenancy app/schemas_backends.py
at 1fd51454 (AgentLease 296-297, AgentLeaseOut 300-309, AgentVaultOut 278-283).

Gates (local, py3.12): ruff format --check, ruff check, pytest 50 passed,
bash -n packaging/install.sh.

Doc-Drift: DD-0524 fixed
Doc-Drift: DD-0526 fixed
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AW3QqEpwLV69KHn24Re45Q
2026-09-06 23:13:09 -07:00
monky-deployd-merger b39a60af46 Merge branch 'fix/tenancy-intercept-port' into 'main'
fix: tenancy.port is the intercept port (443); clear sdk dial error; 0.1.5

See merge request tikali/applications/monky/monky-deployd!7
2026-09-07 06:12:42 +00:00
mdella fcf2dcb1eb fix(bundle): ${VAR} inside comment lines is not a reference
The renderer's .env.template header literally says "substitutes every
${VAR}", which the refusal check counted as an unresolved variable
(ENV_INCOMPLETE: unresolved: VAR) — env-qa-02's first bundle was refused.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KLB7jieMNRkTsJ2epr4Ds1
2026-09-07 06:09:08 +00:00
mdella 31586c3058 fix: tenancy.port is the intercept port (443); clear sdk dial error; 0.1.5
install.sh wrote tenancy.port 8081 (the in-pod port) but the
monky.tenancy.deploy intercept is 443, so the SDK found no intercept and the
check-in failed with 'service not available' followed by the SDK's bare
TypeError. Also wrap non-OSError SDK failures in a TransportError that names
the host:port and what to check.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KLB7jieMNRkTsJ2epr4Ds1
2026-09-07 06:05:21 +00:00
monky-deployd-merger 41541629ec Merge branch 'fix/tunnel-execstartpre' into 'main'
fix(install.sh): clear ExecStartPre in the run-host drop-in; config/grant before aborting; 0.1.4

See merge request tikali/applications/monky/monky-deployd!6
2026-09-07 06:00:16 +00:00
mdella 744041ca1e fix(install.sh): clear ExecStartPre in the run-host drop-in; don't abort before config/grant land; 0.1.4
On env-qa-02 (ziti-edge-tunnel 1.18.7) the package's ExecStartPre script is
not executable by user ziti (203/EXEC), so the unit never started, and under
set -e `systemctl enable --now` aborted install.sh before config.yaml, ACLs,
the bootstrap grant and the timer were written — a second kit was needed.

- run-host drop-in clears ExecStartPre= (run-host does not auto-enrol JWTs)
- a tunneler start failure is a WARNING until everything else is installed,
  then a hard error at the end (re-run needs no new kit)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KLB7jieMNRkTsJ2epr4Ds1
2026-09-07 05:58:57 +00:00
monky-deployd-merger b08e0adf82 Merge branch 'fix/install-version-clobber-docker' into 'main'
fix(install.sh): --version clobbered by os-release; Docker Engine install; 0.1.3

See merge request tikali/applications/monky/monky-deployd!5
2026-09-06 16:20:12 +00:00
mdella 837a7e5bb1 fix(install.sh): --version clobbered by /etc/os-release; install Docker Engine when absent; 0.1.3
The first real kit run (env-qa-02) downloaded
monky-deployd_26.04 LTS (Resolute Raccoon)_amd64.deb: sourcing /etc/os-release
inline overwrote VERSION. Script vars are now DEPLOYD_VERSION and os-release is
read in a subshell.

Per the operator's rule (end users run ONE script), install.sh now installs
Docker Engine when absent — Docker's apt suite for the host codename, falling
back to Ubuntu's docker.io — with --docker-data-root <dir> to place the
data-root before first start. Existing Docker is left untouched.

bash -n + shellcheck -S warning clean.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KLB7jieMNRkTsJ2epr4Ds1
2026-09-06 16:18:47 +00:00
monky-deployd-merger f8a35947db Merge branch 'feat/deploy-token-download' into 'main'
feat(install): read-only deploy token for the private package registry (v0.1.2)

See merge request tikali/applications/monky/monky-deployd!4
2026-09-05 19:10:24 +00:00
mdella 6336012b74 feat(install): read-only deploy token for the private package registry (v0.1.2)
The GitLab project is private (its parent groups are private, so it cannot be
made public): the v0.1.1 one-liner answered 401 anonymously. install.sh gains
--token / MONKY_DEPLOYD_TOKEN and sends `DEPLOY-TOKEN: <token>` (a GitLab deploy
token, scope read_package_registry only, revocable) on every registry download,
the script itself included; the token goes through a 0600 curl -K file (never the
command line, the log or an xtrace). The grant is taken via --bootstrap-file when
the script is piped (stdin IS the script). Ansible: monky_deployd_download_token
(vaulted) -> DEPLOY-TOKEN header, no_log. Docs explain why, the token's scope and
the --source gitea alternative (split-horizon Gitea, cbs/iac#102).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KLB7jieMNRkTsJ2epr4Ds1
2026-09-05 19:03:25 +00:00
30 changed files with 851 additions and 131 deletions
+3 -2
View File
@@ -10,8 +10,9 @@
# GITEA_TOKEN — Gitea API token (write:repository) for `release:gitea`; without it the job is manual # GITEA_TOKEN — Gitea API token (write:repository) for `release:gitea`; without it the job is manual
# #
# Release assets are published to BOTH the GitLab generic package registry (the installer's primary # Release assets are published to BOTH the GitLab generic package registry (the installer's primary
# source — the project is public; gitea.cbs.tikali.net is split-horizon inside the estate, cbs/iac#102) # source — the project is PRIVATE (parent groups are private), so install.sh sends a read-only deploy
# and the Gitea mirror release (`install.sh --source gitea`, off-estate). # token (read_package_registry) as DEPLOY-TOKEN; gitea.cbs.tikali.net is split-horizon inside the
# estate, cbs/iac#102) and the Gitea mirror release (`install.sh --source gitea`, off-estate).
# GITLAB_TRANSLATE_TOKEN — used by the translate-docs component (group level) # GITLAB_TRANSLATE_TOKEN — used by the translate-docs component (group level)
include: include:
+93
View File
@@ -1,6 +1,99 @@
<!-- xlate:verbatim-fences --> <!-- xlate:verbatim-fences -->
# Changelog # Changelog
## 0.1.8 — onboarding: keep the identity readable, refuse a full disk (2026-09-09)
Three faults from one onboarding (env-dev-08, agent-managed, 2026-09-09), each of which sent the
diagnosis somewhere other than the fault.
- **The agent's read on the ziti identity now survives a tunneller rewrite.** `install.sh` granted it
with a POSIX ACL. `ziti-edge-tunnel` rewrites the identity file whenever the controller sends a config
update, and the rewrite drops the ACL — the agent had applied cleanly and was failing every tick six
minutes later. The durable grant is **group membership**: `install.sh` and the package's postinstall
now add `monky-deployd` to the identity file's group (`ziti`) when it exists, and a **default ACL** on
the identity directory carries the grant onto a freshly created file. The explicit ACLs stay.
- **An unusable identity says so.** `openziti.load()` accepts an unreadable or malformed identity, logs
"configuration is invalid" at the C layer and hands back a context that fails LATER, at dial, as a
bare `TypeError` — which the transport reported as a missing intercept or a policy gap. The SDK
transport now reads and parses the identity itself first, and names the real fault (absent, not
readable by this uid, or not JSON) before anything touches the mesh.
- **A full disk is refused before the pull, and the right disk is measured.** The pre-flight ran only
when the bundle declared `disk_need_bytes`; a bundle without one sailed past it and died mid-pull with
containerd's `no space left on device`, which reads as a registry fault. A bundle that declares no size
now has to clear the headroom floor. The pre-flight also measures **containerd's root as well as the
docker data-root**: docker 29 keeps image layers in the containerd image store, and on env-dev-08 those
were on different filesystems — 93 GiB free where the agent looked, 2.8 GiB where the pull wrote.
## 0.1.7 — registry credential + a pull error you can read (2026-09-08)
- **No hand `docker login` on a new box** (monky-design-docs !225, doc 24 §4a). A manifest entry marked
`use: registry-auth` is the Harbor pull credential, not an env var: it is written to
`<state_dir>/docker/config.json` (0600, in a directory the agent owns) and the docker CLI is pointed at it
with an explicit **`DOCKER_CONFIG`**. That last part is the trap — the unit runs as `monky-deployd`, so a
`docker login` by a human or by root is invisible to the agent and looks exactly like no credential at all
(env-dev-01, 2026-09-08). Both seeded shapes are accepted: a JSON object, or `username:password` with the
registry from the new `registry_host` config.
- **`compose pull` failures carry the registry's own message** instead of `rc=1`. "no basic auth
credentials", "manifest unknown" and DNS failures now reach the journal and the report to tenancy.
## 0.1.6 — 2026-09-07
- **A valid lease token is reused across applies.** Every apply requested a new lease, so a failing deploy
retried by the 60 s timer burned tenancy's 5-leases-per-hour budget and then failed on
`LEASE_RATE_LIMITED` forever (env-qa-02 pilot). Now: reuse a lease-derived token while `lookup-self`
says it is valid; swap the bootstrap token for a lease once; and if tenancy rate-limits the lease while a
working token exists, apply with it and defer the swap instead of failing the deploy.
## 0.1.5 — 2026-09-07
- **`tenancy.port` defaults to 443 everywhere** — the ansible role default and the `TenancyCfg` default now
match `install.sh`/`config.example.yaml` (0.1.5 only changed those two); PROTOCOL.md states the intercept
port separately from the in-pod 8081, that the broker does not add `#monky-deploy-agent` yet, that the jwt
mount keeps no `jti` state, and that `release:gitea` has not run (doc-drift DD-0523/0525/0527/0528).
- **`${VAR}` inside comment lines is not a reference.** The renderer's `.env.template` header literally says
"substitutes every ${VAR}", which the refusal check counted as an unresolved variable
(`ENV_INCOMPLETE: unresolved: VAR`) — the first bundle on env-qa-02 was refused for it.
- **config: `tenancy.port` is the service's intercept port (443), not the in-pod 8081.** With 8081 the SDK
found no intercept and the dial failed (`service not available`, then a bare `TypeError` from the SDK's
fallback). `install.sh` now writes 443; `config.example.yaml` updated.
- **transport sdk: clear error instead of a TypeError** when an address has no intercept or the identity
has no dial policy for the service (`TransportError` names the host:port and what to check).
## 0.1.4 — 2026-09-07
- **install.sh: `ziti-edge-tunnel.service` failed to start after enrolment** (`203/EXEC`: the package's
`ExecStartPre` script is not executable by user `ziti` on ziti-edge-tunnel 1.18.x) and, because the
unit was enabled with `--now` under `set -e`, the script aborted BEFORE writing config/ACLs/grant/timer —
a second kit was needed. The run-host drop-in now clears `ExecStartPre=`; a tunneler start failure is a
WARNING until everything else is in place, then a hard error at the end.
## 0.1.3 — 2026-09-06
- **install.sh: `--version` was clobbered by `/etc/os-release`** (it defines `VERSION`), so the kit tried to download
`monky-deployd_26.04 LTS (Resolute Raccoon)_amd64.deb` (env-qa-02 pilot, first real kit run). Script variables are now
`DEPLOYD_VERSION`; os-release is read in a subshell.
- **install.sh installs Docker Engine when absent** (Docker's apt suite for the codename → fallback `docker.io`), with
`--docker-data-root <dir>` to place the data-root before first start. One script for the end user, per the operator's rule.
## v0.1.2 — 2026-09-05
- **The GitLab project is private** (its parent groups are private, so it cannot be made public;
found when the v0.1.1 one-liner returned 401 anonymously). `install.sh` gains `--token <deploy-token>`
/ `MONKY_DEPLOYD_TOKEN`: every download from the generic package registry — the script itself
included — sends `DEPLOY-TOKEN: <token>`, a read-only GitLab **deploy token** (scope
`read_package_registry` only, revocable), seeded in OpenBao at `monky/monky-tenancy/deployd-download`
(key `token`) and handed to the box by the monky-tenancy install kit. The token never reaches the
command line, the log or an xtrace (curl `-K` config file, 0600, deleted after the download;
`set +x` forced). The one-liner now fetches `install.sh` from the registry
(`…/monky-deployd/<ver>/install.sh`) instead of `-/raw/main`, and takes the bootstrap grant via
`--bootstrap-file` — with `curl … | bash -s --` stdin IS the script, so it is no longer read for
the grant in that mode. `--source gitea` (no token) stays the off-estate alternative.
- Ansible role: `monky_deployd_download_token` (vaulted) → `DEPLOY-TOKEN` header on both fetches,
`no_log: true`.
- README / OPERATIONS: why (private project; split-horizon Gitea, cbs/iac#102), token scope, revocation.
No agent behaviour change.
## v0.1.1 — 2026-09-05 ## v0.1.1 — 2026-09-05
- `install.sh` / ansible role / README: the **primary download is the public GitLab project's generic - `install.sh` / ansible role / README: the **primary download is the public GitLab project's generic
+3 -2
View File
@@ -75,8 +75,9 @@ systemd-analyze verify packaging/systemd/*.service # when available
Bump `monky_deployd.__version__` + `pyproject.toml` + `packaging/install.sh` `DEFAULT_VERSION` + Bump `monky_deployd.__version__` + `pyproject.toml` + `packaging/install.sh` `DEFAULT_VERSION` +
`CHANGELOG.md`, merge to `main`, tag `vX.Y.Z` (protected `v*`). The tag pipeline builds the `.deb`, `CHANGELOG.md`, merge to `main`, tag `vX.Y.Z` (protected `v*`). The tag pipeline builds the `.deb`,
publishes the **GitLab generic package registry + release** the installer downloads from (public publishes the **GitLab generic package registry + release** the installer downloads from (PRIVATE
project; the Gitea name is split-horizon inside the estate, cbs/iac#102) and — with `GITEA_TOKEN` — project the installer sends a read-only deploy token as `DEPLOY-TOKEN`; the Gitea name is
split-horizon inside the estate, cbs/iac#102) and — with `GITEA_TOKEN` —
the **Gitea release** (`install.sh --source gitea`, off-estate; `docs/OPERATIONS.md` has the manual recipe). `lint` refuses a tag whose version the **Gitea release** (`install.sh --source gitea`, off-estate; `docs/OPERATIONS.md` has the manual recipe). `lint` refuses a tag whose version
differs from `__version__`. differs from `__version__`.
+38 -15
View File
@@ -18,12 +18,14 @@ Verified on **Ubuntu 26.04**.
## Install (one-liner, from the enrolment kit) ## Install (one-liner, from the enrolment kit)
An admin reveals the kit once in the console (`GET /v1/backends/{id}/agent/install`); it hands An admin reveals the kit once in the console (`GET /v1/backends/{id}/agent/install`); it hands
you the enrolment JWT and a one-time **bootstrap deploy grant**. On the box: you the enrolment JWT, a one-time **bootstrap deploy grant** and the read-only **download token**
(the kit runs the line below for you). By hand, on the box:
```sh ```sh
curl -fsSL https://scm.tikali.ai/tikali/applications/monky/monky-deployd/-/raw/main/packaging/install.sh \ T=<deploy token> # read-only GitLab deploy token (read_package_registry); the kit carries it
| sudo bash -s -- --env env-qa-02 --site cbs --enrol-jwt ./monky-host.env-qa-02.jwt < bootstrap.jwt curl -sSf -H "DEPLOY-TOKEN: $T" https://scm.tikali.ai/api/v4/projects/69/packages/generic/monky-deployd/0.1.6/install.sh \
# [--transport sdk|proxy|system] [--version 0.1.1] [--laptop] [--bao-ca openbao-ca.pem] [--source gitlab|gitea] | sudo bash -s -- --env env-qa-02 --site cbs --token "$T" --bootstrap-file bootstrap.jwt --enrol-jwt ./monky-host.env-qa-02.jwt
# [--transport sdk|proxy|system] [--version 0.1.6] [--docker-data-root /home/docker-data] [--laptop] [--bao-ca openbao-ca.pem] [--source gitlab|gitea]
``` ```
`install.sh` installs `ziti-edge-tunnel` (OpenZiti `jammy` suite) and `docker-compose-plugin` if `install.sh` installs `ziti-edge-tunnel` (OpenZiti `jammy` suite) and `docker-compose-plugin` if
@@ -32,11 +34,21 @@ absent, downloads the pinned `.deb` + `.sha256` from the [scm.tikali.ai generic
enrols `monky-host.<env>` if the identity is missing, switches the tunneler to `run-host`, enrols `monky-host.<env>` if the identity is missing, switches the tunneler to `run-host`,
writes `/etc/monky-deployd/config.yaml`, grants the agent read access to the identity (ACL), writes `/etc/monky-deployd/config.yaml`, grants the agent read access to the identity (ACL),
stages the bootstrap grant (0600), enables `monky-deployd.timer`, runs one tick and deletes the stages the bootstrap grant (0600), enables `monky-deployd.timer`, runs one tick and deletes the
JWT. The GitLab project is **public**, so anonymous downloads work from scm.tikali.ai. Why not the JWT.
Gitea mirror: inside the estate `gitea.cbs.tikali.net` is split-horizon to jump1's RED EIP
(`10.10.0.175`), which has no HTTP ingress, so backend boxes cannot reach it (cbs/iac#102); **Why a token.** The GitLab project is **private** and cannot be made public (its parent groups
`--source gitea` (or `MONKY_DEPLOYD_SOURCE=gitea`) keeps the are private), so anonymous downloads answer 401. Every registry fetch — the script itself
[Gitea release](https://gitea.cbs.tikali.net/mdella/monky-deployd/releases) as the off-estate alternative. included — therefore sends `DEPLOY-TOKEN: <token>`: a GitLab **deploy token** with the single
scope `read_package_registry` (it can download packages and nothing else: no code, no API, no
write; revoke it in the project's *Settings → Repository → Deploy tokens* and issue a new one).
The operator seeds it in OpenBao at `monky/monky-tenancy/deployd-download` (key `token`); the
monky-tenancy install kit reads it from there and passes `--token` (`MONKY_DEPLOYD_TOKEN` also
works). The script never prints it (curl `-K` config file, 0600, deleted after the download;
`set -x` is switched off). Why not the Gitea mirror: inside the estate `gitea.cbs.tikali.net` is
split-horizon to jump1's RED EIP (`10.10.0.175`), which has no HTTP ingress, so backend boxes
cannot reach it (cbs/iac#102); `--source gitea` (or `MONKY_DEPLOYD_SOURCE=gitea`, no token) keeps
the [Gitea release](https://gitea.cbs.tikali.net/mdella/monky-deployd/releases) as the
off-estate alternative.
## Transports ## Transports
@@ -86,11 +98,13 @@ monky-deployd version
## Security model ## Security model
- **Identity = the box's ziti host identity.** Only identities with `#monky-deploy-agent` can dial - **Identity = the box's ziti host identity.** Only identities with `#monky-deploy-agent` can dial
tenancy's agent entrypoint; `#openbao-client` reaches OpenBao. The agent reads the identity tenancy's agent entrypoint; `#openbao-client` reaches OpenBao — tenancy sends both to the broker
as `host_extra_attrs` at register (monky-tenancy !29 / monky-ziti `ac258352`, 2026-09-07), so a
backend registered since then dials without hand-added attrs (see PROTOCOL.md §Where and how). The agent reads the identity
through an ACL (`setfacl -m u:monky-deployd:r`), never owns it. through an ACL (`setfacl -m u:monky-deployd:r`), never owns it.
- **The bearer to tenancy is the agent's own OpenBao token**, minted by OpenBao from a - **The bearer to tenancy is the agent's own OpenBao token**, minted by OpenBao from a
tenancy-signed ES256 deploy grant (`aud openbao-see-env`, `kind deploy-grant`, 1 h, single-use tenancy-signed ES256 deploy grant (`aud openbao-see-env`, `kind deploy-grant`, 1 h; its `jti`
`jti`). Tenancy verifies it with `auth/token/lookup`, pins `meta.env_id`, and refuses a token becomes the token's `meta.grant_jti` — the mount keeps no replay state, tenancy does). Tenancy verifies it with `auth/token/lookup`, pins `meta.env_id`, and refuses a token
whose `meta.grant_jti` was superseded (kit re-reveal, retire) → `401 AGENT_UNAUTHENTICATED`. whose `meta.grant_jti` was superseded (kit re-reveal, retire) → `401 AGENT_UNAUTHENTICATED`.
**No AppRole, nothing to unwrap** (Gate 1 result, 2026-09-05). **No AppRole, nothing to unwrap** (Gate 1 result, 2026-09-05).
- **The agent never receives a secret from tenancy.** Bundles carry placeholders; the agent reads - **The agent never receives a secret from tenancy.** Bundles carry placeholders; the agent reads
@@ -145,13 +159,22 @@ systemd-analyze verify packaging/systemd/*.service # where systemd is availabl
egress to github.com and pypi.org) and `package` builds the `.deb` with `nfpm` (binary from GitHub egress to github.com and pypi.org) and `package` builds the `.deb` with `nfpm` (binary from GitHub
releases, goreleaser apt repo as fallback). Both were proven on the v0.1.0 tag pipeline and are releases, goreleaser apt repo as fallback). Both were proven on the v0.1.0 tag pipeline and are
blocking on `main`/tags (manual on MRs); the `.deb` always ships the SDK wheel. On a `v*` tag blocking on `main`/tags (manual on MRs); the `.deb` always ships the SDK wheel. On a `v*` tag
`release` uploads to the GitLab generic package registry + release (**the primary download**, public `release` uploads to the GitLab generic package registry + release (**the primary download**; the
project), and `release:gitea` publishes the same assets on the Gitea mirror (the `--source gitea` project is private, so the installer sends the read-only deploy token), and `release:gitea` publishes the same assets on the Gitea mirror (the `--source gitea`
alternative; automatic when `GITEA_TOKEN` is set, manual otherwise — see `docs/OPERATIONS.md` for alternative; automatic when `GITEA_TOKEN` is set, manual otherwise — see `docs/OPERATIONS.md` for
the by-hand recipe). Both locations keep being published. the by-hand recipe). The GitLab registry is published by every tag pipeline; the Gitea mirror only
when `release:gitea` runs — automatically once `GITEA_TOKEN` is set in CI, by hand otherwise — so
check the Gitea release page before pointing an installer at it.
## See also ## See also
- `docs/PROTOCOL.md`, `docs/OPERATIONS.md`, `CHANGELOG.md` - `docs/PROTOCOL.md`, `docs/OPERATIONS.md`, `CHANGELOG.md`
- monky-tenancy `docs/usage.md` (agent protocol), `app/api/agent.py`, `app/schemas_backends.py` - monky-tenancy `docs/usage.md` (agent protocol), `app/api/agent.py`, `app/schemas_backends.py`
- monky-deploy (the renderer whose `render_files` produces the bundle) - monky-deploy (the renderer whose `render_files` produces the bundle)
### Docker Engine (since 0.1.3)
`install.sh` installs Docker Engine when it is absent (Docker's apt suite for the host's Ubuntu codename, falling back to
Ubuntu's `docker.io`), so an end user runs exactly one script. `--docker-data-root <dir>` writes `/etc/docker/daemon.json`
before the daemon first starts (pilot VMs keep the data-root on a dedicated disk, e.g. `/home/docker-data`). A pre-existing
Docker is left untouched. macOS/Windows laptops still need Docker Desktop from the user.
+6 -2
View File
@@ -4,7 +4,9 @@ Installs and configures [monky-deployd](https://scm.tikali.ai/tikali/application
(the Monky backend pull agent, MONKY-ADR-0028) on a docker host that already carries an enrolled (the Monky backend pull agent, MONKY-ADR-0028) on a docker host that already carries an enrolled
host identity (`roles/ziti_tunneler`, run-host mode). Skeleton for **osg1-07**; copy it there. host identity (`roles/ziti_tunneler`, run-host mode). Skeleton for **osg1-07**; copy it there.
What it does: pin + download the `.deb` from the scm.tikali.ai package registry (sha256 verified; `monky_deployd_base_url`/`_deb_url` switch to the Gitea release off-estate) → ACL What it does: pin + download the `.deb` from the scm.tikali.ai package registry (sha256 verified; the
project is **private**, so the fetches send the read-only deploy token `monky_deployd_download_token`
as `DEPLOY-TOKEN`, `no_log`; `monky_deployd_base_url`/`_deb_url` switch to the Gitea release off-estate) → ACL
`u:monky-deployd:r` on the identity (`rx` on the dir) → `/etc/monky-deployd/config.yaml` from the `u:monky-deployd:r` on the identity (`rx` on the dir) → `/etc/monky-deployd/config.yaml` from the
template → optional openbao-ca PEM → optional `monky-deployd-proxy.service` (`transport: proxy`) template → optional openbao-ca PEM → optional `monky-deployd-proxy.service` (`transport: proxy`)
→ the **one-time bootstrap deploy grant** from a vault var (0600, `no_log`) → `monky-deployd.timer` → the **one-time bootstrap deploy grant** from a vault var (0600, `no_log`) → `monky-deployd.timer`
@@ -14,7 +16,8 @@ template → optional openbao-ca PEM → optional `monky-deployd-proxy.service`
| var | note | | var | note |
|---|---| |---|---|
| `monky_deployd_version` | pinned release, e.g. `0.1.1` | | `monky_deployd_version` | pinned release, e.g. `0.1.6` |
| `monky_deployd_download_token` | **vaulted**: GitLab deploy token, scope `read_package_registry` only (revocable) — the registry is private; seeded in OpenBao at `monky/monky-tenancy/deployd-download` key `token` (path/key are the operator's choice). Empty = no header (only works with the Gitea `base_url`) |
| `monky_deployd_env_id` / `_site` / `_transport` | per host (`env-dev-06`, `cbs`, `sdk`) | | `monky_deployd_env_id` / `_site` / `_transport` | per host (`env-dev-06`, `cbs`, `sdk`) |
| `monky_deployd_bootstrap_grant` | tenancy-minted deploy grant (1 h) — `ansible-vault` or a lookup at play time; empty keeps the existing token | | `monky_deployd_bootstrap_grant` | tenancy-minted deploy grant (1 h) — `ansible-vault` or a lookup at play time; empty keeps the existing token |
| `monky_deployd_bao_ca_pem` | the `openbao-ca` certificate (PEM) | | `monky_deployd_bao_ca_pem` | the `openbao-ca` certificate (PEM) |
@@ -30,6 +33,7 @@ template → optional openbao-ca PEM → optional `monky-deployd-proxy.service`
- role: monky_deployd - role: monky_deployd
vars: vars:
monky_deployd_bootstrap_grant: "{{ lookup('pipe', 'tenancy-mint-grant ' ~ inventory_hostname) }}" monky_deployd_bootstrap_grant: "{{ lookup('pipe', 'tenancy-mint-grant ' ~ inventory_hostname) }}"
monky_deployd_download_token: "{{ vault_monky_deployd_download_token }}" # ansible-vault / OpenBao lookup
``` ```
Rollout order (plan §E): pilot env-qa-02 → env-dev-06..09 → env-dev-01 last (after its live key moves into Bao). Rollout order (plan §E): pilot env-qa-02 → env-dev-06..09 → env-dev-01 last (after its live key moves into Bao).
+13 -5
View File
@@ -1,16 +1,24 @@
--- ---
# monky_deployd — install and configure the Monky backend pull agent (MONKY-ADR-0028 §D). # monky_deployd — install and configure the Monky backend pull agent (MONKY-ADR-0028 §D).
# Copy this role into osg1-07 (roles/monky_deployd) and roll to env-dev-06..09 after the pilot. # Copy this role into osg1-07 (roles/monky_deployd) and roll to env-dev-06..09 after the pilot.
monky_deployd_version: "0.1.1" monky_deployd_version: "0.1.6"
monky_deployd_deb: "monky-deployd_{{ monky_deployd_version }}_amd64.deb" monky_deployd_deb: "monky-deployd_{{ monky_deployd_version }}_amd64.deb"
# PRIMARY download = the public GitLab project's generic package registry on scm.tikali.ai. Inside the # PRIMARY download = the GitLab project's generic package registry on scm.tikali.ai. Inside the estate
# estate gitea.cbs.tikali.net is split-horizon to jump1's RED EIP (10.10.0.175, no HTTP ingress), so # gitea.cbs.tikali.net is split-horizon to jump1's RED EIP (10.10.0.175, no HTTP ingress), so backend
# backend boxes cannot fetch from the Gitea mirror (cbs/iac#102). Off-estate alternative (Gitea release): # boxes cannot fetch from the Gitea mirror (cbs/iac#102). The project is PRIVATE (its parent groups are
# private), so the registry needs a read-only GitLab deploy token (scope read_package_registry only,
# revocable) sent as the `DEPLOY-TOKEN` header — `monky_deployd_download_token`, a VAULTED var
# (ansible-vault or an OpenBao lookup: KV `monky/monky-tenancy/deployd-download`, key `token`).
# Off-estate alternative (Gitea release, no token):
# monky_deployd_base_url: "https://gitea.cbs.tikali.net/mdella/monky-deployd" # monky_deployd_base_url: "https://gitea.cbs.tikali.net/mdella/monky-deployd"
# monky_deployd_deb_url: "{{ monky_deployd_base_url }}/releases/download/v{{ monky_deployd_version }}/{{ monky_deployd_deb }}" # monky_deployd_deb_url: "{{ monky_deployd_base_url }}/releases/download/v{{ monky_deployd_version }}/{{ monky_deployd_deb }}"
# monky_deployd_download_token: ""
monky_deployd_base_url: "https://scm.tikali.ai/api/v4/projects/69/packages/generic/monky-deployd" monky_deployd_base_url: "https://scm.tikali.ai/api/v4/projects/69/packages/generic/monky-deployd"
monky_deployd_deb_url: "{{ monky_deployd_base_url }}/{{ monky_deployd_version }}/{{ monky_deployd_deb }}" monky_deployd_deb_url: "{{ monky_deployd_base_url }}/{{ monky_deployd_version }}/{{ monky_deployd_deb }}"
monky_deployd_deb_sha256_url: "{{ monky_deployd_deb_url }}.sha256" monky_deployd_deb_sha256_url: "{{ monky_deployd_deb_url }}.sha256"
monky_deployd_download_token: ""
# the header the download tasks send (empty map when no token)
monky_deployd_download_headers: "{{ {'DEPLOY-TOKEN': monky_deployd_download_token} if monky_deployd_download_token | length > 0 else {} }}"
# per host (inventory / host_vars) # per host (inventory / host_vars)
monky_deployd_env_id: "{{ inventory_hostname }}" # env-dev-06 ... monky_deployd_env_id: "{{ inventory_hostname }}" # env-dev-06 ...
@@ -25,7 +33,7 @@ monky_deployd_disk_factor: 1.5
monky_deployd_disk_headroom_bytes: 2147483648 monky_deployd_disk_headroom_bytes: 2147483648
monky_deployd_tenancy_service: monky.tenancy.deploy monky_deployd_tenancy_service: monky.tenancy.deploy
monky_deployd_tenancy_port: 8081 monky_deployd_tenancy_port: 443 # the intercept port of monky.tenancy.deploy (host.v1 forwards to 8081 in the pod)
monky_deployd_bao_addr: "https://bao.cbs.tikali.net:8200" monky_deployd_bao_addr: "https://bao.cbs.tikali.net:8200"
monky_deployd_bao_auth_mount: jwt-tenancy monky_deployd_bao_auth_mount: jwt-tenancy
monky_deployd_bao_role: see-env monky_deployd_bao_role: see-env
@@ -22,20 +22,26 @@
changed_when: false changed_when: false
failed_when: false failed_when: false
# The registry is PRIVATE: both fetches send `DEPLOY-TOKEN: {{ monky_deployd_download_token }}`
# (read_package_registry only) and run no_log so the header never reaches the play output.
- name: monky_deployd | download .deb + sha256 from the scm.tikali.ai package registry - name: monky_deployd | download .deb + sha256 from the scm.tikali.ai package registry
when: monky_deployd_installed.stdout != monky_deployd_version when: monky_deployd_installed.stdout != monky_deployd_version
block: block:
- name: monky_deployd | fetch sha256 - name: monky_deployd | fetch sha256
ansible.builtin.uri: ansible.builtin.uri:
url: "{{ monky_deployd_deb_sha256_url }}" url: "{{ monky_deployd_deb_sha256_url }}"
headers: "{{ monky_deployd_download_headers }}"
return_content: true return_content: true
register: monky_deployd_sha register: monky_deployd_sha
no_log: true
- name: monky_deployd | fetch .deb (checksum verified) - name: monky_deployd | fetch .deb (checksum verified)
ansible.builtin.get_url: ansible.builtin.get_url:
url: "{{ monky_deployd_deb_url }}" url: "{{ monky_deployd_deb_url }}"
headers: "{{ monky_deployd_download_headers }}"
dest: "/var/cache/apt/archives/{{ monky_deployd_deb }}" dest: "/var/cache/apt/archives/{{ monky_deployd_deb }}"
checksum: "sha256:{{ monky_deployd_sha.content.split()[0] }}" checksum: "sha256:{{ monky_deployd_sha.content.split()[0] }}"
mode: "0644" mode: "0644"
no_log: true
- name: monky_deployd | install .deb - name: monky_deployd | install .deb
ansible.builtin.apt: ansible.builtin.apt:
deb: "/var/cache/apt/archives/{{ monky_deployd_deb }}" deb: "/var/cache/apt/archives/{{ monky_deployd_deb }}"
+6 -2
View File
@@ -1,4 +1,4 @@
# /etc/monky-deployd/config.yaml — monky-deployd v0.1.1 (MONKY-ADR-0028 §D) # /etc/monky-deployd/config.yaml — monky-deployd v0.1.6 (MONKY-ADR-0028 §D)
# Written by packaging/install.sh (or the ansible role monky_deployd). YAML *subset*: maps, scalars, # Written by packaging/install.sh (or the ansible role monky_deployd). YAML *subset*: maps, scalars,
# simple lists, comments. Keys not listed here are a config error. # simple lists, comments. Keys not listed here are a config error.
@@ -10,7 +10,7 @@ identity: /opt/openziti/etc/identities/monky-host.env-qa-02.json # the box's h
tenancy: tenancy:
service: monky.tenancy.deploy # ziti service bound by the tenancy sidecar -> 127.0.0.1:8081 (agent entrypoint) service: monky.tenancy.deploy # ziti service bound by the tenancy sidecar -> 127.0.0.1:8081 (agent entrypoint)
host: monky.tenancy.deploy # intercept host (sdk/system); defaults to `service` host: monky.tenancy.deploy # intercept host (sdk/system); defaults to `service`
port: 8081 port: 443 # intercept port of monky.tenancy.deploy (in-pod 8081)
scheme: http # plain HTTP inside the mesh; the mesh is the transport security scheme: http # plain HTTP inside the mesh; the mesh is the transport security
proxy_addr: 127.0.0.1:18443 # transport: proxy proxy_addr: 127.0.0.1:18443 # transport: proxy
timeout_s: 30 timeout_s: 30
@@ -44,3 +44,7 @@ laptop_mode: false # true: offline exits 0 quietly; run withou
# compose_project: monky-env-qa-02 # docker compose project name # compose_project: monky-env-qa-02 # docker compose project name
# docker_bin: docker # docker_bin: docker
# log_level: INFO # log_level: INFO
# The registry the bundle pulls from. Used only when the seeded pull credential is a bare
# `username:password` (a JSON credential names its own registry). doc 24 §4a.
registry_host: harbor.tikali.net
+30 -9
View File
@@ -22,7 +22,13 @@ monky-deployd status --json | jq .
Files: `/etc/monky-deployd/config.yaml` (0640 root:monky-deployd), `/etc/monky-deployd/openbao-ca.pem`, Files: `/etc/monky-deployd/config.yaml` (0640 root:monky-deployd), `/etc/monky-deployd/openbao-ca.pem`,
`/etc/monky-deployd/bootstrap.jwt` (only until the first login), `/var/lib/monky-deployd/{bao.token,state.json,lock}`, `/etc/monky-deployd/bootstrap.jwt` (only until the first login), `/var/lib/monky-deployd/{bao.token,state.json,lock}`,
`/var/lib/monky-deployd/<env>/releases/<sha>/` + `current` (the compose project dir, `.env` 0600), `/var/lib/monky-deployd/<env>/releases/<sha>/` + `current` (the compose project dir, `.env` 0600),
`/opt/openziti/etc/identities/monky-host.<env>.json` (ziti:ziti 0600 + ACL `u:monky-deployd:r`). `/opt/openziti/etc/identities/monky-host.<env>.json` (ziti:ziti 0640; the agent reads it as a member of
the `ziti` **group**, plus an explicit ACL `u:monky-deployd:r` and a default ACL on the directory).
> The group is the grant that matters. `ziti-edge-tunnel` rewrites the identity file whenever the
> controller sends a config update, and the rewrite drops POSIX ACLs — an ACL-only box works until the
> first refresh and then fails every tick (env-dev-08, 2026-09-09). On a host where the tunneller was
> installed independently of the kit, check `id monky-deployd` for `ziti` before anything else.
## Reading the journal ## Reading the journal
@@ -31,9 +37,10 @@ Files: `/etc/monky-deployd/config.yaml` (0640 root:monky-deployd), `/etc/monky-d
| `checkin: action=none desired=… applied=…` then `healthy; heartbeat reported` | converged | | `checkin: action=none desired=… applied=…` then `healthy; heartbeat reported` | converged |
| `checkin: action=apply …``read 3 secret(s): GEMINI_API_KEY, …``promoted release …``applied …` | a deploy | | `checkin: action=apply …``read 3 secret(s): GEMINI_API_KEY, …``promoted release …``applied …` | a deploy |
| `refused: ENV_INCOMPLETE: unresolved: X` | the bundle needs a variable no manifest entry supplies — fix the descriptor / set the secret in the console; nothing was started | | `refused: ENV_INCOMPLETE: unresolved: X` | the bundle needs a variable no manifest entry supplies — fix the descriptor / set the secret in the console; nothing was started |
| `refused: DISK_INSUFFICIENT: docker data-root has N MiB free, bundle needs M MiB` | free space (the env-dev-09 lesson): grow the data-root disk or prune | | `refused: DISK_INSUFFICIENT: image storage (…) has N MiB free, …` | free space: grow the disk or prune. The paths in the message are every filesystem a pull can fill — docker's data-root AND containerd's root, which docker 29 fills with image layers and which is often a different disk (env-dev-08). A bundle that declares no size must still clear `disk.headroom_bytes` |
| `refused: PRIVILEGED_REFUSED` / `ROLLBACK_REFUSED` | the bundle needs `allow_privileged` / `allow_rollback` in its `agent` profile | | `refused: PRIVILEGED_REFUSED` / `ROLLBACK_REFUSED` | the bundle needs `allow_privileged` / `allow_rollback` in its `agent` profile |
| `temporary network failure` (exit 75) | mesh/tenancy unreachable — check `ziti-edge-tunnel`, the identity's terminators, `monky.tenancy.deploy` health | | `temporary network failure` (exit 75) | mesh/tenancy unreachable — check `ziti-edge-tunnel`, the identity's terminators, `monky.tenancy.deploy` health |
| `transport sdk: identity … is not readable by this user` | the agent lost its read on the identity file (usually a tunneller rewrite dropping the ACL) — `usermod -a -G ziti monky-deployd` |
| `AGENT_UNAUTHENTICATED: bearer refused` (exit 1) | the grant was superseded (kit re-revealed / retire) or the token revoked → re-run the install kit | | `AGENT_UNAUTHENTICATED: bearer refused` (exit 1) | the grant was superseded (kit re-revealed / retire) or the token revoked → re-run the install kit |
| `AGENT_ENV_MISMATCH` (exit 78) | the token belongs to another env than `config.yaml` — fix the config or re-issue the identity; the timer keeps firing but every tick exits 78 immediately (no storm) | | `AGENT_ENV_MISMATCH` (exit 78) | the token belongs to another env than `config.yaml` — fix the config or re-issue the identity; the timer keeps firing but every tick exits 78 immediately (no storm) |
| `failed: docker compose pull failed (rc=1)` | registry/pull problem; compose output is in the report's tail and in the journal | | `failed: docker compose pull failed (rc=1)` | registry/pull problem; compose output is in the report's tail and in the journal |
@@ -88,14 +95,28 @@ the picker, nothing is retired automatically.
## Where the installer downloads from ## Where the installer downloads from
The primary source is the **scm.tikali.ai generic package registry** of this (public) project: The primary source is the **scm.tikali.ai generic package registry** of this project:
`https://scm.tikali.ai/api/v4/projects/69/packages/generic/monky-deployd/<ver>/<file>` for `https://scm.tikali.ai/api/v4/projects/69/packages/generic/monky-deployd/<ver>/<file>` for
`monky-deployd_<ver>_amd64.deb`, `.sha256` and `install.sh`; the one-liner fetches the script from `monky-deployd_<ver>_amd64.deb`, `.sha256` and `install.sh`. The project is **private** (its parent
`https://scm.tikali.ai/tikali/applications/monky/monky-deployd/-/raw/main/packaging/install.sh`. groups are private, so it cannot be made public): every fetch, the script itself included, sends the
Inside the estate `gitea.cbs.tikali.net` is split-horizon to jump1's RED EIP (`10.10.0.175`), which `DEPLOY-TOKEN` header with a read-only GitLab **deploy token** — scope `read_package_registry` only,
has no HTTP ingress, so backend boxes cannot fetch from the Gitea mirror (cbs/iac#102). Off-estate, nothing else (no repository, no API, no write); revocable at any time in the project's *Settings →
`install.sh --source gitea` (ansible: `monky_deployd_base_url`/`_deb_url`) uses the Gitea release Repository → Deploy tokens*. It lives in OpenBao at `monky/monky-tenancy/deployd-download` (key
instead. The tag pipeline publishes to both (`release`, `release:gitea`). `token`); the monky-tenancy install kit carries it and passes `--token`, the ansible role sends it
from the vaulted `monky_deployd_download_token`. The one-liner:
```sh
curl -sSf -H "DEPLOY-TOKEN: $T" https://scm.tikali.ai/api/v4/projects/69/packages/generic/monky-deployd/<ver>/install.sh \
| sudo bash -s -- --env <id> --site <site> --token "$T" --bootstrap-file bootstrap.jwt
```
`install.sh` never prints the token (it goes through a 0600 curl `-K` file that is deleted after the
download; xtrace is switched off). A 401 on the download means the token is missing, revoked or
lacks the scope. Inside the estate `gitea.cbs.tikali.net` is split-horizon to jump1's RED EIP
(`10.10.0.175`), which has no HTTP ingress, so backend boxes cannot fetch from the Gitea mirror
(cbs/iac#102). Off-estate, `install.sh --source gitea` (ansible: `monky_deployd_base_url`/`_deb_url`,
no token) uses the Gitea release instead. The tag pipeline always publishes to GitLab (`release`); `release:gitea` is automatic only with
`GITEA_TOKEN` in CI and a manual job otherwise — it has not run on a tag pipeline yet.
## Publishing a release to Gitea by hand ## Publishing a release to Gitea by hand
+36 -19
View File
@@ -10,8 +10,15 @@ The agent side of the backend lifecycle agent protocol — [MONKY-ADR-0028](http
- **Where:** tenancy's **agent entrypoint** — a second container (`app.agent_main`, - **Where:** tenancy's **agent entrypoint** — a second container (`app.agent_main`,
`127.0.0.1:8081`) bound to the ziti service **`monky.tenancy.deploy`** by a `ziti-edge-tunnel `127.0.0.1:8081`) bound to the ziti service **`monky.tenancy.deploy`** by a `ziti-edge-tunnel
run-host` sidecar. Unreachable from the public ingress. Only host identities carrying run-host` sidecar. Unreachable from the public ingress. **The agent dials the service's
`#monky-deploy-agent` may dial (the broker adds the attr when the identity is created at kit reveal). INTERCEPT port, `tcp/443`** (`monky.tenancy.deploy.intercept.v1` in openziti
`state/overlay/configs.json` — the authority for the port); the sidecar's `host.v1` forwards to
`127.0.0.1:8081` inside the pod. `tenancy.port` in the config, `install.sh` and the ansible role all
default to 443 (0.1.5). Only host identities carrying `#monky-deploy-agent` may dial
(`monky.tenancy.deploy.dial`, openziti). Tenancy sends that attr and `#openbao-client` to the
broker as `host_extra_attrs` when it registers an agent-backed backend (monky-tenancy !29,
monky-ziti `ac258352` — both 2026-09-07), so `monky-host.<env>` carries them from creation;
identities created before that (env-qa-02's) had them added on the controller by hand.
- **Transport:** plain HTTP inside the mesh (`transport: sdk` dials by service name; - **Transport:** plain HTTP inside the mesh (`transport: sdk` dials by service name;
`proxy``127.0.0.1:18443`). OpenBao is the existing `openbao` ziti service (`#openbao-client`), `proxy``127.0.0.1:18443`). OpenBao is the existing `openbao` ziti service (`#openbao-client`),
dialled as `https://bao.cbs.tikali.net:8200` (an intercept name), TLS validated against the dialled as `https://bao.cbs.tikali.net:8200` (an intercept name), TLS validated against the
@@ -29,7 +36,7 @@ The agent side of the backend lifecycle agent protocol — [MONKY-ADR-0028](http
``` ```
kit reveal (admin, once) tenancy signs a bootstrap DEPLOY GRANT: ES256 JWT kit reveal (admin, once) tenancy signs a bootstrap DEPLOY GRANT: ES256 JWT
iss <tenancy issuer>, aud openbao-see-env, sub agent:<env_id>, iss <tenancy issuer>, aud openbao-see-env, sub agent:<env_id>,
env_id, kind deploy-grant, jti (single-use, recorded), exp now+1h env_id, kind deploy-grant, jti (→ token meta grant_jti; tenancy records it), exp now+1h
install.sh stages it at /etc/monky-deployd/bootstrap.jwt (0600) install.sh stages it at /etc/monky-deployd/bootstrap.jwt (0600)
first tick POST https://bao…/v1/auth/jwt-tenancy/login {"role":"see-env","jwt":<grant>} first tick POST https://bao…/v1/auth/jwt-tenancy/login {"role":"see-env","jwt":<grant>}
→ auth.client_token (ttl 24 h, max 30 d, policy see-env, → auth.client_token (ttl 24 h, max 30 d, policy see-env,
@@ -49,10 +56,13 @@ OpenBao side (Terraform, `tikali/services/security/openbao`): mount `jwt-tenancy
`monky/data/{{identity.entity.aliases.<accessor>.metadata.env_id}}/see/*` read + metadata list + `monky/data/{{identity.entity.aliases.<accessor>.metadata.env_id}}/see/*` read + metadata list +
`auth/token/{renew-self,lookup-self}`. **Gate 1 v2 passed 2026-09-05 07:40Z** (distinct entity per env). `auth/token/{renew-self,lookup-self}`. **Gate 1 v2 passed 2026-09-05 07:40Z** (distinct entity per env).
**Second-reveal semantics.** A kit re-reveal (or a retire) supersedes every earlier grant of the **Second-reveal semantics.** A kit re-reveal (or a retire) supersedes every earlier bootstrap grant
backend and revokes the token accessors tenancy knows. The old kit's grant fails at login of the backend (tenancy marks their `backend_leases` rows revoked) and revokes the token accessors
(unknown/used `jti`), and a token already minted from it is refused at the next call with tenancy knows. The `jwt-tenancy` mount keeps **no** replay state — it validates signature, `aud`,
`401 AGENT_UNAUTHENTICATED` (its `meta.grant_jti` is superseded) — **not** `AGENT_ENV_MISMATCH`. `bound_claims` and `exp` only (openbao `terraform/jwt-tenancy.tf`, the `see_env` role) — so a
superseded grant still logs in until its 1 h `exp`; the token it yields, like a token already minted
from it, is refused by tenancy at the first bearer call with `401 AGENT_UNAUTHENTICATED` (its
`meta.grant_jti` is superseded) — **not** `AGENT_ENV_MISMATCH`.
The agent then deletes its token; if a fresh `bootstrap.jwt` is on disk it bootstraps again in the The agent then deletes its token; if a fresh `bootstrap.jwt` is on disk it bootstraps again in the
same tick, otherwise it exits 1 and says "re-run the install kit". same tick, otherwise it exits 1 and says "re-run the install kit".
@@ -70,7 +80,8 @@ same tick, otherwise it exits 1 and says "re-run the install kit".
```json ```json
{"env_id": "env-qa-02", "desired_sha": "7a10…", "action": "apply", "purge_volumes": false, {"env_id": "env-qa-02", "desired_sha": "7a10…", "action": "apply", "purge_volumes": false,
"bundle_url": "/v1/agent/bundle/env-qa-02/7a10…", "checkin_interval_s": 60, "bundle_url": "/v1/agent/bundle/env-qa-02/7a10…", "checkin_interval_s": 60,
"vault": {"addr": "https://bao.cbs.tikali.net:8200", "mount": "monky", "prefix": "env-qa-02/see"}} "vault": {"addr": "https://bao.cbs.tikali.net:8200", "mount": "monky", "prefix": "env-qa-02/see",
"auth_mount": "jwt-tenancy", "auth_role": "see-env"}}
``` ```
`action`: `apply` (desired ≠ applied), `none` (converged → heartbeat), `down` (retire; `purge_volumes` `action`: `apply` (desired ≠ applied), `none` (converged → heartbeat), `down` (retire; `purge_volumes`
is only meaningful here). `vault.mount` is the **KV** mount; the agent adopts it if it differs from is only meaningful here). `vault.mount` is the **KV** mount; the agent adopts it if it differs from
@@ -109,12 +120,16 @@ address, pull, renderer, images_policy, secrets_provider, agent{…}, files[]`.
### `POST /v1/agent/lease` ### `POST /v1/agent/lease`
```json ```json
{"env_id": "env-qa-02", "reason": "apply"} // reason: apply | renew {"env_id": "env-qa-02"}
``` ```
(The agent also sends `"reason": "apply" | "renew"` for its own logs; tenancy's `AgentLease` is
`{env_id}` and ignores unknown fields.)
```json ```json
{"env_id": "env-qa-02", "login_jwt": "eyJ…", "ttl_s": 3600, "mount": "jwt-tenancy", "role": "see-env", {"env_id": "env-qa-02", "login_jwt": "eyJ…", "ttl_s": 3600, "mount": "jwt-tenancy", "role": "see-env",
"vault": {"addr": "https://bao.cbs.tikali.net:8200", "mount": "monky"}} "addr": "https://bao.cbs.tikali.net:8200"}
``` ```
`addr` is the OpenBao address for the login below, top-level (tenancy `AgentLeaseOut`); the KV
mount/prefix come from `checkin`'s `vault`, not from the lease.
Then `POST /v1/auth/{mount}/login {"role": "{role}", "jwt": "{login_jwt}"}` on OpenBao. A body with Then `POST /v1/auth/{mount}/login {"role": "{role}", "jwt": "{login_jwt}"}` on OpenBao. A body with
`wrapping_token` / `role_id` (the pre-Gate-1 AppRole lease) is refused with `LEASE_SHAPE` → report `wrapping_token` / `role_id` (the pre-Gate-1 AppRole lease) is refused with `LEASE_SHAPE` → report
`failed`. `429 LEASE_RATE_LIMITED` → exit 75. `failed`. `429 LEASE_RATE_LIMITED` → exit 75.
@@ -133,15 +148,17 @@ first 2 KiB land in the audit log — it has been through the redactor.
## Divergences (2026-09-05) ## Divergences (2026-09-05)
- **monky-tenancy `main` (MR !15) still implements the AppRole lease and install kit** - **Resolved 2026-09-05 (before v0.1.0 was tagged):** monky-tenancy !17 (`288df791`) landed the
(`AgentLeaseOut{wrapping_token, role_id}`, `bootstrap.wrap`, `bao.approle` in the kit's config). The tenancy side of the Gate 1 RESULT / ADR-0028 amendment — `AgentLeaseOut{env_id, login_jwt, ttl_s,
binding design is the plan's Gate 1 RESULT / ADR-0028 amendment: `{login_jwt, ttl_s, mount, role}` mount, role, addr}`, the ES256 deploy-grant signer (`app/agent_keys.py`), `GET
and `POST /v1/auth/jwt-tenancy/login`. This agent implements the latter; against an un-migrated /.well-known/agent-jwks.json`, no AppRole and no response wrapping anywhere in tenancy. The
tenancy it reports `failed` with `LEASE_SHAPE` and refuses `bao.approle` in its config. The tenancy `LEASE_SHAPE` refusal of a `wrapping_token`/`role_id` body stays in this agent as a guard against a
follow-up (deploy-grant signer, JWKS, `lease` shape, kit → `bootstrap.jwt`) is tracked on stale tenancy, not as a description of `main`.
monky-tenancy. - **Resolved 2026-09-05:** the kit no longer generates a config file. Since monky-tenancy !22
- The kit's generated config uses `tenancy.base_url: http://monky.tenancy.deploy:8081` — accepted as (`61bd0281`) the one-time install script stages the bootstrap grant and runs `install.sh --env …
an alias for `tenancy.{scheme,host,port}`. --site … --version … --bootstrap-file …`, and `install.sh` writes `/etc/monky-deployd/config.yaml`
(tenancy service name, OpenBao address and `jwt-tenancy`/`see-env`). `tenancy.base_url:` remains an
accepted alias for `tenancy.{scheme,host,port}` (`config.py`) for hand-written configs.
- `report` gains an optional `detail` (doc 24 §3.3); tenancy's `AgentReport` ignores unknown fields - `report` gains an optional `detail` (doc 24 §3.3); tenancy's `AgentReport` ignores unknown fields
today — if `strict` bodies land, `detail` folds into `log_tail`. today — if `strict` bodies land, `detail` folds into `log_tail`.
- Bundle sha header: tenancy sends `X-Bundle-Sha`, doc 24 says `X-Bundle-Sha256`; the agent reads - Bundle sha header: tenancy sends `X-Bundle-Sha`, doc 24 says `X-Bundle-Sha256`; the agent reads
+1 -1
View File
@@ -4,4 +4,4 @@ Dials monky-tenancy over the mesh with the box's host identity, fetches the rend
leases a deploy grant, logs in to OpenBao, reads its own secrets, runs `docker compose`, leases a deploy grant, logs in to OpenBao, reads its own secrets, runs `docker compose`,
reports. Stdlib only; the optional `openziti` SDK is the `sdk` transport.""" reports. Stdlib only; the optional `openziti` SDK is the `sdk` transport."""
__version__ = "0.1.1" __version__ = "0.1.8"
+67 -9
View File
@@ -24,6 +24,7 @@ from pathlib import Path
from monky_deployd import __version__ from monky_deployd import __version__
from monky_deployd import bundle as bundlemod from monky_deployd import bundle as bundlemod
from monky_deployd import registry as registrymod
from monky_deployd import state as statemod from monky_deployd import state as statemod
from monky_deployd.bao import BaoClient, BaoError, BaoToken, ManifestPathError, kv_data_path from monky_deployd.bao import BaoClient, BaoError, BaoToken, ManifestPathError, kv_data_path
from monky_deployd.bundle import Bundle, BundleError from monky_deployd.bundle import Bundle, BundleError
@@ -86,7 +87,7 @@ class Agent:
def __init__(self, cfg: Config, *, prune: bool = False, docker: Docker | None = None): def __init__(self, cfg: Config, *, prune: bool = False, docker: Docker | None = None):
self.cfg = cfg self.cfg = cfg
self.prune = prune self.prune = prune
self.docker = docker or Docker(cfg.docker_bin) self.docker = docker or Docker(cfg.docker_bin, docker_config=str(cfg.docker_config_dir))
self.transport = build(cfg) self.transport = build(cfg)
self.tenancy_http = HttpClient( self.tenancy_http = HttpClient(
self.transport, cfg.tenancy.scheme, cfg.tenancy.host, cfg.tenancy.port, timeout=cfg.tenancy.timeout_s self.transport, cfg.tenancy.scheme, cfg.tenancy.host, cfg.tenancy.port, timeout=cfg.tenancy.timeout_s
@@ -274,13 +275,29 @@ class Agent:
if hdr_sha and hdr_sha != b.sha: if hdr_sha and hdr_sha != b.sha:
log.warning("bundle header sha %s disagrees with content %s", _short(hdr_sha), _short(b.sha)) log.warning("bundle header sha %s disagrees with content %s", _short(hdr_sha), _short(b.sha))
self._refusal_checks(b) self._refusal_checks(b)
# secrets: lease -> login -> reads (values never logged; names only) # secrets: lease -> login -> reads (values never logged; names only). An entry marked
# `use: registry-auth` is NOT an env var — it is the pull credential, written to the
# agent's own Docker config instead of `.env` (doc 24 §4a).
entries = b.manifest.get("entries", []) entries = b.manifest.get("entries", [])
registry_entries = [e for e in entries if e.get("use") == registrymod.USE]
env_entries = [e for e in entries if e.get("use") != registrymod.USE]
if entries: if entries:
token = self._lease_login("apply") token = self._token_for_apply()
for e in entries: for e in env_entries:
self._values[e["var"]] = self.bao.kv_read(token, e["path"], cfg.env_id, e.get("version")) self._values[e["var"]] = self.bao.kv_read(token, e["path"], cfg.env_id, e.get("version"))
log.info("read %d secret(s): %s", len(entries), ", ".join(sorted(self._values))) for e in registry_entries:
raw = self.bao.kv_read(token, e["path"], cfg.env_id, e.get("version"))
try:
auth = registrymod.parse(raw, default_registry=cfg.registry_host)
registrymod.write_docker_config(cfg.docker_config_dir, auth)
except registrymod.RegistryAuthError as exc:
log.warning("registry credential unusable (%s) — private images will not pull", exc)
log.info(
"read %d secret(s): %s%s",
len(entries),
", ".join(sorted(self._values)),
" (+ registry credential)" if registry_entries else "",
)
env_text = bundlemod.render_env(b.env_template, self._values) env_text = bundlemod.render_env(b.env_template, self._values)
leftover = bundlemod.referenced_vars(env_text) leftover = bundlemod.referenced_vars(env_text)
if leftover: if leftover:
@@ -288,7 +305,17 @@ class Agent:
release = self._promote(self._stage(b, env_text)) release = self._promote(self._stage(b, env_text))
compose = self._compose() compose = self._compose()
log.info("compose pull") log.info("compose pull")
try:
compose.pull() compose.pull()
except ComposeError as exc:
# the registry's own message is the diagnosis ("no basic auth credentials", "manifest
# unknown", a DNS failure). Swallowing it behind `rc=1` cost an SSH hunt on env-dev-01.
detail = " | ".join(line.strip() for line in (exc.output or "").splitlines() if line.strip())[-600:]
self.state.last_result = "failed"
self.state.last_error = f"compose pull failed: {detail or exc}"
log.error("compose pull failed (rc=%s): %s", exc.rc, detail or "(no output)")
self._report("failed", b.sha, detail=f"compose pull: {detail or exc}"[:900])
return EX_FAIL
log.info("compose up -d --remove-orphans") log.info("compose up -d --remove-orphans")
compose.up() compose.up()
ok, containers = compose.wait_healthy(cfg.healthy_timeout_s) ok, containers = compose.wait_healthy(cfg.healthy_timeout_s)
@@ -326,15 +353,23 @@ class Agent:
raise Refusal("PRIVILEGED_REFUSED", ", ".join(findings) + " (bundle.json allow_privileged is not set)") raise Refusal("PRIVILEGED_REFUSED", ", ".join(findings) + " (bundle.json allow_privileged is not set)")
if self.state.is_rollback(b.sha) and not b.flag("allow_rollback"): if self.state.is_rollback(b.sha) and not b.flag("allow_rollback"):
raise Refusal("ROLLBACK_REFUSED", f"{b.sha[:12]} was applied before; allow_rollback is not set") raise Refusal("ROLLBACK_REFUSED", f"{b.sha[:12]} was applied before; allow_rollback is not set")
# A bundle that declares its size gets the full sum; one that does not still has to clear
# the headroom floor. Without the floor a nearly-full box sails past this check and dies
# mid-pull with containerd's "no space left on device", which reads as a registry fault
# and costs an SSH hunt (env-dev-08, 2026-09-09).
need = b.disk_need_bytes need = b.disk_need_bytes
if need:
free = self.docker.free_bytes() free = self.docker.free_bytes()
required = int(need * cfg.disk.factor + cfg.disk.headroom_bytes) required = int(need * cfg.disk.factor + cfg.disk.headroom_bytes) if need else cfg.disk.headroom_bytes
if free is not None and free < required: if free is not None and free < required:
sized = (
f"bundle needs {required // 2**20} MiB "
f"({need // 2**20} MiB x {cfg.disk.factor} + {cfg.disk.headroom_bytes // 2**20} MiB headroom)"
if need
else f"a pull needs at least {required // 2**20} MiB headroom (bundle declares no size)"
)
raise Refusal( raise Refusal(
"DISK_INSUFFICIENT", "DISK_INSUFFICIENT",
f"docker data-root has {free // 2**20} MiB free, bundle needs {required // 2**20} MiB " f"image storage ({', '.join(self.docker.storage_paths())}) has {free // 2**20} MiB free, {sized}",
f"({need // 2**20} MiB x {cfg.disk.factor} + {cfg.disk.headroom_bytes // 2**20} MiB headroom)",
) )
# -- credentials ------------------------------------------------------------------------------------- # -- credentials -------------------------------------------------------------------------------------
@@ -392,6 +427,29 @@ class Agent:
if old and old != token: if old and old != token:
self.bao.revoke_self(old) self.bao.revoke_self(old)
def _token_for_apply(self) -> str:
"""The OpenBao token to read this bundle's secrets with. A lease-derived token that is
still valid is REUSED (a lease per apply burned tenancy's 5/h budget on every retry —
env-qa-02 pilot, 2026-09-07); the bootstrap token is swapped for a lease once; and if
tenancy rate-limits the lease while we hold a working token, apply with what we have
and swap later rather than fail the deploy."""
cfg = self.cfg
st = self.state.token
if self.token and st is not None and st.source == "lease":
try:
info = self.bao.lookup_self(self.token)
if int(info.get("ttl") or 0) > cfg.bao.renew_below_s:
return self.token
except BaoError as exc:
log.info("lease token no longer valid (%s); re-leasing", exc)
try:
return self._lease_login("apply")
except RateLimited as exc:
if self.token:
log.warning("lease rate-limited (%s); applying with the current token, swap deferred", exc)
return self.token
raise
def _lease_login(self, reason: str) -> str: def _lease_login(self, reason: str) -> str:
assert self.tenancy is not None assert self.tenancy is not None
lease = self.tenancy.lease(reason) lease = self.tenancy.lease(reason)
+8 -2
View File
@@ -144,12 +144,18 @@ def parse(data: bytes, *, max_bytes: int = 4 * 1024 * 1024) -> Bundle:
# --- refusal checks (pure; names only, never values) ----------------------------------------- # --- refusal checks (pure; names only, never values) -----------------------------------------
def _code_lines(text: str) -> str:
"""Drop comment lines: a `# … ${VAR} …` remark in .env.template (the renderer writes one)
is not a reference. Compose/dotenv comments start with `#` after optional whitespace."""
return "\n".join(ln for ln in text.splitlines() if not ln.lstrip().startswith("#"))
def referenced_vars(text: str) -> set[str]: def referenced_vars(text: str) -> set[str]:
return {m.group(1) for m in _VAR_RE.finditer(text)} return {m.group(1) for m in _VAR_RE.finditer(_code_lines(text))}
def defaulted_vars(text: str) -> set[str]: def defaulted_vars(text: str) -> set[str]:
return {m.group(1) for m in _VAR_DEFAULTED_RE.finditer(text)} return {m.group(1) for m in _VAR_DEFAULTED_RE.finditer(_code_lines(text))}
def unresolved_vars(bundle: Bundle, provided: set[str]) -> list[str]: def unresolved_vars(bundle: Bundle, provided: set[str]) -> list[str]:
+39 -4
View File
@@ -12,6 +12,10 @@ import time
from dataclasses import dataclass from dataclasses import dataclass
from pathlib import Path from pathlib import Path
# containerd's default root: docker 29's image store lives here, often on another filesystem
# than DockerRootDir. Both are checked before a pull (see Docker.storage_paths).
CONTAINERD_ROOTS = ("/var/lib/containerd",)
log = logging.getLogger("monky-deployd.compose") log = logging.getLogger("monky-deployd.compose")
@@ -41,9 +45,13 @@ class Container:
class Docker: class Docker:
def __init__(self, docker_bin: str = "docker", timeout_s: int = 600): def __init__(self, docker_bin: str = "docker", timeout_s: int = 600, docker_config: str | None = None):
self.bin = docker_bin self.bin = docker_bin
self.timeout_s = timeout_s self.timeout_s = timeout_s
# doc 24 §4a: registry credentials live in a directory the AGENT owns, named explicitly
# rather than inherited from $HOME. The unit runs as `monky-deployd`, so a `docker login`
# by a human or by root is invisible here — which is exactly what cost env-dev-01 an hour.
self.docker_config = docker_config
def available(self) -> bool: def available(self) -> bool:
return shutil.which(self.bin) is not None return shutil.which(self.bin) is not None
@@ -59,7 +67,7 @@ class Docker:
capture_output=True, capture_output=True,
text=True, text=True,
timeout=timeout or self.timeout_s, timeout=timeout or self.timeout_s,
env={**os.environ, "COMPOSE_INTERACTIVE_NO_CLI": "1"}, env=self._env(),
) )
except FileNotFoundError as exc: except FileNotFoundError as exc:
raise ComposeError(what, 127, f"{self.bin} not found") from exc raise ComposeError(what, 127, f"{self.bin} not found") from exc
@@ -70,6 +78,12 @@ class Docker:
raise ComposeError(what, p.returncode, out.strip()) raise ComposeError(what, p.returncode, out.strip())
return out return out
def _env(self) -> dict:
env = {**os.environ, "COMPOSE_INTERACTIVE_NO_CLI": "1"}
if self.docker_config:
env["DOCKER_CONFIG"] = self.docker_config
return env
# -- facts ------------------------------------------------------------------------------------ # -- facts ------------------------------------------------------------------------------------
def version(self) -> str | None: def version(self) -> str | None:
try: try:
@@ -90,8 +104,23 @@ class Docker:
root = "" root = ""
return root or "/var/lib/docker" return root or "/var/lib/docker"
def free_bytes(self, path: str | None = None) -> int | None: def storage_paths(self) -> list[str]:
p = path or self.data_root() """Every filesystem a `compose pull` can fill.
docker 29 keeps IMAGE layers in the containerd image store (containerd's own root,
/var/lib/containerd by default), NOT under DockerRootDir. On a box where those two sit
on different filesystems, measuring only the data-root reports plenty of room while the
pull dies with "no space left on device" (env-dev-08, 2026-09-09: 93 GiB free on the
data-root, 2.8 GiB on the root filesystem that held containerd).
"""
paths = [self.data_root()]
for extra in CONTAINERD_ROOTS:
if os.path.isdir(extra):
paths.append(extra)
return paths
def _free_at(self, path: str) -> int | None:
p = path
while p and not os.path.exists(p): while p and not os.path.exists(p):
p = os.path.dirname(p) p = os.path.dirname(p)
try: try:
@@ -100,6 +129,12 @@ class Docker:
return None return None
return st.f_bavail * st.f_frsize return st.f_bavail * st.f_frsize
def free_bytes(self, path: str | None = None) -> int | None:
"""Free bytes on `path`, or the TIGHTEST of the image-storage filesystems."""
paths = [path] if path else self.storage_paths()
seen = [v for v in (self._free_at(p) for p in paths) if v is not None]
return min(seen) if seen else None
def image_prune(self) -> None: def image_prune(self) -> None:
try: try:
self.run(["image", "prune", "-f"], timeout=300) self.run(["image", "prune", "-f"], timeout=300)
+11 -1
View File
@@ -129,7 +129,7 @@ def parse_yaml_subset(text: str) -> dict:
class TenancyCfg: class TenancyCfg:
service: str = "monky.tenancy.deploy" service: str = "monky.tenancy.deploy"
host: str = "" # intercept host; defaults to `service` host: str = "" # intercept host; defaults to `service`
port: int = 8081 port: int = 443 # the service's INTERCEPT port (openziti); the pod-side listener is 127.0.0.1:8081
scheme: str = "http" scheme: str = "http"
proxy_addr: str = "127.0.0.1:18443" proxy_addr: str = "127.0.0.1:18443"
timeout_s: int = 30 timeout_s: int = 30
@@ -175,6 +175,9 @@ class Config:
healthy_timeout_s: int = 300 healthy_timeout_s: int = 300
compose_project: str = "" compose_project: str = ""
docker_bin: str = "docker" docker_bin: str = "docker"
# doc 24 §4a: the registry the bundle pulls from, used when the seeded credential is a bare
# `username:password` (a JSON credential names its own registry).
registry_host: str = "harbor.tikali.net"
log_level: str = "INFO" log_level: str = "INFO"
path: str = DEFAULT_CONFIG_PATH path: str = DEFAULT_CONFIG_PATH
@@ -191,6 +194,13 @@ class Config:
def lock_path(self) -> Path: def lock_path(self) -> Path:
return Path(self.state_dir) / "lock" return Path(self.state_dir) / "lock"
@property
def docker_config_dir(self) -> Path:
"""Where the agent keeps its OWN registry credentials (`DOCKER_CONFIG`). Not `$HOME`: the
unit runs as `monky-deployd`, and a human's or root's `docker login` must not be what the
agent depends on (doc 24 §4a)."""
return Path(self.state_dir) / "docker"
@property @property
def is_prod(self) -> bool: def is_prod(self) -> bool:
if self.prod is not None: if self.prod is not None:
+80
View File
@@ -0,0 +1,80 @@
"""The registry credential (monky-design-docs doc 24 §4a).
The bundle's manifest carries one entry marked `use: registry-auth` — the estate-wide read-only
Harbor robot, copied by monky-tenancy into this environment's own prefix so the agent can read it
with the OpenBao policy it already has. It is NOT an env var (an entry in `.env` would put the
registry password into every container's environment), so it never reaches the compose file: it is
written to a Docker config **the agent owns**, and `DOCKER_CONFIG` points the docker CLI at it.
That last part is the whole point. The unit runs as `monky-deployd`, whose home is the state dir,
so a `docker login` performed by a human or by root is invisible to it — the failure looks exactly
like "no credentials at all" (env-dev-01, 2026-09-08).
Accepted shapes for the secret's value, because the seeded robot has been written both ways:
* a JSON object: `{"registry": …, "username": …, "password": …}`
* a `username:password` string, with the registry taken from `registry_host` config
"""
from __future__ import annotations
import base64
import json
import logging
from dataclasses import dataclass
from pathlib import Path
log = logging.getLogger("monky-deployd.registry")
USE = "registry-auth"
class RegistryAuthError(Exception):
pass
@dataclass
class RegistryAuth:
registry: str
username: str
password: str
def docker_config(self) -> dict:
token = base64.b64encode(f"{self.username}:{self.password}".encode()).decode()
return {"auths": {self.registry: {"auth": token}}}
def parse(value: str, *, default_registry: str) -> RegistryAuth:
"""`value` is whatever the KV entry held; never logged, never echoed."""
text = (value or "").strip()
if not text:
raise RegistryAuthError("empty registry credential")
if text.startswith("{"):
try:
data = json.loads(text)
except json.JSONDecodeError as exc:
raise RegistryAuthError("registry credential is not valid JSON") from exc
user, pw = data.get("username"), data.get("password")
registry = data.get("registry") or default_registry
if not user or not pw:
raise RegistryAuthError("registry credential JSON needs username + password")
return RegistryAuth(registry=str(registry), username=str(user), password=str(pw))
if ":" not in text:
raise RegistryAuthError("registry credential is neither JSON nor username:password")
user, _, pw = text.partition(":")
if not default_registry:
raise RegistryAuthError("username:password credential needs a configured registry host")
return RegistryAuth(registry=default_registry, username=user, password=pw)
def write_docker_config(dir_path: Path, auth: RegistryAuth) -> Path:
"""0600 `config.json` in a directory the agent owns; DOCKER_CONFIG points the CLI at it."""
dir_path = Path(dir_path)
dir_path.mkdir(parents=True, exist_ok=True)
dir_path.chmod(0o700)
target = dir_path / "config.json"
tmp = dir_path / "config.json.tmp"
tmp.write_text(json.dumps(auth.docker_config(), indent=2) + "\n")
tmp.chmod(0o600)
tmp.replace(target)
log.info("registry credential in place for %s (%s)", auth.registry, auth.username)
return target
+4 -2
View File
@@ -4,7 +4,7 @@ Bearer = the agent's OpenBao token (from the `jwt-tenancy` login). Tenancy pins
the token's `meta.env_id` (403 AGENT_ENV_MISMATCH -> exit 78, never retried) and refuses a token the token's `meta.env_id` (403 AGENT_ENV_MISMATCH -> exit 78, never retried) and refuses a token
whose deploy grant was superseded (401 AGENT_UNAUTHENTICATED -> re-bootstrap or re-run the kit). whose deploy grant was superseded (401 AGENT_UNAUTHENTICATED -> re-bootstrap or re-run the kit).
Lease shape of record (Gate 1 v2, 2026-09-05): `{login_jwt, ttl_s, mount, role, vault}`. An Lease shape of record (Gate 1 v2, 2026-09-05): `{env_id, login_jwt, ttl_s, mount, role, addr}`. An
AppRole-era body (`wrapping_token`, `role_id`) is refused loudly — there is nothing to unwrap.""" AppRole-era body (`wrapping_token`, `role_id`) is refused loudly — there is nothing to unwrap."""
from __future__ import annotations from __future__ import annotations
@@ -158,13 +158,15 @@ class TenancyClient:
"(ADR-0028 amendment 2026-09-05)", "(ADR-0028 amendment 2026-09-05)",
) )
raise TenancyError(200, "LEASE_SHAPE", "lease response carries no login_jwt") raise TenancyError(200, "LEASE_SHAPE", "lease response carries no login_jwt")
# shape of record (tenancy AgentLeaseOut): `addr` is top-level; a pre-0.1.x `vault{}` object
# is still read as a fallback so an older fake or tenancy does not break the lease
v = js.get("vault") or {} v = js.get("vault") or {}
return Lease( return Lease(
login_jwt=str(js["login_jwt"]), login_jwt=str(js["login_jwt"]),
ttl_s=int(js.get("ttl_s") or 3600), ttl_s=int(js.get("ttl_s") or 3600),
mount=str(js.get("mount") or "jwt-tenancy"), mount=str(js.get("mount") or "jwt-tenancy"),
role=str(js.get("role") or "see-env"), role=str(js.get("role") or "see-env"),
vault=Vault(addr=v.get("addr"), mount=v.get("mount"), prefix=v.get("prefix")), vault=Vault(addr=js.get("addr") or v.get("addr"), mount=v.get("mount"), prefix=v.get("prefix")),
) )
def report( def report(
+37
View File
@@ -16,6 +16,7 @@ from __future__ import annotations
import http.client import http.client
import json import json
import logging import logging
import os
import socket import socket
import ssl import ssl
from dataclasses import dataclass from dataclasses import dataclass
@@ -35,6 +36,27 @@ class TransportError(Exception):
class Transport: class Transport:
name = "base" name = "base"
def _check_identity_readable(self) -> None:
path = self.identity_path
try:
with open(path, "rb") as fh:
raw = fh.read()
except FileNotFoundError as exc:
raise TransportError(f"transport sdk: identity {path} does not exist") from exc
except PermissionError as exc:
raise TransportError(
f"transport sdk: identity {path} is not readable by this user "
f"(uid {os.geteuid()}) — the file is owned by the tunneller and rewritten on every "
"config refresh, which drops POSIX ACLs; add the agent's user to the file's group "
"(usually `ziti`) so the grant survives"
) from exc
except OSError as exc:
raise TransportError(f"transport sdk: identity {path} is unreadable: {exc}") from exc
try:
json.loads(raw)
except ValueError as exc:
raise TransportError(f"transport sdk: identity {path} is not valid JSON: {exc}") from exc
def connect(self, host: str, port: int, timeout: float) -> socket.socket: # pragma: no cover def connect(self, host: str, port: int, timeout: float) -> socket.socket: # pragma: no cover
raise NotImplementedError raise NotImplementedError
@@ -87,6 +109,13 @@ class SdkTransport(Transport):
"transport sdk: the openziti module is not installed in this venv; " "transport sdk: the openziti module is not installed in this venv; "
"use transport: proxy (monky-deployd-proxy.service) or system" "use transport: proxy (monky-deployd-proxy.service) or system"
) from exc ) from exc
# openziti.load() does NOT raise on an unreadable or malformed identity: the C SDK logs
# "Failed to load Ziti Identity ...: configuration is invalid" and hands back a context
# that fails LATER, at dial, as a bare TypeError — which reads as a missing intercept or
# a policy gap and sends you hunting the mesh instead of the file (env-dev-08,
# 2026-09-09: ziti-edge-tunnel rewrote the identity and dropped the agent's ACL).
# So check the file ourselves first and name the real fault.
self._check_identity_readable()
try: try:
self._ctx = openziti.load(self.identity_path) self._ctx = openziti.load(self.identity_path)
except Exception as exc: except Exception as exc:
@@ -104,6 +133,14 @@ class SdkTransport(Transport):
return socket.create_connection((host, port), timeout=timeout) return socket.create_connection((host, port), timeout=timeout)
except OSError as exc: except OSError as exc:
raise TransportError(f"transport sdk: dial {host}:{port} failed: {exc}") from exc raise TransportError(f"transport sdk: dial {host}:{port} failed: {exc}") from exc
except Exception as exc: # noqa: BLE001 - the SDK raises bare Exception((code, msg)) and TypeError
# openziti-sdk-py: an address with NO matching intercept falls through to
# PySocket.connect(tuple) → TypeError; a matching intercept the identity may not dial
# raises Exception((-18, 'service not available')) — env-qa-02 pilot, 2026-09-07.
raise TransportError(
f"transport sdk: dial {host}:{port} failed: {exc} — no intercept for that host:port, or this "
"identity has no dial policy for the service (check the intercept port and the identity's attrs)"
) from exc
def describe(self) -> str: def describe(self) -> str:
return f"sdk(identity={self.identity_path})" return f"sdk(identity={self.identity_path})"
+112 -37
View File
@@ -1,18 +1,26 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# monky-deployd installer — Ubuntu 26.04 (verified target). # monky-deployd installer — Ubuntu 26.04 (verified target).
# #
# curl -fsSL https://scm.tikali.ai/tikali/applications/monky/monky-deployd/-/raw/main/packaging/install.sh \ # T=<read-only GitLab deploy token, scope read_package_registry> # from the install kit / OpenBao
# | sudo bash -s -- --env env-qa-02 --site cbs [--transport sdk|proxy|system] [--version 0.1.1] \ # curl -sSf -H "DEPLOY-TOKEN: $T" \
# [--enrol-jwt /path/monky-host.env-qa-02.jwt] [--laptop] [--source gitlab|gitea] < bootstrap.jwt # https://scm.tikali.ai/api/v4/projects/69/packages/generic/monky-deployd/0.1.6/install.sh \
# | sudo bash -s -- --env env-qa-02 --site cbs --token "$T" --bootstrap-file bootstrap.jwt \
# [--transport sdk|proxy|system] [--version 0.1.6] [--enrol-jwt /path/monky-host.env-qa-02.jwt] \
# [--laptop] [--source gitlab|gitea] [--docker-data-root /home/docker-data]
# #
# stdin (or --bootstrap-file): the ONE-TIME bootstrap deploy grant from the install kit # --token / MONKY_DEPLOYD_TOKEN: the GitLab project is PRIVATE (its parent groups are private, so it
# (GET /v1/backends/{id}/agent/install). The enrolment JWT is read from --enrol-jwt or # cannot be made public); every download from the generic package registry — this script included —
# /etc/monky-deployd/enrol.jwt and is only needed when the host identity is not enrolled yet. # sends `DEPLOY-TOKEN: <token>`. The token is read-only (read_package_registry), revocable, and is
# never printed by this script. Not needed with --source gitea.
# --bootstrap-file (or stdin when the script runs from a FILE): the ONE-TIME bootstrap deploy grant
# from the install kit (GET /v1/backends/{id}/agent/install). The enrolment JWT is read from
# --enrol-jwt or /etc/monky-deployd/enrol.jwt and is only needed when the host identity is not
# enrolled yet.
# #
# What it does (idempotent): # What it does (idempotent):
# 1. apt: ziti-edge-tunnel (OpenZiti `jammy` suite) if absent, docker-compose-plugin, acl # 1. apt: ziti-edge-tunnel (OpenZiti `jammy` suite) if absent, docker-compose-plugin, acl
# 2. downloads the pinned monky-deployd_<ver>_amd64.deb + .sha256 from the scm.tikali.ai package registry # 2. downloads the pinned monky-deployd_<ver>_amd64.deb + .sha256 from the scm.tikali.ai package registry
# (--source gitea: the Gitea release, off-estate), verifies, installs # (DEPLOY-TOKEN header; --source gitea: the Gitea release, off-estate), verifies, installs
# 3. enrols /opt/openziti/etc/identities/monky-host.<env>.json if absent (ziti-edge-tunnel enroll), # 3. enrols /opt/openziti/etc/identities/monky-host.<env>.json if absent (ziti-edge-tunnel enroll),
# chown ziti:ziti 0600, switches ziti-edge-tunnel.service to `run-host` via a drop-in # chown ziti:ziti 0600, switches ziti-edge-tunnel.service to `run-host` via a drop-in
# 4. writes /etc/monky-deployd/config.yaml, ACLs so user monky-deployd can read the identity, # 4. writes /etc/monky-deployd/config.yaml, ACLs so user monky-deployd can read the identity,
@@ -21,22 +29,25 @@
set -euo pipefail set -euo pipefail
umask 077 umask 077
DEFAULT_VERSION="0.1.1" DEFAULT_VERSION="0.1.6"
# Download source. PRIMARY is the public GitLab project's generic package registry on scm.tikali.ai: # Download source. PRIMARY is the GitLab project's generic package registry on scm.tikali.ai: inside
# inside the estate gitea.cbs.tikali.net is split-horizon to jump1's RED EIP (10.10.0.175), which has # the estate gitea.cbs.tikali.net is split-horizon to jump1's RED EIP (10.10.0.175), which has no HTTP
# no HTTP ingress, so backend boxes cannot reach the Gitea mirror (cbs/iac#102); scm.tikali.ai they can. # ingress, so backend boxes cannot reach the Gitea mirror (cbs/iac#102); scm.tikali.ai they can. The
# `--source gitea` keeps the Gitea release as the off-estate alternative. --base-url / MONKY_DEPLOYD_BASE_URL # project is PRIVATE (parent groups are private), so the registry needs the read-only deploy token
# override the base for whichever layout is selected. # (--token / MONKY_DEPLOYD_TOKEN, sent as the DEPLOY-TOKEN header). `--source gitea` keeps the Gitea
# release as the off-estate alternative (no token). --base-url / MONKY_DEPLOYD_BASE_URL override the
# base for whichever layout is selected.
GITLAB_BASE_URL="https://scm.tikali.ai/api/v4/projects/69/packages/generic/monky-deployd" GITLAB_BASE_URL="https://scm.tikali.ai/api/v4/projects/69/packages/generic/monky-deployd"
GITEA_BASE_URL="https://gitea.cbs.tikali.net/mdella/monky-deployd" GITEA_BASE_URL="https://gitea.cbs.tikali.net/mdella/monky-deployd"
SOURCE="${MONKY_DEPLOYD_SOURCE:-gitlab}" SOURCE="${MONKY_DEPLOYD_SOURCE:-gitlab}"
BASE_URL="${MONKY_DEPLOYD_BASE_URL:-}" BASE_URL="${MONKY_DEPLOYD_BASE_URL:-}"
TOKEN="${MONKY_DEPLOYD_TOKEN:-}"
OPENZITI_SUITE="${OPENZITI_SUITE:-jammy}" OPENZITI_SUITE="${OPENZITI_SUITE:-jammy}"
IDENTITY_DIR="/opt/openziti/etc/identities" IDENTITY_DIR="/opt/openziti/etc/identities"
ETC="/etc/monky-deployd" ETC="/etc/monky-deployd"
ENV_ID="" SITE="" TRANSPORT="sdk" VERSION="$DEFAULT_VERSION" ENROL_JWT="" BOOTSTRAP_FILE="" NO_RUN="" FORCE_CONFIG="" BAO_CA="" LAPTOP="false" ENV_ID="" SITE="" TRANSPORT="sdk" DOCKER_DATA_ROOT="" TUNNEL_FAILED="" DEPLOYD_VERSION="$DEFAULT_VERSION" ENROL_JWT="" BOOTSTRAP_FILE="" NO_RUN="" FORCE_CONFIG="" BAO_CA="" LAPTOP="false"
usage() { sed -n '2,20p' "$0"; exit "${1:-0}"; } usage() { if [ -f "$0" ]; then sed -n '2,28p' "$0"; else echo "monky-deployd install.sh — see README.md (Install)"; fi; exit "${1:-0}"; }
die() { echo "install.sh: $*" >&2; exit 1; } die() { echo "install.sh: $*" >&2; exit 1; }
log() { echo "==> $*"; } log() { echo "==> $*"; }
@@ -45,12 +56,14 @@ while [ $# -gt 0 ]; do
--env) ENV_ID="$2"; shift 2 ;; --env) ENV_ID="$2"; shift 2 ;;
--site) SITE="$2"; shift 2 ;; --site) SITE="$2"; shift 2 ;;
--transport) TRANSPORT="$2"; shift 2 ;; --transport) TRANSPORT="$2"; shift 2 ;;
--version) VERSION="$2"; shift 2 ;; --version) DEPLOYD_VERSION="$2"; shift 2 ;;
--docker-data-root) DOCKER_DATA_ROOT="$2"; shift 2 ;;
--enrol-jwt) ENROL_JWT="$2"; shift 2 ;; --enrol-jwt) ENROL_JWT="$2"; shift 2 ;;
--bootstrap-file) BOOTSTRAP_FILE="$2"; shift 2 ;; --bootstrap-file) BOOTSTRAP_FILE="$2"; shift 2 ;;
--bao-ca) BAO_CA="$2"; shift 2 ;; --bao-ca) BAO_CA="$2"; shift 2 ;;
--base-url) BASE_URL="$2"; shift 2 ;; --base-url) BASE_URL="$2"; shift 2 ;;
--source) SOURCE="$2"; shift 2 ;; --source) SOURCE="$2"; shift 2 ;;
--token) TOKEN="$2"; shift 2 ;;
--laptop) LAPTOP="true"; shift ;; --laptop) LAPTOP="true"; shift ;;
--no-run) NO_RUN=1; shift ;; --no-run) NO_RUN=1; shift ;;
--force-config) FORCE_CONFIG=1; shift ;; --force-config) FORCE_CONFIG=1; shift ;;
@@ -67,13 +80,20 @@ SITE="${SITE,,}"
[[ "$SITE" =~ ^(cbs|pdx)$ ]] || die "site must be cbs|pdx" [[ "$SITE" =~ ^(cbs|pdx)$ ]] || die "site must be cbs|pdx"
[[ "$TRANSPORT" =~ ^(sdk|proxy|system)$ ]] || die "transport must be sdk|proxy|system" [[ "$TRANSPORT" =~ ^(sdk|proxy|system)$ ]] || die "transport must be sdk|proxy|system"
[[ "$SOURCE" =~ ^(gitlab|gitea)$ ]] || die "source must be gitlab|gitea" [[ "$SOURCE" =~ ^(gitlab|gitea)$ ]] || die "source must be gitlab|gitea"
# never let the token leak through xtrace / the environment of children
{ set +x; } 2>/dev/null
export -n MONKY_DEPLOYD_TOKEN 2>/dev/null || true
IDENTITY="$IDENTITY_DIR/monky-host.$ENV_ID.json" IDENTITY="$IDENTITY_DIR/monky-host.$ENV_ID.json"
export DEBIAN_FRONTEND=noninteractive export DEBIAN_FRONTEND=noninteractive
if [ -r /etc/os-release ]; then if [ -r /etc/os-release ]; then
. /etc/os-release # read os-release in a SUBSHELL: it defines VERSION/NAME/ID/... and sourcing it inline clobbered
if [ "${ID:-}" != "ubuntu" ] || [ "${VERSION_ID:-}" != "26.04" ]; then # this script's own variables (0.1.2 → "26.04 LTS (Resolute Raccoon)" on the env-qa-02 pilot)
echo "WARNING: verified on Ubuntu 26.04; this is ${PRETTY_NAME:-unknown}. Continuing." >&2 OS_ID=$(. /etc/os-release; printf '%s' "${ID:-}")
OS_VERSION_ID=$(. /etc/os-release; printf '%s' "${VERSION_ID:-}")
OS_CODENAME=$(. /etc/os-release; printf '%s' "${VERSION_CODENAME:-${UBUNTU_CODENAME:-}}")
if [ "$OS_ID" != "ubuntu" ] || [ "$OS_VERSION_ID" != "26.04" ]; then
echo "WARNING: verified on Ubuntu 26.04 only (this is ${OS_ID:-?} ${OS_VERSION_ID:-?}); continuing" >&2
fi fi
fi fi
@@ -93,7 +113,34 @@ fi
if [ "$TRANSPORT" = "proxy" ] && ! command -v ziti >/dev/null 2>&1; then if [ "$TRANSPORT" = "proxy" ] && ! command -v ziti >/dev/null 2>&1; then
need_pkgs+=(openziti) # the `ziti` CLI (ziti tunnel proxy) from the same repo need_pkgs+=(openziti) # the `ziti` CLI (ziti tunnel proxy) from the same repo
fi fi
command -v docker >/dev/null 2>&1 || die "docker is not installed; install Docker Engine first (https://docs.docker.com/engine/install/ubuntu/)" if ! command -v docker >/dev/null 2>&1; then
# Docker Engine from Docker's apt repo (the suite for this Ubuntu codename); falls back to Ubuntu's
# docker.io if Docker has no suite for the codename yet. --docker-data-root puts the data-root on
# a dedicated disk BEFORE the daemon first starts (pilot VMs mount one at /home/docker-data).
log "docker is not installed — installing Docker Engine"
export DEBIAN_FRONTEND=noninteractive
install -d -m 0755 /etc/apt/keyrings
if curl -fsSL --max-time 20 "https://download.docker.com/linux/ubuntu/dists/${OS_CODENAME:-noble}/Release" -o /dev/null 2>/dev/null; then
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg
chmod a+r /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu ${OS_CODENAME:-noble} stable" > /etc/apt/sources.list.d/docker.list
apt-get update -qq
docker_pkgs="docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin"
else
log "Docker has no apt suite for '${OS_CODENAME:-?}' yet — using Ubuntu's docker.io"
apt-get update -qq
docker_pkgs="docker.io docker-compose-v2"
fi
if [ -n "$DOCKER_DATA_ROOT" ]; then
install -d -m 0710 "$DOCKER_DATA_ROOT"
install -d -m 0755 /etc/docker
[ -f /etc/docker/daemon.json ] || printf '{ "data-root": "%s" }\n' "$DOCKER_DATA_ROOT" > /etc/docker/daemon.json
fi
# shellcheck disable=SC2086
apt-get install -y -qq --no-install-recommends $docker_pkgs
systemctl enable --now docker
log "installed $(docker --version) (data-root $(docker info -f '{{.DockerRootDir}}' 2>/dev/null || echo default))"
fi
docker compose version >/dev/null 2>&1 || need_pkgs+=(docker-compose-plugin) docker compose version >/dev/null 2>&1 || need_pkgs+=(docker-compose-plugin)
python3 -c 'import sys; sys.exit(0 if sys.version_info >= (3, 12) else 1)' 2>/dev/null || die "python3 >= 3.12 is required" python3 -c 'import sys; sys.exit(0 if sys.version_info >= (3, 12) else 1)' 2>/dev/null || die "python3 >= 3.12 is required"
if [ "${#need_pkgs[@]}" -gt 0 ]; then if [ "${#need_pkgs[@]}" -gt 0 ]; then
@@ -104,19 +151,28 @@ fi
# --- 2. the pinned .deb ----------------------------------------------------------------------------- # --- 2. the pinned .deb -----------------------------------------------------------------------------
installed="$(dpkg-query -W -f='${Version}' monky-deployd 2>/dev/null || true)" installed="$(dpkg-query -W -f='${Version}' monky-deployd 2>/dev/null || true)"
if [ "$installed" = "$VERSION" ]; then if [ "$installed" = "$DEPLOYD_VERSION" ]; then
log "monky-deployd $VERSION already installed" log "monky-deployd $DEPLOYD_VERSION already installed"
else else
tmp="$(mktemp -d)"; trap 'rm -rf "$tmp"' EXIT tmp="$(mktemp -d)"; trap 'rm -rf "$tmp"' EXIT
deb="monky-deployd_${VERSION}_amd64.deb" deb="monky-deployd_${DEPLOYD_VERSION}_amd64.deb"
# the deploy token travels in a 0600 curl config file (-K), never on the command line, so neither
# `ps` nor an xtrace shows it; the file dies with $tmp
curlrc="$tmp/curlrc"; : > "$curlrc"; chmod 0600 "$curlrc"
if [ "$SOURCE" = gitlab ]; then if [ "$SOURCE" = gitlab ]; then
url="${BASE_URL:-$GITLAB_BASE_URL}/${VERSION}" url="${BASE_URL:-$GITLAB_BASE_URL}/${DEPLOYD_VERSION}"
if [ -n "$TOKEN" ]; then
printf 'header = "DEPLOY-TOKEN: %s"\n' "$TOKEN" > "$curlrc"
else else
url="${BASE_URL:-$GITEA_BASE_URL}/releases/download/v${VERSION}" echo "WARNING: no --token/MONKY_DEPLOYD_TOKEN: the scm.tikali.ai project is private, the download will 401 (use --source gitea off-estate)" >&2
fi fi
log "downloading $deb from $url" else
curl -fsSL -o "$tmp/$deb" "$url/$deb" url="${BASE_URL:-$GITEA_BASE_URL}/releases/download/v${DEPLOYD_VERSION}"
curl -fsSL -o "$tmp/$deb.sha256" "$url/$deb.sha256" fi
log "downloading $deb from $url${TOKEN:+ (DEPLOY-TOKEN)}"
curl -fsSL -K "$curlrc" -o "$tmp/$deb" "$url/$deb" || die "download of $deb failed (private registry: is the deploy token set and valid?)"
curl -fsSL -K "$curlrc" -o "$tmp/$deb.sha256" "$url/$deb.sha256" || die "download of $deb.sha256 failed"
rm -f "$curlrc"
(cd "$tmp" && sha256sum -c "$deb.sha256") || die "sha256 mismatch on $deb" (cd "$tmp" && sha256sum -c "$deb.sha256") || die "sha256 mismatch on $deb"
apt_update_once || true apt_update_once || true
apt-get install -y -qq "$tmp/$deb" apt-get install -y -qq "$tmp/$deb"
@@ -138,20 +194,34 @@ chmod 0600 "$IDENTITY"
# ziti-edge-tunnel as a HOST (bind side, no tun/DNS) — the agent dials with the SDK or the proxy # ziti-edge-tunnel as a HOST (bind side, no tun/DNS) — the agent dials with the SDK or the proxy
install -d /etc/systemd/system/ziti-edge-tunnel.service.d install -d /etc/systemd/system/ziti-edge-tunnel.service.d
cat > /etc/systemd/system/ziti-edge-tunnel.service.d/run-host.conf <<'DROPIN' cat > /etc/systemd/system/ziti-edge-tunnel.service.d/run-host.conf <<'DROPIN'
# monky-deployd: run-host mode (no tun, no DNS); identities from the standard directory # monky-deployd: run-host mode (no tun, no DNS); identities from the standard directory.
# ExecStartPre is cleared: the package's ziti-edge-tunnel.sh (auto-enrol *.jwt in the identity dir)
# is not executable by user ziti on 1.18.x (203/EXEC, env-qa-02 pilot) and run-host does not need it.
[Service] [Service]
ExecStartPre=
ExecStart= ExecStart=
ExecStart=/opt/openziti/bin/ziti-edge-tunnel run-host --identity-dir=/opt/openziti/etc/identities ExecStart=/opt/openziti/bin/ziti-edge-tunnel run-host --identity-dir=/opt/openziti/etc/identities
DROPIN DROPIN
systemctl daemon-reload systemctl daemon-reload
systemctl enable --now ziti-edge-tunnel.service systemctl enable ziti-edge-tunnel.service >/dev/null 2>&1 || true
systemctl restart ziti-edge-tunnel.service || true if ! systemctl restart ziti-edge-tunnel.service; then
# do NOT abort here: config, ACLs, grant and timer below must land so a re-run needs no new kit
echo "WARNING: ziti-edge-tunnel.service failed to start — see: journalctl -u ziti-edge-tunnel; continuing" >&2
TUNNEL_FAILED=1
fi
# --- 4. config, ACLs, bootstrap grant ----------------------------------------------------------------- # --- 4. config, ACLs, bootstrap grant -----------------------------------------------------------------
install -d -m 0750 -o root -g monky-deployd "$ETC" install -d -m 0750 -o root -g monky-deployd "$ETC"
install -d -m 0700 -o monky-deployd -g monky-deployd /var/lib/monky-deployd install -d -m 0700 -o monky-deployd -g monky-deployd /var/lib/monky-deployd
# The identity file is the agent's only credential for the mesh. ziti-edge-tunnel rewrites it on
# every controller config update and the rewrite DROPS these ACLs, so the durable grant is group
# membership (the file stays <ziti>:<ziti> 0640 across rewrites); the ACLs below still cover the
# window before the agent's next login and hosts where the file has another group.
# A default ACL on the directory carries the grant onto a freshly created identity file.
if getent group ziti >/dev/null; then usermod -a -G ziti monky-deployd || true; fi
setfacl -m u:monky-deployd:r "$IDENTITY" setfacl -m u:monky-deployd:r "$IDENTITY"
setfacl -m u:monky-deployd:rx "$IDENTITY_DIR" setfacl -m u:monky-deployd:rx "$IDENTITY_DIR"
setfacl -d -m u:monky-deployd:r "$IDENTITY_DIR" 2>/dev/null || true
setfacl -m u:monky-deployd:x /opt/openziti/etc 2>/dev/null || true setfacl -m u:monky-deployd:x /opt/openziti/etc 2>/dev/null || true
if [ -n "$BAO_CA" ]; then if [ -n "$BAO_CA" ]; then
install -m 0644 "$BAO_CA" "$ETC/openbao-ca.pem" install -m 0644 "$BAO_CA" "$ETC/openbao-ca.pem"
@@ -162,7 +232,7 @@ if [ -s "$ETC/config.yaml" ] && [ -z "$FORCE_CONFIG" ]; then
log "keeping existing $ETC/config.yaml (use --force-config to rewrite)" log "keeping existing $ETC/config.yaml (use --force-config to rewrite)"
else else
cat > "$ETC/config.yaml" <<CFG cat > "$ETC/config.yaml" <<CFG
# written by install.sh $(date -u +%FT%TZ) — monky-deployd $VERSION # written by install.sh $(date -u +%FT%TZ) — monky-deployd $DEPLOYD_VERSION
env_id: $ENV_ID env_id: $ENV_ID
site: $SITE site: $SITE
transport: $TRANSPORT transport: $TRANSPORT
@@ -170,7 +240,7 @@ identity: $IDENTITY
tenancy: tenancy:
service: monky.tenancy.deploy service: monky.tenancy.deploy
host: monky.tenancy.deploy host: monky.tenancy.deploy
port: 8081 port: 443 # the service's INTERCEPT port (host.v1 forwards to 8081 inside the pod); plain HTTP inside the mesh
scheme: http scheme: http
proxy_addr: 127.0.0.1:18443 proxy_addr: 127.0.0.1:18443
bao: bao:
@@ -194,10 +264,11 @@ if [ "$TRANSPORT" = "proxy" ]; then
printf 'ZITI_IDENTITY=%s\n' "$IDENTITY" > "$ETC/proxy.env"; chmod 0644 "$ETC/proxy.env" printf 'ZITI_IDENTITY=%s\n' "$IDENTITY" > "$ETC/proxy.env"; chmod 0644 "$ETC/proxy.env"
systemctl enable --now monky-deployd-proxy.service systemctl enable --now monky-deployd-proxy.service
fi fi
# bootstrap grant: stdin (the kit pipes it) or --bootstrap-file; 0600, owned by the agent so it can consume it # bootstrap grant: --bootstrap-file, or stdin — but ONLY when this script runs from a file: with
# `curl … | bash -s --` stdin IS the script, so reading it here would swallow the rest of it
grant="" grant=""
if [ -n "$BOOTSTRAP_FILE" ]; then grant="$(tr -d '[:space:]' < "$BOOTSTRAP_FILE")" if [ -n "$BOOTSTRAP_FILE" ]; then grant="$(tr -d '[:space:]' < "$BOOTSTRAP_FILE")"
elif [ ! -t 0 ]; then grant="$(tr -d '[:space:]' </dev/stdin || true)"; fi elif [ ! -t 0 ] && [ -f "$0" ]; then grant="$(tr -d '[:space:]' </dev/stdin || true)"; fi
if [ -n "$grant" ]; then if [ -n "$grant" ]; then
printf '%s\n' "$grant" > "$ETC/bootstrap.jwt" printf '%s\n' "$grant" > "$ETC/bootstrap.jwt"
chown monky-deployd:monky-deployd "$ETC/bootstrap.jwt"; chmod 0600 "$ETC/bootstrap.jwt" chown monky-deployd:monky-deployd "$ETC/bootstrap.jwt"; chmod 0600 "$ETC/bootstrap.jwt"
@@ -207,7 +278,7 @@ elif [ -s /var/lib/monky-deployd/bao.token ]; then
else else
echo "WARNING: no bootstrap grant on stdin and no bao.token: the agent cannot check in until you provide one" >&2 echo "WARNING: no bootstrap grant on stdin and no bao.token: the agent cannot check in until you provide one" >&2
fi fi
unset grant unset grant TOKEN
# --- 5. timer + first tick ---------------------------------------------------------------------------------- # --- 5. timer + first tick ----------------------------------------------------------------------------------
systemctl daemon-reload systemctl daemon-reload
@@ -221,7 +292,7 @@ rm -f "$ETC/enrol.jwt"
cat <<CHECK cat <<CHECK
monky-deployd $VERSION installed for $ENV_ID ($SITE, transport $TRANSPORT). Verification checklist (Ubuntu 26.04): monky-deployd $DEPLOYD_VERSION installed for $ENV_ID ($SITE, transport $TRANSPORT). Verification checklist (Ubuntu 26.04):
python3 --version # >= 3.12 python3 --version # >= 3.12
docker compose version # compose plugin present docker compose version # compose plugin present
systemctl is-active ziti-edge-tunnel # run-host mode (drop-in run-host.conf) systemctl is-active ziti-edge-tunnel # run-host mode (drop-in run-host.conf)
@@ -235,3 +306,7 @@ monky-deployd $VERSION installed for $ENV_ID ($SITE, transport $TRANSPORT). Veri
CHECK CHECK
CHECK_STATUS="$(monky-deployd status 2>&1 || true)" CHECK_STATUS="$(monky-deployd status 2>&1 || true)"
echo "$CHECK_STATUS" | sed 's/^/ | /' echo "$CHECK_STATUS" | sed 's/^/ | /'
if [ -n "$TUNNEL_FAILED" ] && ! systemctl is-active --quiet ziti-edge-tunnel.service; then
echo "ERROR: ziti-edge-tunnel.service is not running; the agent cannot reach the mesh until it is. Everything else is installed — fix the unit and 'systemctl restart ziti-edge-tunnel monky-deployd'." >&2
exit 1
fi
+5
View File
@@ -13,6 +13,11 @@ install -d -m 0700 -o monky-deployd -g monky-deployd /var/lib/monky-deployd
install -d -m 0750 -o root -g monky-deployd /etc/monky-deployd install -d -m 0750 -o root -g monky-deployd /etc/monky-deployd
# the agent drives docker compose: docker group membership (no root) # the agent drives docker compose: docker group membership (no root)
if getent group docker >/dev/null; then usermod -a -G docker monky-deployd || true; fi if getent group docker >/dev/null; then usermod -a -G docker monky-deployd || true; fi
# the agent reads the host's ziti identity (owned by the tunneller, mode 0640). ziti-edge-tunnel
# REWRITES that file whenever the controller sends a config update, and the rewrite drops any
# POSIX ACL granting the agent read — group membership is the grant that survives it.
# (env-dev-08, 2026-09-09: the agent went from applied to "no intercept" 6 minutes after a refresh.)
if getent group ziti >/dev/null; then usermod -a -G ziti monky-deployd || true; fi
# the venv is relocatable only to the path it was built at; refuse a broken interpreter early # the venv is relocatable only to the path it was built at; refuse a broken interpreter early
/opt/monky-deployd/venv/bin/python -c 'import monky_deployd' || { echo "monky-deployd: venv unusable (python3 mismatch?)" >&2; exit 1; } /opt/monky-deployd/venv/bin/python -c 'import monky_deployd' || { echo "monky-deployd: venv unusable (python3 mismatch?)" >&2; exit 1; }
if [ -d /run/systemd/system ]; then if [ -d /run/systemd/system ]; then
+1 -1
View File
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project] [project]
name = "monky-deployd" name = "monky-deployd"
version = "0.1.1" version = "0.1.8"
description = "Monky backend pull agent: checkin -> bundle -> lease -> OpenBao -> docker compose -> report, over the ziti mesh (MONKY-ADR-0028)" description = "Monky backend pull agent: checkin -> bundle -> lease -> OpenBao -> docker compose -> report, over the ziti mesh (MONKY-ADR-0028)"
readme = "README.md" readme = "README.md"
requires-python = ">=3.12" requires-python = ">=3.12"
+3
View File
@@ -52,6 +52,9 @@ def fake_docker(tmp_path, monkeypatch):
} }
) )
) )
# the real agent also measures containerd's root; on a CI runner that path belongs to the
# runner's own docker and would make every disk assertion depend on the runner's free space.
monkeypatch.setattr("monky_deployd.compose.CONTAINERD_ROOTS", ())
monkeypatch.setenv("PATH", f"{FAKEBIN}:{os.environ['PATH']}") monkeypatch.setenv("PATH", f"{FAKEBIN}:{os.environ['PATH']}")
monkeypatch.setenv("FAKE_DOCKER_LOG", str(log)) monkeypatch.setenv("FAKE_DOCKER_LOG", str(log))
monkeypatch.setenv("FAKE_DOCKER_STATE", str(state)) monkeypatch.setenv("FAKE_DOCKER_STATE", str(state))
+6 -3
View File
@@ -190,9 +190,11 @@ class FakeBao:
return h._send(400, {"errors": ["error validating token: expired"]}) return h._send(400, {"errors": ["error validating token: expired"]})
if body.get("role") != self.role: if body.get("role") != self.role:
return h._send(400, {"errors": [f"role {body.get('role')!r} could not be found"]}) return h._send(400, {"errors": [f"role {body.get('role')!r} could not be found"]})
meta = self.grants.pop(body.get("jwt", ""), None) # single-use grant # the jwt mount keeps no jti/replay state: a known grant logs in until it expires;
# a SUPERSEDED grant is refused by tenancy (FakeTenancy.superseded_jtis), not here
meta = self.grants.get(body.get("jwt", ""))
if meta is None: if meta is None:
return h._send(400, {"errors": ["error validating token: unknown or already used grant"]}) return h._send(400, {"errors": ["error validating token: unknown grant"]})
m = json.loads(meta) m = json.loads(meta)
tok = self.mint(m["env_id"]) tok = self.mint(m["env_id"])
self.tokens[tok]["grant_jti"] = m["jti"] self.tokens[tok]["grant_jti"] = m["jti"]
@@ -354,7 +356,8 @@ class FakeTenancy:
"ttl_s": 3600, "ttl_s": 3600,
"mount": self.bao.mount, "mount": self.bao.mount,
"role": self.bao.role, "role": self.bao.role,
"vault": {"addr": "https://bao.cbs.tikali.net:8200", "mount": self.kv_mount}, # tenancy AgentLeaseOut: `addr` is top-level; KV mount/prefix come from checkin
"addr": "https://bao.cbs.tikali.net:8200",
}, },
headers={"Cache-Control": "no-store"}, headers={"Cache-Control": "no-store"},
) )
+52 -5
View File
@@ -107,7 +107,7 @@ def test_rollback_refused_unless_allowed(bootstrapped, tenancy, fake_docker):
tenancy.set_files(first) tenancy.set_files(first)
assert tick(cfg) == EX_FAIL assert tick(cfg) == EX_FAIL
assert tenancy.reports[-1]["result"] == "failed" and "ROLLBACK_REFUSED" in tenancy.reports[-1]["detail"] assert tenancy.reports[-1]["result"] == "failed" and "ROLLBACK_REFUSED" in tenancy.reports[-1]["detail"]
assert len(tenancy.leases) == 2 # a refused bundle never leases assert len(tenancy.leases) == 1 # a refused bundle never leases; the 2nd apply reused the lease token
tenancy.set_files(make_files(meta={"agent": {"allow_rollback": True}})) tenancy.set_files(make_files(meta={"agent": {"allow_rollback": True}}))
assert tick(cfg) == EX_OK assert tick(cfg) == EX_OK
@@ -237,6 +237,30 @@ def test_disk_refusal(bootstrapped, tenancy, fake_docker):
assert tenancy.leases == [] assert tenancy.leases == []
def test_disk_refusal_without_a_declared_size(bootstrapped, tenancy, fake_docker, monkeypatch):
"""A bundle that declares no size still has to clear the headroom floor.
Without the floor a nearly-full box passes the pre-flight and dies mid-pull with
containerd's "no space left on device", which reads as a registry fault (env-dev-08).
"""
cfg = bootstrapped
cfg.disk.headroom_bytes = 2 * 1024**3
monkeypatch.setattr("monky_deployd.compose.Docker.free_bytes", lambda self, path=None: 32 * 2**20)
tenancy.set_files(make_files())
assert tick(cfg) == EX_FAIL
detail = tenancy.reports[-1]["detail"]
assert "DISK_INSUFFICIENT" in detail and "declares no size" in detail
assert tenancy.leases == []
def test_ample_disk_without_a_declared_size_applies(bootstrapped, tenancy, fake_docker, monkeypatch):
cfg = bootstrapped
monkeypatch.setattr("monky_deployd.compose.Docker.free_bytes", lambda self, path=None: 500 * 2**30)
tenancy.set_files(make_files())
assert tick(cfg) == EX_OK
assert tenancy.reports[-1]["result"] == "applied"
def test_legacy_approle_lease_is_refused_loudly(bootstrapped, tenancy, fake_docker): def test_legacy_approle_lease_is_refused_loudly(bootstrapped, tenancy, fake_docker):
cfg = bootstrapped cfg = bootstrapped
tenancy.lease_shape = "approle" tenancy.lease_shape = "approle"
@@ -246,10 +270,9 @@ def test_legacy_approle_lease_is_refused_loudly(bootstrapped, tenancy, fake_dock
assert "compose up" not in fake_docker.subcommands() assert "compose up" not in fake_docker.subcommands()
def test_lease_rate_limited_is_temporary(bootstrapped, tenancy, fake_docker): # (test_lease_rate_limited_is_temporary was retired in 0.1.6: a rate-limited lease is only a
cfg = bootstrapped # temporary failure when NO working token exists; with one, the agent applies and defers the swap —
tenancy.lease_limit = 0 # see test_rate_limited_lease_does_not_block_an_apply_when_a_token_exists.)
assert tick(cfg) == EX_TEMPFAIL
def test_unhealthy_after_up_reports_failed_with_compose_logs(bootstrapped, tenancy, fake_docker): def test_unhealthy_after_up_reports_failed_with_compose_logs(bootstrapped, tenancy, fake_docker):
@@ -320,3 +343,27 @@ def test_write_private_mode(tmp_path):
p = tmp_path / "d" / "f" p = tmp_path / "d" / "f"
statemod.write_private(p, b"x") statemod.write_private(p, b"x")
assert oct(p.stat().st_mode & 0o777) == "0o600" and not any(n.startswith(".f.") for n in os.listdir(p.parent)) assert oct(p.stat().st_mode & 0o777) == "0o600" and not any(n.startswith(".f.") for n in os.listdir(p.parent))
def test_second_apply_reuses_the_lease_token(bootstrapped, tenancy, bao, fake_docker):
"""A valid lease-derived token is reused: a new bundle does NOT lease again (5/h budget —
env-qa-02 pilot: a retried deploy re-leased every 60 s and hit LEASE_RATE_LIMITED forever)."""
cfg = bootstrapped
assert tick(cfg) == EX_OK
assert len(tenancy.leases) == 1
tenancy.set_files(make_files(manifest=make_manifest(versions={"gemini_api_key": 1})))
assert tick(cfg) == EX_OK
assert len(tenancy.leases) == 1 # reused
st = statemod.load(cfg.state_path, ENV)
assert st.applied_sha == tenancy.desired_sha and st.token.source == "lease"
def test_rate_limited_lease_does_not_block_an_apply_when_a_token_exists(bootstrapped, tenancy, bao, fake_docker):
"""tenancy 429 on lease while the bootstrap token still works → apply with it, swap deferred."""
cfg = bootstrapped
tenancy.lease_limit = 0
assert tick(cfg) == EX_OK
assert tenancy.leases == []
st = statemod.load(cfg.state_path, ENV)
assert st.applied_sha == tenancy.desired_sha and st.token.source == "bootstrap"
assert tenancy.reports[-1]["result"] == "applied"
+13
View File
@@ -79,3 +79,16 @@ def test_disk_need_bytes_spellings():
assert b.parse(tar_bytes(make_files(meta={"agent": {"disk_need_bytes": 5}}))).disk_need_bytes == 5 assert b.parse(tar_bytes(make_files(meta={"agent": {"disk_need_bytes": 5}}))).disk_need_bytes == 5
assert b.parse(tar_bytes(make_files(meta={"disk": {"need_bytes": 7}}))).disk_need_bytes == 7 assert b.parse(tar_bytes(make_files(meta={"disk": {"need_bytes": 7}}))).disk_need_bytes == 7
assert b.parse(tar_bytes(make_files())).disk_need_bytes == 0 assert b.parse(tar_bytes(make_files())).disk_need_bytes == 0
def test_placeholders_in_comment_lines_are_not_references():
"""The renderer's .env.template header says '... substitutes every ${VAR} ...' — that must not
become an unresolved 'VAR' (env-qa-02 pilot: ENV_INCOMPLETE: unresolved: VAR)."""
from monky_deployd.bundle import defaulted_vars, referenced_vars
text = (
"# The on-box agent substitutes every ${VAR} from OpenBao per secrets.manifest.json.\n"
" # ${ALSO_COMMENT}\nGEMINI_API_KEY=${GEMINI_API_KEY}\nPG=${PGPASSWORD:-x}\n"
)
assert referenced_vars(text) == {"GEMINI_API_KEY", "PGPASSWORD"}
assert defaulted_vars(text) == {"PGPASSWORD"}
+3 -1
View File
@@ -79,6 +79,8 @@ def test_sdk_transport_uses_openziti_monkeypatch(monkeypatch, tmp_path):
srv = socket.socket() srv = socket.socket()
srv.bind(("127.0.0.1", 0)) srv.bind(("127.0.0.1", 0))
srv.listen(1) srv.listen(1)
# the transport pre-flights the identity file before handing it to the SDK
(tmp_path / "id.json").write_text('{"ztAPI": "https://example.invalid"}')
t = SdkTransport(str(tmp_path / "id.json")) t = SdkTransport(str(tmp_path / "id.json"))
s = t.connect("127.0.0.1", srv.getsockname()[1], 2) s = t.connect("127.0.0.1", srv.getsockname()[1], 2)
s.close() s.close()
@@ -93,7 +95,7 @@ def test_proxy_transport_refuses_unmapped_hosts():
cfg = from_dict({"env_id": "env-dev-06", "site": "cbs", "transport": "proxy"}) cfg = from_dict({"env_id": "env-dev-06", "site": "cbs", "transport": "proxy"})
t = build(cfg) t = build(cfg)
assert t.mapping[("monky.tenancy.deploy", 8081)] == ("127.0.0.1", 18443) assert t.mapping[("monky.tenancy.deploy", 443)] == ("127.0.0.1", 18443)
assert t.mapping[("bao.cbs.tikali.net", 8200)] == ("127.0.0.1", 18200) assert t.mapping[("bao.cbs.tikali.net", 8200)] == ("127.0.0.1", 18200)
try: try:
t.connect("example.com", 443, 1) t.connect("example.com", 443, 1)
+3 -3
View File
@@ -10,7 +10,7 @@ transport: sdk
identity: /opt/openziti/etc/identities/monky-host.env-qa-02.json identity: /opt/openziti/etc/identities/monky-host.env-qa-02.json
tenancy: tenancy:
service: monky.tenancy.deploy service: monky.tenancy.deploy
base_url: http://monky.tenancy.deploy:8081 base_url: http://monky.tenancy.deploy:443
bao: bao:
service: openbao service: openbao
addr: https://bao.cbs.tikali.net:8200 # intercept, not public DNS addr: https://bao.cbs.tikali.net:8200 # intercept, not public DNS
@@ -27,7 +27,7 @@ volumes_on_absent: keep
def test_yaml_subset_parses_nested_maps_and_types(): def test_yaml_subset_parses_nested_maps_and_types():
d = c.parse_yaml_subset(KIT) d = c.parse_yaml_subset(KIT)
assert d["env_id"] == "env-qa-02" assert d["env_id"] == "env-qa-02"
assert d["tenancy"]["base_url"] == "http://monky.tenancy.deploy:8081" assert d["tenancy"]["base_url"] == "http://monky.tenancy.deploy:443"
assert d["bao"]["addr"] == "https://bao.cbs.tikali.net:8200" assert d["bao"]["addr"] == "https://bao.cbs.tikali.net:8200"
assert d["interval_s"] == 60 and d["laptop_mode"] is False assert d["interval_s"] == 60 and d["laptop_mode"] is False
@@ -46,7 +46,7 @@ def test_yaml_subset_refuses_flow_style_and_tabs():
def test_config_defaults_and_derivations(): def test_config_defaults_and_derivations():
cfg = c.from_dict(c.parse_yaml_subset(KIT)) cfg = c.from_dict(c.parse_yaml_subset(KIT))
assert cfg.tenancy.host == "monky.tenancy.deploy" and cfg.tenancy.port == 8081 and cfg.tenancy.scheme == "http" assert cfg.tenancy.host == "monky.tenancy.deploy" and cfg.tenancy.port == 443 and cfg.tenancy.scheme == "http"
assert cfg.bao_url == ("https", "bao.cbs.tikali.net", 8200) assert cfg.bao_url == ("https", "bao.cbs.tikali.net", 8200)
assert cfg.deploy_dir == "/var/lib/monky-deployd/env-qa-02" assert cfg.deploy_dir == "/var/lib/monky-deployd/env-qa-02"
assert cfg.compose_project == "monky-env-qa-02" assert cfg.compose_project == "monky-env-qa-02"
+53
View File
@@ -0,0 +1,53 @@
"""The registry credential (doc 24 §4a): parsed from either seeded shape, written to a Docker
config the AGENT owns, never an env var, never logged."""
from __future__ import annotations
import base64
import json
import pytest
from monky_deployd import registry as registrymod
def test_parses_json_and_user_colon_password():
a = registrymod.parse(
json.dumps({"registry": "harbor.tikali.net", "username": "robot$pull", "password": "p4ss"}),
default_registry="ignored.example",
)
assert (a.registry, a.username, a.password) == ("harbor.tikali.net", "robot$pull", "p4ss")
b = registrymod.parse("robot$pull:p4ss", default_registry="harbor.tikali.net")
assert (b.registry, b.username, b.password) == ("harbor.tikali.net", "robot$pull", "p4ss")
# a password containing a colon survives (partition on the FIRST one)
c = registrymod.parse("robot$pull:p4:ss", default_registry="h")
assert c.password == "p4:ss"
@pytest.mark.parametrize("bad", ["", " ", "no-colon-here", '{"username": "u"}', "{not json"])
def test_refuses_what_it_cannot_use(bad):
with pytest.raises(registrymod.RegistryAuthError):
registrymod.parse(bad, default_registry="harbor.tikali.net")
def test_docker_config_is_written_0600_in_a_directory_the_agent_owns(tmp_path):
auth = registrymod.parse("robot$pull:p4ss", default_registry="harbor.tikali.net")
target = registrymod.write_docker_config(tmp_path / "docker", auth)
assert target.exists()
assert oct(target.stat().st_mode)[-3:] == "600"
assert oct(target.parent.stat().st_mode)[-3:] == "700"
cfg = json.loads(target.read_text())
token = cfg["auths"]["harbor.tikali.net"]["auth"]
assert base64.b64decode(token).decode() == "robot$pull:p4ss"
# rewriting is idempotent (the agent does it every tick)
registrymod.write_docker_config(tmp_path / "docker", auth)
assert json.loads(target.read_text()) == cfg
def test_docker_runner_points_the_cli_at_that_directory():
from monky_deployd.compose import Docker
d = Docker("docker", docker_config="/var/lib/monky-deployd/docker")
assert d._env()["DOCKER_CONFIG"] == "/var/lib/monky-deployd/docker"
# unset when no directory is configured — never silently fall back to a human's $HOME
assert "DOCKER_CONFIG" not in Docker("docker")._env()
+59
View File
@@ -0,0 +1,59 @@
"""Free-space accounting across the filesystems a pull can fill.
docker 29 keeps image layers in containerd's store, which is frequently on a different
filesystem than DockerRootDir. env-dev-08 (2026-09-09) had 93 GiB free on the data-root and
2.8 GiB on the filesystem containerd was writing to; the pull died with "no space left on
device" after the pre-flight had reported plenty of room.
"""
from __future__ import annotations
import monky_deployd.compose as compose_mod
from monky_deployd.compose import Docker
class _FixedDocker(Docker):
def __init__(self, root: str):
super().__init__()
self._root = root
def data_root(self) -> str:
return self._root
def test_storage_paths_include_containerd_root_when_present(tmp_path, monkeypatch):
root = tmp_path / "docker"
containerd = tmp_path / "containerd"
root.mkdir()
containerd.mkdir()
monkeypatch.setattr(compose_mod, "CONTAINERD_ROOTS", (str(containerd),))
assert _FixedDocker(str(root)).storage_paths() == [str(root), str(containerd)]
def test_storage_paths_skip_a_containerd_root_that_does_not_exist(tmp_path, monkeypatch):
root = tmp_path / "docker"
root.mkdir()
monkeypatch.setattr(compose_mod, "CONTAINERD_ROOTS", (str(tmp_path / "absent"),))
assert _FixedDocker(str(root)).storage_paths() == [str(root)]
def test_free_bytes_reports_the_tightest_filesystem(tmp_path, monkeypatch):
root = tmp_path / "docker"
containerd = tmp_path / "containerd"
root.mkdir()
containerd.mkdir()
monkeypatch.setattr(compose_mod, "CONTAINERD_ROOTS", (str(containerd),))
d = _FixedDocker(str(root))
monkeypatch.setattr(d, "_free_at", lambda p: 900 if p == str(root) else 5)
assert d.free_bytes() == 5
# an explicit path is still measured on its own
assert d.free_bytes(str(root)) == 900
def test_free_bytes_is_none_when_nothing_can_be_measured(tmp_path, monkeypatch):
root = tmp_path / "docker"
root.mkdir()
monkeypatch.setattr(compose_mod, "CONTAINERD_ROOTS", ())
d = _FixedDocker(str(root))
monkeypatch.setattr(d, "_free_at", lambda p: None)
assert d.free_bytes() is None
+54
View File
@@ -0,0 +1,54 @@
"""The SDK transport names an unusable identity file instead of failing at dial.
openziti.load() accepts an unreadable or malformed identity, logs "configuration is invalid"
at the C layer and returns a context that only fails when something dials through it — as a
bare TypeError that reads like a missing intercept. env-dev-08 (2026-09-09) lost its identity
ACL to a tunneller rewrite and spent every tick reporting a mesh fault it did not have.
"""
from __future__ import annotations
import json
import pytest
from monky_deployd.transport import SdkTransport, TransportError
def _sdk(path):
t = SdkTransport(str(path))
return t
def test_missing_identity_is_named(tmp_path):
with pytest.raises(TransportError) as e:
_sdk(tmp_path / "absent.json")._check_identity_readable()
assert "does not exist" in str(e.value)
def test_malformed_identity_is_named(tmp_path):
p = tmp_path / "id.json"
p.write_text("{not json")
with pytest.raises(TransportError) as e:
_sdk(p)._check_identity_readable()
assert "not valid JSON" in str(e.value)
def test_unreadable_identity_points_at_the_group_grant(tmp_path, monkeypatch):
p = tmp_path / "id.json"
p.write_text(json.dumps({"ztAPI": "https://example.invalid"}))
def _denied(*a, **k):
raise PermissionError(13, "Permission denied")
monkeypatch.setattr("builtins.open", _denied)
with pytest.raises(TransportError) as e:
_sdk(p)._check_identity_readable()
msg = str(e.value)
assert "not readable" in msg and "group" in msg
def test_a_good_identity_passes(tmp_path):
p = tmp_path / "id.json"
p.write_text(json.dumps({"ztAPI": "https://example.invalid"}))
_sdk(p)._check_identity_readable() # no raise