Merge branch 'fix/reuse-lease-token' into 'main'

fix(agent): reuse a valid lease token; rate-limited lease no longer blocks a deploy; 0.1.6

See merge request tikali/applications/monky/monky-deployd!11

CHANGELOG.md

@@ -1,8 +1,20 @@
 <!-- xlate:verbatim-fences -->
 # Changelog
 
+## 0.1.6 — 2026-09-07
+
+- **A valid lease token is reused across applies.** Every apply requested a new lease, so a failing deploy
+  retried by the 60 s timer burned tenancy's 5-leases-per-hour budget and then failed on
+  `LEASE_RATE_LIMITED` forever (env-qa-02 pilot). Now: reuse a lease-derived token while `lookup-self`
+  says it is valid; swap the bootstrap token for a lease once; and if tenancy rate-limits the lease while a
+  working token exists, apply with it and defer the swap instead of failing the deploy.
+
 ## 0.1.5 — 2026-09-07
 
+- **`tenancy.port` defaults to 443 everywhere** — the ansible role default and the `TenancyCfg` default now
+  match `install.sh`/`config.example.yaml` (0.1.5 only changed those two); PROTOCOL.md states the intercept
+  port separately from the in-pod 8081, that the broker does not add `#monky-deploy-agent` yet, that the jwt
+  mount keeps no `jti` state, and that `release:gitea` has not run (doc-drift DD-0523/0525/0527/0528).
 - **`${VAR}` inside comment lines is not a reference.** The renderer's `.env.template` header literally says
   "substitutes every ${VAR}", which the refusal check counted as an unresolved variable
   (`ENV_INCOMPLETE: unresolved: VAR`) — the first bundle on env-qa-02 was refused for it.

README.md

@@ -23,9 +23,9 @@ you the enrolment JWT, a one-time **bootstrap deploy grant** and the read-only *
 
 ```sh
 T=<deploy token>   # read-only GitLab deploy token (read_package_registry); the kit carries it
-curl -sSf -H "DEPLOY-TOKEN: $T" https://scm.tikali.ai/api/v4/projects/69/packages/generic/monky-deployd/0.1.5/install.sh \
+curl -sSf -H "DEPLOY-TOKEN: $T" https://scm.tikali.ai/api/v4/projects/69/packages/generic/monky-deployd/0.1.6/install.sh \
   | sudo bash -s -- --env env-qa-02 --site cbs --token "$T" --bootstrap-file bootstrap.jwt --enrol-jwt ./monky-host.env-qa-02.jwt
-#   [--transport sdk|proxy|system] [--version 0.1.5] [--docker-data-root /home/docker-data] [--laptop] [--bao-ca openbao-ca.pem] [--source gitlab|gitea]
+#   [--transport sdk|proxy|system] [--version 0.1.6] [--docker-data-root /home/docker-data] [--laptop] [--bao-ca openbao-ca.pem] [--source gitlab|gitea]
 ```
 
 `install.sh` installs `ziti-edge-tunnel` (OpenZiti `jammy` suite) and `docker-compose-plugin` if
@@ -98,11 +98,13 @@ monky-deployd version
 ## Security model
 
 - **Identity = the box's ziti host identity.** Only identities with `#monky-deploy-agent` can dial
-  tenancy's agent entrypoint; `#openbao-client` reaches OpenBao. The agent reads the identity
+  tenancy's agent entrypoint; `#openbao-client` reaches OpenBao — tenancy sends both to the broker
+  as `host_extra_attrs` at register (monky-tenancy !29 / monky-ziti `ac258352`, 2026-09-07), so a
+  backend registered since then dials without hand-added attrs (see PROTOCOL.md §Where and how). The agent reads the identity
   through an ACL (`setfacl -m u:monky-deployd:r`), never owns it.
 - **The bearer to tenancy is the agent's own OpenBao token**, minted by OpenBao from a
-  tenancy-signed ES256 deploy grant (`aud openbao-see-env`, `kind deploy-grant`, 1 h, single-use
-  `jti`). Tenancy verifies it with `auth/token/lookup`, pins `meta.env_id`, and refuses a token
+  tenancy-signed ES256 deploy grant (`aud openbao-see-env`, `kind deploy-grant`, 1 h; its `jti`
+  becomes the token's `meta.grant_jti` — the mount keeps no replay state, tenancy does). Tenancy verifies it with `auth/token/lookup`, pins `meta.env_id`, and refuses a token
   whose `meta.grant_jti` was superseded (kit re-reveal, retire) → `401 AGENT_UNAUTHENTICATED`.
   **No AppRole, nothing to unwrap** (Gate 1 result, 2026-09-05).
 - **The agent never receives a secret from tenancy.** Bundles carry placeholders; the agent reads
@@ -160,7 +162,9 @@ blocking on `main`/tags (manual on MRs); the `.deb` always ships the SDK wheel.
 `release` uploads to the GitLab generic package registry + release (**the primary download**; the
 project is private, so the installer sends the read-only deploy token), and `release:gitea` publishes the same assets on the Gitea mirror (the `--source gitea`
 alternative; automatic when `GITEA_TOKEN` is set, manual otherwise — see `docs/OPERATIONS.md` for
-the by-hand recipe). Both locations keep being published.
+the by-hand recipe). The GitLab registry is published by every tag pipeline; the Gitea mirror only
+when `release:gitea` runs — automatically once `GITEA_TOKEN` is set in CI, by hand otherwise — so
+check the Gitea release page before pointing an installer at it.
 
 ## See also
 

ansible/roles/monky_deployd/README.md

@@ -16,7 +16,7 @@ template → optional openbao-ca PEM → optional `monky-deployd-proxy.service`
 
 | var | note |
 |---|---|
-| `monky_deployd_version` | pinned release, e.g. `0.1.5` |
+| `monky_deployd_version` | pinned release, e.g. `0.1.6` |
 | `monky_deployd_download_token` | **vaulted**: GitLab deploy token, scope `read_package_registry` only (revocable) — the registry is private; seeded in OpenBao at `monky/monky-tenancy/deployd-download` key `token` (path/key are the operator's choice). Empty = no header (only works with the Gitea `base_url`) |
 | `monky_deployd_env_id` / `_site` / `_transport` | per host (`env-dev-06`, `cbs`, `sdk`) |
 | `monky_deployd_bootstrap_grant` | tenancy-minted deploy grant (1 h) — `ansible-vault` or a lookup at play time; empty keeps the existing token |

ansible/roles/monky_deployd/defaults/main.yml

@@ -1,7 +1,7 @@
 ---
 # monky_deployd — install and configure the Monky backend pull agent (MONKY-ADR-0028 §D).
 # Copy this role into osg1-07 (roles/monky_deployd) and roll to env-dev-06..09 after the pilot.
-monky_deployd_version: "0.1.5"
+monky_deployd_version: "0.1.6"
 monky_deployd_deb: "monky-deployd_{{ monky_deployd_version }}_amd64.deb"
 # PRIMARY download = the GitLab project's generic package registry on scm.tikali.ai. Inside the estate
 # gitea.cbs.tikali.net is split-horizon to jump1's RED EIP (10.10.0.175, no HTTP ingress), so backend
@@ -33,7 +33,7 @@ monky_deployd_disk_factor: 1.5
 monky_deployd_disk_headroom_bytes: 2147483648
 
 monky_deployd_tenancy_service: monky.tenancy.deploy
-monky_deployd_tenancy_port: 8081
+monky_deployd_tenancy_port: 443   # the intercept port of monky.tenancy.deploy (host.v1 forwards to 8081 in the pod)
 monky_deployd_bao_addr: "https://bao.cbs.tikali.net:8200"
 monky_deployd_bao_auth_mount: jwt-tenancy
 monky_deployd_bao_role: see-env

config.example.yaml

@@ -1,4 +1,4 @@
-# /etc/monky-deployd/config.yaml — monky-deployd v0.1.5 (MONKY-ADR-0028 §D)
+# /etc/monky-deployd/config.yaml — monky-deployd v0.1.6 (MONKY-ADR-0028 §D)
 # Written by packaging/install.sh (or the ansible role monky_deployd). YAML *subset*: maps, scalars,
 # simple lists, comments. Keys not listed here are a config error.
 

docs/OPERATIONS.md

@@ -108,7 +108,8 @@ download; xtrace is switched off). A 401 on the download means the token is miss
 lacks the scope. Inside the estate `gitea.cbs.tikali.net` is split-horizon to jump1's RED EIP
 (`10.10.0.175`), which has no HTTP ingress, so backend boxes cannot fetch from the Gitea mirror
 (cbs/iac#102). Off-estate, `install.sh --source gitea` (ansible: `monky_deployd_base_url`/`_deb_url`,
-no token) uses the Gitea release instead. The tag pipeline publishes to both (`release`, `release:gitea`).
+no token) uses the Gitea release instead. The tag pipeline always publishes to GitLab (`release`); `release:gitea` is automatic only with
+`GITEA_TOKEN` in CI and a manual job otherwise — it has not run on a tag pipeline yet.
 
 ## Publishing a release to Gitea by hand
 

docs/PROTOCOL.md

@@ -10,8 +10,15 @@ The agent side of the backend lifecycle agent protocol — [MONKY-ADR-0028](http
 
 - **Where:** tenancy's **agent entrypoint** — a second container (`app.agent_main`,
   `127.0.0.1:8081`) bound to the ziti service **`monky.tenancy.deploy`** by a `ziti-edge-tunnel
-  run-host` sidecar. Unreachable from the public ingress. Only host identities carrying
-  `#monky-deploy-agent` may dial (the broker adds the attr when the identity is created at kit reveal).
+  run-host` sidecar. Unreachable from the public ingress. **The agent dials the service's
+  INTERCEPT port, `tcp/443`** (`monky.tenancy.deploy.intercept.v1` in openziti
+  `state/overlay/configs.json` — the authority for the port); the sidecar's `host.v1` forwards to
+  `127.0.0.1:8081` inside the pod. `tenancy.port` in the config, `install.sh` and the ansible role all
+  default to 443 (0.1.5). Only host identities carrying `#monky-deploy-agent` may dial
+  (`monky.tenancy.deploy.dial`, openziti). Tenancy sends that attr and `#openbao-client` to the
+  broker as `host_extra_attrs` when it registers an agent-backed backend (monky-tenancy !29,
+  monky-ziti `ac258352` — both 2026-09-07), so `monky-host.<env>` carries them from creation;
+  identities created before that (env-qa-02's) had them added on the controller by hand.
 - **Transport:** plain HTTP inside the mesh (`transport: sdk` dials by service name;
   `proxy` → `127.0.0.1:18443`). OpenBao is the existing `openbao` ziti service (`#openbao-client`),
   dialled as `https://bao.cbs.tikali.net:8200` (an intercept name), TLS validated against the
@@ -29,7 +36,7 @@ The agent side of the backend lifecycle agent protocol — [MONKY-ADR-0028](http
 ```
 kit reveal (admin, once)     tenancy signs a bootstrap DEPLOY GRANT: ES256 JWT
                               iss <tenancy issuer>, aud openbao-see-env, sub agent:<env_id>,
-                              env_id, kind deploy-grant, jti (single-use, recorded), exp now+1h
+                              env_id, kind deploy-grant, jti (→ token meta grant_jti; tenancy records it), exp now+1h
 install.sh                   stages it at /etc/monky-deployd/bootstrap.jwt (0600)
 first tick                   POST https://bao…/v1/auth/jwt-tenancy/login {"role":"see-env","jwt":<grant>}
                               → auth.client_token (ttl 24 h, max 30 d, policy see-env,
@@ -49,10 +56,13 @@ OpenBao side (Terraform, `tikali/services/security/openbao`): mount `jwt-tenancy
 `monky/data/{{identity.entity.aliases.<accessor>.metadata.env_id}}/see/*` read + metadata list +
 `auth/token/{renew-self,lookup-self}`. **Gate 1 v2 passed 2026-09-05 07:40Z** (distinct entity per env).
 
-**Second-reveal semantics.** A kit re-reveal (or a retire) supersedes every earlier grant of the
-backend and revokes the token accessors tenancy knows. The old kit's grant fails at login
-(unknown/used `jti`), and a token already minted from it is refused at the next call with
-`401 AGENT_UNAUTHENTICATED` (its `meta.grant_jti` is superseded) — **not** `AGENT_ENV_MISMATCH`.
+**Second-reveal semantics.** A kit re-reveal (or a retire) supersedes every earlier bootstrap grant
+of the backend (tenancy marks their `backend_leases` rows revoked) and revokes the token accessors
+tenancy knows. The `jwt-tenancy` mount keeps **no** replay state — it validates signature, `aud`,
+`bound_claims` and `exp` only (openbao `terraform/jwt-tenancy.tf`, the `see_env` role) — so a
+superseded grant still logs in until its 1 h `exp`; the token it yields, like a token already minted
+from it, is refused by tenancy at the first bearer call with `401 AGENT_UNAUTHENTICATED` (its
+`meta.grant_jti` is superseded) — **not** `AGENT_ENV_MISMATCH`.
 The agent then deletes its token; if a fresh `bootstrap.jwt` is on disk it bootstraps again in the
 same tick, otherwise it exits 1 and says "re-run the install kit".
 

monky_deployd/__init__.py

@@ -4,4 +4,4 @@ Dials monky-tenancy over the mesh with the box's host identity, fetches the rend
 leases a deploy grant, logs in to OpenBao, reads its own secrets, runs `docker compose`,
 reports. Stdlib only; the optional `openziti` SDK is the `sdk` transport."""
 
-__version__ = "0.1.5"
+__version__ = "0.1.6"

monky_deployd/agent.py

@@ -277,7 +277,7 @@ class Agent:
         # secrets: lease -> login -> reads (values never logged; names only)
         entries = b.manifest.get("entries", [])
         if entries:
-            token = self._lease_login("apply")
+            token = self._token_for_apply()
             for e in entries:
                 self._values[e["var"]] = self.bao.kv_read(token, e["path"], cfg.env_id, e.get("version"))
             log.info("read %d secret(s): %s", len(entries), ", ".join(sorted(self._values)))
@@ -392,6 +392,29 @@ class Agent:
         if old and old != token:
             self.bao.revoke_self(old)
 
+    def _token_for_apply(self) -> str:
+        """The OpenBao token to read this bundle's secrets with. A lease-derived token that is
+        still valid is REUSED (a lease per apply burned tenancy's 5/h budget on every retry —
+        env-qa-02 pilot, 2026-09-07); the bootstrap token is swapped for a lease once; and if
+        tenancy rate-limits the lease while we hold a working token, apply with what we have
+        and swap later rather than fail the deploy."""
+        cfg = self.cfg
+        st = self.state.token
+        if self.token and st is not None and st.source == "lease":
+            try:
+                info = self.bao.lookup_self(self.token)
+                if int(info.get("ttl") or 0) > cfg.bao.renew_below_s:
+                    return self.token
+            except BaoError as exc:
+                log.info("lease token no longer valid (%s); re-leasing", exc)
+        try:
+            return self._lease_login("apply")
+        except RateLimited as exc:
+            if self.token:
+                log.warning("lease rate-limited (%s); applying with the current token, swap deferred", exc)
+                return self.token
+            raise
+
     def _lease_login(self, reason: str) -> str:
         assert self.tenancy is not None
         lease = self.tenancy.lease(reason)

monky_deployd/config.py

@@ -129,7 +129,7 @@ def parse_yaml_subset(text: str) -> dict:
 class TenancyCfg:
     service: str = "monky.tenancy.deploy"
     host: str = ""  # intercept host; defaults to `service`
-    port: int = 8081
+    port: int = 443  # the service's INTERCEPT port (openziti); the pod-side listener is 127.0.0.1:8081
     scheme: str = "http"
     proxy_addr: str = "127.0.0.1:18443"
     timeout_s: int = 30

packaging/install.sh

@@ -3,9 +3,9 @@
 #
 #   T=<read-only GitLab deploy token, scope read_package_registry>   # from the install kit / OpenBao
 #   curl -sSf -H "DEPLOY-TOKEN: $T" \
-#     https://scm.tikali.ai/api/v4/projects/69/packages/generic/monky-deployd/0.1.5/install.sh \
+#     https://scm.tikali.ai/api/v4/projects/69/packages/generic/monky-deployd/0.1.6/install.sh \
 #     | sudo bash -s -- --env env-qa-02 --site cbs --token "$T" --bootstrap-file bootstrap.jwt \
-#     [--transport sdk|proxy|system] [--version 0.1.5] [--enrol-jwt /path/monky-host.env-qa-02.jwt] \
+#     [--transport sdk|proxy|system] [--version 0.1.6] [--enrol-jwt /path/monky-host.env-qa-02.jwt] \
 #     [--laptop] [--source gitlab|gitea] [--docker-data-root /home/docker-data]
 #
 # --token / MONKY_DEPLOYD_TOKEN: the GitLab project is PRIVATE (its parent groups are private, so it
@@ -29,7 +29,7 @@
 set -euo pipefail
 umask 077
 
-DEFAULT_VERSION="0.1.5"
+DEFAULT_VERSION="0.1.6"
 # Download source. PRIMARY is the GitLab project's generic package registry on scm.tikali.ai: inside
 # the estate gitea.cbs.tikali.net is split-horizon to jump1's RED EIP (10.10.0.175), which has no HTTP
 # ingress, so backend boxes cannot reach the Gitea mirror (cbs/iac#102); scm.tikali.ai they can. The

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "monky-deployd"
-version = "0.1.5"
+version = "0.1.6"
 description = "Monky backend pull agent: checkin -> bundle -> lease -> OpenBao -> docker compose -> report, over the ziti mesh (MONKY-ADR-0028)"
 readme = "README.md"
 requires-python = ">=3.12"

tests/fakes.py

@@ -190,9 +190,11 @@ class FakeBao:
                 return h._send(400, {"errors": ["error validating token: expired"]})
             if body.get("role") != self.role:
                 return h._send(400, {"errors": [f"role {body.get('role')!r} could not be found"]})
-            meta = self.grants.pop(body.get("jwt", ""), None)  # single-use grant
+            # the jwt mount keeps no jti/replay state: a known grant logs in until it expires;
+            # a SUPERSEDED grant is refused by tenancy (FakeTenancy.superseded_jtis), not here
+            meta = self.grants.get(body.get("jwt", ""))
             if meta is None:
-                return h._send(400, {"errors": ["error validating token: unknown or already used grant"]})
+                return h._send(400, {"errors": ["error validating token: unknown grant"]})
             m = json.loads(meta)
             tok = self.mint(m["env_id"])
             self.tokens[tok]["grant_jti"] = m["jti"]

tests/test_agent.py

@@ -107,7 +107,7 @@ def test_rollback_refused_unless_allowed(bootstrapped, tenancy, fake_docker):
     tenancy.set_files(first)
     assert tick(cfg) == EX_FAIL
     assert tenancy.reports[-1]["result"] == "failed" and "ROLLBACK_REFUSED" in tenancy.reports[-1]["detail"]
-    assert len(tenancy.leases) == 2  # a refused bundle never leases
+    assert len(tenancy.leases) == 1  # a refused bundle never leases; the 2nd apply reused the lease token
     tenancy.set_files(make_files(meta={"agent": {"allow_rollback": True}}))
     assert tick(cfg) == EX_OK
 
@@ -246,10 +246,9 @@ def test_legacy_approle_lease_is_refused_loudly(bootstrapped, tenancy, fake_dock
     assert "compose up" not in fake_docker.subcommands()
 
 
-def test_lease_rate_limited_is_temporary(bootstrapped, tenancy, fake_docker):
-    cfg = bootstrapped
-    tenancy.lease_limit = 0
-    assert tick(cfg) == EX_TEMPFAIL
+# (test_lease_rate_limited_is_temporary was retired in 0.1.6: a rate-limited lease is only a
+# temporary failure when NO working token exists; with one, the agent applies and defers the swap —
+# see test_rate_limited_lease_does_not_block_an_apply_when_a_token_exists.)
 
 
 def test_unhealthy_after_up_reports_failed_with_compose_logs(bootstrapped, tenancy, fake_docker):
@@ -320,3 +319,27 @@ def test_write_private_mode(tmp_path):
     p = tmp_path / "d" / "f"
     statemod.write_private(p, b"x")
     assert oct(p.stat().st_mode & 0o777) == "0o600" and not any(n.startswith(".f.") for n in os.listdir(p.parent))
+
+
+def test_second_apply_reuses_the_lease_token(bootstrapped, tenancy, bao, fake_docker):
+    """A valid lease-derived token is reused: a new bundle does NOT lease again (5/h budget —
+    env-qa-02 pilot: a retried deploy re-leased every 60 s and hit LEASE_RATE_LIMITED forever)."""
+    cfg = bootstrapped
+    assert tick(cfg) == EX_OK
+    assert len(tenancy.leases) == 1
+    tenancy.set_files(make_files(manifest=make_manifest(versions={"gemini_api_key": 1})))
+    assert tick(cfg) == EX_OK
+    assert len(tenancy.leases) == 1  # reused
+    st = statemod.load(cfg.state_path, ENV)
+    assert st.applied_sha == tenancy.desired_sha and st.token.source == "lease"
+
+
+def test_rate_limited_lease_does_not_block_an_apply_when_a_token_exists(bootstrapped, tenancy, bao, fake_docker):
+    """tenancy 429 on lease while the bootstrap token still works → apply with it, swap deferred."""
+    cfg = bootstrapped
+    tenancy.lease_limit = 0
+    assert tick(cfg) == EX_OK
+    assert tenancy.leases == []
+    st = statemod.load(cfg.state_path, ENV)
+    assert st.applied_sha == tenancy.desired_sha and st.token.source == "bootstrap"
+    assert tenancy.reports[-1]["result"] == "applied"

tests/test_cli.py

@@ -93,7 +93,7 @@ def test_proxy_transport_refuses_unmapped_hosts():
 
     cfg = from_dict({"env_id": "env-dev-06", "site": "cbs", "transport": "proxy"})
     t = build(cfg)
-    assert t.mapping[("monky.tenancy.deploy", 8081)] == ("127.0.0.1", 18443)
+    assert t.mapping[("monky.tenancy.deploy", 443)] == ("127.0.0.1", 18443)
     assert t.mapping[("bao.cbs.tikali.net", 8200)] == ("127.0.0.1", 18200)
     try:
         t.connect("example.com", 443, 1)

tests/test_config.py

@@ -10,7 +10,7 @@ transport: sdk
 identity: /opt/openziti/etc/identities/monky-host.env-qa-02.json
 tenancy:
   service: monky.tenancy.deploy
-  base_url: http://monky.tenancy.deploy:8081
+  base_url: http://monky.tenancy.deploy:443
 bao:
   service: openbao
   addr: https://bao.cbs.tikali.net:8200   # intercept, not public DNS
@@ -27,7 +27,7 @@ volumes_on_absent: keep
 def test_yaml_subset_parses_nested_maps_and_types():
     d = c.parse_yaml_subset(KIT)
     assert d["env_id"] == "env-qa-02"
-    assert d["tenancy"]["base_url"] == "http://monky.tenancy.deploy:8081"
+    assert d["tenancy"]["base_url"] == "http://monky.tenancy.deploy:443"
     assert d["bao"]["addr"] == "https://bao.cbs.tikali.net:8200"
     assert d["interval_s"] == 60 and d["laptop_mode"] is False
 
@@ -46,7 +46,7 @@ def test_yaml_subset_refuses_flow_style_and_tabs():
 
 def test_config_defaults_and_derivations():
     cfg = c.from_dict(c.parse_yaml_subset(KIT))
-    assert cfg.tenancy.host == "monky.tenancy.deploy" and cfg.tenancy.port == 8081 and cfg.tenancy.scheme == "http"
+    assert cfg.tenancy.host == "monky.tenancy.deploy" and cfg.tenancy.port == 443 and cfg.tenancy.scheme == "http"
     assert cfg.bao_url == ("https", "bao.cbs.tikali.net", 8200)
     assert cfg.deploy_dir == "/var/lib/monky-deployd/env-qa-02"
     assert cfg.compose_project == "monky-env-qa-02"