mdella/monky-deployd
1
0
Fork 0
mirror of https://scm.tikali.ai/tikali/applications/monky/monky-deployd.git synced 2026-09-18 04:36:15 +00:00
Code Issues Packages Projects Releases 1 Wiki Activity

feat: monky-deployd v0.1.0 — pull agent over the mesh (ADR-0028)

Browse Source 
Stdlib-only Python 3.12 agent for docker VMs and laptops: flock → checkin
(bearer = the agent's OpenBao token, bootstrapped from the install kit's
jwt-tenancy deploy grant) → action apply|none|down → bundle (sha256
verified) → refusal checks (unresolved ${VAR} names only, manifest paths
pinned to monky/data/<env>/see/, privileged/host-network, rollback, disk
need×1.5+headroom) → lease → POST /v1/auth/jwt-tenancy/login → KV reads →
.env 0600 → promote → compose pull/up → wait healthy → report; finally
renew-self / re-lease before max TTL, scrub. Exit 0/75/78/1. Redactor log
filter. Transports sdk (openziti) / proxy (ziti tunnel proxy 18443/18200) /
system. Laptop mode.

Packaging: hardened oneshot + 60 s timer + proxy unit, nfpm .deb with
/opt/monky-deployd/venv, install.sh for Ubuntu 26.04 (Gitea release
download, enrol, ACLs, bootstrap from stdin), ansible role skeleton for
osg1-07. CI: lint/test on every change; wheel (openziti on ubuntu:26.04) and
package (nfpm) allow_failure until runner egress is proven; GitLab release +
release:gitea on v* tags. Docs: README, PROTOCOL, OPERATIONS, CHANGELOG,
CLAUDE/AGENTS.

Divergence noted: monky-tenancy main (MR !15) still ships the AppRole lease
and kit; this agent implements the plan's Gate 1 RESULT (login_jwt, no
unwrap) and refuses an AppRole lease loudly (LEASE_SHAPE).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KLB7jieMNRkTsJ2epr4Ds1
This commit is contained in:
Marcos Della
2026-09-05 08:01:36 +00:00
parent c72d6c0227
commit 1c42e913a8
48 changed files with 4932 additions and 64 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+122 -64
View File
@@ -1,93 +1,151 @@
<!-- xlate:verbatim-fences -->
# monky-deployd
## Getting started
To make it easy for you to get started with GitLab, here's a list of recommended next steps.
Already a pro? Just edit this README.md and make it your own. Want to make it easy? [Use the template at the bottom](#editing-this-readme)!
## Add your files
* [Create](https://docs.gitlab.com/user/project/repository/web_editor/#create-a-file) or [upload](https://docs.gitlab.com/user/project/repository/web_editor/#upload-a-file) files
* [Add files using the command line](https://docs.gitlab.com/topics/git/add_files/#add-files-to-a-git-repository) or push an existing Git repository with the following command:
**The on-box pull agent for Monky backends** ([MONKY-ADR-0028](https://scm.tikali.ai/tikali/applications/monky/monky-design-docs/-/blob/develop/adr/0028-backend-lifecycle-ownership-and-execution.md), [design doc 24](https://scm.tikali.ai/tikali/applications/monky/monky-design-docs/-/blob/develop/docs/24-backend-lifecycle.md)).
A docker VM or a laptop that hosts a backend environment runs this agent; every minute it dials
**monky-tenancy over the OpenZiti mesh with the box's own host identity**, asks what the backend
should be running, and converges: fetch the rendered bundle, lease a deploy grant, log in to
OpenBao, read its own secrets, `docker compose up`, report. Nothing pushes into the box, no SSH,
no credentials for the box are held anywhere else. The same check-in is the liveness heartbeat.
```
cd existing_repo
git remote add origin https://scm.tikali.ai/tikali/applications/monky/monky-deployd.git
git branch -M main
git push -uf origin main
box / laptop ◄── mesh ──► monky-deployd: checkin → bundle → lease → OpenBao → compose up → report
```
## Integrate with your tools
Python 3.12+, **stdlib only** (the `openziti` SDK is optional and vendored into the `.deb`).
Verified on **Ubuntu 26.04**.
* [Set up project integrations](https://scm.tikali.ai/tikali/applications/monky/monky-deployd/-/settings/integrations)
## Install (one-liner, from the enrolment kit)
## Collaborate with your team
An admin reveals the kit once in the console (`GET /v1/backends/{id}/agent/install`); it hands
you the enrolment JWT and a one-time **bootstrap deploy grant**. On the box:
* [Invite team members and collaborators](https://docs.gitlab.com/user/project/members/)
* [Create a new merge request](https://docs.gitlab.com/user/project/merge_requests/creating_merge_requests/)
* [Automatically close issues from merge requests](https://docs.gitlab.com/user/project/issues/managing_issues/#closing-issues-automatically)
* [Enable merge request approvals](https://docs.gitlab.com/user/project/merge_requests/approvals/)
* [Set auto-merge](https://docs.gitlab.com/user/project/merge_requests/auto_merge/)
```sh
curl -fsSL https://gitea.cbs.tikali.net/mdella/monky-deployd/raw/branch/main/packaging/install.sh \
| sudo bash -s -- --env env-qa-02 --site cbs --enrol-jwt ./monky-host.env-qa-02.jwt < bootstrap.jwt
# [--transport sdk|proxy|system] [--version 0.1.0] [--laptop] [--bao-ca openbao-ca.pem]
```
## Test and Deploy
`install.sh` installs `ziti-edge-tunnel` (OpenZiti `jammy` suite) and `docker-compose-plugin` if
absent, downloads the pinned `.deb` + `.sha256` from the [Gitea release](https://gitea.cbs.tikali.net/mdella/monky-deployd/releases),
enrols `monky-host.<env>` if the identity is missing, switches the tunneler to `run-host`,
writes `/etc/monky-deployd/config.yaml`, grants the agent read access to the identity (ACL),
stages the bootstrap grant (0600), enables `monky-deployd.timer`, runs one tick and deletes the
JWT. The GitLab project is private, so **the public download is the Gitea mirror**.
Use the built-in continuous integration in GitLab.
## Transports
* [Get started with GitLab CI/CD](https://docs.gitlab.com/ci/quick_start/)
* [Analyze your code for known vulnerabilities with Static Application Security Testing (SAST)](https://docs.gitlab.com/user/application_security/sast/)
* [Deploy to Kubernetes, Amazon EC2, or Amazon ECS using Auto Deploy](https://docs.gitlab.com/topics/autodevops/requirements/)
* [Use pull-based deployments for improved Kubernetes management](https://docs.gitlab.com/user/clusters/agent/)
* [Set up protected environments](https://docs.gitlab.com/ci/environments/protected_environments/)
| `transport` | how | when |
|---|---|---|
| `sdk` (default) | the OpenZiti Python SDK dials `monky.tenancy.deploy` / `openbao` by service name with the same host identity `ziti-edge-tunnel run-host` uses; no tun, no root beyond the docker group | VMs and laptops with the vendored wheel |
| `proxy` | `monky-deployd-proxy.service` runs `ziti tunnel proxy -i <identity> monky.tenancy.deploy:18443 openbao:18200` as user `ziti`; the agent talks to `127.0.0.1:18443/18200` (TLS SNI + cert check still `bao.cbs.tikali.net`) | the wheel is unavailable; `ziti` CLI present |
| `system` | plain DNS/TCP | laptops whose tunneler runs in `run` mode (tun + DNS) |
***
## Commands and exit codes
# Editing this README
```
monky-deployd run --once [--prune] # one tick (what the timer runs); --prune drops old releases + docker image prune
monky-deployd run # loop (laptop mode: no timer); SIGTERM stops it
monky-deployd status [--json] # token present? applied vs desired sha, last checkin/report, compose ps
monky-deployd bootstrap [--force] # log in to OpenBao with /etc/monky-deployd/bootstrap.jwt, store the token
monky-deployd version
```
When you're ready to make this README your own, just edit this file and use the handy template below (or feel free to structure it however you want - this is just a starting point!). Thanks to [makeareadme.com](https://www.makeareadme.com/) for this template.
| exit | meaning |
|---|---|
| `0` | converged / heartbeat sent — or offline in `laptop_mode` |
| `75` | temporary network failure (mesh down, tenancy unreachable, `429`); the next tick retries — `SuccessExitStatus=75` in the unit |
| `78` | `AGENT_ENV_MISMATCH`: the token is pinned to another environment than `config.yaml` says; **not retried** (loop mode exits) |
| `1` | refusal or failure (`ENV_INCOMPLETE`, `PRIVILEGED_REFUSED`, `DISK_INSUFFICIENT`, `ROLLBACK_REFUSED`, `BUNDLE_SHA_MISMATCH`, compose failure, `AGENT_UNAUTHENTICATED`, no credentials) — reported to tenancy as `failed` with the code and secret *names* only |
## Suggestions for a good README
## What a tick does
Every project is different, so consider which of these sections apply to yours. The sections used in the template are suggestions for most open source projects. Also keep in mind that while a README can be too long and detailed, too long is better than too short. If you think your README is too long, consider utilizing another form of documentation rather than cutting out information.
1. `flock` (a second concurrent tick exits 0) → load `state.json`.
2. Bearer = the OpenBao token at `/var/lib/monky-deployd/bao.token` (0600). Absent → log in to
`jwt-tenancy` with the kit's bootstrap grant (`/etc/monky-deployd/bootstrap.jwt`), then delete it.
3. `POST /v1/agent/checkin` `{env_id, agent_version, applied_sha, host, containers}` → `action`:
- **`down`** → `docker compose down --remove-orphans` (`-v` only if `purge_volumes` was
requested or `volumes_on_absent: purge`, **never on prod**) → report `down`.
- **`none`** → `compose ps`; healthy → report `applied` (heartbeat); unhealthy → report `failed`.
- **`apply`** → `GET /v1/agent/bundle/{env}/{sha}`, verify **sha256** over the files, then refuse on:
unresolved `${VAR}` (names only), manifest paths outside `monky/data/<env>/see/`, `privileged`/host
network/`SYS_ADMIN` unless `bundle.json` `allow_privileged`, a rollback unless `allow_rollback`,
docker data-root free space `< need × 1.5 + 2 GiB` → `POST /v1/agent/lease` → OpenBao
`POST /v1/auth/jwt-tenancy/login {"role":"see-env","jwt":<grant>}` → KV reads per
`secrets.manifest.json` (pinned versions) → `.env` written atomically 0600 into a staging dir →
promoted to `releases/<sha>` + `current` → `compose pull` → `up -d --remove-orphans` → wait
healthy → report `applied` (or `failed` with the compose log tail).
4. `finally`: renew-self when the TTL runs low, re-lease before max TTL, scrub secrets from memory,
remove staging dirs, save state.
## Name
Choose a self-explaining name for your project.
## Security model
## Description
Let people know what your project can do specifically. Provide context and add a link to any reference visitors might be unfamiliar with. A list of Features or a Background subsection can also be added here. If there are alternatives to your project, this is a good place to list differentiating factors.
- **Identity = the box's ziti host identity.** Only identities with `#monky-deploy-agent` can dial
tenancy's agent entrypoint; `#openbao-client` reaches OpenBao. The agent reads the identity
through an ACL (`setfacl -m u:monky-deployd:r`), never owns it.
- **The bearer to tenancy is the agent's own OpenBao token**, minted by OpenBao from a
tenancy-signed ES256 deploy grant (`aud openbao-see-env`, `kind deploy-grant`, 1 h, single-use
`jti`). Tenancy verifies it with `auth/token/lookup`, pins `meta.env_id`, and refuses a token
whose `meta.grant_jti` was superseded (kit re-reveal, retire) → `401 AGENT_UNAUTHENTICATED`.
**No AppRole, nothing to unwrap** (Gate 1 result, 2026-09-05).
- **The agent never receives a secret from tenancy.** Bundles carry placeholders; the agent reads
`monky/data/<env_id>/see/*` itself, and the OpenBao policy is templated on the token's entity
(one entity per env), so another env's path is a 403 — and refused locally before any read.
- **Journald never carries a value.** A `Redactor` filter masks token shapes (`hvs.*`, JWTs,
`Authorization: Bearer`) and every value the agent has read; the report's `log_tail` goes through
the same scrubber. Secret *names* are logged.
- **Hardened oneshot**: user `monky-deployd` (+ `docker` group), `NoNewPrivileges`,
`ProtectSystem=strict`, `ReadWritePaths` only the state dir, `/etc/monky-deployd` (to consume the
grant) and the docker socket, `UMask=0077`, no capabilities.
- **Prod is special**: `-v` is never passed on a prod env, and tenancy never auto-changes prod state.
## Badges
On some READMEs, you may see small images that convey metadata, such as whether or not all the tests are passing for the project. You can use Shields to add some to your README. Many services also have instructions for adding a badge.
## Ubuntu 26.04 verified checklist (from design doc 24 §2.1a / plan §D 2.8)
## Visuals
Depending on what you are making, it can be a good idea to include screenshots or even a video (you'll frequently see GIFs rather than actual videos). Tools like ttygif can help, but check out Asciinema for a more sophisticated method.
```
python3 --version # >= 3.12 (26.04 ships 3.14)
docker compose version # compose plugin
systemctl is-active ziti-edge-tunnel # run-host (drop-in run-host.conf)
ziti tunnel proxy --help # fallback transport present (transport proxy)
/opt/monky-deployd/venv/bin/python -c 'import openziti' # SDK import (transport sdk)
monky-deployd status # token present; applied == desired
journalctl -u monky-deployd -n 50 # "checkin: action=..." then "applied ..."
df -h $(docker info -f '{{.DockerRootDir}}') # free >= bundle need x 1.5 + headroom
docker compose -p monky-<env> ps # healthy
curl monky.percept.<env>:47283/health # from another mesh member -> 200
```
## Installation
Within a particular ecosystem, there may be a common way of installing things, such as using Yarn, NuGet, or Homebrew. However, consider the possibility that whoever is reading your README is a novice and would like more guidance. Listing specific steps helps remove ambiguity and gets people to using your project as quickly as possible. If it only runs in a specific context like a particular programming language version or operating system or has dependencies that have to be installed manually, also add a Requirements subsection.
## Repository layout
## Usage
Use examples liberally, and show the expected output if you can. It's helpful to have inline the smallest example of usage that you can demonstrate, while providing links to more sophisticated examples if they are too long to reasonably include in the README.
```
monky_deployd/ cli.py config.py transport.py tenancy.py bao.py bundle.py compose.py state.py redact.py agent.py
packaging/ install.sh systemd/{monky-deployd.service,.timer,monky-deployd-proxy.service} nfpm.yaml scripts/
ansible/roles/monky_deployd/ role skeleton for osg1-07 (env-dev-06..09 rollout)
docs/ PROTOCOL.md (the four calls + the grant flow) OPERATIONS.md (systemd, logs, retire, laptops)
tests/ hermetic: fake tenancy + fake OpenBao HTTP servers, a stub `docker` on PATH
```
## Support
Tell people where they can go to for help. It can be any combination of an issue tracker, a chat room, an email address, etc.
## Development
## Roadmap
If you have ideas for releases in the future, it is a good idea to list them in the README.
```sh
ruff check . && ruff format --check .
pytest # no network, no docker: fakes only
bash -n packaging/install.sh
systemd-analyze verify packaging/systemd/*.service # where systemd is available
```
## Contributing
State if you are open to contributions and what your requirements are for accepting them.
### CI notes
For people who want to make changes to your project, it's helpful to have some documentation on how to get started. Perhaps there is a script that they should run or some environment variables that they need to set. Make these steps explicit. These instructions could also be useful to your future self.
`lint` and `test` run on every MR/branch. `wheel` builds the `openziti` wheel on `ubuntu:26.04`
(PyPI ships an sdist that fetches **ziti-sdk-c from github.com** at build time — the runner needs
egress to github.com and pypi.org) and `package` builds the `.deb` with `nfpm` (binary from GitHub
releases, goreleaser apt repo as fallback). Both are `allow_failure: true` until proven on this
runner; without the wheel the `.deb` still works with `transport: proxy|system`. On a `v*` tag
`release` uploads to the GitLab generic package registry + release, and `release:gitea` publishes
the same assets on the public Gitea mirror (automatic when `GITEA_TOKEN` is set, manual otherwise
— see `docs/OPERATIONS.md` for the by-hand recipe).
You can also document commands to lint the code or run tests. These steps help to ensure high code quality and reduce the likelihood that the changes inadvertently break something. Having instructions for running tests is especially helpful if it requires external setup, such as starting a Selenium server for testing in a browser.
## See also
## Authors and acknowledgment
Show your appreciation to those who have contributed to the project.
## License
For open source projects, say how it is licensed.
## Project status
If you have run out of energy or time for your project, put a note at the top of the README saying that development has slowed down or stopped completely. Someone may choose to fork your project or volunteer to step in as a maintainer or owner, allowing your project to keep going. You can also make an explicit request for maintainers.
- `docs/PROTOCOL.md`, `docs/OPERATIONS.md`, `CHANGELOG.md`
- monky-tenancy `docs/usage.md` (agent protocol), `app/api/agent.py`, `app/schemas_backends.py`
- monky-deploy (the renderer whose `render_files` produces the bundle)