{"protocol":"clank-doc/1","frameworkVersion":"0.8.0","slug":"distributed-deployment","title":"Durable distributed deployment","description":"Clank deployment coordination is persisted in the control database. Process local maps are still used as an optimization, but correctness is protected by authenticated leases and monotonic fencing tokens.","group":{"id":"deploy","title":"Deploy and operate"},"url":"https://docs.clank.run/docs/distributed-deployment","source":"docs/distributed-deployment.md","headings":["Platform behavior","Agent loop","Failure semantics"],"tableOfContents":[{"id":"platform-behavior","title":"Platform behavior","level":2},{"id":"agent-loop","title":"Agent loop","level":2},{"id":"failure-semantics","title":"Failure semantics","level":2}],"markdown":"# Durable distributed deployment\n\nClank deployment coordination is persisted in the control database. Process-local maps are still used as an optimization, but correctness is protected by authenticated leases and monotonic fencing tokens.\n\n`openDeploymentOrchestrator` provides four durable contracts:\n\n1. **Distributed project leases** serialize deploy, rollback, backup restore, and other destructive project operations across control-plane workers.\n2. **Node sessions** authenticate deployment agents, publish region/capacity/labels, support draining, and expire without heartbeats.\n3. **Desired placement state** records release, running/stopped state, assigned node, and a monotonically increasing generation.\n4. **Operations** use idempotency keys, explicit queued/leased/retry/succeeded/failed states, retry timing, attempt limits, lease expiry, and fencing.\n\n```ts\nconst orchestrator = openDeploymentOrchestrator(controlDatabase);\n\nconst agent = await orchestrator.registerNode({\n  id: \"iad-node-01\",\n  region: \"iad\",\n  capacity: 100,\n  labels: { runtime: \"node24\", isolation: \"microvm\" },\n});\n\nconst desired = await orchestrator.setDesired({\n  projectId,\n  releaseId,\n  state: \"running\",\n  region: \"iad\",\n});\n\nconst [operation] = await orchestrator.claim(agent.node.id, agent.token);\n```\n\nAn agent must renew a leased operation before expiry. Completion and failure compare the node, token digest, lease expiry, and fence. If a worker resumes after another worker has reclaimed the operation, its stale completion is rejected.\n\nDesired-state observations are generation checked. A late report for generation 4 cannot overwrite generation 5.\n\n## Platform behavior\n\nThe built-in platform acquires a durable `project:<id>` lease in addition to its local queue. It renews the lease during long operations and returns `PROJECT_BUSY` if another control worker owns it. A lost lease is surfaced as `PROJECT_LEASE_LOST` rather than silently claiming coordinated success.\n\n## Agent loop\n\nA production deployment agent should:\n\n1. register or load its node credential;\n2. heartbeat before half of the node TTL;\n3. stop claiming new work while draining;\n4. claim a bounded operation batch;\n5. renew long-running operation leases;\n6. make runtime changes using the operation fence;\n7. report desired generation observations; and\n8. complete or fail with a bounded, non-secret result.\n\nAgent credentials and operation lease tokens are shown only to the worker and stored as digests. Control-plane database access remains privileged and should not be exposed to application processes.\n\n## Failure semantics\n\n- Duplicate API requests converge through idempotency keys.\n- Crashed workers leave leased operations that become reclaimable.\n- Expired nodes become offline for placement.\n- Draining nodes keep current work but receive no new desired placements.\n- Retry delay is exponential and bounded; exhausted operations enter `failed`.\n- Node capacity is placement based, deterministic, and region aware.\n\nSQLite coordination is suitable for multiple workers on one durable host. Multi-region control planes should bind the same orchestration semantics to the external transactional control database described in the data-plane guide.\n\n"}