Durable distributed deployment

Clank deployment coordination is persisted in the control database. Process-local maps are still used as an optimization, but correctness is protected by authenticated leases and monotonic fencing tokens.

openDeploymentOrchestrator provides four durable contracts:

1. Distributed project leases serialize deploy, rollback, backup restore, and other destructive project operations across control-plane workers.
2. Node sessions authenticate deployment agents, publish region/capacity/labels, support draining, and expire without heartbeats.
3. Desired placement state records release, running/stopped state, assigned node, and a monotonically increasing generation.
4. Operations use idempotency keys, explicit queued/leased/retry/succeeded/failed states, retry timing, attempt limits, lease expiry, and fencing.

const orchestrator = openDeploymentOrchestrator(controlDatabase);

const agent = await orchestrator.registerNode({
  id: "iad-node-01",
  region: "iad",
  capacity: 100,
  labels: { runtime: "node24", isolation: "microvm" },
});

const desired = await orchestrator.setDesired({
  projectId,
  releaseId,
  state: "running",
  region: "iad",
});

const [operation] = await orchestrator.claim(agent.node.id, agent.token);

An agent must renew a leased operation before expiry. Completion and failure compare the node, token digest, lease expiry, and fence. If a worker resumes after another worker has reclaimed the operation, its stale completion is rejected.

Desired-state observations are generation checked. A late report for generation 4 cannot overwrite generation 5.

#Platform behavior

The built-in platform acquires a durable project:<id> lease in addition to its local queue. It renews the lease during long operations and returns PROJECT_BUSY if another control worker owns it. A lost lease is surfaced as PROJECT_LEASE_LOST rather than silently claiming coordinated success.

#Agent loop

A production deployment agent should:

1. register or load its node credential;
2. heartbeat before half of the node TTL;
3. stop claiming new work while draining;
4. claim a bounded operation batch;
5. renew long-running operation leases;
6. make runtime changes using the operation fence;
7. report desired generation observations; and
8. complete or fail with a bounded, non-secret result.

Agent credentials and operation lease tokens are shown only to the worker and stored as digests. Control-plane database access remains privileged and should not be exposed to application processes.

#Failure semantics

• Duplicate API requests converge through idempotency keys.
• Crashed workers leave leased operations that become reclaimable.
• Expired nodes become offline for placement.
• Draining nodes keep current work but receive no new desired placements.
• Retry delay is exponential and bounded; exhausted operations enter failed.
• Node capacity is placement based, deterministic, and region aware.

SQLite coordination is suitable for multiple workers on one durable host. Multi-region control planes should bind the same orchestration semantics to the external transactional control database described in the data-plane guide.