Chaos and failure testing
Clank tests failures as state machine behavior, not only happy path output. The deterministic chaos suite runs under Node's test runner as part of every npm test and npm run check .
Clank tests failures as state-machine behavior, not only happy-path output. The deterministic chaos suite runs under Node's test runner as part of every npm test and npm run check.
Automated scenarios
| Fault | Expected invariant | Evidence |
|---|---|---|
| Deployment worker disappears after claiming work | Expired work is reclaimed with a higher fence; the stale worker cannot commit | tests/chaos.test.mjs, orchestration tests |
| Encrypted backup is corrupted | Authentication fails before replacement; the live database remains unchanged | Chaos and recovery tests |
| Application upstream becomes unreachable | Requests fail generically, the circuit opens, and a later probe recovers | Chaos and data-plane tests |
| Candidate startup/health fails | Prior data and active release are restored | Platform tests and packaged conformance |
| Migration or artifact is malformed | Intake fails before activation and does not escape its project boundary | Migration, deploy, platform, and conformance tests |
| Concurrent/stale state is written | Version/fence/idempotency checks reject the stale operation | Backend, jobs, orchestration, and platform tests |
| Credential payload is hostile | Body, base64url, CBOR collection, and nesting limits fail closed | Security, auth, and WebAuthn tests |
Run only the deterministic chaos file after building:
npm run build
node --test tests/chaos.test.mjsRun the release-level lifecycle:
npm run conformanceStaging drills
Before public beta and at least quarterly:
- Kill a deployment worker after it claims an operation; verify another worker reclaims it and the old fence cannot commit.
- Stop the active app during traffic; verify health alerts, bounded failures, restart policy, and ingress recovery.
- Publish an artifact whose health check fails after a pending migration; verify data and code return to the previous release.
- Restore an encrypted off-host backup into a clean directory with the original key; compare integrity, revision, migrations, and representative application queries.
- Remove access to email, object storage, webhook targets, and external database APIs; verify timeouts, retries, idempotency, dead letters, and redacted logs.
- Drain a node and expire its heartbeat; verify desired placement is reassigned without accepting a stale observation.
- Rotate a project secret and scoped CLI token; verify the prior values stop working and audit records remain readable.
- Simulate disk-full and read-only filesystem conditions in an isolated environment; verify no partial release is activated.
Safety rules
- Never run destructive drills against the only copy of production data.
- Capture the exact version, topology, fault injection, expected invariant, observed recovery time, and follow-up owner.
- Use synthetic accounts and credentials.
- Treat an unexpected successful stale write, cross-tenant read, unauthenticated restore, plaintext backup, or secret log entry as a release blocker.
- Keep drills deterministic in CI; reserve network partitions, process kills, storage faults, and regional failures for staging.