{"protocol":"clank-doc/1","frameworkVersion":"0.8.0","slug":"database","title":"Database revisions and correctness","description":"Clank uses Node's built in SQLite as a transactional document store. Application fields live as validated JSON, while identity, ownership, creation time, and document version live in dedicated columns.","group":{"id":"full-stack","title":"Full stack"},"url":"https://docs.clank.run/docs/database","source":"docs/database.md","headings":["Physical layout","Commit sequence","Document versions and lost update protection","Consistent reads","Dependency and ownership invalidation","Multiple server processes","Auth revisions","Durability and file safety","Migrations, backup, and restore","Important options","Guarantees and boundaries"],"tableOfContents":[{"id":"physical-layout","title":"Physical layout","level":2},{"id":"commit-sequence","title":"Commit sequence","level":2},{"id":"document-versions-and-lost-update-protection","title":"Document versions and lost-update protection","level":2},{"id":"consistent-reads","title":"Consistent reads","level":2},{"id":"dependency-and-ownership-invalidation","title":"Dependency and ownership invalidation","level":2},{"id":"multiple-server-processes","title":"Multiple server processes","level":2},{"id":"auth-revisions","title":"Auth revisions","level":2},{"id":"durability-and-file-safety","title":"Durability and file safety","level":2},{"id":"migrations-backup-and-restore","title":"Migrations, backup, and restore","level":2},{"id":"important-options","title":"Important options","level":2},{"id":"guarantees-and-boundaries","title":"Guarantees and boundaries","level":2}],"markdown":"# Database revisions and correctness\n\nClank uses Node's built-in SQLite as a transactional document store. Application fields live as validated JSON, while identity, ownership, creation time, and document version live in dedicated columns.\n\nThere are two different revision concepts:\n\n| Value | Scope | Meaning |\n| --- | --- | --- |\n| `runtime.version` / live `version` | Whole database | The latest committed change transaction observed by this runtime |\n| `document._version` | One document | The number of content versions committed for that document |\n\nIf a page shows `revision 36`, it means the database has committed 36 change-producing transactions since its revision counter began. It does not mean there are 36 records, 36 users, or 36 browser connections. The number is an internal synchronization cursor, not an authentication state or a user-facing progress metric.\n\nNew interfaces should normally display `synced` or `reconnecting` and keep the numeric revision in diagnostics.\n\n## Physical layout\n\nFor a declared `todos` table, Clank creates `clank_todos`:\n\n```sql\nCREATE TABLE clank_todos (\n  _id TEXT PRIMARY KEY,\n  _owner_id TEXT,\n  _creation_time INTEGER NOT NULL,\n  _version INTEGER NOT NULL,\n  _data TEXT NOT NULL CHECK (json_valid(_data))\n);\n```\n\n`_data` is canonical JSON derived from the table schema. Owned tables store `_owner_id` outside JSON so an application patch cannot change ownership.\n\nFramework state uses:\n\n- `clank_meta`: the persisted global revision;\n- `clank_changes`: the bounded cross-process change journal;\n- `clank_auth_users` and `clank_auth_sessions`: built-in auth;\n- `clank_migrations`: immutable SQL migration history.\n\nThe `clank_` and legacy `proact_` SQL namespaces are reserved. Safe application migrations cannot modify them.\n\n## Commit sequence\n\nEvery mutation uses one synchronous `BEGIN IMMEDIATE` transaction:\n\n```mermaid\nflowchart LR\n  A[\"Validate arguments\"] --> B[\"BEGIN IMMEDIATE\"]\n  B --> C[\"Run synchronous handler\"]\n  C --> D[\"Validate writes and output\"]\n  D --> E[\"Increment document versions\"]\n  E --> F[\"Increment global revision once\"]\n  F --> G[\"Write change-journal records\"]\n  G --> H[\"COMMIT\"]\n  H --> I[\"Invalidate affected queries\"]\n  I --> J[\"Publish current snapshots\"]\n```\n\nThe application writes, global revision, and change records commit together. Any handler error, schema failure, stale-version conflict, invalid output, non-JSON output, or oversized output rolls back all of them.\n\nObservers run after `COMMIT`. An observer exception is reported through `onError` and cannot turn a successful commit into a failed mutation response.\n\nA transaction that changes several documents advances the global revision once. A transaction that makes no logical change does not advance it.\n\n## Document versions and lost-update protection\n\nInserted documents start at `_version: 1`. A content-changing patch or replacement increments that document only. Deleting a document removes it. A no-op patch or replacement returns the existing immutable document without changing either revision.\n\nPass the version the user actually saw:\n\n```ts\nconst saved = db.table(\"todos\").patch(\n  todo._id,\n  { title: nextTitle },\n  { ifVersion: todo._version },\n);\n```\n\nIf another browser changed or deleted the document first, Clank throws `DatabaseConflictError`. RPC converts it to HTTP `409 VERSION_CONFLICT` with the expected and actual versions. The stale mutation commits nothing.\n\nThis is optimistic concurrency control. It avoids holding a lock while a person edits and prevents last-write-wins data loss.\n\nSingleton records, such as one profile per user, should accept `version: number | null`. `null` means “I observed no record”; the mutation checks and inserts inside one transaction, so simultaneous creates cannot silently overwrite one another.\n\n## Consistent reads\n\nQueries run inside a short deferred SQLite read transaction. The global revision is read in the same snapshot as the application rows, so a query cannot combine rows from different commits.\n\nReturned documents, cached query outputs, and change metadata are immutable at runtime. One subscriber cannot accidentally alter the snapshot delivered to another subscriber.\n\nBefore a cached one-shot query is used, the runtime synchronizes its persisted revision cursor. Correctness therefore does not depend on waiting for the background poll.\n\n## Dependency and ownership invalidation\n\nClank records what each query reads:\n\n- `get(id)` depends on one table/document pair;\n- `query()` and `collect()` depend on the table;\n- owned-table dependencies also include the authenticated owner.\n\nEach committed journal record contains a table, document ID, and optional owner ID. Only intersecting cache entries become dirty. Bob's private todo mutation does not rerun or republish Alice's todo query.\n\nThe numeric global revision still advances for every committed transaction. It is an operational cursor, not a per-user counter, which is another reason not to present it as product data.\n\n## Multiple server processes\n\nFile-backed runtimes poll `clank_changes` every 100 ms by default. Local commits publish immediately. If several commits accumulated, a runtime combines their affected records and publishes one snapshot at the newest revision because SQLite can only read the current database state.\n\nThe journal retains 10,000 revisions by default. If a process was offline long enough to miss retained history, Clank performs a conservative full cache invalidation. Authenticated live streams are closed and reconnect with freshly resolved authorization.\n\nBrowser `EventSource` reconnects receive a complete current snapshot. The client ignores payloads older than the snapshot it already holds.\n\nThis supports several processes sharing one SQLite file on one host. It is not a distributed multi-host database protocol. Network filesystems and independently replicated SQLite files are outside the supported consistency model.\n\n## Auth revisions\n\nAuth user and session writes use the same transaction/revision journal. Role changes, disabling a user, password changes, logout, and session revocation invalidate the affected identity.\n\nAcross processes, affected live streams close as soon as the other runtime observes the journal record. Long-lived server callers refresh their session record before each operation, so a role downgrade is not left cached.\n\n## Durability and file safety\n\nFile databases default to:\n\n- WAL journal mode;\n- `synchronous = FULL`;\n- foreign keys enabled;\n- extension loading disabled;\n- `trusted_schema = OFF`;\n- recursive triggers disabled;\n- secure deletion enabled;\n- a five-second busy timeout;\n- startup `quick_check` and foreign-key validation.\n\nDatabase, WAL, SHM, backup, and restored files are restricted to mode `0600` where the operating system supports POSIX modes. Final database paths cannot be symbolic links. Corrupt databases and semantically inconsistent revision journals stop startup instead of being used.\n\nSet `durability: \"normal\"` only after accepting weaker power-loss durability. Set `integrityCheck: \"full\"` for SQLite's more expensive full startup check, or `false` only when another verified operational process owns integrity checking.\n\n## Migrations, backup, and restore\n\nOrdered SQL migrations are checksummed and immutable. All pending files and ledger rows run inside one `BEGIN IMMEDIATE`. Safe mode rejects database attachment, extension loading, PRAGMAs, transaction control, and every `clank_` or `proact_` table reference.\n\nBackup and restore:\n\n1. reject symbolic-link and non-file sources;\n2. verify SQLite integrity and foreign keys;\n3. write a private temporary file;\n4. verify the completed copy;\n5. atomically replace the destination;\n6. remove stale WAL/SHM sidecars.\n\nThe deployment platform stops the application before migration or restore. A failed migration, startup, or health check restores the verified pre-release snapshot.\n\n## Important options\n\n```ts\nconst runtime = await openBackend(backend, {\n  path: \"./data/app.sqlite\",\n  durability: \"full\",\n  integrityCheck: \"quick\",\n  busyTimeout: 5_000,\n  changePollIntervalMs: 100,\n  changeRetentionRevisions: 10_000,\n  maxRequestBytes: 64 * 1024,\n  maxResponseBytes: 4 * 1024 * 1024,\n  maxLivePayloadBytes: 4 * 1024 * 1024,\n  onError(error) {\n    logger.error(error);\n  },\n});\n```\n\nAll numeric resource limits must be positive integers. Slow SSE consumers are disconnected instead of being allowed to build an unbounded queue; EventSource reconnect then restores the current snapshot.\n\n## Guarantees and boundaries\n\nClank guarantees atomic local commits, consistent SQLite snapshots, optimistic conflict detection when `ifVersion` is used, owner-scoped invalidation, and persisted same-host process synchronization.\n\nApplication code must still:\n\n- pass `_version` for user edits that could race;\n- keep network calls and other asynchronous side effects outside mutations;\n- authorize non-owned/public data explicitly;\n- keep database files outside public static roots;\n- run scheduled off-host backups;\n- use an external database when requiring multi-host writes or continuous zero-downtime schema changes.\n"}