{"protocol":"clank-doc/1","frameworkVersion":"0.24.0","slug":"durable-data-workflows","title":"Durable data workflows","description":"Clank includes authenticated services and DOM controls for shared document editing, server search, bulk changes, shared views, and resumable CSV imports. They use SQLite and the existing auth, origin, and CSRF checks. Browser helpers use sa","group":{"id":"full-stack","title":"Full stack"},"url":"https://docs.clank.run/docs/durable-data-workflows","source":"docs/durable-data-workflows.md","headings":["Mount services beside your backend","Preview and apply a bulk edit","Persist collaborative document edits","Search authorized records","Publish shared saved views","Resume large CSV imports","Database upgrades and rollback"],"tableOfContents":[{"id":"mount-services-beside-your-backend","title":"Mount services beside your backend","level":2},{"id":"preview-and-apply-a-bulk-edit","title":"Preview and apply a bulk edit","level":2},{"id":"persist-collaborative-document-edits","title":"Persist collaborative document edits","level":2},{"id":"search-authorized-records","title":"Search authorized records","level":2},{"id":"publish-shared-saved-views","title":"Publish shared saved views","level":2},{"id":"resume-large-csv-imports","title":"Resume large CSV imports","level":2},{"id":"database-upgrades-and-rollback","title":"Database upgrades and rollback","level":2}],"markdown":"# Durable data workflows\n\nClank includes authenticated services and DOM controls for shared document editing, server search,\nbulk changes, shared views, and resumable CSV imports. They use SQLite and the existing auth,\norigin, and CSRF checks. Browser helpers use same-origin cookies and accept the same `auth` client\nas `createSyncClient`. Every `mount…` function returns a cleanup function; call it on navigation,\nworkspace changes, or logout.\n\n## Mount services beside your backend\n\nThe server supplies the database schema, editable fields, and authorization rules. Browser input\ncannot select a different table or policy. Use the same SQLite path and auth definition as your\napplication. Pass its schema when authorization needs to read application records:\n\n```ts\nimport { openBulkEditor } from \"@clank.run/framework/bulk-edit\";\nimport { openDurableImport } from \"@clank.run/framework/durable-import\";\n\n// auth, schema, and application are your existing server definitions/runtime.\n// This example's records table is .owned(), with title:string and score:integer.\nconst bulk = await openBulkEditor({\n  path: \"app.sqlite\", auth, schema, table: \"records\", fields: [\"title\", \"score\"],\n});\nconst imports = await openDurableImport({\n  path: \"app.sqlite\", auth, schema, table: \"records\", fields: [\"title\", \"score\"],\n  uniqueBy: [\"title\"], batchSize: 100,\n});\nconst services = new Map([\n  [\"/__clank/bulk\", bulk], [\"/__clank/imports\", imports],\n]);\nasync function handle(request: Request): Promise<Response> {\n  const pathname = new URL(request.url).pathname;\n  for (const [prefix, service] of services) {\n    if (pathname === prefix || pathname.startsWith(`${prefix}/`)) {\n      return service.handle(request);\n    }\n  }\n  return application.handle(request);\n}\n// On shutdown: close each service, then your application runtime.\n```\n\nOther default mount prefixes are `/__clank/documents`, `/__clank/search`, and\n`/__clank/shared-views`. Set a service's `prefix` and the browser client's `url` together when\nchanging them. Route feature prefixes before the application's general `/__clank` handler.\nThese services are not automatically published as agent tools.\n\nAll authorization callbacks are **synchronous** and must return exactly `true` to grant access.\nUse the provided `context.db` to evaluate current membership and row permissions in the same\ntransaction as the operation. An async callback returns a Promise and is denied. Query caching is\ndisabled for these services, so each HTTP read reevaluates current access, including policies\nbacked by external state. External permission changes do not proactively erase data already\nrendered in another browser; refresh, polling, and subsequent operations discover revocation.\n\nOwned tables retain the backend's account isolation. Bulk edits and imports targeting an unowned\ntable require an explicit `authorize` callback. For example, a policy for an application with an\nunowned `memberships` table indexed by workspace/user can read it synchronously:\n\n```ts\nimport type { AuthRequest, ReadDatabase } from \"@clank.run/framework\";\n\nfunction canWrite(\n  { auth, db }: { auth: AuthRequest; db: ReadDatabase<typeof schema> },\n  record: Readonly<Record<string, unknown>>,\n) {\n  if (typeof record.workspaceId !== \"string\") return false;\n  const member = db.table(\"memberships\").query()\n    .where(\"workspaceId\", record.workspaceId)\n    .where(\"userId\", auth.requireUser().id).first();\n  return member?.role === \"owner\" || member?.role === \"editor\";\n}\n```\n\nSupply that policy to both services when the target is shared. Include `workspaceId` in imported\nfields when it belongs to the target schema, and validate it through the policy. Do not authorize\nfrom a browser-supplied role or an assumed current workspace. The backend's other application\nmutation handlers are not called by these services: keep all required field constraints in the\nschema and all required record rules in the supplied service policy.\n\n## Preview and apply a bulk edit\n\n```ts\nimport { createBulkEditClient, mountBulkEditor } from \"@clank.run/framework/bulk-edit\";\nconst client = createBulkEditClient({ auth: browserAuth });\nconst dispose = mountBulkEditor(bulkContainer, client, {\n  selection: () => selectedRecordIds,\n  changes: () => ({ score: Number(scoreInput.value) }),\n  applied: () => refreshRecords(),\n});\n```\n\nThe control displays the before/after records and requires a separate apply action. The service\nvalidates every selected record, editable field, schema value, permission, and reviewed version.\nApply rechecks all of them inside one transaction before writing any row. A denied or stale row\nrejects the entire batch. The default limit is 200 records; `maxRecords` allows up to 1,000.\nPreview includes each record's declared fields, so preview authorization must grant record read\naccess as well as permission to propose the change.\n\nFor custom UI, call `preview(ids, changes)` and pass its result to `apply(preview)`. A\n`BULK_PREVIEW_STALE` response requires a fresh preview. If a response is lost, refresh current\nrecords before retrying: the transaction may have committed even though the browser received no\nconfirmation. This service edits database records; enqueue external effects through your own\napplication's durable job flow when they are required.\n\n## Persist collaborative document edits\n\n```ts\nimport { openCollaborativeDocuments } from \"@clank.run/framework/collaborative-documents\";\nimport { s } from \"@clank.run/framework\";\nconst documents = await openCollaborativeDocuments({\n  path: \"app.sqlite\", auth, schema,\n  authorize({ auth, db }, documentId, operation) {\n    const record = db.table(\"records\").get(s.id(\"records\").parse(documentId));\n    // .owned() applies the current account; replace with a shared membership policy as needed.\n    return record !== null;\n  },\n});\n// Mount documents.handle under /__clank/documents as above.\n```\n\n```ts\nimport {\n  createCollaborativeDocumentsClient, mountCollaborativeEditor, textEdit,\n} from \"@clank.run/framework/collaborative-documents\";\nconst documents = createCollaborativeDocumentsClient({ auth: browserAuth });\n// Create once after authorizing the corresponding application record.\nawait documents.create(recordId, \"Initial text\");\nconst dispose = mountCollaborativeEditor(editorContainer, documents, recordId);\n// A custom editor can submit a bounded splice using a stable ID across network retries:\nconst before = await documents.read(recordId);\nconst operation = {\n  documentId: recordId, operationId: crypto.randomUUID(), baseRevision: before.revision,\n  ...textEdit(before.text, \"Revised text\"),\n};\nawait documents.edit(operation);\n```\n\nText, revisions, edit history, and operation fingerprints survive server restart. Concurrent edits\nto disjoint ranges are transformed against retained operations. Overlapping ranges or a revision\nolder than the retained window return `COLLAB_EDIT_CONFLICT`. Positions count JavaScript UTF-16\ncode units, matching string slicing. The editor retains unsaved text when remote edits arrive,\nshows the latest server text, and offers explicit discard or rebase before saving again.\n\nA retry must reuse the same operation ID and exact input. Reuse with different content is rejected;\nan exact retry returns the current document and the original `acceptedRevision`. The default text\nlimit is 200,000 code units (maximum 1,000,000); `retainedOperations` defaults to 1,000 (maximum\n10,000). Operation fingerprints remain durable after transform history is pruned. Plan retention\nand database capacity for this metadata; there is no automatic document/receipt deletion API.\n\n`subscribe(id, listener, intervalMs)` polls every second by default and rechecks authorization on\neach read. It calls `listener(null, error)` on denied/unavailable reads; the built-in editor clears\ntext and disables editing then. Cleanup cancels further deliveries. There is no instantaneous\npush guarantee for externally managed membership changes. Use the separate\n[presence service](collaboration.md) when cursors or typing indicators are also useful.\n\n## Search authorized records\n\n```ts\nimport { openSearch } from \"@clank.run/framework/search\";\nimport { s } from \"@clank.run/framework\";\nconst search = await openSearch({\n  path: \"app.sqlite\", auth, schema,\n  authorize: ({ auth }, scope) => scope === auth.requireUser().id,\n  authorizeRecord: ({ db }, indexed) => db.table(\"records\").get(s.id(\"records\").parse(indexed.id)) !== null,\n});\n// Trusted server code, after loading the authoritative record:\nsearch.upsert({ scope: ownerId, id: record._id, title: record.title, body: record.description });\n// On deletion: search.remove(ownerId, recordId).\n```\n\n```ts\nimport { createSearchClient, mountSearch } from \"@clank.run/framework/search\";\nconst search = createSearchClient({ auth: browserAuth });\nconst dispose = mountSearch(searchContainer, search, {\n  scope: () => currentUserId,\n  open: id => navigateToRecord(id),\n});\n```\n\nSQLite FTS5 persists the index. Index updates are trusted server methods, never exposed browser\nmutations. Integrate `upsert`/`remove` with a durable application outbox or indexing worker;\nindex writes have their own transaction, so indexing is eventually consistent with a separate\nbusiness write. Keep `authorizeRecord` tied to current authoritative records when deletion or\nrecord permissions can change independently of the index.\n\nScope authorization runs before candidate retrieval, and optional record authorization runs\nbefore fetching text, scoring, or generating snippets. Scores use only each accessible record's\ntext, not statistics from hidden documents. Queries accept 1–10 literal words, combined with AND;\nuser-supplied FTS operators are not executed. Results contain text snippets, rendered using\n`textContent` by the control.\n\nA request returns up to 100 hits (20 by default), scans at most 5,000 candidates by default\n(`maxCandidates` up to 50,000), and stops after 16 MiB of authorized text. `total` counts authorized\nmatches within that budget, not the entire corpus; `truncated` tells the UI to narrow the search.\nIndexed titles are limited to 1,000 UTF-8 bytes and bodies to 1 MiB each.\n\n## Publish shared saved views\n\n```ts\nimport { openSharedSavedViews } from \"@clank.run/framework/saved-views\";\nconst views = await openSharedSavedViews({\n  path: \"app.sqlite\", auth, schema, fields: [\"title\", \"score\"],\n  authorize({ auth, db }, workspaceId, operation) {\n    const member = db.table(\"memberships\").query()\n      .where(\"workspaceId\", workspaceId).where(\"userId\", auth.requireUser().id).first();\n    if (!member) return false;\n    if (operation === \"read\") return true;\n    if (operation === \"default\") return member.role === \"owner\";\n    return member.role === \"owner\" || member.role === \"editor\";\n  },\n});\n```\n\n```ts\nimport { createSharedViewsClient, mountSharedSavedViews } from \"@clank.run/framework/saved-views\";\nconst views = createSharedViewsClient({ auth: browserAuth, workspaceId });\nconst dispose = mountSharedSavedViews(viewsContainer, views, {\n  current: () => currentViewDefinition,\n  apply: definition => updateView(definition),\n});\n```\n\nEvery view is visible to currently authorized workspace readers. Its author chooses who may edit:\n`owner` (default) or `workspace` editors permitted by the policy. Only its author can change that\nchoice. Authors still require current workspace membership. Updating/deleting requires the\nreviewed `expectedRevision`; stale writes return `VIEW_CHANGED`. Setting a workspace default\nrequires the separate `default` policy, and one transaction keeps at most one default.\n\nThe control exposes update/delete/default controls according to the server's current capabilities.\nEach workspace allows 50 views by default (`maxViews` up to 200). Declared fields bound filter,\nsort, and column definitions. A saved filter never grants access to data: apply it only to records\nreturned by an authorized query. Existing account-private `openSavedViews` storage and behavior\nremain available separately.\n\n## Resume large CSV imports\n\n```ts\nimport { createDurableImportClient, mountDurableImporter } from \"@clank.run/framework/durable-import\";\nconst imports = createDurableImportClient({ auth: browserAuth });\nconst columns = [\n  { source: \"Title\", target: \"title\", type: \"text\", required: true },\n  { source: \"Score\", target: \"score\", type: \"integer\", required: true },\n] as const;\nconst dispose = mountDurableImporter(importContainer, imports, { columns });\n```\n\nThe control uploads, shows the durable import ID, starts/resumes execution, refreshes progress,\nretries a failed batch, and cancels remaining rows. Save the ID to resume after browser restart.\nFor custom UI, use `uploadCsv(file, columns, { id, progress })` to resume an upload with the same\nfile, then `run(id, { progress, signal })`. The CSV reader streams files up to 100 MiB, including\nquoted newlines, through bounded chunks instead of buffering the entire file. Each record is\nbounded to 1 MiB of text; encoded JSON chunks are bounded to 4 MiB and 500 records. A server job\nallows up to 1,000,000 rows, configurable downward with `maxRows`.\n\nChunks and progress live in owned SQLite tables. Reuploading starts from the file's beginning to\ncompare every previously stored normalized chunk; changed content returns `IMPORT_CHUNK_CHANGED`.\nJob creation also accepts a caller-supplied stable key for retrying creation. Another account\ncannot inspect, append to, execute, or cancel the job.\n\n`run` drives bounded server transactions from the client; there is no hidden background worker.\nThe default apply batch is 100 rows (`batchSize` up to 500). Each batch reevaluates the current\nschema, access rules, and optional `uniqueBy` constraint for every row before any target insert.\nAny failed row persists safe row-number/error-code diagnostics and leaves the batch cursor and\nall target records unchanged. Prior successful batches remain committed. Duplicate rows fail by\ndefault; opt into `duplicates: \"skip\"` to count and skip them.\n\nInsertions and cursor advances commit together. Retrying the same processed-row cursor after a\nlost response returns durable progress without inserting a second copy. `retry` reopens only a\nfailed job after the underlying permission/duplicate problem is resolved; uploaded chunks are\nimmutable. Cancel prevents remaining batches and keeps already imported rows. Aborting or\nunmounting the browser stops future requests, but a dispatched batch may still commit.\n\nAt most 20 unfinished imports are allowed per account. Completed/cancelled job data and uploaded\nchunks are retained; include them in normal database capacity, access, backup, and retention plans.\nThe existing small-file CSV planner remains available for previews and simpler workflows.\n\n## Database upgrades and rollback\n\nThese services add their internal tables on first open; they do not rewrite existing application\nrecords or automatically backfill an index. Reserved names are `sharedSavedViews`,\n`collaborativeDocs`, `collaborativeOperations`, `collaborativeReceipts`, `durableImportJobs`,\n`durableImportChunks`, and `clank_search_fts`. Search without an application schema also uses\n`searchServiceState`. Keep one compatible schema/auth definition for each shared SQLite file.\n\nTake a consistent SQLite backup before enabling services or changing target schemas. For imports,\nfinish/cancel active jobs before incompatible schema changes; stored rows are validated again on\nexecution and may require a new job. Rollback can unmount the feature endpoints and controls;\nkeep their tables so a later compatible version can resume safely. Do not remove receipt or\ncursor metadata and then replay old requests. Already committed bulk edits, imports, and document\nedits require your application's data history/restore policy or a coordinated database restore;\nunmounting a service does not reverse writes.\n"}