Observability

One createObservability instance provides structured logs, W3C trace propagation, metrics, and readiness checks without a runtime package.

import {
  createObservability,
  createOtlpHttpSpanExporter,
} from "@clank.run/framework/observability";

const observability = createObservability({
  serviceName: "orbit-tasks",
  serviceVersion: "1.0.0",
  environment: process.env.NODE_ENV,
  exporter: process.env.OTLP_TRACES_URL
    ? createOtlpHttpSpanExporter({
        url: process.env.OTLP_TRACES_URL,
        headers: { authorization: `Bearer ${process.env.OTLP_TOKEN}` },
        serviceName: "orbit-tasks",
      })
    : undefined,
});

const app = createApp()
  .use(observability.middleware())
  .get("/healthz", () => observability.health.response());

Generated apps install the middleware and a database readiness check automatically.

#Logs

Logs are one JSON object per event and automatically include service, trace, span, and request IDs. Keys containing password, secret, token, authorization, cookie, or API-key patterns are recursively redacted. Values, nesting, arrays, event counts, and attribute counts are bounded.

Use child loggers for stable context:

const log = observability.logger.child({ component: "reminders" });
log.info("Reminder queued.", { taskId });

Do not attach email addresses, request bodies, query strings, session IDs, or unbounded user identifiers as metric labels.

#Traces

Incoming traceparent headers are validated according to W3C Trace Context. Server spans preserve sampled trace IDs, create a new span ID, return traceparent, and attach a safe request ID.

await observability.tracer.trace("reminder.send", async (span) => {
  span.setAttribute("reminder.channel", "email");
  await sendReminder();
});

Exceptions become bounded span events and error status. The OTLP/HTTP JSON exporter uses protocol byte encoding, HTTPS, explicit headers, a timeout, and redirect rejection.

#Metrics

The registry supports counters, gauges, and histograms and renders Prometheus text:

const sent = observability.metrics.counter(
  "app_reminders_sent_total",
  "Reminders accepted by the delivery service.",
  ["channel"],
);

sent.add(1, { channel: "email" });

Metric and label names are validated, label sets must match their declaration, histograms use cumulative buckets, and a global series limit prevents accidental memory exhaustion. HTTP middleware normalizes ID-like route segments before using the path as a label.

Expose observability.metrics.response() only on an operator-protected endpoint or private network.

#Deployment ingress metrics

Clank Deploy separately records per-project traffic that passes through managed ingress. Minute rows contain status classes, 5xx errors, fixed method counters, duration sum/max, cumulative latency buckets, and bounded byte counts. The dashboard returns bounded 15m, 1h, 24h, 7d, and 30d current/previous series, rates, percentiles, and distributions without storing host, path, IP, user, email, query-string, or user-agent labels.

Ingress latency ends when upstream response headers arrive; it is not streamed-response completion time. Response size is known only when Content-Length is present and a response can carry a body. Limit-denial responses remain visible in this operational series.

For a provider-hosted project, the Performance view also requests a generation-bound private resource sample: Docker memory/limit, CPU, PIDs, cumulative network I/O, and cumulative block I/O across its web/worker/scheduler topology. These counters reset on a new generation and are not stored as time series. The Logs view merges a bounded provider-process memory tail with durable control-plane lifecycle events after configured project-secret redaction. No container identity, environment, path, or raw Docker failure crosses the provider boundary.

A separate monthly ledger records only admitted requests, their request bytes, declared response bytes, and traffic-limit rejections. It has longer configurable retention and drives /usage, GET /api/usage, and clank usage; it is not a trace, total-egress estimate, or billing record. See Usage accounting and traffic limits and Deployment dashboard, quotas, and domains.

#Jobs and cron

runtime.jobs.stats() returns bounded queue-state counts, due work, and oldest-due time. Per-job history is available through events(id), while release logs identify worker[n] and scheduler streams. Platform memory diagnostics report web, worker, and scheduler processes separately.

Use stable job name, queue, state, and attempt fields in logs and traces. Do not use job IDs, arguments, user IDs, error messages, group keys, or idempotency keys as metric labels. Alert on:

• dead jobs greater than zero;
• oldest-due age beyond the queue's objective;
• repeated lease expiry or timeout events;
• background process restart loops; and
• scheduler last-error or a future occurrence that stops advancing.

Read Durable jobs and cron for inspection, retention, and OpenTelemetry interoperability guidance.

#Readiness

Critical checks determine the HTTP status. Optional dependencies remain visible without taking the service out of rotation:

observability.health.register("database", checkDatabase);
observability.health.register("email", checkMail, { critical: false });

Checks run concurrently with individual timeouts. Responses use no-store and return 503 when a critical dependency fails.

#Request-to-job timelines

Pass the same tracer to HTTP instrumentation and the backend. Backend queries and mutations create metadata-only spans. Enqueued jobs persist the current trace context in the same durable row as their work; a worker can restore it after a process restart. Each attempt gets its own consumer span, linked to the original mutation. Retrying or deduplicating a job preserves its original parent. Workflow runs retain their starting context for later dependent steps.

import { createObservability, openBackend } from "@clank.run/framework";
import { createTraceTimeline } from "@clank.run/framework/trace-timeline";
import { createDevtools, serveDevtools } from "@clank.run/framework/devtools";

const timeline = createTraceTimeline({ maxSpans: 500 });
const telemetry = createObservability({ serviceName: "my-app", exporter: timeline });
const backend = await openBackend(definition, { path: "dev.sqlite", tracer: telemetry.tracer });
const inspector = createDevtools({ timeline: () => timeline.snapshot() });
const panel = await serveDevtools(inspector);
// Install telemetry.middleware() on your app, or wrap its handler with telemetry.instrument().

A separate worker uses its own createObservability() instance and passes that tracer to openBackend() or openJobs(). With an OTLP exporter, the web and worker spans meet in the configured collector using the persisted trace ID. The local in-memory timeline shows only spans exported to that instance; it is not a cross-process collector. snapshot(traceId) filters one trace, and the DevTools panel displays operation, request ID, parent span, timing, job ID, and attempt. Job IDs are trace attributes, never metric labels.

The timeline excludes raw URL paths, arbitrary custom span names, all unrecognized attributes, arguments, results, and exception details. HTTP spans are labelled “HTTP request”; framework query/mutation/job names remain visible. It retains at most 500 spans by default (configurable 1–5,000), labels truncated history, and renders escaped HTML. It is a local operator tool with no public application endpoint or implied per-user access control. Use static operation names and opaque request IDs; the framework's ordinary OTLP exporter retains its existing richer metadata.

Sampling still applies. Legacy jobs with no context start an independent trace; malformed persisted contexts are ignored. Cancellation, timeout, failure, and lost-lease attempts are not reported as successful spans. Job completion spans measure the attempt through settlement, not time spent waiting in the queue. Stop the inspector, backend, and telemetry when shutting down.

#Release error inbox

openErrorInbox(database) from @clank.run/framework/error-inbox persists bounded operator-only error metadata in SQLite. Capture failures from application error hooks with an explicit release identity and, when available, the request's trace ID:

const inbox = await openErrorInbox(database);
inbox.capture(error, { release: "0.22.1", code: "TICKET_SAVE_FAILED", traceId });
const inspector = createDevtools({ errorInbox: () => inbox.snapshot() });

The loopback DevTools view groups occurrences, counts them by release, and displays recent frame locations and trace IDs. Integrate the snapshot with an existing trace timeline using those IDs. inbox.resolve(fingerprint, release) records the intended fixing release. An occurrence captured after resolution changes the group to regressed; this includes older instances that are still running, so inspect the occurrence's actual release. Silence alone is not proof of a fix—compare traffic and observation windows.

Register a matching version-3 source map with inbox.registerSourceMap(release, generatedFile, sourceMap) before capture. Mapping uses Node's built-in source-map reader; maps are bound to a release and normalized filename and remain in memory, so reload them after restart. Generated and original locations are one-based. Only the last source directory and filename are retained; maps sharing that suffix must not be registered for the same release. The report marks unmapped frames explicitly. Grouping hashes the declared error code and first three normalized source locations, so moving source lines can create a new group. Source contents are discarded from registered maps.

Capture stores at most eight frames, release ID, timestamp, code, and an optional validated trace ID. Messages, raw stacks, URL queries, source contents, request data, and absolute deployment roots are excluded. Use stable application error codes; never put user data in release/code/file names. The default retention is 1,000 occurrences over seven days, with configurable bounds of 10,000 events and 90 days. Counts and first/last-seen times describe the retained window.

The API is for trusted server/operator use. Mount it only behind operator authorization or the existing loopback DevTools server; no public ingestion or unauthenticated production route is created. Call capture outside an existing database transaction, and keep diagnostics failures separate from the application's error response.