Observability signals

Source: apps/api/src/observability/logger.ts, apps/api/src/observability/metrics.ts, apps/api/src/observability/otel.ts, apps/api/src/observability/otel-logs.ts, probe handlers in apps/api/src/server/api.ts. Metric-recording and span call sites also live in libs/engine-core/src/engine/interpreter.ts and apps/api/src/restate/services.ts, cited below where an instrument or span is actually emitted. For setup steps, see how-to/enable-observability.md.

Processes and service identity

Both processes call initLogger(...) and startOtel(...) once, at the top of their entrypoint, before any other module reads process.env.

Structured logs

Log record envelope

Every log line is one JSON object (pino), or pino-pretty formatted text when pretty is active. Fields present on every line:

Level and format control

Full variable catalogue: reference/environment-variables.md.

Redaction

baseOptions.redact in logger.ts censors these paths on every logger (root and children), replacing the value with "[redacted]":

*.authorization
*.apiKey
*.api_key
*.password
*.token
req.headers.authorization
config.apiKey

No event or patient payload is ever passed to the logger by design (events are logged by type/eventId/patientId only) — this redaction list is defense in depth, not the primary control. See explanation/security-posture.md.

Component loggers

componentLogger(component) (logger.ts) returns a lazily-derived child of the current root (root.child({ component })), re-derived on first use after any root change. Components instantiated in the current code:

Log call sites (selected)

Message strings are reproduced verbatim (as emitted, untranslated).

route above is normalized before logging (/api/runs/{id}, /api/records/{kind}/{id}) — no run or subject identifier appears as a route label; the identifier itself is logged separately as runId where relevant.

Metrics

Instruments are created lazily on first use (metrics.ts, instruments() cache) so that an instrument created before startOtel registers the global MeterProvider never freezes as a permanent no-op; without startOtel running at all, every recorder call is a zero-cost no-op (OTel API default). All instruments share one meter: metrics.getMeter("meridian").

Instrument catalogue

Durations are computed as performance.now() deltas in milliseconds, matching the declared unit: "ms".

clinical.nodes.executed's status attribute type allows done or skipped, but every current call site (libs/engine-core/src/engine/interpreter.ts, standard node and map-body node execution) passes "done" only; skipped nodes are recorded via the observation/log trace (node-skipped-after, node-skipped-missing-input), not via this counter.

Human-task nodes still call recordNodeExecuted({ ..., status: "done" }) after resolution, but are not wrapped in a duration span or histogram (see Span inventory).

Shared tracer re-export

metrics.ts re-exports tracer, withSpan, injectTraceContext, and runWithRemoteContext from @meridian/shared/observability/trace.ts so that proxy, engine, and plugin call sites can import them from ../observability/metrics.js without a second dependency.

Traces

Tracer and span helper

libs/shared/src/observability/trace.ts defines the tracer shared by every process and plugin: trace.getTracer("meridian"). withSpan(name, attributes, fn) runs fn inside tracer.startActiveSpan(...): on throw it calls span.recordException(e) and sets SpanStatusCode.ERROR, then always calls span.end(); the active span becomes the implicit parent of any span created inside fn.

Span inventory

Human-task nodes are not wrapped in a span: a human-task suspends the Restate invocation, and a span left open across that suspension would either span the entire wait or be recreated on replay; only the clinical.nodes.executed counter is recorded for them.

Plugin adapters emit further child spans under the node span that calls them, using the same shared tracer, outside the files this document otherwise sources from:

Trace propagation across the Restate boundary

• injectTraceContext() (trace.ts): W3C carrier via propagation.inject. Used twice for the same run, belt-and-suspenders: as HTTP headers on the ingress fetch call, and as the trace field of the run/send payload (apps/api/src/server/api.ts, opSendEvent) — the payload field is the one the engine actually consumes, since it is journaled by Restate.
• runWithRemoteContext(carrier, fn) (trace.ts): propagation.extract + context.with; empty/undefined carrier runs fn under the current context unchanged. Used once, around workflow.run in apps/api/src/restate/services.ts, so engine-side spans become children of the proxy's request trace.

See contributing/explanation/restate-integration.md for how the Restate invocation itself is structured.

OTel bootstrap (otel.ts)

Activation

startOtel(serviceName) is a no-op if OTEL_EXPORTER_OTLP_ENDPOINT is unset, or if already called once (module-level sdk guard — idempotent).

Resource and exporters

All three exporters are constructed with no arguments; endpoint resolution beyond OTEL_EXPORTER_OTLP_ENDPOINT follows the OTel JS SDK's own defaults.

Lifecycle

• On successful start, logs export OTel démarré (traces + métriques + logs) with field endpoint (component otel).
• Registers SIGINT/SIGTERM handlers that call sdk.shutdown().
• flushOtel() calls sdk.shutdown() and clears the module-level sdk reference — for short-lived scripts that need buffers flushed before exit.

Logs → OTel bridge (otel-logs.ts)

Activation condition

shipLogs in logger.ts: !pretty && Boolean(OTEL_EXPORTER_OTLP_ENDPOINT). When true, makeOtelLogStream() is added as a second stream in pino's multistream, alongside process.stdout (stdout is always written to, regardless of shipping).

Mechanism

makeOtelLogStream() returns a Writable that, per pino JSON line:

1. Parses the line; on parse failure, silently drops it (never throws).

2. Maps the pino numeric level to an OTel SeverityNumber/text via the first matching threshold, highest first:

   pino level ≥	SeverityNumber	severityText
   60	FATAL	FATAL
   50	ERROR	ERROR
   40	WARN	WARN
   30	INFO	INFO
   20	DEBUG	DEBUG
   0	TRACE	TRACE

   (defaults to level 30/INFO if rec.level isn't a number)

3. Builds OTel log attributes from every JSON field except the pino-meta set level, time, msg, pid, hostname, v, trace_id, span_id, trace_flags; object/null values are JSON.stringify-ed, everything else is passed through as string/number/boolean.

4. If rec.trace_id and rec.span_id are both strings, reconstructs a SpanContext (trace_flags ?? TraceFlags.SAMPLED, isRemote: false) and attaches it as the emitted record's context, so the log correlates to its trace even though the write happens outside the originating span's active context.

5. Emits via logs.getLogger("clinical").emit({ severityNumber, severityText, body: rec.msg ?? "", attributes, context?, timestamp: rec.time }). The OTel logger is resolved on every write (not cached), since the stream is constructed before startOtel runs and must not freeze a no-op logger.

HTTP probes (apps/api/src/server/api.ts)

Both checks run on every /readyz call (no caching); either failing independently flips ready to false while leaving the other's boolean observable in checks. The engine process (apps/api/src/restate/server.ts) exposes neither route.

Related

• how-to/enable-observability.md — wiring steps, the LGTM stack, and probe configuration.
• reference/environment-variables.md — full variable catalogue.
• reference/topology.md — where the proxy and engine sit, and what probes them.
• reference/run-store.md — the run index checked by /readyz's checks.runStore.
• explanation/why-durable-execution.md — why spans can be re-emitted across a suspend/resume replay.
• explanation/security-posture.md — what is and isn't allowed to reach logs and traces.
• contributing/explanation/restate-integration.md — the Restate invocation workflow.run and its span sit inside.
• contributing/explanation/interpreter-internals.md — node execution and map-body iteration that node <id> / node <id>#<i> spans wrap.