Observability signals

Source: apps/api/src/observability/logger.ts, apps/api/src/observability/metrics.ts, apps/api/src/observability/otel.ts, apps/api/src/observability/otel-logs.ts, probe handlers in apps/api/src/server/routes/health.ts. Metric-recording and span call sites also live in libs/engine-core/src/engine/interpreter.ts, apps/api/src/restate/services.ts and apps/api/src/restate/run-outcome.ts, cited below where an instrument or span is actually emitted. For setup steps, see how-to/enable-observability.md.

Processes and service identity

Both processes call initLogger(...) and startOtel(...) once, at the top of their entrypoint, before any other module reads process.env, then install the shutdown path (exitOnSignals, apps/api/src/shutdown.ts) described in Shutdown.

Structured logs

Log record envelope

Every log line is one JSON object (pino), or pino-pretty formatted text when pretty is active. Fields present on every line:

Level and format control

Full variable catalogue: reference/environment-variables.md.

Redaction

baseOptions.redact in logger.ts censors these paths on every logger (root and children), replacing the value with "[redacted]":

*.authorization
*.apiKey
*.api_key
*.password
*.token
req.headers.authorization
config.apiKey

No event or patient payload is ever passed to the logger by design: events are logged by eventType, eventId and the context ids of subjects only. The contract messages of a declared event name the field and its expected type, never the value received (libs/shared/src/engine/event-contract.ts). This redaction list is defense in depth, not the primary control. See explanation/security-posture.md.

Component loggers

componentLogger(component) (logger.ts) returns a lazily-derived child of the current root (root.child({ component })), re-derived on first use after any root change. Components instantiated in the current code:

Log call sites (selected)

Message strings are reproduced verbatim (as emitted, untranslated).

The engine messages that end a run's attempt are listed in Run outcome.

route above is normalized before logging (normalizeRoute, apps/api/src/server/http-server.ts): /api/runs/{id}, /api/internal/runs/{id}/progress, /api/resume/{runId}/{nodeId}, /api/agents/{id}/runs/{runId}, /api/agents/{id}/invoke, /api/agents/{id}/schema, /api/records/{kind}/{id}, /api/memory/{kind}/{id}, /api/hooks/{id}, /api/credentials/{id}, /api/connections/{id}, /api/ingress/{id} — no run or subject identifier appears as a route label; the identifier itself is logged separately as runId where relevant. The same label is the http.route of the HTTP span and of meridian.http.server.duration.

The two ne peut pas tourner errors are logged each time the proxy loads the workflows, for each active workflow that cannot run: a node type is missing from the catalogue, or the event of its trigger is declared by no loaded plugin (Événement reçu configured on an event whose plugin is gone or that was renamed). plugins lists the loaded plugins; the workflow's new runs fail. Any other validation error of an active workflow is logged at warn (« <file> » invalide (<n> erreur(s)) — cf. GET /api/workflows.).

sourceEventId is the id of the event as received; runId is <eventId>:<slug> for a run started by a domain event. match is the reason the workflow received the event: selector, unfiltered, targeted or pinned.

Event source fhir-poll

Messages of the fhir-poll source of @posos/fhir (external-plugins/fhir/sources/poll.ts), logged by the sources component at info with the field connection (the identifier of the pulled inbound connection, such as fhir-poll). <instant> is the cursor of the feed: the time of its last processed record, or, before any record is processed, the initial cursor (start time minus startOverlapSeconds, or the epoch for a since: epoch feed).

Event ingestion

Logged by the proxy's root logger (apps/api/src/server/ingest.ts) while it routes an event to the workflows that receive it.

Run outcome

Logged by the engine component at the end of each attempt of the run handler (apps/api/src/restate/run-outcome.ts, called by apps/api/src/restate/services.ts), together with meridian.runs.executed, and by the endpoint's SDK logger transport (apps/api/src/restate/sdk-logger.ts) for a replay that diverges without an error.

Restate replays the journal of a run in flight at each resumption: a webhook resume, a decided validation, a restart of the endpoint. When the endpoint no longer runs the code or the plugins that wrote the journal (the plugin that declares the event of the trigger unloaded, a node type gone), the replay throws before the end of the journal. The write of the failed status that follows does not match the journal, so the SDK abandons the attempt (Restate error RT0016, Journal mismatch) and Restate retries the invocation with backoff. Nothing is persisted: the run keeps its last status (waiting, suspended…) in the run index and on its page, and goes on at the first attempt that matches the journal again. The endpoint tells this case from a failure of the run by whether the attempt was still replaying the journal when the error was raised (isProcessing() of the SDK's restate.internal.ContextInternal, read before the status is written). error is then the error that stopped the replay.

A replay can also diverge without an error: for a run of journal format 1, a plugin version that projects the trigger's event differently, or that changes the title of a task the run waits on, writes a different state where the journal holds another (a run of format 2 reads them from its journal, see Journal formats). The SDK abandons the attempt with the same RT0016 error. The endpoint's logger transport recognizes the SDK's error (a RestateError of code 570 on the run invocation of meridianWorkflow) and logs the same warning, with error = the first line of the SDK's message ((570) Found a mismatch between the code paths taken during the previous execution and the paths taken during this execution.) and command = the journal command that differs (set state [1]). The compared values, which carry the run's state, are not logged. The end of an attempt and the transport share the registry of the reported invocations: an invocation is reported once, whichever of the two sees it first, and stays in the registry (bounded at 1000 invocations).

After the mismatch the handler can still reach its end: the SDK serves the durable operations that follow as pending, and a bounded wait, such as a validation with a deadline, expires at once. That end is not the run's: the attempt is abandoned and Restate retries it. The transport marks the attempt, and the end of an attempt that is marked, or after which isProcessing() stays false, is neither logged run terminé nor counted.

To unblock such a run, restore the endpoint and plugin versions that started it. To end it, kill the invocation invocationId in Restate: a cancel is not processed while the replay fails, since Restate delivers it to the next attempt, whose replay fails before reading it; only a kill ends the invocation. Restate lists the invocation as backing-off, with RT0016 as its last failure, and the SDK writes its own [restate] lines (Error when processing a Restate context operation., then RestateError: (570) Found a mismatch between the code paths…) to the endpoint's standard error at each attempt. See Recover runs blocked by a missing plugin.

A record port that fails at the publication or the closing of a validation request does not change the run: the endpoint writes publication de la demande de validation échouée : <error> or clôture de la demande de validation échouée : <kind> : <error> to the console as a Restate SDK line, at warn, once (the steps are journaled). The closing exists for the runs of journal format 2.

When the settlement withdraws the request of a killed run and the record port keeps failing, the record service retries for up to one hour, then logs clôture de la demande de validation abandonnée après les relances : la clore dans le dossier (component engine, warn, with runId, pendingId, kind and error) and the invocation fails.

The transport writes every line of the SDK to the console as the SDK's default transport does, which the SDK does not export: nothing while replaying, nothing below RESTATE_LOGGING (info by default), console.info and console.debug (standard output) for info and debug, the standard error for warn, error and trace, each prefixed [restate][<time>][<target>][<invocationId>].

Run settlement

Logged by the runs component of the proxy when it freezes the terminal status of a run whose invocation ended in Restate without a terminal snapshot (apps/api/src/server/runs.ts, run-settlement.ts; see Run index — Final status), together with meridian.runs.settled.

Plugin load warnings

Logged by the plugins component when the API loads the plugins (apps/api/src/plugins.ts), in both processes. The load problems share the fields plugin, kind, name (event type, node id, binding name, module path for module, key path for key, dependency name for dependency, or plugin name for contract and sdk) and issue (the verbatim problem, ending with the fix). They are logged at warn, except the filter note, logged at info, and appended to LoadedPlugins.issues:

The Plugins page marks dropped nodes, receptions, bindings and events ignored at load. An adapter, event source or port sandbox left out by module non chargé, and the warnings that drop nothing, appear only here (see Inspect loaded plugins). The plugins of the repository load without any of these warnings or notes (apps/api/tests/unit/declared-events-load.test.ts).

Other load messages:

Metrics

Instruments are created lazily on first use (metrics.ts, instruments() cache) so that an instrument created before startOtel registers the global MeterProvider never freezes as a permanent no-op; without startOtel running at all, every recorder call is a zero-cost no-op (OTel API default). All instruments share one meter, metrics.getMeter("meridian"), and their names carry the meridian. prefix.

Instrument catalogue

Durations are computed as performance.now() deltas in milliseconds, matching the declared unit: "ms".

Contract outcomes: valid — no issue; invalid — at least one issue, and the event was refused (mode: enforce, also counted rejected in meridian.events.ingested) or let through (mode: observe). An event of a type without a contract is not counted, nor any event when EVENT_CONTRACTS=off. The warning log carries the number of issues, never the event's content; the issue messages are in the recent events (GET /api/events/recent, contract).

Settled endings: completed and failed — the status the run wrote at its end, or the result its run handler returned; killed — the invocation was killed (409) without a terminal snapshot, frozen failed; errored — the invocation ended in another failure returned by Restate, frozen failed; lost — the Restate of the run no longer knows the invocation, frozen failed. Sources: read — the runs list, a run's detail, the settling of an ingestion, of a decision or of a resume; sweep — the settlement pass (RUN_SETTLE_INTERVAL_MS). workflow is the run's workflow name, as for meridian.runs.started. A lasting lost rate means the proxy freezes runs whose trace is gone; a completed or failed rate from sweep counts the runs that ended while nobody read them.

Ingestion outcomes: accepted — at least one run created; duplicate — every workflow that receives the event already had it; filtered — workflows receive the event type but no trigger filter keeps the event; rejected — refused (400); unavailable — at least one run could not be submitted to Restate (503, the other deliveries were tried).

meridian.events.generated counts one outcome per binding at each delivery attempt of a message or of a source record, with the same values. unavailable: a run could not be submitted to Restate; the reception delivers the message or record again, and the retry starts only the missing runs. For a source record, rejected also counts a projection that failed and a record whose required fields are empty; source is the name of the source (fhir-poll), not of the connection.

meridian.nodes.executed's status attribute type allows done or skipped, but every current call site (libs/engine-core/src/engine/interpreter.ts, standard node and map-body node execution) passes "done" only; skipped nodes are recorded via the observation/log trace (node-skipped-after, node-skipped-missing-input), not via this counter.

Human-task nodes still call recordNodeExecuted({ ..., status: "done" }) after resolution, but are not wrapped in a duration span or histogram (see Span inventory).

Shared tracer re-export

metrics.ts re-exports tracer, withSpan, injectTraceContext, and runWithRemoteContext from @meridian/shared/observability/trace.ts so that proxy, engine, and plugin call sites can import them from ../observability/metrics.js without a second dependency.

Traces

Tracer and span helper

libs/shared/src/observability/trace.ts defines the tracer shared by every process and plugin: trace.getTracer("meridian"). withSpan(name, attributes, fn) runs fn inside tracer.startActiveSpan(...): on throw it calls span.recordException(e) and sets SpanStatusCode.ERROR, then always calls span.end(); the active span becomes the implicit parent of any span created inside fn.

Span inventory

Human-task nodes are not wrapped in a span: a human-task suspends the Restate invocation, and a span left open across that suspension would either span the entire wait or be recreated on replay; only the meridian.nodes.executed counter is recorded for them.

Plugin adapters emit further child spans under the node span that calls them, using the same shared tracer, outside the files this document otherwise sources from:

Trace propagation across the Restate boundary

• injectTraceContext() (trace.ts): W3C carrier via propagation.inject. Used twice for the same run, belt-and-suspenders: as HTTP headers on the ingress fetch call, and as the trace field of the run/send payload (apps/api/src/server/ingestion.ts, startRun) — the payload field is the one the engine actually consumes, since it is journaled by Restate. An event received by several workflows starts several runs: the proxy's trace holds one workflow.run span per run.
• runWithRemoteContext(carrier, fn) (trace.ts): propagation.extract + context.with; empty/undefined carrier runs fn under the current context unchanged. Used once, around workflow.run in apps/api/src/restate/services.ts, so engine-side spans become children of the proxy's request trace.

See contributing/explanation/restate-integration.md for how the Restate invocation itself is structured.

OTel bootstrap (otel.ts)

Activation

startOtel(serviceName) is a no-op if OTEL_EXPORTER_OTLP_ENDPOINT is unset, or if already called once (module-level sdk guard — idempotent).

Resource and exporters

All three exporters are constructed with no arguments; endpoint resolution beyond OTEL_EXPORTER_OTLP_ENDPOINT follows the OTel JS SDK's own defaults.

Lifecycle

• On successful start, logs export OTel démarré (traces + métriques + logs) with field endpoint (component otel).
• Registers no signal handler: each entry point flushes through flushOtel() after its stop() (see Shutdown).
• flushOtel() calls sdk.shutdown() and clears the module-level sdk reference; a call during a flush in progress awaits that flush.

Shutdown

On SIGINT/SIGTERM, each entry point (apps/api/src/server/api.ts, apps/api/src/restate/server.ts) runs one sequence (exitOnSignals, apps/api/src/shutdown.ts): log arrêt demandé, the application's stop(), flushOtel(), then exit with code 0.

• Proxy stop(): memory sweep timer, run settlement timer, schedule campaign (its wait between two attempts is interrupted), then, together, the inbound reception (pulled sources included) and a settlement pass in progress (it stops between two runs, its ingress request interrupted), HTTP listener, then the six stores. A close that fails is logged fermeture en échec with its component; the other closes still run, then stop() throws an AggregateError.
• Endpoint stop(): agent refresh timer and the registration retries. The HTTP/2 listener is the Restate SDK's (listen()), which has no close: it closes when the process exits. An invocation cut by the exit is retried by Restate, which replays it from its journal on the next endpoint.

A stop() that throws is logged arrêt en échec and the process exits with code 1 after the flush. The whole sequence is bounded by SHUTDOWN_TIMEOUT_MS (default 10 000): past it, arrêt interrompu : délai dépassé and exit code 1 — an OTLP collector that accepts connections without answering cannot keep the process, or its port, alive.

A signal received during boot:

• before start() is called (configuration, instance and plugin loading, store opening): the process exits right away after the OpenTelemetry flush, with code 0;
• once start() runs: the stop waits for start() to settle, then follows the same sequence. A start() that fails first stops what it had started (timers, schedule campaign, inbound reception, listener).

Logs → OTel bridge (otel-logs.ts)

Activation condition

shipLogs in logger.ts: !pretty && Boolean(OTEL_EXPORTER_OTLP_ENDPOINT). When true, makeOtelLogStream() is added as a second stream in pino's multistream, alongside process.stdout (stdout is always written to, regardless of shipping).

Mechanism

makeOtelLogStream() returns a Writable that, per pino JSON line:

1. Parses the line; on parse failure, silently drops it (never throws).

2. Maps the pino numeric level to an OTel SeverityNumber/text via the first matching threshold, highest first:

   pino level ≥	SeverityNumber	severityText
   60	FATAL	FATAL
   50	ERROR	ERROR
   40	WARN	WARN
   30	INFO	INFO
   20	DEBUG	DEBUG
   0	TRACE	TRACE

   (defaults to level 30/INFO if rec.level isn't a number)

3. Builds OTel log attributes from every JSON field except the pino-meta set level, time, msg, pid, hostname, v, trace_id, span_id, trace_flags; object/null values are JSON.stringify-ed, everything else is passed through as string/number/boolean.

4. If rec.trace_id and rec.span_id are both strings, reconstructs a SpanContext (trace_flags ?? TraceFlags.SAMPLED, isRemote: false) and attaches it as the emitted record's context, so the log correlates to its trace even though the write happens outside the originating span's active context.

5. Emits via logs.getLogger("meridian").emit({ severityNumber, severityText, body: rec.msg ?? "", attributes, context?, timestamp: rec.time }). The OTel logger is resolved on every write (not cached), since the stream is constructed before startOtel runs and must not freeze a no-op logger.

HTTP probes (apps/api/src/server/routes/health.ts)

Both checks run on every /readyz call (no caching); either failing independently flips ready to false while leaving the other's boolean observable in checks. The engine process (apps/api/src/restate/server.ts) exposes neither route.

Related

• how-to/enable-observability.md — wiring steps, the LGTM stack, and probe configuration.
• reference/environment-variables.md — full variable catalogue.
• reference/topology.md — where the proxy and engine sit, and what probes them.
• reference/run-store.md — the run index checked by /readyz's checks.runStore.
• explanation/why-durable-execution.md — why spans can be re-emitted across a suspend/resume replay.
• explanation/security-posture.md — what is and isn't allowed to reach logs and traces.
• contributing/explanation/restate-integration.md — the Restate invocation workflow.run and its span sit inside.
• contributing/explanation/interpreter-internals.md — node execution and map-body iteration that node <id> / node <id>#<i> spans wrap.