Documentation / Exploitation / Guides pratiques / Enable logs, traces and metrics

Enable logs, traces and metrics

The proxy and the engine always emit structured logs to stdout. Traces, metrics, and OTLP log export are opt-in: point OTEL_EXPORTER_OTLP_ENDPOINT at a collector and they turn on; leave it unset and every instrument stays a no-op at zero cost (apps/api/src/observability/otel.ts). This how-to wires that export, brings up the bundled local visualization stack, and sets liveness/readiness probes. For the full catalogue of log fields, span names and metric instruments, see reference/observability-signals.md; for the complete variable list see reference/environment-variables.md.

Before you start

  • Observability env vars are read once at process boot (initLogger/startOtel are called at the top of apps/api/src/server/api.ts for the proxy and apps/api/src/restate/server.ts for the engine). Changing any variable below requires restarting the corresponding process to take effect.
  • The proxy and the engine are two separate processes with independent service names (meridian-proxy / meridian-endpoint) but share the same environment variable names — set them on whichever process(es) you are instrumenting.
  • Only the proxy serves /healthz and /readyz (port 4000); the engine has no HTTP probe endpoint of its own.

1. Configure structured logs

Logs are JSON-on-stdout by default outside an interactive terminal (pino, apps/api/src/observability/logger.ts), one object per line, with a service field (meridian-proxy / meridian-endpoint) and a component field per subsystem (plugins, instance, http, sources, engine, …).

  1. Set the verbosity with LOG_LEVEL (trace | debug | info | warn | error, default info).
  2. Set LOG_PRETTY to force a format: 1 forces human-readable (pino-pretty), 0 forces JSON. Leave it unset to get the default (pretty on an interactive TTY outside production, JSON otherwise — e.g. under Docker or any collector).
  3. Optionally set SERVICE_NAME to override the service field baked in by each entrypoint (meridian-proxy / meridian-endpoint).
  4. Watch the plugins component at boot. Besides plugins chargés, it logs a warning for each plugin or contribution it drops or signals: « contrat plugin incompatible — plugin ignoré », « événement signalé », « exemple d'événement retiré », « déclencheur signalé », « nœud signalé », « binding hors contrat », « plage sdk insuffisante », « module non chargé »; some drop nothing and appear nowhere else. See Plugin load warnings.

Secret-looking fields (authorization, apiKey, api_key, password, token, req.headers.authorization, config.apiKey) are redacted automatically as defense in depth. Events themselves are only ever logged by type and id — no patient payload reaches the logs, by design; see explanation/security-posture.md.

2. Export traces, metrics and logs to a collector

  1. Set OTEL_EXPORTER_OTLP_ENDPOINT to your collector's OTLP/HTTP endpoint, e.g. http://localhost:4318. This single variable enables traces + metrics + logs together — there is no per-signal toggle.
  2. Optionally set OTEL_SERVICE_NAME to override the resource service.name reported to the collector (defaults to meridian-proxy / meridian-endpoint).
  3. Restart the proxy and/or the engine. startOtel() is idempotent, so re-running it on an already-started process is harmless, but the environment variable is only read once at boot.

With the endpoint set, every HTTP request into the proxy, every Restate ingress call, and every workflow node execution produces a span in a single end-to-end trace per event, with the W3C trace context propagated from the proxy to the engine over the Restate invocation payload:

POST /api/events (proxy)
└─ restate.ingress (proxy)            (one per workflow that receives the event)
   └─ workflow.run (engine)
      ├─ node <id>            (one span per executed node; map body: node <id>#<i>)
      │  └─ fhir GET|POST|PUT  /  posos.graphql   (outbound I/O, under the calling node)
      └─ …

Platform metrics, all under the meridian. prefix (meridian.events.generated, meridian.runs.executed, meridian.nodes.duration, …), and HTTP/ingress latency histograms are exported the same way; see reference/observability-signals.md for the full instrument list and attributes (apps/api/src/observability/metrics.ts).

A human-task validation suspends the run — there is no span for the wait itself. On Restate replay after resumption, spans for nodes that already ran once may be re-emitted; single-pass workflows are unaffected. See explanation/why-durable-execution.md.

3. Correlate logs with traces

Every log line emitted while a span is active already carries trace_id / span_id (a pino mixin, apps/api/src/observability/logger.ts). To also ship those logs to your collector as OTLP log records — so you can pivot from a log line to its trace in Grafana — both of these must hold at once:

  • output is JSON, i.e. LOG_PRETTY is 0 or unset on a non-TTY process (pretty is false); and
  • OTEL_EXPORTER_OTLP_ENDPOINT is set.

When both hold, logs are shipped in-process through the pino → OTel bridge (apps/api/src/observability/otel-logs.ts) with their trace context attached natively. This is always the case for containerized deployments; in a bare dev process with LOG_PRETTY=1 (or an interactive TTY), logs stay stdout-only even with the endpoint set.

4. Run the bundled LGTM stack locally

For local, all-in-one visualization use the grafana/otel-lgtm image (Tempo + Prometheus + Loki + Grafana, pre-wired) defined under the obs compose profile in docker-compose.yml.

  1. Start it alongside the containerized stack:

    OTEL_EXPORTER_OTLP_ENDPOINT=http://lgtm:4318 \
  docker compose --profile stack --profile obs up -d --build

  2. Open Grafana at http://localhost:3001 (admin / admin).

  3. Go to Explore and pick a datasource:

    • Tempo — search by service meridian-proxy or meridian-endpoint.
    • Prometheus — query the clinical_* metrics.
    • Loki — query {service_name="meridian-endpoint"} (or -proxy), then click a log line's trace_id to jump straight into its Tempo trace.

  4. Running the proxy/engine on the host instead of in containers: start only the collector (docker compose --profile obs up -d lgtm) and export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 before pnpm run endpoint / pnpm run proxy.

5. Wire liveness and readiness probes

The proxy exposes two probes (apps/api/src/server/routes/health.ts):

  • GET /healthz — liveness: returns 200 { ok: true } as soon as the process is up. No dependency checks.
  • GET /readyz — readiness: 200 { ready: true, checks } only when both the Restate ingress (GET {RESTATE_URL}/restate/health) and the run index respond; otherwise 503 { ready: false, checks } with a per-dependency boolean in checks (restate, runStore) telling you which one failed.
  1. Point your orchestrator's liveness probe at GET /healthz on the proxy (port 4000).
  2. Point its readiness probe at GET /readyz on the same port. The bundled docker-compose.yml already does this for the proxy service's Docker healthcheck — mirror the same path as livenessProbe.httpGet.path / readinessProbe.httpGet.path for any other orchestrator (Kubernetes, systemd, …); see reference/topology.md for where the proxy sits in the deployment.
  3. The engine (apps/api/src/restate/server.ts) has no /healthz or /readyz of its own — its health is observed through Restate's own admin healthcheck (restate:9070, already probed by the restate service in the compose file) and through the proxy's checks.restate.

6. Watch event fan-out

Each event starts one run per published workflow that receives it. Two signals follow that behaviour (see reference/observability-signals.md):

  • meridian.events.deliveries — runs created per accepted event, by eventType. Alert on a change of its level after a publication: a new workflow on a busy event type multiplies the runs (and their effects) of that type.
  • meridian.selector.errors — trigger filters that failed to evaluate, by workflow. A non-zero rate means a workflow is not starting on events its author expects it to receive.

meridian.events.ingested counts each event once, by outcome:

outcome Meaning
accepted At least one run was created.
duplicate Every workflow that receives the event already had it.
filtered Workflows receive the event type, but no trigger filter keeps the event.
rejected Refused (400), for example: no trigger claims the type, no published workflow receives it, or a type with its own address arrived without a designated workflow.
unavailable At least one run could not be submitted to Restate (503); the other deliveries were tried.

After unavailable, the inbound reception delivers the message again, whether it came through a transport or from an event source. A POST /api/events caller gets 503 with failed[] and resends the same eventId. In every case only the missing runs start. The console's Emit an event composer completes the payload with that eventId for this purpose (see Event ingestion API). meridian.events.generated counts the same outcomes per source and feed, at each delivery attempt (see Records that fail to process).

Verify

  1. Confirm the probes respond:

    curl -i http://localhost:4000/healthz
curl -i http://localhost:4000/readyz

    /readyz should return 200 with "ready": true once Restate and the run index are both reachable.

  2. Trigger any event or run (see tutorials/first-instance.md) and confirm a log line appears on stdout with matching trace_id/span_id fields.

  3. With the LGTM stack up, find that same trace in Grafana's Tempo Explore view, and the corresponding log line in Loki via the trace_id pivot.

97 documents9 sectionssource : /docs · généré au build