Documentation / Exploitation / Guides pratiques / Enable logs, traces and metrics

Enable logs, traces and metrics

The proxy and the engine always emit structured logs to stdout. Traces, metrics, and OTLP log export are opt-in: point OTEL_EXPORTER_OTLP_ENDPOINT at a collector and they turn on; leave it unset and every instrument stays a no-op at zero cost (apps/api/src/observability/otel.ts). This how-to wires that export, brings up the bundled local visualization stack, and sets liveness/readiness probes. For the full catalogue of log fields, span names and metric instruments, see reference/observability-signals.md; for the complete variable list see reference/environment-variables.md.

Before you start

  • Observability env vars are read once at process boot (initLogger/startOtel are called at the top of apps/api/src/server/api.ts for the proxy and apps/api/src/restate/server.ts for the engine). Changing any variable below requires restarting the corresponding process to take effect.
  • The proxy and the engine are two separate processes with independent service names (meridian-proxy / meridian-endpoint) but share the same environment variable names — set them on whichever process(es) you are instrumenting.
  • Only the proxy serves /healthz and /readyz (port 4000); the engine has no HTTP probe endpoint of its own.

1. Configure structured logs

Logs are JSON-on-stdout by default outside an interactive terminal (pino, apps/api/src/observability/logger.ts), one object per line, with a service field (meridian-proxy / meridian-endpoint) and a component field per subsystem (plugins, instance, http, sources, engine, …).

  1. Set the verbosity with LOG_LEVEL (trace | debug | info | warn | error, default info).
  2. Set LOG_PRETTY to force a format: 1 forces human-readable (pino-pretty), 0 forces JSON. Leave it unset to get the default (pretty on an interactive TTY outside production, JSON otherwise — e.g. under Docker or any collector).
  3. Optionally set SERVICE_NAME to override the service field baked in by each entrypoint (meridian-proxy / meridian-endpoint).

Secret-looking fields (authorization, apiKey, api_key, password, token, req.headers.authorization, config.apiKey) are redacted automatically as defense in depth. Events themselves are only ever logged by type and id — no patient payload reaches the logs, by design; see explanation/security-posture.md.

2. Export traces, metrics and logs to a collector

  1. Set OTEL_EXPORTER_OTLP_ENDPOINT to your collector's OTLP/HTTP endpoint, e.g. http://localhost:4318. This single variable enables traces + metrics + logs together — there is no per-signal toggle.
  2. Optionally set OTEL_SERVICE_NAME to override the resource service.name reported to the collector (defaults to meridian-proxy / meridian-endpoint).
  3. Restart the proxy and/or the engine. startOtel() is idempotent, so re-running it on an already-started process is harmless, but the environment variable is only read once at boot.

With the endpoint set, every HTTP request into the proxy, every Restate ingress call, and every workflow node execution produces a span in a single end-to-end trace per event, with the W3C trace context propagated from the proxy to the engine over the Restate invocation payload:

POST /api/events (proxy)
└─ restate.ingress (proxy)
   └─ workflow.run (engine)
      ├─ node <id>            (one span per executed node; map body: node <id>#<i>)
      │  └─ fhir GET|POST|PUT  /  posos.graphql   (outbound I/O, under the calling node)
      └─ …

Business metrics (clinical.events.generated, clinical.runs.executed, clinical.nodes.duration, …) and HTTP/ingress latency histograms are exported the same way; see reference/observability-signals.md for the full instrument list and attributes (apps/api/src/observability/metrics.ts).

A human-task validation suspends the run — there is no span for the wait itself. On Restate replay after resumption, spans for nodes that already ran once may be re-emitted; single-pass workflows are unaffected. See explanation/why-durable-execution.md.

3. Correlate logs with traces

Every log line emitted while a span is active already carries trace_id / span_id (a pino mixin, apps/api/src/observability/logger.ts). To also ship those logs to your collector as OTLP log records — so you can pivot from a log line to its trace in Grafana — both of these must hold at once:

  • output is JSON, i.e. LOG_PRETTY is 0 or unset on a non-TTY process (pretty is false); and
  • OTEL_EXPORTER_OTLP_ENDPOINT is set.

When both hold, logs are shipped in-process through the pino → OTel bridge (apps/api/src/observability/otel-logs.ts) with their trace context attached natively. This is always the case for containerized deployments; in a bare dev process with LOG_PRETTY=1 (or an interactive TTY), logs stay stdout-only even with the endpoint set.

4. Run the bundled LGTM stack locally

For local, all-in-one visualization use the grafana/otel-lgtm image (Tempo + Prometheus + Loki + Grafana, pre-wired) defined under the obs compose profile in docker-compose.yml.

  1. Start it alongside the containerized stack:

    OTEL_EXPORTER_OTLP_ENDPOINT=http://lgtm:4318 \
  docker compose --profile stack --profile obs up -d --build

  2. Open Grafana at http://localhost:3001 (admin / admin).

  3. Go to Explore and pick a datasource:

    • Tempo — search by service meridian-proxy or meridian-endpoint.
    • Prometheus — query the clinical_* metrics.
    • Loki — query {service_name="meridian-endpoint"} (or -proxy), then click a log line's trace_id to jump straight into its Tempo trace.

  4. Running the proxy/engine on the host instead of in containers: start only the collector (docker compose --profile obs up -d lgtm) and export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 before pnpm run endpoint / pnpm run proxy.

5. Wire liveness and readiness probes

The proxy exposes two probes (apps/api/src/server/api.ts):

  • GET /healthz — liveness: returns 200 { ok: true } as soon as the process is up. No dependency checks.
  • GET /readyz — readiness: 200 { ready: true, checks } only when both the Restate ingress (GET {RESTATE_URL}/restate/health) and the run index respond; otherwise 503 { ready: false, checks } with a per-dependency boolean in checks (restate, runStore) telling you which one failed.
  1. Point your orchestrator's liveness probe at GET /healthz on the proxy (port 4000).
  2. Point its readiness probe at GET /readyz on the same port. The bundled docker-compose.yml already does this for the proxy service's Docker healthcheck — mirror the same path as livenessProbe.httpGet.path / readinessProbe.httpGet.path for any other orchestrator (Kubernetes, systemd, …); see reference/topology.md for where the proxy sits in the deployment.
  3. The engine (apps/api/src/restate/server.ts) has no /healthz or /readyz of its own — its health is observed through Restate's own admin healthcheck (restate:9070, already probed by the restate service in the compose file) and through the proxy's checks.restate.

Verify

  1. Confirm the probes respond:

    curl -i http://localhost:4000/healthz
curl -i http://localhost:4000/readyz

    /readyz should return 200 with "ready": true once Restate and the run index are both reachable.

  2. Trigger any event or run (see tutorials/first-instance.md) and confirm a log line appears on stdout with matching trace_id/span_id fields.

  3. With the LGTM stack up, find that same trace in Grafana's Tempo Explore view, and the corresponding log line in Loki via the trace_id pivot.

75 documents8 sectionssource : /docs · généré au build