Why durable execution

Meridian workflows are not short-lived request handlers. A single run can read a patient record, call an LLM agent, and then suspend for hours or days while it waits for a clinician to validate a proposed decision. It must survive process restarts, deployments, and node failures without losing its place, and it must never execute the same irreversible step twice. That set of guarantees — crash recovery, exactly-once steps, durable waiting, and an auditable history — is what "durable execution" means here, and it is the reason the platform is built on a dedicated control plane rather than an ordinary application server.

This page explains why that choice was made and how responsibility is split, at the level an operator needs to run and reason about a deployment. For the engineering detail behind the seam and the interpreter, follow the links into the contributing area.

The problem: workflows that outlive the process

Consider a discharge-prescription review that pauses for human validation. Between the moment the workflow proposes a change and the moment a pharmacist accepts or rejects it, arbitrary things happen: the API host is redeployed, a container is recreated, the machine reboots. When the decision finally arrives, the workflow has to resume exactly where it left off — same intermediate values, same pending validation, without re-reading the record or re-running the agent that produced the proposal.

Solving this by hand means building, per workflow, a persisted state machine: a schema for every intermediate value, save/restore logic at each step, a retry policy that avoids double-writes, timers for validation deadlines, and a way to correlate an incoming human decision back to the suspended run. That is a large, error-prone body of infrastructure that has nothing to do with clinical logic — and it would have to be re-derived every time someone authors a new workflow.

Durable execution moves all of that below the application. Authors describe what the workflow does; the engine guarantees that the description keeps running correctly across failures.

What Restate buys you

Meridian's control plane is Restate, run as a single binary alongside the platform. Restate is the only execution engine — an earlier in-process LocalRuntime existed as a teaching maquette and has been removed (apps/api/src/restate/services.ts). Concretely, it provides four things the platform would otherwise have to reinvent:

Exactly-once steps and deterministic replay

Every side-effecting step runs inside a journaled action. Restate records the step's result the first time it runs; on replay it returns the recorded result instead of running the step again. Recovery after a crash is therefore just replaying the journal to rebuild the workflow's in-memory state, then continuing from the first step that had not completed. The step that already wrote to the patient record does not write twice.

Durable suspension for human-in-the-loop

When a workflow waits for a human decision, it does not hold a thread or a database transaction open. It parks on a durable promise keyed by a deterministic id; the run is genuinely suspended and consumes no resources. A clinician's decision arrives later as a signal that resolves that promise, and the run wakes up. This is what lets Human-in-the-Loop (HITL) validations exist without a bespoke state machine per workflow. See Add human validation for the authoring side and Run and resolve for how a pending decision is answered.

Durable timers for SLAs and escalation

A validation can carry a deadline. If no human answers in time, the run resumes on an escalation branch instead of waiting forever. The timeout is a durable timer owned by the control plane, so it fires correctly even if the whole platform was down for part of the deadline window.

An event history you can trust

Because every step, suspension, resolution, and timeout is journaled, the run's history is the audit trail — it is not a best-effort log written on the side. The platform layers structured trace codes and a per-run snapshot on top of this for the monitoring UI (see Monitor executions), but the underlying source of truth for a live run is Restate's own state.

The three planes

Responsibility is split across three planes. Keeping them separate is what lets the control plane specialise in durability while the rest stays simple and mostly stateless.

The configuration plane is where workflows are drawn and edited. It never touches durability; it produces workflow specs.

The control plane is Restate. It is the only component that must be treated as stateful and durable: it holds the journal, the event history, and the per-run state. If you back up one thing for run continuity, it is this — see Back up and restore.

The execution plane is the API host process. It registers the workflow and record services with Restate, loads plugins, and holds the port adapters that talk to external systems (the FHIR warehouse, terminology servers, the LLM agent runner). Between steps it keeps no authoritative state of its own, so it can be restarted or redeployed freely — Restate replays the journal to bring any affected run back to where it was.

Two consequences worth internalising as an operator:

• The HTTP API is a proxy. POST /api/events and the read endpoints forward to Restate's ingress; the platform's API server is not itself the orchestrator.
• Subject records live in a durable service. Reads and the live record snapshot (GET /api/records/:kind/:id) are served from a Restate record service, not from an in-memory buffer in the API process.

For the internal architecture behind this split, see The three planes and the overall architecture.

The seam: one workflow, mapped onto Restate

Workflow authors and the generic interpreter never call the Restate SDK directly. They target a small abstraction, WorkflowContext (libs/engine-core/src/runtime/context.ts), and the platform maps that seam onto Restate primitives in apps/api/src/restate/context.ts:

The value of the seam is that the same generic interpreter and the same YAML specs run on the real durable engine — there is no separate "production" code path to keep in sync. The engineering rationale lives in The WorkflowContext seam.

One service for all workflows

There is a single generic durable workflow service (meridianWorkflow) that interprets whatever spec it is handed. The spec travels as part of the invocation argument, so Restate journals it: deterministic replay uses the spec frozen at launch, even if the YAML is edited afterwards. A run started against last week's version of a workflow keeps replaying against last week's version.

The operational payoff: authoring or changing a workflow adds no Restate service and requires no control-plane deployment. New clinical logic ships as plugin code and workflow specs, not as new orchestration endpoints.

The costs and trade-offs

Durable execution is not free, and the design made deliberate choices:

• You run a stateful control plane. Restate is an additional component with its own storage and lifecycle. It is the piece that must be sized, monitored, and backed up. See Sizing, Enable observability and Observability signals.

• Step bodies must be replay-safe. Because Restate reconstructs state by replaying the journal, anything non-deterministic or side-effecting must live inside a ctx.run step so its result is recorded once and replayed. This is a constraint on how behaviors are written, handled by the engine and the seam rather than by authors, but it is why the model works.

• Business errors are made terminal on purpose. Restate retries failed invocations by default. Expected failures — an unknown patient, an unreachable FHIR or terminology server — are converted to terminal errors so Restate does not retry into an endless back-off loop; only genuinely transient failures are left retryable. This keeps a misconfiguration from masquerading as a stuck run.

• Restate is the source of truth for a live run, not your query database. For listing and monitoring across many runs there is a separate run index that can be backed by SQLite or Postgres (see Run store and Use a Postgres run index). That index is a projection for querying; it does not replace Restate's journal.

Alternatives considered

The seam was intentionally kept engine-agnostic — its primitives map just as cleanly onto other durable-execution engines such as Temporal — precisely so the platform is not wedded to one vendor. Restate was chosen for its operational simplicity: a single binary to run rather than a multi-service cluster, which suits the deployment footprint Meridian targets. The removed in-process LocalRuntime was never a real alternative for production; it existed only to illustrate the seam without any infrastructure.

Where to go next

• Component topology and how the planes are wired: Topology.
• What each environment variable controls (including the registry and instance selection): Environment variables.
• Backing up the durable state and restoring it: Back up and restore.
• Upgrading without losing in-flight runs: Upgrade the platform.
• Vocabulary used throughout: Glossary.