Recover runs blocked by a missing plugin

A run in flight replays its journal each time it resumes: a decided validation, a timer, a webhook resume, a restart of the endpoint. The replay needs the code that wrote the journal. When the Restate endpoint no longer loads it, the run is blocked:

• the plugin that provides one of the workflow's node types is no longer loaded, or, for a run of journal format 1, the plugin that provides its trigger or declares the event its « Événement reçu » trigger receives (a run of format 2 reads its trigger from its journal);
• for a run of journal format 1, started with RUN_JOURNAL_FORMAT=1 (the default) or by a proxy older than format 2: a plugin version gives the run's event other trigger outputs than the version that started the run (a field added to or removed from the declared event, an output renamed), or gives a task the run has reached another title (a changed translation, another instance locale). A run of format 2 reads its trigger's outputs and the texts and outputs of its tasks from its journal;
• for a run of format 2 whose validation was decided or expired: the loaded plugin version no longer proposes that validation, so it cannot compute the task's outputs. The step <node>/outputs fails with « Nœud « » : la version chargée du plugin ne propose plus cette validation. Rétablir la version du plugin qui a démarré la tâche. », which Restate shows as the last failure of the invocation;
• the endpoint was rolled back below a core node type the run uses, or below the journal format of the run (see Workflows that use a new core node and Journal formats).

Restate retries the invocation with backoff and without a maximum number of attempts (worker.invoker.retry-policy of Restate 1.4): after a few attempts, one attempt every 10 to 12 seconds. The run keeps its last persisted state, a validation already decided included, and goes on at the first attempt after the endpoint loads the same code again. While the plugin is missing, a new run of the workflow is not blocked: it fails at its trigger.

This recipe finds the blocked runs, then restores the plugin or ends the runs, and closes the validation requests a run leaves open in the record, such as a Task in the FHIR warehouse. The log messages are described in Observability signals — Run outcome.

Before you start

• Run the Restate commands inside the stack. The Restate server image (docker.restate.dev/restatedev/restate:1.4) ships the restate CLI at the server's version, which reaches the admin API on localhost:9070 without configuration: docker exec <restate-container> restate …, or kubectl exec in the Restate pod. The admin API answers the same requests over HTTP on port 9070. Neither has authentication: never publish port 9070 (see Security posture).
• npx @restatedev/restate downloads the CLI from the npm registry: it does not run offline.
• Keep the endpoint and proxy logs of the last boots: the plugins chargés line lists the name@version of each plugin a process loaded.

1. Detect the blocked runs

In Restate

List the run invocations that Restate retries:

restate invocations list --service meridianWorkflow --status backing-off

  ❯ [2026-09-28 19:19:43.337 +00:00] inv_1kOHVCNtkCO4665x9By7fCZXnBG7iQWGft
     Target:      meridianWorkflow/<eventId>:<slug>/run
     Status:      backing-off  (9 seconds and 675 ms. Retried 8 time(s). Next retry in …)
     Deployment:  dp_15MB0NX6tNqSdY5gr4b06VH [required] using restate-sdk-typescript/1.14.5
     Error:       [570 Journal mismatch] Found a mismatch between the code paths taken during
                  the previous execution and the paths taken during this execution.
                  - The mismatch happened while executing 'set state' (index '1')
                  - Difference:
                     value: '{"status":"suspended",…}' != '{"status":"failed",…,"error":"Événement « <type> » non déclaré par les plugins chargés."}'

The key between meridianWorkflow/ and /run is the Meridian run id. The same rows, one per invocation, with SQL:

restate sql "SELECT id, target_service_key AS run, retry_count, last_failure_error_code AS code FROM sys_invocation WHERE target_service_name = 'meridianWorkflow' AND target_handler_name = 'run' AND status = 'backing-off'"

 ID                                      RUN                         RETRY_COUNT  CODE
 inv_1kOHVCNtkCO4665x9By7fCZXnBG7iQWGft  <eventId>:<slug>            13           RT0016

Through the admin API, POST /query runs the same query and answers {"rows": [...]}:

curl -s -X POST http://<restate-admin>:9070/query \
  -H 'content-type: application/json' -H 'accept: application/json' \
  -d "{\"query\": \"SELECT id, target_service_key, retry_count, last_failure_error_code, last_failure FROM sys_invocation WHERE target_service_name = 'meridianWorkflow' AND target_handler_name = 'run' AND status = 'backing-off'\"}"

restate invocations describe <invocationId> shows the journal of one invocation and its last failure.

Read the last failure:

The Difference gives the state the run holds (left) and the state the endpoint tried to write (right). When the replay fails the run, the right side is a failed state with the error: « Événement « » non déclaré par les plugins chargés. » for « Événement reçu », « Trigger invalide : » for a trigger whose plugin is gone, « Type de nœud inconnu : (nœud ) » for another node type. When both sides have the same status and differ in values, such as the trigger's outputs or the title of the pending validation, a plugin version gives the event of a run of format 1 other outputs, or one of its tasks another title. Both sides are the run's state, with workflow data, and Restate writes the same text to its server log at each attempt: handle it as clinical data.

A run that is only waiting is not blocked yet: suspended on a validation, a timer or a webhook resume while the plugin is missing, it stays suspended in Restate and in Meridian, and blocks when it resumes. An invocation stays attached to the endpoint for the inactivity timeout (1 minute by default) before Restate suspends it: a restart of the endpoint within that minute replays it at once, and blocks it without any decision.

In Meridian

• Endpoint log, component engine: rejeu interrompu : non-concordance avec le journal du run. Restate retente l'invocation, le run garde son dernier état. Rétablir les versions de l'endpoint et des plugins qui ont démarré le run, ou tuer l'invocation (kill). at warn, once per invocation and per endpoint process, with workflow, runId and invocationId. When the replay throws (plugin or node type missing), error is the error that stopped it, and later attempts are logged at debug. A replay that diverges without an error, such as a run of format 1 on a plugin version that projects the event differently or changes a task title, gets the same line with error = the first line of the SDK's error ((570) Found a mismatch between the code paths…) and command = the journal command that differs (set state [1]), without the compared values. The SDK also writes its own lines to the endpoint's standard error at each attempt ([restate]…WARN: Error when processing a Restate context operation., then RestateError: (570) Found a mismatch between the code paths…).
• Endpoint boot log: the plugin is missing from plugins chargés, or listed at another version.
• Proxy log, each time the proxy loads the workflows, at error: « <file> » ne peut pas tourner : événement « <type> » non déclaré par les plugins chargés. Vérifier les plugins résolus. or « <file> » ne peut pas tourner : type(s) de nœud absent(s) du catalogue (<type>). Vérifier les plugins résolus. These lines appear only when the proxy lacks the plugin too. GET /api/workflows then serves valid: false and a triggerProblem (event-unknown, trigger-unknown) for these workflows, and POST /api/events refuses their event with 400 « Aucun déclencheur ne revendique l'événement « ». ».
• Runs: GET /api/runs and the Executions view (« Exécutions ») keep the blocked run at its last status: suspended with its validation (counted by the to resolve tag, « à trancher »), waiting or running. The decision that resumes a suspended run is accepted: 200 after about 6 seconds, validation tranchée in the proxy log, and the run still suspended with the same validation. A second decision answers 409 « validation déjà tranchée ».
• New runs that reach the endpoint while the proxy still loads the plugin (a partial rollout, two manifests) end failed at once, the trigger node failed with the error, and the endpoint logs run échoué.
• Metrics: meridian.runs.executed counts runs that complete or fail, not the attempts of a blocked run. No metric counts blocked runs: watch the rejeu interrompu warning and the backing-off list.

Tell a blocked run from a failed run

2. Decide

• Restore the code that wrote the journals: the plugin at the version the runs started with, or the platform image. Nothing is lost: each blocked run replays at its next attempt and goes on, and a decision taken meanwhile is applied. Published plugin versions are immutable, so the registry still holds that version.
• End the runs when the plugin must not come back or the workflow must not go on. Only a kill ends a blocked run. The steps after the blocking point never run: the pending clinical work (a decided validation, a notification) is handled by other means.

The version to restore is in the endpoint's plugins chargés line of the boot before the change. meridian-plugin versions <@scope/name> --registry <url> lists the published versions (see Set up a plugin registry).

3. Restore the plugin

1. Pin the version in the instance manifest of the endpoint and of the proxy:

   plugins:
     "@acme/vitals": "1.4.2"   # the version that wrote the journals
   

   A range can resolve to another version at the next boot. For a plugin loaded through PLUGINS_PATH, put the same folder back: a plugin listed in plugins: that is also found on PLUGINS_PATH loads from there, whatever the pin, and the boot only warns when that copy is outside it. See Declare and install plugins.

2. Restart the endpoint, then the proxy. The endpoint registers again at the same address (RESTATE_ENDPOINT_URL): Restate updates the deployment in place, and the blocked invocations, bound to it (Deployment: … [required]), run there.

3. Check the endpoint log: plugin présent en cache or plugin téléchargé du registre with the version, then plugins chargés listing <name>@<version>. At its next attempt, within about 12 seconds, each blocked run replays: run terminé when it completes, nothing when it reaches its next wait. restate invocations list --service meridianWorkflow --status backing-off no longer lists it, and GET /api/runs/<runId> serves its new status, with the decision taken during the blockage in its trace (validation-resolved).

4. The runs that failed at their trigger during the gap stay failed. Sending their event again with the same eventId starts nothing (200, created: false). To run one workflow on it, send the event with a new eventId and the workflow's file in __workflowFile; the other workflows that receive the event are not started again:

   curl -s -X POST http://<proxy>:4000/api/events -H 'content-type: application/json' \
     -d '{"type": "<type>", "eventId": "<new-id>", "__workflowFile": "<slug>.workflow.yaml", "patientId": "<id>", …}'
   

   In the console: Executions → Emit an event (« Émettre un événement »), with the workflow in Target workflow (« Workflow cible »).

4. End the runs

1. Kill each blocked invocation, by target or by invocation id:

   restate invocations cancel --kill --yes meridianWorkflow/<runId>/run
   

    ID                                      TARGET                                 STATUS
    inv_14DFsDTTZXby16ZYgrYrGANFhULWcSlMUy  meridianWorkflow/<runId>/run           backing-off
   
   Are you sure you want to kill these invocations? Auto-confirming --yes is set.
   
   [OK]: Request was sent successfully
   

   curl -s -X PATCH http://<restate-admin>:9070/invocations/<invocationId>/kill
   

   The admin API answers 200 with an empty body.

2. Check the validation requests these runs leave open in the record (step 5).

Restate then lists the invocation completed, with [409 Aborted] killed as completion_failure:

restate sql "SELECT target_service_key AS run, status, completion_failure FROM sys_invocation WHERE target_service_key = '<runId>' AND target_handler_name = 'run'"

 RUN        STATUS     COMPLETION_FAILURE
 <runId>    completed  [409 Aborted] killed

The proxy reads how each live run's invocation ended (GET /restate/workflow/meridianWorkflow/<runId>/output on the ingress, 409 killed for a killed one) and freezes the killed run in the run index:

• when: at the next settlement pass (RUN_SETTLE_INTERVAL_MS, every minute by default, see Environment variables). When the run's last state is suspended or waiting, also at once when its detail is read (GET /api/runs/<runId>, the run in Executions), a decision is sent for it or its wait is resumed. A run killed while its last state is running waits for the pass; with RUN_SETTLE_INTERVAL_MS=0 it is never frozen;
• status failed, error « Invocation arrêtée dans Restate (killed). ». The run's last state, the validation it waited on included, is archived with the status, so its detail stays readable after Restate removes the workflow, one day after the kill (workflow_completion_retention of the meridianWorkflow service);
• the proxy logs run arrêté dans Restate : statut figé en échec (component runs, info) with runId, workflow and failure: killed, and meridian.runs.settled counts it (ending: killed);
• when the run's last state waits on a validation, the proxy withdraws its request before it freezes the run, whatever the run's journal format: one call per subject of the run whose kind has a record port, sent one way to the endpoint's record service (closeValidationRequest, outcome withdrawn) with the idempotency key withdraw:<runId>:<pendingId>:<kind>, and retried by Restate for up to one hour. A record adapter that implements closeValidationRequest closes the request; one that does not, such as the fhir adapter of @posos/fhir 0.1.5, leaves it open (step 5). A call the ingress refuses leaves the run live in the index: the proxy logs retrait de la demande de validation impossible : run non figé, repris au règlement suivant (component runs, warn) with runId, workflow and kind, and sends again at the next pass.

GET /api/runs, GET /api/runs/<runId> and Executions then show the run failed, without a pending validation: it leaves the to resolve tag (« à trancher »), and Accept (« Accepter ») and Reject (« Refuser ») are no longer offered. A decision through POST /api/runs/<runId>/resolve answers 409 « run terminé : aucune validation en attente », and no signal reaches Restate; a decision feed rejects it and logs résolution rejetée with « décision sur un run terminé : validation non appliquée ». A resume of its wait through POST /api/resume/<runId>/<nodeId> answers 409 « run terminé : aucune attente en cours ».

A purge (restate invocations purge <invocationId>, PATCH /invocations/<invocationId>/purge) removes the invocation and its state at once. Purge only after the proxy has frozen the run (its status is failed in GET /api/runs). Otherwise the proxy that started the run, the one whose RESTATE_URL is recorded on the run (restate_url in the run index), freezes it failed with the error « Invocation absente de Restate : issue du run inconnue. », without its last state (stateLost: true), and logs run absent de Restate : statut figé en échec at warn. A run started before the release that records restate_url is never frozen that way: the proxy counts it in <n> run(s) d'un autre Restate ou sans Restate d'origine, non réglé(s) at each pass (see Upgrade the platform).

A cancel (restate invocations cancel <invocationId> without --kill, PATCH /invocations/<invocationId>/cancel, answer 202) does not end a blocked run. Restate delivers the cancellation to the next attempt, whose replay fails before reading it, and the invocation stays backing-off. While the plugin is missing, a cancel sent to a run that is only suspended resumes it into the same retry loop. When the plugin comes back, a pending cancel ends a run still waiting on its validation (failed, error Cancelled), and a run whose validation was decided before the cancel goes on to its end. A run of format 2 withdraws its validation request before the cancel ends it.

5. Close the validation requests left open

A run publishes each validation it waits on to the record of its subjects, through the record port of each kind (publishValidationRequest). The platform closes that request through the same port (closeValidationRequest) when the validation ends outside the record:

• a run of journal format 2 closes it itself, in a durable step, after a decision taken in the console or through the API (accepted or rejected, with its author, time and comment), after its deadline (expired), and when a cancel reaches it while it waits (withdrawn);
• the proxy withdraws it when it freezes a run killed, or ended in failure, in Restate while the run waited on it, whatever the run's format (step 4).

A request stays open in the record, to close by hand, in these cases:

• the run is of format 1 and its validation ended outside the record: a decision with Accept (« Accepter ») or Reject (« Refuser ») or through POST /api/runs/<runId>/resolve, an expired deadline, a processed cancel;
• the record adapter does not implement closeValidationRequest: the fhir adapter of @posos/fhir 0.1.5 does not;
• the run was frozen without its last state (stateLost: true, « Invocation absente de Restate : issue du run inconnue. »): the proxy does not know the validation it waited on, and withdraws nothing;
• the closing failed: in a run, the endpoint writes clôture de la demande de validation échouée : <kind> : <error> to its console (a Restate SDK line, warn), and the run goes on. After a kill, the closeValidationRequest invocation of the record service is backing-off while it retries (restate invocations list --service record --status backing-off), then ends in failure after one hour and leaves that list: the endpoint logs clôture de la demande de validation abandonnée après les relances : la clore dans le dossier (component engine, warn) with runId, pendingId and kind;
• the run of format 2 is blocked after its decision because the loaded plugin version no longer proposes the validation (see the list at the top): the run closed the request as decided before it computes the task's outputs. Restore the plugin version that started the task (Restore the plugin) and the run applies the decision. When you kill the run instead, the Task keeps the decision and the workflow does not apply it: tell the care team, as for a completed or rejected Task below.

When the patient port uses the fhir adapter, each validation a run waits on is a FHIR Task (code urn:ehr-lab:validation|decision), identified by urn:ehr-lab:validation-run|<runId> and urn:ehr-lab:validation-pending|<pendingId> (see ADR 0002). For each run in one of these cases, find its Tasks:

curl -s '<fhir-base>/Task?identifier=urn:ehr-lab:validation-run|<runId>'

• requested: nobody decided in ehr-lab. After a kill, a cancel or a decision taken elsewhere, ehr-lab still lists the validation, and a decision taken there reaches no run. Set the Task to cancelled: read it, change status, write it back.

  curl -s '<fhir-base>/Task/<id>' > task.json
  # "status": "requested" → "status": "cancelled"
  curl -s -X PUT '<fhir-base>/Task/<id>' -H 'content-type: application/fhir+json' --data-binary @task.json
  

  ehr-lab lists only requested Tasks, and the fhir-validation-decided binding reads only completed and rejected: a cancelled Task leaves both.

• completed or rejected: the decision was taken, and the run did not apply it. Tell the care team: the actions the workflow takes after that decision did not happen.

For a run of format 1, a decision taken in the console or through POST /api/runs/<runId>/resolve does not change the Task: a validation decided with Accept or Reject keeps its Task requested. Check the Tasks of every run of format 1 you end, whatever the way its validation was decided, and of every run whose record adapter does not close its requests.

6. Prevent

• Keep a plugin on every endpoint replica while runs of the workflows that use its triggers, events or nodes are in flight. Before removing it, or before rolling the endpoint back to an image without one of the core node types a workflow uses:
  1. unpublish those workflows (Unpublish, « Dépublier », in the workflow's version panel, or POST /api/workflows/<slug>/unpublish); this stops new runs only;
  2. wait until none of their runs is running, suspended or waiting in Executions, or cancel them in Restate while the endpoint still loads the plugin (restate invocations cancel --yes meridianWorkflow/<runId>/run): the cancel is then processed, and the run ends failed with the error Cancelled. A run of format 2 withdraws its validation request; close the requests of the cancelled runs of format 1 (step 5);
  3. remove the plugin, or roll the endpoint back.
• Publish no workflow on a new trigger or event before every endpoint replica loads the plugin that provides it: a replica without it fails the new runs at their trigger.
• Pin plugin versions exactly in plugins:, and give the endpoint and the proxy the same manifest. A run of format 2 keeps its trigger's outputs and the texts and outputs of its tasks: a plugin version that changes the declared event, the trigger's outputs or a task title applies to the runs that start after it. A run of format 1 is blocked by such a version when its trigger's outputs change, or the title of a task it has reached: deploy it when no run of format 1 of the workflows concerned is in flight, or roll it back as in Restore the plugin. Every run started with RUN_JOURNAL_FORMAT=2 is of format 2.
• Keep every endpoint replica on a release that knows the journal format of the runs in flight: set RUN_JOURNAL_FORMAT=2 on the proxy once every endpoint replica knows format 2, and before rolling the endpoint back below it, set RUN_JOURNAL_FORMAT=1 and wait until no run of format 2 is in flight (see Journal formats).
• Watch the rejeu interrompu warning, and run the backing-off query after each deployment of the endpoint.

The rollout order is in Upgrade the platform.

Related

• Observability signals — Run outcome — the log messages of a run's attempts.
• Upgrade the platform — rollout order, core node types, roll back.
• Declare and install plugins and Set up a plugin registry — plugins:, pinned versions, the registry.
• Why durable execution — journal and replay.
• Declare an event — renaming or removing a declared event, from the plugin author's side.
• ADR 0007: Trigger configured by a declared event — the regime of a vanished event.
• ADR 0008: A run keeps its projection and task texts — journal format 2, and the closing of validation requests.