The Orchestrator

Business process management is not new. Software for tracking a case, recording where it sits, and managing how it executes has existed for decades, and durable workflow state is not new either. What is new is the prominence of the word in front of it — durable — in agent tooling, and the word that needs defining is not execution but orchestration, because what an orchestrator runs is not one program. It is a system of components, each holding state of its own.

A durable orchestration engine records enough workflow state to resume work after a failure or a long wait. The broader question for an orchestrated system is what else must survive: agent state, memory versions, retrieved evidence, and the context needed to continue. External resources do not freeze when the workflow pauses. Credentials may expire, records may change, and the restored system must check its current authority before acting. Can you recover the work and reconstruct the earlier decision? Those are related guarantees, and a design should say which one it provides.

What kind of work is this? Not the calls — the processes. The work shows up as business processes with a shape in time: a customer onboarding that takes a week; an insurance claim that takes a month; a loan application waiting on an appraisal; a government permit waiting on a third-party response. This is the machinery of banking, government, healthcare and any other domain where a customer or citizen interacts with an organization and the interaction cannot finish in one sitting. What makes it hard is not any single step. It is that complicated context has to be maintained across all of the steps: what this customer filed, which documents have arrived, who reviewed what, what is still outstanding, what happens next. An agent that can wake up and respond to an event is not enough here — the context belongs to one particular customer or situation, and it has to be tracked reliably for weeks.

The naive way to build this is a scheduled job plus a state table: run the process a step at a time, write where it is into a database, and write code to pick it back up after each wait. Every step needs resume logic. Every wait needs a poller. Every crash needs recovery code. The glue is where it breaks, and the glue is most of the program. A durable execution engine does that work once, for every process it runs.

Temporal records the workflow as a durable history of events: started, called this, got that result, timer set, signal received. The workflow code runs as ordinary code, but it does not hold its own state — the history does. If the process dies mid-wait, the engine starts the code again on another machine and replays the history into it, handing back each recorded result as the code asks for it, until the code arrives exactly where it stopped. A three-day wait for a human review is not a process that stays alive for three days; it is a timer recorded in the history, and when the timer fires, the workflow resumes on whatever machine is available. From the developer’s point of view, the function pauses at that line and resumes weeks later with every variable intact, as if the failure never happened.⁴²

Run the shape against something concrete: a loan application, from arrival to decision. Day one, the workflow starts, verifies the applicant’s data, and requests documents. Day three, the documents arrive, recorded as events delivered into the running workflow. Day four, the workflow requests a human underwriter review and sets a timer. Day nine, the review is recorded, the workflow orders a third-party appraisal, and sets another timer. Three weeks in, the appraisal returns, the decision step runs, the customer is notified, and the workflow closes. In between every one of those steps, machines rebooted, deployments shipped, instances died and were replaced. The workflow did not notice, because its state was never in any of those machines.

Temporal and Restate already provide durable execution; Restate also documents persistent agent sessions, approvals, and recovery controls.⁴³ Gas Town persists task state, hooks, agent identity, history, and orchestration state in Beads and Git-backed storage, so worker restarts do not erase everything either. The remaining design work is at the boundaries: which memory version belonged to the run, which external evidence must be retained, which dependencies can be rebuilt, and which credentials must be renewed. No workflow engine can restore a customer’s changed record or make an expired grant current by replaying history. An orchestrator has to decide what is recorded for reconstruction and what is checked again before resuming.

This is the last infrastructure layer in the survey — the sections after it describe assembled agents, not components — and it is the layer that keeps state distinct from memory. Memory (Section 5.5) is what the system knows — facts, decisions, corrections. Durable orchestration is where the system is — which step, waiting on what, what happens next. A system can have one without the other; an orchestrated business process needs both. And durability is the floor of this layer, not the ceiling: once a system’s state can be stood up at will, replaying a decision for an auditor, evaluating a system against its own history, and improving it over months rather than sessions all become buildable. Everything this book describes above this layer assumes it will exist.

1.9.1 Where durable orchestration shows up for an orchestrator

The encounters are decisions about what runs on this layer and what does not. Choosing the boundary: which parts of an orchestrated flow need durable guarantees (the approval wait, the payment step, the permit application) and which can stay ephemeral. Wiring the waits: the multi-day pause for human review is a durable timer, not a script that sleeps. And replaying: after a failure or an audit, the history is a first-class record of what step the workflow was on, what it had already done, and what it was waiting for, queryable step by step. The one discipline to know before adopting these engines: when a workflow is replayed to rebuild its state, the code has to make exactly the same decisions it made the first time. In Temporal, use workflow-safe APIs for time and randomness, and activities for external work such as model or service calls. Replay returns the recorded results; an activity that has not completed may be retried. External effects therefore need idempotency or deduplication, so resuming the work does not issue the same payment twice.

This layer also carries the audit burden asymmetry that Chapter 7 develops: operational work — the runs that touch customers, money, and production — is exactly the work that needs durability most, because it is the work whose interruption is expensive and whose record is demanded later. The question to ask of any long-running design: if this workflow dies at step seven, what is lost, what resumes, and who finds out?