Checkpoints and resume
How a workflow run in this application stops in one HTTP request and continues in a later one. This page is for a developer who wants to follow a paused return from the moment it stops to the moment it finishes, and to know which table holds what in between.
The one workflow that pauses is workflow:return-replace. It stops at an approval gate, a reviewer decides hours later, and a different request (possibly a different process) carries it on from a checkpoint in Postgres. Approvals covers who may decide and how the decision is routed. Return lifecycle covers the optional waits after approval. Tutorial 18, state and checkpoints teaches the framework feature on its own.
What a checkpoint is, and when one is saved
A checkpoint is Microsoft Agent Framework’s snapshot of a workflow graph’s execution state: where the graph was when it stopped, not what was said in the conversation (concepts: state, memory and sessions).
The framework saves one checkpoint per superstep whenever a run is given a checkpoint storage. PrePurchaseMode and ReturnReplaceMode (agents/python/orchestrator/modes/workflow_mode.py) attach one to every run, taken from shared.factory.get_checkpoint_storage(). The backend is MAF_CHECKPOINT_BACKEND: postgres (the default), file or memory.
The framework’s event stream does not report a save. So both modes wrap the storage in RecordingCheckpointStorage (agents/python/shared/checkpoint_storage.py), which records each saved id. The mode drains that list between events and emits its own checkpoint event, which the chat stream forwards to the browser as an event: checkpoint frame.
What is stored, and where
Two tables, both in the baseline schema (docker/postgres/init.sql).
workflow_checkpoints holds the snapshots. PostgresCheckpointStorage writes them.
| Column | What it holds |
|---|---|
checkpoint_id | UUID primary key |
workflow_name | The graph’s name, return-and-replace for a return |
payload | The checkpoint as JSONB, encoded with the framework’s own encode_checkpoint_value |
usage_log_id | The run (usage_logs row) this checkpoint belongs to |
session_id, parent_checkpoint_id, seq | Used by the .NET store only; see What differs on .NET |
hitl_requests holds one row per pause. It is the index that says which checkpoint and which framework request belong to which paused run.
| Column | What it holds |
|---|---|
workflow_run_id | The run that paused |
request_id | The framework’s own resume token for the pending request |
checkpoint_id | The checkpoint to reload |
user_email | The requester. The resumed steps run as this user |
kind | return_approval for the gate; carrier_scan and return_inspection for the lifecycle waits |
status | pending, processing while a resume holds the claim, then approved or rejected |
claimed_at | The claim’s fencing token |
approval_id | The linked row in the staff inbox (approvals) |
response | The recorded outcome, including who decided |
After a run ends, _link_run_artifacts (agents/python/orchestrator/routes/chat.py) sets usage_log_id on the run’s latest checkpoint. If the run paused for approval it also inserts the hitl_requests row and then opens the inbox approvals row. The inbox row is best effort: if it is missing, approvals_service.ensure_for_hitl_request creates it when someone first decides.
The pause
The graph is check-eligibility → policy-gate → initiate-return → search-replacements → apply-discount → finalize (agents/python/workflows/return_replace.py).
_PolicyGateExecutor.run asks the approval policy. A route of auto goes straight on. Any other route pauses: the executor yields a snapshot of the state, then calls ctx.request_info with a ReturnApprovalRequest and expects a ReturnDecision back.
Two properties of the pause matter later:
- Nothing has been created. The
returnsrow comes frominitiate-return, the step after the gate. A rejected or expired request leaves no return and the order untouched. - The request carries the whole
WorkflowState. The resumed run restores its state from the request itself, so nothing depends on process memory.
ReturnReplaceMode.run ends the request with a run_completed event that carries pending_approval: true, the framework’s request_id, the policy’s route and reasons, and latest_checkpoint_id. The customer sees “Return for order … needs approval … Nothing has been changed yet.”
The resume
%%{init: {'theme':'base','themeVariables':{'actorBkg':'#0891b2','actorBorder':'#0e7490','actorTextColor':'#ffffff','signalColor':'#94a3b8','signalTextColor':'#64748b','noteBkgColor':'#fbbf24','noteBorderColor':'#d97706','noteTextColor':'#1f2937'}}}%%
sequenceDiagram
accTitle: The resume
participant C as Customer
participant O as Orchestrator
participant W as Workflow (MAF)
participant DB as Postgres
participant S as Staff reviewer
C->>O: POST /api/chat/stream (mode workflow:return-replace)
O->>W: run(state, checkpoint_storage)
W->>DB: save checkpoint (each superstep)
W-->>O: request_info (request_id) from policy-gate
O->>DB: link checkpoint to run, insert hitl_requests (pending), open approval
O-->>C: "needs approval, nothing has been changed yet"
Note over O,W: The request ends. The workflow object is gone.
S->>O: POST /api/approvals/{id}/decide (or /api/orchestration/{run_id}/resume)
O->>DB: claim hitl_requests (pending to processing, claimed_at)
O->>W: new workflow: run(responses={request_id: decision}, checkpoint_id)
W->>DB: load checkpoint
Note over W: Runs as the original requester
W->>DB: initiate-return, then the remaining steps
W-->>O: run_completed
O->>DB: hitl_requests approved or rejected, if the claim still holds
O-->>S: result
Staff decide through POST /api/approvals/{id}/decide or the older POST /api/orchestration/{run_id}/resume (agents/python/orchestrator/routes/orchestration.py, resume_run). The second finds the run’s latest hitl_requests row and calls the same approvals_service.decide. The requester can never be the one who decides.
approvals_service.resume_workflow does the work, in this order:
- Claim. One
UPDATEmoves thehitl_requestsrow frompendingtoprocessingand stampsclaimed_at. A second click finds nothing to claim and gets a409. Aprocessingrow older thanHITL_CLAIM_TTL_SECONDS(default 600) is treated as a crashed resume and can be taken over. - Become the requester. It sets
current_user_emailto the row’suser_emailandcurrent_user_roleto that user’s role. Tools read identity from these ContextVars, so the return is created as the customer. The reviewer is recorded separately. - Resume.
ReturnReplaceMode.resumebuilds a fresh workflow object; the one that paused lived in an earlier request. It callsrun(responses={request_id: decision}, checkpoint_id=..., checkpoint_storage=...). The gate’son_decisionhandler restores the state from the original request. On approval it sends the state on toinitiate-return; on rejection it ends the run with an error and creates nothing. - Record. The row becomes
approvedorrejected, but only ifclaimed_atstill matches. A resume whose claim was taken over records nothing and answers409. The new latest checkpoint is linked to the same run.
The resume has a hard limit of HITL_RESUME_TIMEOUT_SECONDS (default 300, validated to be shorter than the claim TTL). Past it the caller gets a 504 and the approval is still pending. Any exit without a completed resume, cancellation included, hands the claim back. A row of another kind is refused with a 400, and a row with no request_id or checkpoint_id with a 409.
What happens on timeout
A pending approval has a deadline from the policy file: 24 hours for return_approval, with on_timeout: expire. The outbox worker reminds, escalates a support review to a manager once, and at the top level calls approvals_service.expire (approvals: the outbox worker).
expire uses the same resume path. It calls resume_workflow with ReturnDecision(decision="reject", decided_by="system"), so the paused run finishes as a rejection and nothing is created. The approval then becomes expired.
With RETURN_LIFECYCLE_ENABLED, an approved return pauses again for the carrier’s scan and for inspection. Those waits use the same checkpoint and hitl_requests mechanism with their own timers; see Return lifecycle.
Seeing it in the UI
Open /runs. The page calls GET /api/runs/{id}/checkpoints for every visible run. The response lists the checkpoints linked to the run and its latest hitl_requests row. Admins can read any run; everyone else only their own.
- A run with a pending request shows a Needs approval badge without expanding the row.
- Staff (
supportoradmin) see Approve and Reject, which callPOST /api/orchestration/{run_id}/resume. Everyone else sees “Awaiting staff review”. - A return in a lifecycle wait shows its timeline from
GET /api/runs/{id}/timeline.
What differs on .NET
The .NET orchestrator pauses and resumes the same workflow over the same two tables, and serves the same routes. The differences:
- The store.
PostgresJsonCheckpointStore(agents/dotnet/src/ECommerceAgents.Shared/Checkpoints/) implements the framework’sICheckpointStore<JsonElement>. It keys on(session_id, checkpoint_id), reads the index ordered byseq, and mints the UUID itself so thehitl_requests.checkpoint_idforeign key holds.CheckpointStorageFactorytakes the samememory | file | postgresvalues. - The payload format. A .NET checkpoint is MAF .NET’s own JSON, which Python’s storage does not decode. A return paused by the .NET orchestrator is resumed by the .NET orchestrator.
- The pause. The gate pauses on a
RequestPortnode; MAF .NET has noctx.request_infocall from inside an arbitrary executor (parity matrix, row 8). - The resume.
ReturnReplaceMode.ResumeAsyncalso builds a fresh workflow.ReturnAndReplaceWorkflow.ResumeFromCheckpointAsyncrestores the run, waits for the framework to raise the pending request again, and answers that. The storedrequest_idis a correlation check, not the source of the response. - Timeout. Only a stack can resume its own checkpoint, so the .NET orchestrator applies the timeout for a paused return it opened: its
ApprovalTimeoutWorkerhosted service drains thoseapproval.timeout_applyrows and the Python worker skips them (parity matrix, row 36; added after release 1.4.0). The lifecycle waits are not ported (row 41).
Source: docs/checkpoints-and-resume.md — this page is generated from the repository.