Designing Reliable Agent Workflows
Make long agent tasks survive reality: idempotency, checkpoints, resumability, and explicit failure states.
Assume the process will die
A long-running agent workflow will be interrupted: network blip, rate limit, container restart, user closing the laptop. Design for interruption from the start, because retrofitting resumability into a linear script is a rewrite.
The core question for every step: if this crashes right here, what happens on retry?
Idempotency makes retry safe
An operation is idempotent when running it twice equals running once. This single property is what makes retries safe instead of dangerous.
| Not idempotent | Make it idempotent |
|---|---|
create_report(id) | create_report(id, dedupe_key=task_id) |
append_row(...) | upsert_row(row_id, ...) |
send_email(to, body) | send_email(idempotency_key=hash(to+body+date)) |
increment_counter() | set_counter(computed_value) |
Attach a deterministic key derived from the task, not a random one — randomness defeats the purpose.
Checkpoints: durable state outside the context
Write progress to durable storage after each meaningful step, not in the model’s memory:
run-2026-10-16-a1/
├── manifest.json # inputs, task ids, started_at
├── steps/
│ ├── 01-extract.json # status, output pointers, duration
│ ├── 02-transform.json
│ └── 03-review.json
└── artifacts/
On resume, read the manifest, find the first incomplete step, and continue there. Completed steps should be skipped by checking their output exists, not by trusting a status field — status fields lie when a process dies mid-write.
Failure states must be explicit
“Failed” is not a diagnosis. Distinguish, because each has a different remedy:
| State | Meaning | Response |
|---|---|---|
retryable | Transient (timeout, 429) | Backoff and retry |
validation_failed | Output malformed | Retry with the error fed back |
needs_input | Missing information required | Ask the human, do not guess |
blocked | External dependency unavailable | Park the task, notify |
fatal | Unrecoverable | Stop cleanly, keep artifacts for debugging |
A workflow that treats every failure as retryable burns budget and eventually corrupts state. A workflow that treats everything as fatal gives up prematurely.
Never let the model silently absorb a failure. If a step did not succeed, the workflow must know — not merely have a vaguer memory of the attempt.
Budget tracking
Long tasks run out of money before they run out of ideas. Track per-run:
- Tokens consumed vs. ceiling
- Tool calls vs. ceiling
- Wall-clock elapsed
- Distinct retries per step
Check the budget before starting each step, not after discovering the overage. When the ceiling is hit, stop with interrupted and a resumable checkpoint — never with half-applied changes.
The resume checklist
- Read manifest → restore task identity and inputs
- Discover completed steps by checking artifacts exist
- Re-validate any step output from a previous run (do not assume it is intact)
- Continue from first incomplete step
- Re-check budget before each subsequent step
That third item is the one teams skip. A checkpoint written moments before a crash can be truncated; verifying it costs one read and saves hours of confusion.