Agentic AI systems are often presented as reasoning loops around a language model. In production, the difficult problems usually appear outside the model: jobs outlive HTTP requests, workers fail halfway through expensive tasks, model outputs violate schemas, tools return partial results, memory becomes stale, and debugging crosses several services.
A reliable backend must treat an agentic task as a durable workflow rather than a long chat completion. Execution state must be persisted, side effects must be idempotent, outputs must be validated, memory must remain inspectable, and every operation must share one distributed trace.
The language model can select actions and synthesize results. It should not be responsible for durability, arithmetic correctness, retry safety, or the integrity of application state.
The Long-Running Task Problem
Consider a request to process a batch of business documents, reconcile invoice line items with a product catalog, retrieve contract conditions, calculate exceptions, and produce a sourced summary. The workflow may require OCR, database queries, several model calls, external tools, and human review.
Keeping the original HTTP connection open for ten minutes is fragile. Proxies and clients can time out, the user cannot reliably resume after disconnecting, and a worker failure can discard the entire task.
The standard pattern is:
submit → job_id → background execution → status/progress → result notification
The API accepts the request, validates it, persists a job, enqueues work, and returns immediately.
An Asynchronous API Contract
A task submission endpoint can return 202 Accepted:
POST /analyses
Idempotency-Key: 8cf92ad7-7bf4-47c1-a472-65452d5c7190
Content-Type: application/json
{
"document_ids": ["invoice-2026-0148", "catalog-2026-03"],
"analysis_type": "invoice_catalog_reconciliation"
}
HTTP/1.1 202 Accepted
Location: /analyses/job-7f82a1
{
"job_id": "job-7f82a1",
"status": "queued",
"created_at": "2026-07-16T09:15:00Z"
}
The job resource becomes the stable interface:
GET /analyses/job-7f82a1
{
"job_id": "job-7f82a1",
"status": "running",
"current_step": "validate_line_items",
"completed_steps": 4,
"total_steps": 7,
"progress": 0.57,
"updated_at": "2026-07-16T09:16:42Z"
}
Clients can poll this endpoint. Server-Sent Events or WebSockets can provide live progress. A webhook or push notification can announce completion. These delivery mechanisms do not change the underlying job model.
Job State as a State Machine
A durable job should move through explicit states:
queued → running → waiting_for_review → completed
↘ retrying
↘ failed
↘ cancelled
retrying is a sub-state entered from running after a step failure, not a terminal branch.
Every transition should be persisted with a timestamp and reason. Invalid transitions should be rejected. For example, a completed job cannot return to running, and cancellation should distinguish between “requested” and “confirmed stopped.”
A waiting_for_review state should carry an expiration policy: an SLA timer that escalates, reminds a reviewer, or automatically fails the job if no response arrives, so a job cannot remain silently stuck indefinitely.
Step-level state makes progress meaningful and supports resumption. Storing only a percentage does not reveal which side effects have already occurred.
Queue and Workflow Engine Choices
A task queue such as Celery with Redis can be sufficient for short, independent background jobs. It provides asynchronous execution, retries, and worker scaling with relatively low complexity.
A durable workflow engine such as Temporal is better suited to long, multi-step processes that need timers, compensation, human pauses, durable retries, and resumption across worker restarts. Workflow code describes the sequence, while the engine persists execution history.
Durable workflow code must remain deterministic across deployments. When workflow logic changes while executions are still in flight, the engine must distinguish old and new code paths (e.g., versioning or patch markers) or replay will fail against historical event history. Deploying a workflow change without a migration strategy is a common source of production incidents, not an edge case.
The choice should follow workflow characteristics:
- Use a simple queue when tasks are short, mostly independent, and easy to retry from the beginning.
- Use a durable workflow engine when tasks are expensive, stateful, long-running, or contain non-repeatable side effects.
Adopting a durable workflow engine adds operational cost: running or subscribing to the engine, enforcing deterministic workflow code, and training the team on its execution model. This cost should be weighed against the failure modes it prevents, not adopted by default.
A language-model orchestration framework can manage reasoning state, but it does not automatically replace a durable execution engine. Checkpointing the agent graph and durably coordinating external activities are related but distinct concerns.
Idempotency
Users retry requests. Browsers resend submissions. Queues can deliver messages more than once. A reliable system assumes duplicate execution is possible.
An idempotency key associates equivalent submissions with one job. If the same key and payload arrive twice, the API should return the existing job rather than start another analysis.
Idempotency is also required inside the workflow. Writing a result, calling an external service, or sending a webhook should use stable operation IDs. A retried step must not create duplicate records or notifications.
Database constraints can enforce uniqueness for critical side effects. The workflow should prefer upserts, compare-and-set transitions, and transactional outbox patterns over “check, then insert” logic that can race.
Checkpointing and Resumption
A ten-step analysis should not restart from step one because a worker failed at step seven. Each completed step should persist:
- Input state
- Validated output
- Tool and model versions
- Source references
- Side effects performed
- Retry count
- Completion timestamp
After a restart, the workflow loads the latest valid checkpoint and continues from the next safe step. Checkpoints must represent validated state. Persisting malformed model output simply makes an error durable.
Some steps are safe to repeat, while others require compensation. A read-only retrieval can be retried freely. A step that updates an external system needs idempotency or a compensating action.
Testing Durable Workflows
Reliability claims should be verified, not assumed. Useful practices include: replaying recorded production histories against new workflow code before deploying it, time-skipping test harnesses that simulate long timers without real waiting, fault-injection tests that kill a worker mid-step to confirm resumption behavior, and contract tests that assert schema/business-rule validation rejects known-bad model outputs.
Boundaries: Timeouts, Step Limits, and Cancellation
Agent loops need explicit limits. A production workflow should define:
- Maximum model calls
- Maximum tool calls
- Maximum total runtime
- Per-tool timeout
- Token and cost budget
- Allowed tools by route
- Maximum retry count
- Cancellation behavior
Cancellation should propagate to workers and tools where possible. The system should stop scheduling new steps, mark in-flight operations appropriately, preserve completed state, and record whether any non-reversible side effect occurred.
Without these boundaries, a misunderstood request or repeated tool failure can create an unbounded loop.
Authorization and Tenant Isolation
Each job should carry a scoped authorization context: which tools, data sources, and workspaces it may access. Cross-tenant data must never be reachable from a single job's execution context, and external credentials used by tools should be short-lived and scoped per job rather than shared static secrets.
Treating LLM Output as Untrusted Input
A language model can return malformed JSON, omit required fields, use the wrong type, add unsupported properties, or produce a structurally valid but inconsistent result. Its output should be validated with the same discipline applied to external user input.
A typed schema defines the contract:
from decimal import Decimal
from pydantic import BaseModel, Field
class LineItem(BaseModel):
product_code: str
quantity: int = Field(gt=0)
unit_price: Decimal = Field(ge=0)
line_total: Decimal = Field(ge=0)
source_id: str
class ReconciliationResult(BaseModel):
invoice_number: str
currency: str
items: list[LineItem]
grand_total: Decimal = Field(ge=0)
warnings: list[str] = []
Structured-output capabilities should be used so that the model is constrained toward the schema from the start. Validation is still required because a correct shape does not guarantee correct values.
The Repair, Retry, and Fallback Ladder
Validation failures should follow a bounded recovery sequence.
Mechanical Repair
Cheap deterministic fixes should run first. Examples include removing Markdown fences, repairing minor JSON syntax, normalizing decimal separators, or converting an unambiguous numeric string to a numeric type.
Mechanical repair must not invent missing business values.
Feedback-Guided Retry
If validation still fails, the error can be returned to the model:
Validation failed:
- items[2].source_id is missing
- grand_total must be a number
Return a corrected object using only the supplied evidence.
Retries should have a small fixed limit. Repeating the same prompt indefinitely is not a recovery strategy.
Model Escalation
A stronger model may handle difficult structured output or synthesis when the normal model fails. Escalation should preserve the same schema, evidence, and trace. It should also be measured because frequent escalation can erase expected cost savings.
Safe Failure
If the output remains invalid, the workflow should return a controlled failure or request human review. Incorrect financial values are more damaging than an explicit statement that processing could not be completed safely.
Every validation error should be recorded as a metric by model, prompt version, route, and field. A rising validation-failure rate is often an early signal of model or prompt regression.
Deterministic Business Validation
Schema validation verifies shape and types. Business validation verifies relationships among values.
For invoice reconciliation, deterministic rules can check:
quantity × unit_price = line_total
sum(line totals) + tax - discounts = grand_total
product_code exists in catalog
invoice currency matches contract currency or conversion is explicit
every value has a valid source_id
These checks should produce structured errors with field paths and source references. The workflow can then reprocess one line item instead of repeating the entire analysis.
The final generated explanation should be compared with the validated result object. If the tool state contains 125.50 EUR and the answer states 152.50 EUR, a runtime consistency check should block the response.
Three Layers of Agent Memory
Memory is application state, not an unlimited transcript. Different time horizons require different storage and retention policies.
Short-Term Session Memory
Short-term memory contains the active conversation and task state. Recent messages may remain verbatim, while older messages are summarized. Critical entities should be stored in structured form:
{
"supplier": "Supplier Beta",
"product_group": "Monitoring Products",
"period": "2026-Q1",
"currency": "EUR"
}
Structured state prevents a summary from silently changing identifiers, periods, or units. Session memory can expire when the task ends unless it is promoted to a longer-lived layer.
Episodic Task Memory
Episodic memory stores the outcome of a completed workflow, not every raw reasoning step. A useful record may include:
{
"task": "supplier_spend_analysis",
"entities": ["Supplier Beta", "PRD-482"],
"summary": "Quarterly spend calculated from 14 validated invoice lines.",
"result_reference": "analysis-job-7f82a1",
"sources": [
"agreement-2026-04",
"invoice-2026-0148"
],
"effective_period": "2026-Q1",
"created_at": "2026-07-16T09:18:04Z"
}
This allows a later request to reuse the verified result rather than repeat every tool call.
Long-Term Memory
Long-term memory stores durable preferences and facts that remain useful across sessions, such as preferred currency, reporting format, approved product groups, or recurring analysis scope.
Long-term facts need provenance, timestamps, update rules, and deletion controls. Free-form statements without these properties become an uninspectable source of hidden behavior.
Inspectable Memory and Provenance
Every memory record should answer four questions:
- What information is stored?
- Where did it come from?
- When was it valid?
- Which task or session created it?
Metadata makes memory searchable and correctable. A user or operator can inspect what the system remembers about a supplier analysis, locate the source, and delete or replace an incorrect record.
Memory retrieval can combine entity filters, time filters, and semantic search. For a request referring to “the supplier analysis from last quarter,” the system can filter by task type and period before applying semantic similarity.
The retrieved memory should be confirmed when ambiguity remains. A short confirmation is cheaper than continuing an expensive workflow with the wrong historical record.
Staleness in Memory
Memory is not automatically current. A previous product-price analysis may become outdated after a new catalog or contract takes effect.
Before reusing an episodic result, the system should compare its effective period and source versions with current data. If newer sources exist, the old result can be presented as historical context or used as a baseline while only the changed data is reprocessed.
This approach reduces cost without presenting an old conclusion as a current fact.
Distributed Tracing with One Correlation ID
Agentic workflows cross API services, queues, workers, retrieval systems, model gateways, and databases. A single trace_id should be created when the request enters the system and propagated through every component.
OpenTelemetry spans can describe infrastructure execution:
- API validation
- Queue wait
- Worker execution
- Database query
- Vector search
- External tool call
- Webhook delivery
LLM observability can add semantic spans using OpenTelemetry's GenAI semantic conventions - still evolving as of 2026, so teams should expect attribute names to change and rely on instrumentation libraries rather than hardcoding attribute keys.
- Prompt and model version
- Retrieved chunks and scores
- Tool selection and arguments
- Token use
- Structured-output validation
- Generated answer and citations
The same trace ID should connect both layers. An operator can then determine whether a slow request waited in the queue, spent time in retrieval, retried schema generation, or stalled in an external tool.
Logs should be structured and include job ID, trace ID, step ID, model version, retry number, and status. Sensitive content can be redacted while retaining the identifiers needed for correlation.
Failure Handling and Dead-Letter Queues
Retries should distinguish transient and permanent failures.
Transient failures include timeouts, temporary model unavailability, and short database interruptions. They can use exponential backoff with jitter. Permanent failures include invalid source documents, exhausted validation retries, unauthorized access, and unsupported task types.
Jobs that exhaust their retry policy should move to a dead-letter queue rather than disappear. A dead-letter record should contain the job and trace identifiers, failing step, sanitized error, attempt count, and reprocessing instructions.
Replaying a failed job should use the original idempotency and side-effect controls. Manual replay must not create duplicate notifications or records.
Concurrency, Backpressure, and Resource Isolation
Agentic workflows combine workloads with very different resource profiles. OCR may be GPU-heavy and batch-friendly. Vector retrieval is latency-sensitive. Language-model generation may consume substantial GPU memory for an unpredictable duration. Validation and database updates are usually lightweight but should not wait behind model inference.
Separate queues and worker pools provide isolation:
document queue → OCR workers
embedding queue → embedding workers
agent queue → workflow workers
LLM queue → inference workers
notification queue → delivery workers
Each pool can have its own concurrency, timeout, retry, and scaling policy. A surge of document uploads should not prevent completed jobs from sending notifications or block interactive retrieval requests.
Backpressure should begin at admission. Queue-depth limits, per-workspace quotas, maximum batch sizes, and cost budgets prevent the system from accepting more work than it can process within its service target. When capacity is exhausted, the API can return an explicit overload response or accept the job with a realistic wait estimate.
Batching improves throughput for OCR and embedding inference, but it adds queueing latency. Interactive and batch workloads may therefore need separate queues. Priority should be explicit rather than emerging accidentally from arrival order.
Useful capacity metrics include:
- Queue depth and oldest-job age
- Worker utilization
- GPU memory and batch occupancy
- Throughput by task type
- Time spent waiting versus executing
- Retry and timeout rates
- Token and model cost per job
- Percentage of jobs requiring escalation or review
Resource limits should be part of persisted job policy. A resumed workflow must retain its original maximum steps, token budget, and allowed models rather than silently adopting a more expensive configuration.
Load shedding should preserve correctness. Skipping validation or reducing retrieval quality under pressure creates silent failures. It is safer to delay, reject, or route noncritical work to a lower service tier with an explicit contract.
A Reference Execution Flow
A reliable long-running workflow can follow this sequence:
API submission
→ authenticate and validate
→ resolve idempotency key
→ persist job
→ enqueue workflow
→ load checkpoint
→ execute bounded agent step
→ validate structured output
→ run deterministic business rules
→ persist checkpoint and provenance
→ continue, retry, escalate, review, or fail safely
→ assemble final result
→ runtime consistency check
→ mark completed
→ notify client
Every arrow represents a state transition that should be observable and recoverable.
Reliability Comes from the Boundaries
Agentic systems become dependable when the language model is surrounded by explicit contracts. The API separates submission from execution. The workflow engine persists progress. Idempotency makes retries safe. Schemas and deterministic rules prevent malformed or inconsistent values from entering application state. Memory stores sourced, time-aware records instead of opaque conversation fragments. Distributed tracing connects model behavior to infrastructure behavior.
The model remains important, but it is no longer asked to provide guarantees it cannot provide. Durable execution, validated state, and inspectable provenance belong to the backend. Those boundaries are what allow an agentic workflow to survive real traffic, partial failures, evolving data, and operational scrutiny.