Connecting to a model endpoint takes an afternoon. Teams do it, see a working response, and conclude the integration is mostly done. Then the work that actually determines whether the feature survives contact with users begins - and it has nothing to do with the model.
Integration means the AI capability now participates in a system that already has identity, permissions, transactions, error handling, audit requirements, and users with expectations about latency. The endpoint call is the smallest part of that. Everything expensive lives in how the capability behaves when the surrounding system is under load, when a dependency is slow, when the user is not authorized, and when the answer is wrong.
This article covers the layers a production integration has to address: identity and permissions, the data path, how the capability is called, failure behaviour, and what has to be observable afterwards.
The Endpoint Is Not the Interface
The first architectural decision is to stop calling the provider directly from application code.
Scattered SDK calls across a codebase create three problems that all surface later. Switching providers becomes a search-and-replace across every call site. Cross-cutting behaviour - timeouts, retries, cost accounting, prompt versioning, logging - has to be implemented repeatedly and drifts between implementations. And nobody can answer what the system sends to a third party, because the answer is distributed across the repository.
A single internal interface fixes all three. Application code asks for a capability - "summarize this ticket," "extract these fields" - and the layer behind it decides which provider, which model, which prompt version, and how to handle failure. The prompt lives there as a versioned artifact rather than as a string literal next to business logic.
This is not speculative abstraction. Provider deprecations, price changes, and regional availability requirements arrive on their own schedule, and the boundary converts each of them from a migration into a configuration change.
Identity and Permissions Come First
The most common serious defect in AI integrations is that the model sees data the requesting user is not entitled to.
It happens naturally. The retrieval layer is built with a service account that can read everything, because that made indexing straightforward. The permission check is then applied to the final answer, or to which documents are cited - which is too late, because the model has already read the content and may paraphrase it without a citation.
Permission filtering belongs at retrieval time, evaluated against the end user. The user's identity has to travel through the whole chain: request, retrieval, tool calls, and any downstream service. A service account that acts on behalf of a user must not have more authority than that user.
The practical requirement is that permission state is captured with the content at index time and re-evaluated at query time, because access changes after indexing. A document indexed while a user had access should not remain visible to them after that access is revoked.
Two further points that are easy to miss. Access logs need to record what the model retrieved, not only what the user asked, because from a compliance perspective the retrieval is the access event. And if the same index serves multiple tenants, isolation must be enforced structurally rather than by a filter parameter that a bug could omit.
The Data Path in Both Directions
Integration involves two flows with different characteristics.
Inbound is how content reaches the capability. If it comes from a corpus, there is an ingestion pipeline with a synchronization strategy: real-time on change, scheduled, or on demand. Each has a staleness profile, and the choice should be explicit rather than whatever was easiest. Stale content in a retrieval system is invisible - the system answers confidently from an old version.
If content comes from live systems through tool calls, the concerns are latency budget, rate limits, and what happens when the source is unavailable.
Outbound is what the system does with the output. This is where integrations usually under-invest. A model returns text; your systems expect typed, validated, constrained data. The gap between those has to be closed by code: schema validation, business-rule checks, and a defined path for output that fails validation.
Never write model output directly to a system of record without validation. The failure is not that the model produces nonsense - it is that it produces plausible, well-formed output with one field wrong, which passes casual inspection and is discovered three months later in a reconciliation.
How the Capability Is Called
Latency shapes the integration pattern, and choosing the wrong one is difficult to reverse.
Synchronous works when the user is waiting and the operation completes in a couple of seconds - an inline suggestion, a short answer. It requires a hard timeout and a defined degraded path, because a slow provider must not hold your request thread.
Streaming suits conversational surfaces where perceived latency matters more than total time. It complicates error handling: a failure halfway through a stream has already shown the user part of an answer, and the interface has to handle that state.
Asynchronous is correct for anything longer than a few seconds - document processing, multi-step tasks, batch classification. The request creates a job, returns an identifier, and the result arrives by callback or polling. This requires a job store and a status surface, which teams often discover only after shipping a synchronous version that times out under real documents.
Batch fits volume work with no interactive user, and is usually the cheapest option where it applies.
The mistake worth naming: building synchronously because the prototype was fast on short inputs, then meeting production documents. Migrating to asynchronous afterwards touches the API contract, the interface, and the error model.
Failure Behaviour Is a Design Decision
Model providers have outages, rate limits, and latency spikes. The integration's behaviour during those is a product decision, not an implementation detail.
Decide what the system does when the capability is unavailable. The options are to fail the request, to degrade to a non-AI path such as keyword search or a rule-based response, or to queue the work for later. Each is defensible; leaving it undefined means the answer is whatever the HTTP client does by default.
Retries need care. Retrying a generation is usually safe; retrying an action is not, unless it carries an idempotency key. This distinction causes real incidents: a timeout on a request that actually succeeded, retried, produces the action twice.
Rate limits should be handled with backoff and a queue rather than by failing the user's request, and the limit should be tracked as a capacity metric so you learn about saturation before customers do.
Cost needs a circuit breaker. A runaway loop against a per-token API is a financial incident, and the standard controls - per-request token caps, per-user quotas, and a global daily ceiling with alerting - are cheap to add before they are needed and awkward afterwards.
What Has to Be Observable
Conventional monitoring tells you the endpoint returned 200. That is not enough to tell you the feature is working.
Trace the whole path with one correlation ID, so a single identifier links the user request, the retrieval, the model call, the validation result, and any downstream write. Without this, debugging a specific complaint means correlating timestamps across services by hand.
Record the versions that produced each result: model, prompt, retrieval configuration. Behaviour changes are usually deployments, and without version stamps they look like mysteries.
Track cost per request and per feature. Model spend is one of the few infrastructure costs that scales with usage in a way that can surprise a finance team mid-quarter.
Measure quality, not just availability. Capture user-visible signals - regeneration requests, edits to generated output, thumbs-down, escalation to a human. These are the closest thing to ground truth that arrives for free, and a rising rate is an early warning that something upstream changed.
Sequencing the Work
A build order that avoids the common rework.
Start with the internal interface, even if it wraps a single provider. It is a small amount of code and it is the boundary everything else depends on.
Establish identity flow next, before the retrieval layer is built, because retrofitting permission filtering into an index built without it usually means rebuilding the index.
Add validation on the outbound path before connecting to any system of record, so that unvalidated output never has a route into production data.
Choose the invocation pattern from the realistic worst-case input size rather than the demo input, and build asynchronously if there is any doubt.
Then instrument, and only then optimize. Prompt tuning and model selection produce visible improvements and are tempting to do first, but they are the cheapest parts to change later - and without tracing you cannot tell whether a change helped.
The integration, not the model, is what users experience. A capable model behind a poorly integrated path produces a feature that is slow, occasionally unauthorized, and impossible to debug. A modest model behind a well-built one produces something people can rely on.