Six Layers, One Question Each: Key Takeaways
- Enterprise AI fails for architectural reasons far more often than model reasons.
- Six layers carry the load: data, retrieval, model, orchestration, observability, governance.
- Every unwritten boundary contract has a matching, predictable production failure mode.
- Data shape decides model choice more often than model capability decides data strategy.
- Agent orchestration is a deployment decision, not a capability the platform lacks.
- Governance controls that live in review meetings cannot be evidenced later.
Enterprise AI architecture is the layered system design that runs AI in production: data, retrieval, model, orchestration, observability, and governance. Each boundary carries an explicit contract. When all six are named, the platform survives model replacement without a rewrite, because the model becomes a swappable component rather than the architecture itself.
Why the Failure Rate Is an Architecture Number
Most enterprise AI programmes do not fail at the model. They fail at a boundary nobody wrote down.
Gartner's February 2025 assessment put a number on it: through 2026, organizations will abandon 60% of AI projects unsupported by AI-ready data. The wording matters. Those projects were not abandoned because the model was weak. They were abandoned because the data layer held no contract with the layers above it, so nothing downstream could be reproduced or trusted.
The second pressure is churn. Frontier models get replaced faster than enterprise platforms get rewritten. A system that hard-codes one provider's prompt format, token limits, and tool-calling syntax into its business logic pays a migration cost every time the frontier moves.
Both pressures point at the same discipline. Name the layers, write the contract at each seam, and the platform absorbs change at the seam instead of across the whole codebase. Skip that work and the result is a demo that happens to be running in production.
What Are the Six Layers of an Enterprise AI Architecture?
The layer model is not a taxonomy exercise. Each boundary is a place where a contract has to be written down, and each unwritten contract has a matching failure that shows up in production on a schedule.
| Layer | Boundary contract it owns | Failure mode when the boundary stays implicit |
|---|---|---|
| Data | Schema, lineage, and freshness guarantees per source | Output quality drifts and nobody can name which source moved |
| Retrieval | Query shape, ranking method, recall budget | Answers degrade on the input classes the corpus never indexed |
| Model | Task routing, fallback order, evaluation gates | Every model upgrade turns into a downstream prompt rewrite |
| Orchestration | Step boundaries, retry policy, state ownership | Loops run unbounded and spend becomes unpredictable |
| Observability | Span shape, token accounting, output capture | Regressions get reported by users instead of dashboards |
| Governance | Risk classification, approval path, audit record | Controls live in meetings and cannot be evidenced on request |
Why the boundaries sit where they do
A boundary earns its place when the two sides change at different rates. Data sources change on the provider's schedule. Models change on the vendor's schedule. Business logic changes on the product's schedule. Putting a contract between them means each side can move without dragging the others.
Microsoft's Azure Architecture Center AI/ML baseline separates the data plane from the inference plane for the same reason. The version below goes further by treating orchestration, observability, and governance as first-class layers with their own contracts rather than as operational concerns bolted on after launch.
The layer most often collapsed into another is orchestration. Teams fold step sequencing and state handling into application code, where it becomes invisible to anyone reading the architecture. The result is a system whose control flow can only be recovered by reading the codebase, which is also why nobody can answer what happens on the third retry.
Two layers are commonly deferred rather than collapsed. Observability gets added after the first quality complaint, and governance gets added after the first audit request. Both retrofits are expensive for the same reason: they need data the earlier layers were never instrumented to emit. Retrieved context that was not stored cannot be reconstructed later, and a decision whose prompt version was never recorded cannot be explained after the fact.
Why Does the Data Shape Decide the Model More Often Than the Reverse?
Teams tend to pick a model first, then discover the retrieval layer cannot feed it. The order is backwards. Query distribution and corpus behaviour constrain the retrieval method, and the retrieval method constrains what the model can actually do.
The decision at this boundary is which retrieval method matches the query set, not which one benchmarks best in isolation.
| Approach | Best fit | Cost of owning the boundary | Where it breaks |
|---|---|---|---|
| Vector retrieval | Semantic paraphrase dominates the query set | Embedding refresh on every corpus change | Transliterations, codes, exact identifiers |
| Lexical (BM25) | Queries carry exact names, tokens, or IDs | Analyzer and synonym maintenance | Conceptual questions with no shared terms |
| Hybrid with rerank | Query set mixes both and recall outranks latency | Two indexes plus a ranking policy to own | Tight latency budgets with no caching layer |
| Long context, no retrieval | Small, stable corpus | Token cost scales with corpus on every call | Any corpus that grows or updates daily |
Three data-layer properties decide the outcome more than model size does:
- Update frequency. A corpus that changes daily rules out embedding strategies with slow refresh paths.
- Identifier density. Names, account numbers, and codes reward lexical matching that vector search flattens.
- Residency constraints. Where the data may physically sit decides which inference options remain legal.
Query profiling is the cheap step teams skip. A sample of real queries, classified by whether the answer depends on paraphrase or on an exact token, tells you the index shape before any embedding job runs. Guessing wrong at this boundary is expensive to undo, because the index, the chunking strategy, and the evaluation set all get built on the assumption.
Once those three are fixed, model selection narrows to a shortlist, and domain-specific model selection becomes a routing question rather than a strategic one. The retrieval and data layers are data platform engineering surface, and treating them as AI work rather than data work is how the contract goes unwritten.
Teams mapping these boundaries onto an existing stack can see how AI and machine learning engineering covers the data plane through inference.
Is Agent Orchestration a Capability Decision or a Deployment Decision?
It is a deployment decision. Multi-agent designs do not add capability the model lacks. They add coordination, and coordination has a cost that has to be justified by a real constraint.
Gartner predicted in June 2025 that over 40% of agentic AI projects will be canceled by the end of 2027, with cost and unclear business value named among the drivers. The pattern is consistent: teams reach for agents to solve a problem that a defined workflow with two tool calls would have solved with a bounded spend.
The orchestration boundary owns three things and nothing else:
- Step boundaries. What counts as one unit of work that can be retried safely.
- State ownership. Which component holds conversation, task, and intermediate state, and where it persists.
- Termination policy. The condition under which a loop stops, plus the budget ceiling that stops it anyway.
When the workflow is known in advance, the architectural response is a deterministic pipeline with model calls inside it. When the workflow genuinely branches on content the system cannot enumerate, the architectural response is an agent with an explicit termination policy. The multi-agent orchestration patterns worth deploying are the ones where the branching is real.
What Telemetry Does an AI System Need That a Service Does Not?
A conventional service is observable through latency, error rate, and throughput. All three can look healthy while an AI system produces worse answers than it did last week.
The OpenTelemetry GenAI semantic conventions define spans for model invocations, token counts, and tool calls, which means LLM telemetry now has a standard shape rather than a per-team convention. That standard is the contract for this layer.
Four signals belong here and nowhere else:
- Retrieval trace. Which documents were returned, at what rank, for the query as rewritten.
- Token accounting. Input and output tokens per call, attributed to a tenant and a feature.
- Output capture. The response, the prompt version, and the model version that produced it.
- Evaluation result. A score against a held-out set, run on a schedule rather than at release only.
Without output and prompt-version capture, a regression cannot be reproduced. The team knows quality dropped and cannot say which change did it. That is the failure mode this boundary exists to prevent.
Standard service telemetry misses these regressions structurally, not accidentally. Latency, error rate, and throughput all measure whether the call completed, and an AI system returns a well-formed, fast, wrong answer with a 200 status code. The signal that something changed lives in the retrieval trace and the evaluation score, which means the observability layer has to store content, not just counters. That storage decision carries a retention and access-control requirement, which is where this layer meets the governance layer.
Why Do Compliance Controls Belong in the Architecture Instead of a Review Gate?
Because a procedural control produces a meeting, and an architectural control produces a record. Only one of those can be handed to an auditor.
The NIST AI Risk Management Framework 1.0 (2023) structures its Govern, Map, Measure, and Manage functions around controls that are exercised continuously, not signed off once. ISO/IEC 42001:2023 goes further and treats AI governance as a management system with requirements the platform has to satisfy. Under the EU AI Act, Regulation (EU) 2024/1689, obligations attach to a system's risk classification, which means the classification has to be a property of the architecture rather than a label applied in a document.
The translation into layer terms is direct:
- When the risk policy requires reconstructable decisions, the architectural pattern is immutable versioning of prompt, model, and retrieved context per decision.
- When the policy requires human review on a decision class, the pattern is a routing rule at the orchestration layer, not a note in a runbook.
- When residency requires data to stay in-region, the pattern is per-region inference and index isolation enforced at the boundary.
This is the same discipline that governs auditable AI system design in regulated deployments, and it is why RegTech AI architecture tends to expose these boundaries earlier than other domains. The client's compliance function owns what must be true. The architecture owns how it is enforced and evidenced.
Our Position on Layer Boundaries: Three Calls We Defend
The layer model above came out of building an AI-powered AML screening platform where every boundary was tested by a hard requirement. Three calls held up.
Hybrid retrieval over pure vector similarity. Vector-only search on PostgreSQL with pgvector handled English names well and failed on Arabic and Cyrillic transliterations. We added a phonetic and lexical matching path and reranked across both, chosen over a single vector index because recall on transliterated identifiers was the accuracy driver and no embedding model closed that gap alone.
A caching layer over batch retrieval. The screening SLA was sub-200ms on live payment flows. We put a Redis-backed cache in front of the retrieval layer rather than moving to batch scoring, because batch would have satisfied throughput while breaking the real-time decision path the product existed to serve.
Per-tenant index isolation over shared-index multi-tenancy. Shared indexes are cheaper to run. Regulated clients required data-residency guarantees, so index-level isolation per tenant was the only shape that made residency provable rather than asserted. The same reasoning shaped our watchlist ingestion pipeline design.
Designing the Layer Boundaries with DigiWagon
DigiWagon builds production AI platforms where the layer contracts are written before the first model call. The work covers:
- Data and retrieval layer design with hybrid search and reranking
- Model routing, fallback, and evaluation gates
- Orchestration with bounded state and termination policy
- Observability and governance controls built into the platform
Making the Architecture Outlive the Model
The six layers are not a maturity ladder. They are six places where a contract either exists or does not, and the absence of each one produces a failure that is predictable from the boundary alone. Data without lineage produces drift nobody can attribute. Orchestration without a termination policy produces unbounded spend. Governance without an architectural record produces controls that cannot be evidenced. Get the contracts written and model selection drops to what it should be, a routing decision inside one layer. Platforms built to the six-layer model keep running through the model churn they were designed to absorb.
