Beyond Vector Search: Knowledge Graphs, Structured Retrieval, and Intelligent Routing

Vector search is an effective way to retrieve passages that are semantically similar to a question. It is not a universal interface to every kind of knowledge. Exact identifiers, deterministic calculations, and relationships distributed across multiple documents require different retrieval primitives.

A production knowledge system should begin with the information need, not with the database that happens to be available. Open-ended questions may belong in a vector index. Product codes need an exact-match field, while their surrounding technical descriptions may also benefit from lexical search. Exact totals should come from structured tables. Multi-hop relationship questions may be better represented as graph traversals.

The purpose of a retrieval architecture is therefore not to force every question through one search method. It is to route each question to the representation that can answer it most reliably and combine methods when the request spans several forms of knowledge.

The Structural Limit of Semantic Similarity

Consider a document collection containing procurement agreements, product catalogs, and invoices.

One agreement states that Company A purchases industrial sensors from Supplier Beta and control modules from Supplier Gamma. Separate invoices contain quantities and unit prices for products supplied by Beta and Gamma. A user asks:

What was Company A’s total spend on monitoring products supplied by its approved vendors during the last quarter?

No single passage necessarily resembles that question. The answer requires several operations:

1. Identify Company A’s approved vendors.
2. Identify which products belong to the monitoring category.
3. Find invoice line items connecting those vendors and products to the requested period.
4. Calculate the total from quantity, unit price, discounts, and tax rules.

A vector query may retrieve the procurement agreement because it mentions Company A and suppliers. It may retrieve a general passage about monitoring products. It has no guaranteed mechanism for following the relationship from Company A to each supplier, joining those suppliers to invoice rows, applying a date filter, and computing a precise total.

This is not a weakness that can always be solved by a larger embedding model. The problem is relational and computational rather than purely semantic.

Choosing Among Retrieval Primitives

A practical architecture can expose several complementary retrieval methods behind one policy and planning layer:

Source documents and operational records
                  ↓
       Versioned ingestion pipeline
                  ↓
  ┌───────────────┼───────────────┬───────────────┐
  ↓               ↓               ↓               ↓
Exact fields   Text indexes   Relational data   Knowledge graph
  ↓           ┌────┴────┐          ↓               ↓
Term lookup   BM25    Vectors      SQL          Traversal
  └───────────────┬───────────────┴───────────────┘
                  ↓
      Router, planner, and policy layer
                  ↓
     Validated answer with provenance

Method	Best suited to	Main strength	Main limitation
Exact match	Product codes, invoice numbers, contract IDs, canonical entity IDs	Deterministic identity lookup	Does not handle paraphrases or descriptive intent
BM25 lexical search	Rare terms, technical phrases, names, full-text keyword relevance	Strong token-based ranking without embeddings	Less robust to semantic variation
Vector search	Concepts, paraphrases, explanations, summaries	Retrieves semantically related passages	Weak guarantees for exact identifiers and calculations; hard filters can reduce ANN recall
SQL	Filters, joins, totals, grouped metrics, current state	Typed and deterministic computation	Requires validated structured data and explicit schema
Knowledge graph	Repeated multi-hop relationship queries	Natural path traversal with reusable relationships	Adds entity-resolution, synchronization, and operational cost

Vector Search for Meaning

Vector search is appropriate for questions whose answer depends on concepts, explanations, or paraphrases:

• What risks are described in the supplier agreement?
• How does the product documentation explain calibration?
• Summarize the return conditions for damaged goods.
• Why was the delivery schedule changed?

These questions benefit from semantic matching because the user’s wording may differ from the document.

The lexical/dense split is not the entire retrieval design space. Learned sparse retrieval, such as SPLADE, can be exposed as another retrieval option [TODO-REF]. Late-interaction or multi-vector models, such as ColBERT, offer another cost and quality trade-off [TODO-REF]. Half-precision vectors and binary quantization can reduce index size while changing the cost and quality trade-off [1]. The same routing and fusion policy can expose these methods as optional branches, with selection evaluated by query class rather than treated as a universal recommendation.

Lexical and Exact-Match Retrieval for Identity

Exact identifiers and lexical relevance are related but different retrieval problems. Product codes, invoice numbers, contract IDs, and canonical supplier IDs should normally be stored in non-analyzed keyword fields or equivalent exact-value columns. A term query or database equality predicate can then require the complete normalized identifier rather than rank partially matching tokens. Elasticsearch, for example, distinguishes exact keyword search from analyzed full-text queries and documents that a term query does not analyze its input [2].

Exact-match retrieval is appropriate for:

• Product code PRD-482
• Invoice number INV-2026-0148
• Supplier reference code
• Contract identifier

BM25 belongs in the full-text lexical branch. It ranks analyzed text using term statistics and is particularly useful for rare terminology, technical phrases, abbreviations, product names, and passages where the literal wording matters. It can retrieve a clause containing “differential pressure calibration” even when there is no dedicated identifier field. BM25 is a scoring model for lexical relevance, not a substitute for exact equality [3].

Embedding spaces represent semantic similarity. An opaque identifier may have no useful semantic neighborhood at all. A mixed request such as “Explain the calibration requirements for PRD-482” can therefore use an exact filter for the product code, BM25 for literal technical language, and vector retrieval for explanatory passages. Keeping these branches distinct prevents an analyzer from splitting PRD-482 into misleading partial matches and prevents BM25 score differences from changing a deterministic lookup.

SQL for Deterministic Values

Structured databases should answer questions that require exact filters, aggregations, and calculations:

• What was the unit price of PRD-482 in March?
• How many units were purchased from Supplier Beta?
• What is the total tax amount across a set of invoices?
• Which invoices have a grand total above a threshold?

These answers should not depend on whether a relevant text chunk appears in the top five search results. Once critical fields have been extracted and validated during ingestion, SQL provides explicit filters, repeatable calculations, and predictable types.

Knowledge Graphs for Relationships

A knowledge graph represents facts as relationships among entities. Instead of storing only text passages, it can store triples such as:

(Company A) -[APPROVED_VENDOR]-> (Supplier Beta)
(Supplier Beta) -[SUPPLIES]-> (PRD-482)
(PRD-482) -[BELONGS_TO]-> (Monitoring Products)

Graphs are useful when questions involve paths: suppliers of a company, products supplied by those suppliers, related contracts, or dependencies across several entities. They do not replace source documents. Every relationship should retain a link to the chunk or structured record from which it was extracted.

When a Knowledge Graph Is Worth the Cost

A knowledge graph is not mandatory for every retrieval system. If the relationship is simple, stable, and already represented by foreign keys, an indexed SQL join is often easier to operate and can answer the question with fewer synchronized copies. A company-to-supplier relationship and a supplier-to-product table do not automatically justify a graph database.

A graph becomes more compelling when multi-hop traversal is frequent, the same relationships support many query types, path structure is part of the answer, and entity resolution is reliable enough to prevent duplicate or incorrectly merged nodes. It is also useful when the domain needs reusable relationship provenance or temporal edges across heterogeneous sources.

The trade-off is operational. A graph introduces another schema, query language, storage engine, authorization surface, deployment, backup path, and synchronization process. Edge extraction and entity resolution must be monitored, updates and deletions must propagate, and graph results must remain consistent with SQL and document indexes. If these costs exceed the measured benefit over relational joins, SQL should remain the relationship store.

Building Multiple Representations During Ingestion

Hybrid retrieval begins at ingestion. The same source document can produce several synchronized representations:

• Original document for audit and display
• Parsed chunks for semantic and BM25 lexical search
• Normalized identifier fields for exact-match retrieval
• Embeddings for the vector index
• Validated fields for relational tables
• Normalized entities and relationships for the graph

These representations must share stable identifiers. A relationship extracted from a procurement agreement should include a source document ID, page number, chunk ID, extraction version, and timestamp. A structured invoice row should preserve the same provenance.

For example:

{
  "source_entity": "Company A",
  "relationship": "APPROVED_VENDOR",
  "target_entity": "Supplier Beta",
  "source": {
    "document_id": "agreement-2026-04",
    "page": 12,
    "chunk_id": "agreement-2026-04-p12-c03"
  }
}

Provenance serves two purposes. It allows generated answers to cite evidence, and it allows incorrect graph edges to be traced back to the extraction that created them.

Chunk Boundaries

Chunk boundaries may follow layout structure or fixed-size rules. Layout-aware splitting can preserve headings, clauses, lists, and page regions, while fixed-size splitting provides predictable units but may cut across structure. Tables and identifier blocks should not be split in the middle of a record because the resulting chunks can lose headers, units, or exact identifiers.

Overlap can preserve context near a boundary, but it also creates duplicate-evidence cost that must be handled by the deduplication step in ## Combining Hybrid Retrieval Results. Stable chunk IDs and child-to-parent mappings are required for update propagation and parent-document aggregation. Boundary policy is an evaluated parameter rather than a constant. [AUTHOR: insert measured chunk-size/overlap results for this corpus]

Schema Discipline in the Graph

Graph extraction cannot be an unconstrained language-model task. If the same relationship is stored as SUPPLIER, SUPPLIES_TO, VENDOR_OF, and PROVIDES_FOR, graph queries become inconsistent.

A controlled schema should define:

• Supported entity types
• Canonical relationship names
• Required properties
• Direction of each relationship
• Valid source and target types
• Temporal properties
• Normalization rules for entity names

The extraction model should return structured output that conforms to this schema. Invalid relationship types should be rejected or routed to review. Entity resolution should map spelling variations and aliases to stable IDs rather than creating duplicate nodes.

Temporal information is also important. A supplier relationship may be valid only during a contract period. A product category may change. Graph edges should therefore support effective dates or document-version references when the domain requires historical answers.

Ingestion Lifecycle and Cross-Store Consistency

Creating several representations from one source creates a consistency problem. Production ingestion should therefore be idempotent: replaying the same document version with the same extraction configuration must not create duplicate chunks, invoice rows, entities, or edges. A content hash, stable source ID, document version, and deterministic child IDs allow the pipeline to recognize repeated work and make upserts safe.

Updates and deletions must propagate to every representation. Replacing an agreement should invalidate its old chunks, exact-match fields, embeddings, extracted entities, and graph edges before or atomically with publishing the new version. Deleting an invoice must remove or tombstone its SQL rows and prevent its chunks from remaining searchable. Source deletion is incomplete if an obsolete graph edge can still influence a relationship query.

Every derived record should carry version metadata such as:

{
  "document_id": "agreement-2026-04",
  "document_version": "7",
  "content_hash": "sha256:...",
  "parser_version": "parser-3.2",
  "extraction_model": "relation-extractor-5",
  "extraction_prompt_version": "12",
  "chunker_version": "layout-4",
  "embedding_model": "retrieval-model-a",
  "embedding_revision": "2026-05-18",
  "graph_schema_version": "3",
  "relational_schema_version": "9"
}

A new embedding configuration requires a migration rather than mixing incompatible vectors. The new representation can be backfilled into a parallel index, updated with new writes, evaluated, and activated through an atomic alias or routing change. Schema migrations need the same discipline: migrate stored records and query templates together, validate old and new reads, and preserve rollback until parity checks pass.

Failures should be replayable. Transient parsing, embedding, graph, or database errors can use bounded retries; repeatedly failing events belong in a dead-letter queue with source identity, stage, version, and error details. Backfill and replay jobs should consume the same idempotent handlers as live ingestion so operational recovery does not create a second behavior path.

Perfect distributed atomicity across vector, graph, and SQL stores is often impractical. A versioned publication protocol is a workable alternative: write derived artifacts under a non-serving version, verify expected counts and checksums, then mark that document version active. Readers use only active versions. Reconciliation jobs compare source manifests with each store, detect missing or stale representations, and repair them. Consistency becomes an observable invariant rather than an assumption.

Combining Graph Traversal with Structured Calculation

The earlier total-spend question can be executed as a coordinated workflow.

First, the graph identifies the vendors whose approval overlaps the requested period and the products connected to them. The example treats both the approval interval and requested period as half-open ranges: [valid_from, valid_to) and [start_date, end_date). A missing valid_to means that the approval remains open-ended.

MATCH (:Company {id: $company_id})-[approval:APPROVED_VENDOR]->(s:Supplier)
WHERE approval.valid_from < $end_date
  AND (approval.valid_to IS NULL OR approval.valid_to > $start_date)
MATCH (s)-[:SUPPLIES]->(p:Product)-[:BELONGS_TO]->(g:ProductGroup)
MATCH (g)-[:PART_OF*0..3]->(root:ProductGroup {name: $group_name})
RETURN s.id AS supplier_id, p.code AS product_code

The 0..3 depth bound is deliberate and follows the article’s rule that graph tools use bounded path lengths.

The overlap condition prevents a vendor approved today from being included in a historical quarter. The application should bind $start_date and $end_date as temporal values compatible with the stored properties; Cypher supports chronological comparison of temporal values of the same type [4].

The result is a set of normalized supplier IDs and product codes. Those identifiers become parameters for a structured query. In this example, validated_invoice_lines is a governed view or table that has already resolved duplicate source records and attached the approved reporting-currency rate:

SELECT
    COALESCE(
        SUM(
            CASE WHEN document_type = 'credit_note' THEN -1 ELSE 1 END
            * (line_net_amount + line_tax_amount)
            * fx_rate_to_reporting_currency
        ),
        0
    ) AS total_spend_reporting_currency,
    COUNT(*) AS matched_line_count
FROM validated_invoice_lines
WHERE company_id = :company_id
  AND supplier_id = ANY(:supplier_ids)
  AND product_code = ANY(:product_codes)
  AND invoice_status IN ('posted', 'paid')
  AND is_cancelled = FALSE
  AND is_duplicate = FALSE
  AND invoice_date >= :start_date
  AND invoice_date < :end_date;

Zero spend and no matching data are different answers. matched_line_count distinguishes an empty result from matched lines whose validated amounts sum to zero, allowing the partial-versus-complete policy to preserve that difference.

The familiar expression quantity * unit_price - discount_amount + tax_amount is valid only when those fields have precisely that accounting meaning. A discount may already be reflected in the unit price or net amount, tax may be stored at line or invoice level, and returns may appear as negative quantities or separate credit notes. Blindly applying the formula can double-subtract a discount or double-count tax.

Validated line_net_amount and line_tax_amount fields make the calculation contract clearer, but the surrounding accounting policy still matters. The workflow must define how returns and credit notes change sign, exclude cancelled or draft invoices, deduplicate replicated records, select the appropriate exchange rate and conversion date, and decide which invoice statuses count as recognized spend. Currency and calculation policy should be returned with the total so the result can be reproduced.

The stored fx_rate_to_reporting_currency assumes that a fixed rate type and conversion date were chosen at posting time and attached to each validated line. A system that must reprice at report time should join a governed rate table instead.

The graph resolves time-valid relationships; SQL performs the deterministic calculation. The language model should not mentally add invoice values or invent missing joins. Its role is to understand the request, select the workflow, and explain the validated result with references.

Routing, Planning, and Execution

A router analyzes a request and selects one or more retrieval intents. It may combine deterministic rules, a lightweight classifier, and a language model with structured output. Routing is followed by planning: the router decides what capabilities are required, while the planner turns those requirements into an ordered and authorized execution graph.

User query + authenticated context
                ↓
      Intent and constraint routing
                ↓
     Dependency-aware planning
                ↓
 Exact / BM25 / Vector / SQL / Graph execution
                ↓
 Completeness, policy, and calculation validation
                ↓
       Sourced response or clarification

A single intent field is insufficient for the procurement question because the request is simultaneously relational, aggregative, semantic, and potentially identity-sensitive. A richer contract can look like this:

{
  "intents": [
    {"type": "exact_lookup", "confidence": 0.86},
    {"type": "relationship", "confidence": 0.98},
    {"type": "aggregate", "confidence": 0.99},
    {"type": "semantic", "confidence": 0.74}
  ],
  "confidence": 0.93,
  "requires_clarification": false,
  "entities": {
    "company_name": "Company A",
    "product_group": "Monitoring Products"
  },
  "constraints": {
    "time_range": {"start": "2026-01-01", "end": "2026-04-01"},
    "invoice_status": ["posted", "paid"],
    "reporting_currency": "EUR"
  },
  "authorization_context": {
    "tenant_id": "tenant-42",
    "principal_id": "user-817",
    "scopes": ["procurement:read", "invoice:aggregate"]
  },
  "execution_plan": [
    {
      "step_id": "resolve-company",
      "tool": "entity.resolve_company",
      "depends_on": []
    },
    {
      "step_id": "find-approved-vendors",
      "tool": "graph.find_approved_vendors",
      "depends_on": ["resolve-company"]
    },
    {
      "step_id": "find-monitoring-products",
      "tool": "catalog.find_products",
      "depends_on": ["find-approved-vendors"]
    },
    {
      "step_id": "calculate-spend",
      "tool": "finance.aggregate_spend",
      "depends_on": ["find-approved-vendors", "find-monitoring-products"]
    },
    {
      "step_id": "retrieve-supporting-clauses",
      "tool": "documents.search",
      "depends_on": ["find-approved-vendors"]
    },
    {
      "step_id": "validate-result",
      "tool": "validation.check_completeness",
      "depends_on": ["calculate-spend", "retrieve-supporting-clauses"]
    }
  ]
}

The authorization context is not inferred by the model and should not be accepted from untrusted query text. The application injects it from the authenticated session, and every planned step is checked against it before execution.

The routing policy can follow several principles:

• Relationship language such as supplier, customer, dependency, or ownership may select graph traversal or a relational join.
• Product, invoice, contract, or canonical entity identifiers select exact lookup through keyword fields, term queries, or equality predicates.
• Rare terms and technical phrases select BM25 lexical retrieval.
• Numeric lookup or aggregation selects a governed SQL capability.
• Explanatory, comparative, or summary requests select vector or hybrid text retrieval.
• Requests combining categories preserve all required intents and produce a dependency-aware plan.

Confidence should be calibrated per intent rather than interpreted as a guarantee. If the period, currency, company identity, or accounting basis is ambiguous, requires_clarification should stop execution and produce a focused question. A direct identifier lookup may need one step; a request that discovers relationships, calculates a total, and retrieves explanatory evidence needs several. This distinction controls both cost and reliability.

When to Use an Agent

An agent is justified when the next action depends on an intermediate result. For example:

1. Graph traversal returns a list of approved suppliers.
2. SQL returns invoice totals for those suppliers.
3. Vector search retrieves contract clauses that explain exclusions.
4. A deterministic calculator produces the final comparison.
5. The language model assembles an answer with provenance.

The workflow should still be bounded. It needs a maximum number of steps, an allowed tool list, typed intermediate state, timeouts, and explicit failure behavior. Open-ended reasoning is not a substitute for workflow design.

Hybrid requests can also use predefined workflows instead of a general agent. If a high-volume question type always follows graph → SQL → vector, encoding that sequence directly is cheaper and easier to test.

Model Tiering

Routing can select not only tools but also models. Simple classification, rewriting, and direct lookup responses may run on a smaller model. Complex synthesis across graph, SQL, and text evidence may require a stronger model.

Model tiering should be based on route-specific acceptance thresholds, not on whether a smaller model reproduces every output of a larger model. For a routing task, the smaller model may be sufficient when it meets required intent recall, safety, confidence calibration, latency, and cost thresholds. For synthesis, the relevant thresholds may instead include faithfulness, citation correctness, numerical consistency, and policy compliance.

The thresholds should reflect risk. An exact product lookup can tolerate a different response style as long as identity and authorization remain correct. A financial aggregation route needs stricter completeness and numerical validation. Low confidence, out-of-distribution inputs, safety-sensitive requests, or failed validation can escalate to a stronger model. The cascade succeeds when each tier meets its service contract, not when all tiers produce identical prose.

Combining Hybrid Retrieval Results

Running exact, BM25, and vector retrieval in parallel does not by itself produce one coherent ranking. The system needs an explicit fusion and aggregation policy.

1. Apply metadata and authorization filters. Tenant, access scope, document status, language, product, and temporal constraints should restrict candidates. With pgvector, pre-filtering can cause the planner to skip the HNSW index, which preserves correctness but loses the approximate-search benefit. Post-filtering can instead leave too few surviving rows after the ANN scan. A strict tenant or authorization predicate can therefore reduce recall silently, coupling the security control with the retrieval-quality control. Available levers include indexing the filter column, raising ef_search, using hnsw.iterative_scan in pgvector 0.8.0, or falling back to exact search when the filtered set is small. Recall must be measured against exact search on filtered queries, not only on unfiltered queries [1]. A final policy check is still required after fusion.
2. Build the candidate union. Collect a sufficiently broad result set from each eligible retriever. Exact matches may enter with a deterministic identity flag, while BM25 and vector branches contribute ranked candidates and scores.
3. Remove duplicate chunks. Stable chunk IDs, content hashes, overlap relationships, and source spans prevent the same evidence from receiving artificial support because it appeared in several indexes or overlapping windows.
4. Fuse rankings. Reciprocal Rank Fusion combines rank positions without assuming that BM25 and vector scores share a scale [5]. Weighted RRF can emphasize a branch for a validated query class. Weighted score fusion is another option, but raw scores must first be normalized or calibrated; otherwise a numerically larger but incomparable score range dominates the result.
5. Rerank the reduced set. A cross-encoder can score the query and each candidate together, improving fine-grained relevance at higher request-time cost. Exact identity and trusted metadata boosts may remain deterministic features rather than being left entirely to the reranker.
6. Aggregate by parent document. Several strong child chunks from one agreement should not crowd out all other sources. Parent-document aggregation can combine child evidence, cap per-document contribution, and expand the selected child into its parent section when the answer model needs broader context.

A representative pipeline may union 40 vector candidates, 40 BM25 candidates, and any exact matches; deduplicate and fuse them; rerank the best 20; then select a small, diverse evidence set. These numbers are not defaults. Candidate depth, fusion weights, per-document caps, and reranker depth should be chosen on a query set that includes exact identifiers, technical language, semantic questions, and permission-sensitive cases. [AUTHOR: insert recall@k for exact+BM25+vector fusion vs vector-only on the golden set, and the tuning that produced these depths]

Provenance and Runtime Validation

A multi-source answer should expose how each conclusion was produced. A useful internal result object can separate evidence from calculation:

{
  "result": {
    "total_spend": "184250.75",
    "currency": "EUR"
  },
  "inputs": {
    "supplier_ids": ["SUP-18", "SUP-27"],
    "product_codes": ["PRD-482", "PRD-619"],
    "period": "2026-Q1"
  },
  "sources": [
    "agreement-2026-04-p12-c03",
    "invoice-2026-0148-line-04",
    "invoice-2026-0211-line-07"
  ]
}

Monetary values use exact decimal types end to end and are serialized as strings because binary floating point would silently break the determinism the architecture exists to guarantee.

Before the final answer is returned, deterministic checks should confirm that the displayed amount matches the tool result, every cited source exists, required units are present, and no unsupported entity entered the calculation.

If a tool returns no result, the system should preserve that uncertainty. It should not replace a missing price or relationship with a plausible value generated from model memory.

Security Across Retrieval and Tool Boundaries

Hybrid retrieval expands the security surface because one request can cross document, vector, graph, SQL, cache, and tool boundaries. Authorization must therefore be enforced by every data service rather than treated as a prompt instruction.

Tenant isolation can use separate databases, schemas, indexes, or namespaces when strong physical or operational separation is required. Shared stores need mandatory tenant predicates and policies that cannot be omitted by model-generated arguments. SQL routes should enforce row-level authorization; PostgreSQL row-security policies are one database-level mechanism for limiting which rows a user can read [6]. Graph routes need equivalent node- and edge-level checks so access to one agreement does not automatically expose every connected supplier or invoice.

Retrieved content is untrusted evidence. A contract, web page, or uploaded document can contain prompt-injection text that tells the model to ignore policy or call a tool. Tool output can carry the same attack into the next reasoning step. Retrieved documents and tool results should be clearly separated from trusted instructions, screened according to risk, and prevented from granting permissions or redefining the execution plan. OWASP explicitly treats retrieved content and tool output as indirect prompt-injection surfaces [7].

SQL access should be read-only for retrieval and analytical routes, with parameterized queries rather than model-constructed SQL strings [8]. Higher-risk systems can expose allowlisted query templates, stored procedures, or a governed semantic layer such as finance.aggregate_spend instead of a general SQL executor. Graph tools can similarly restrict relationship types, traversal depth, and returned properties.

Least-privilege tool permissions limit each capability to the data and operation it requires. The document search tool does not need invoice-write access; the aggregation tool does not need arbitrary network access. Rate limits, timeouts, row and result-size caps, and per-tenant quotas reduce both abuse and accidental fan-out. PII should be redacted or minimized before entering prompts and logs. Audit records should connect the authenticated principal, route, tool arguments, policy decision, source IDs, and returned result without copying unnecessary sensitive content.

Security validation continues after execution. The final context and citation set should be checked again for tenant, row, edge, and document authorization. No fusion score or graph path is allowed to override policy.

Evaluating the Router and Tools

Every routing category needs a golden query set, including requests with several valid intents. Evaluation should measure:

• Per-intent precision, recall, and F1
• Exact intent-set accuracy for multi-intent requests
• Confidence calibration and escalation quality
• Clarification precision and recall
• Entity extraction accuracy
• Constraint and authorization-context handling
• Execution-plan validity and dependency correctness
• Tool-call validity
• Retrieval recall within each tool
• Calculation consistency
• Source completeness
• End-to-end answer faithfulness
• Unauthorized retrieval and injection-resistance tests
• Latency and cost by route

[AUTHOR: insert golden-set size, construction method, and per-intent baseline scores]

Mixed queries deserve special attention because an apparently reasonable answer may omit one required branch. A response that summarizes a contract correctly but forgets the SQL aggregation is incomplete even if every sentence is true.

Tool traces should record every selected intent, confidence, clarification decision, extracted entity and constraint, plan dependency, arguments, returned rows or chunks, calculation outputs, policy checks, and final citations. These traces turn vague complaints about wrong answers into specific failures in routing, planning, retrieval, authorization, calculation, or generation.

Model tiers should be evaluated against separate route scorecards. A small router may pass when its multi-intent recall, calibration, safety, p95 latency, and cost remain within target thresholds even if its labels differ from a larger model on harmless edge cases. A synthesis model may need different thresholds for faithfulness, citation quality, and numerical consistency. Deployment decisions should compare each model with the service contract for its route, not require textual equivalence with the largest model.

Failure Boundaries in Multi-Tool Workflows

Combining several retrieval primitives creates new failure modes. A graph can return the correct suppliers while SQL fails to find matching invoice lines. Exact lookup can locate a product code while vector retrieval returns no explanatory passage. A robust workflow must distinguish partial evidence from a complete answer.

Every tool should have a typed request and response contract. The graph tool can return canonical entity IDs and source references. The SQL tool can return typed values, units, aggregation logic, and row provenance. The vector tool can return chunks, scores, and document metadata. The language model should not infer missing fields from another tool’s prose.

Tool execution should enforce several boundaries:

• Every request is validated against the tool’s typed input schema.
• Graph traversals use approved relationship types and bounded path lengths.
• Calculations use deterministic functions with explicit units and currencies.
• Every tool call has a timeout and a maximum result size.
• Returned entity IDs are validated before being passed to another system.
• Intermediate results retain source, schema, and extraction versions.

Partial failure needs an explicit policy. If the graph finds three suppliers but SQL has validated invoice data for only two, the system should not present the partial sum as the complete total. It can return a structured incomplete result:

{
  "status": "partial",
  "calculated_suppliers": ["SUP-18", "SUP-27"],
  "missing_suppliers": ["SUP-31"],
  "total_spend": "184250.75",
  "currency": "EUR"
}

The final response can explain the limitation or request review. Completeness is part of correctness.

Retries should be tool-specific. A temporary SQL timeout can be retried without repeating graph extraction. A graph result rejected by schema validation should not be retried unchanged; it needs corrected extraction or human review. Persisting validated intermediate results reduces cost and prevents successful steps from being repeated unnecessarily.

The security controls defined earlier remain active at every step. Partial execution never broadens tenant, row, edge, or document authorization.

These boundaries keep a hybrid architecture from becoming an opaque agent loop. The language model coordinates capabilities, while typed contracts, policies, and deterministic validators preserve system integrity.

Implementation Note: MCP, Embeddings, and HNSW Are Different Layers

Tool protocols, embedding models, application services, and vector indexes solve different problems. Keeping their boundaries explicit makes the retrieval path easier to secure and replace.

LLM / Application
        ↓
MCP Host
        ↓
MCP Client
        ↓
MCP Server Tool
        ↓
Application Service
        ↓
Database and indexes

The MCP host is the AI application component that manages client connections and permissions. An MCP client maintains the protocol connection to an MCP server. The server publishes tools; a selected tool validates its arguments and calls an application-owned service. That service performs exact lookup, BM25 search, embedding inference, vector search, graph traversal, or SQL. The official MCP architecture describes this host-client-server separation [9].

MCP does not create embeddings, build HNSW indexes, or execute vector similarity by itself. It standardizes tool discovery and invocation. MCP tools declare an inputSchema and may declare an outputSchema, allowing typed, structured arguments and results rather than only a free-text query [10]. A procurement search tool could accept:

{
  "query_text": "calibration requirements",
  "identifiers": {
    "product_codes": ["PRD-482"],
    "contract_ids": []
  },
  "retrieval_modes": ["exact", "bm25", "vector"],
  "filters": {
    "document_types": ["agreement", "product_manual"],
    "valid_at": "2026-03-31"
  },
  "limit": 20
}

The trusted tenant and principal context should come from an application- or host-managed authenticated session, not from model-supplied arguments. Trusted authorization context must never be bound to an MCP protocol-level session. The 2026-07-28 release candidate removes the protocol session and Mcp-Session-Id, carries protocol and client information in request _meta, and leaves application state to explicit handles passed as ordinary tool arguments [11]. The host-client-server separation described above holds across these revisions; only the session mechanics changed.

At the data layer, a pgvector table may look like:

CREATE TABLE document_chunks (
    id BIGSERIAL PRIMARY KEY,
    tenant_id TEXT NOT NULL,
    document_id TEXT NOT NULL,
    document_version TEXT NOT NULL,
    chunk_text TEXT NOT NULL,
    embedding VECTOR(1024),
    metadata JSONB NOT NULL
);

CREATE INDEX document_chunks_tenant_document
ON document_chunks (tenant_id, document_id);

CREATE INDEX document_chunks_embedding_hnsw
ON document_chunks
USING hnsw (embedding vector_cosine_ops);

ALTER TABLE document_chunks ENABLE ROW LEVEL SECURITY;
ALTER TABLE document_chunks FORCE ROW LEVEL SECURITY;

CREATE POLICY document_chunks_tenant_isolation ON document_chunks
USING (tenant_id = current_setting('app.tenant_id', true));

The tenant column, composite index, and row-level security policy are part of the retrieval schema rather than a later add-on [6]. The policy also creates the hard-filter and approximate-index interaction described in the fusion section, so recall must be evaluated on tenant-filtered queries.

The application service creates a compatible query representation and sends it to PostgreSQL; PostgreSQL uses the HNSW index to retrieve approximate neighbors. pgvector documents exact search as the default and HNSW as an optional approximate index that trades some recall for speed [1]. These database mechanics remain the same whether the service is called through MCP, HTTP, a queue, or an internal function.

Query and document embeddings do not always need to come from one identical encoder. They must come from the same encoder or from compatible query and document encoders trained to produce the same latent space. Asymmetric retrieval models may intentionally use different query/document prompts or towers; Sentence Transformers, for example, exposes separate query and document encoding paths for asymmetric semantic search [12].

In the common single-encoder design, compatibility should be enforced by versioning and matching the model name, model revision, vector dimension, normalization policy, query/document prompt or instruction configuration, and similarity function used at indexing and query time. A mismatch can silently make stored vectors incomparable even when the API types still look valid.

The Long-Context and Agentic-Search Objection

The strongest alternative is to avoid most of this retrieval infrastructure. A long-context model paired with an agent that searches files directly can read source material on demand without a precomputed ingestion pipeline, synchronized vector and graph representations, or a cross-store schema. The agent can choose files, inspect their contents, and assemble an answer within one flexible workflow.

That approach can be the better design for small or slow-moving corpora, exploratory questions, low query volume, and prototypes. It is also reasonable when per-query cost and latency are not controlling constraints and when the questions do not require durable structured representations. In those cases, the operational simplicity can outweigh the benefits of a multi-store retrieval architecture.

It does not remove the requirements at the center of the procurement example. A deterministic aggregate is still not a retrieval result, authorization must still be enforced by the data service rather than by the files an agent chose to read, and a reported number still needs reproducible provenance. At volume, per-query processing cost remains an architectural concern, and multi-tenant isolation cannot depend on the model selecting the correct files. [AUTHOR: insert per-query cost comparison]

Choosing the Correct Representation

Vector search remains a core component of knowledge systems, but it should be used for the problem it solves: semantic retrieval over unstructured content. Exact identity, lexical relevance, relationships, and calculations deserve representations designed for them.

A reliable architecture combines exact-match retrieval, BM25, vector search, governed SQL, and, when relationship complexity justifies it, a knowledge graph behind an explicit routing and planning policy. It uses agents only when intermediate results determine subsequent actions. It keeps calculations deterministic, preserves provenance across representations, and exposes every tool as a measurable component.

The result is not merely a larger retrieval stack. It is a system in which the structure of the question determines the structure of the answer path.

References

1. pgvector. “Open-source vector similarity search for Postgres.” Official documentation.
2. Elastic. “Term query.” Elasticsearch Reference.
3. Elastic. “Similarity settings: BM25 similarity.” Elasticsearch Reference.
4. Neo4j. “Equality, ordering, and comparison of value types.” Cypher Manual.
5. Cormack, G. V., Clarke, C. L. A., and Büttcher, S. (2009). “Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods.” SIGIR 2009.
6. PostgreSQL Global Development Group. “Row Security Policies.” PostgreSQL Documentation.
7. OWASP Cheat Sheet Series. “LLM Prompt Injection Prevention.”
8. OWASP Cheat Sheet Series. “SQL Injection Prevention.”
9. Model Context Protocol. “Architecture.” Specification.
10. Model Context Protocol. “Tools.” Specification.
11. Model Context Protocol. “The 2026-07-28 MCP Specification Release Candidate.”
12. Sentence Transformers. “Semantic Search.” Documentation.