How to Build a Context Layer and a Company Brain 如何构建上下文层和公司大脑

TL;DR

• A "context layer" is required to bridge the gap between simple RAG demos and production systems, acting as a continuous mapping and indexing engine rather than just a vector database.
• Ingestion must be a non-terminating reconciliation loop using declarative data maps (datamaps) to handle schema changes, deletions, and identity resolution across diverse sources like SQL warehouses and wikis.
• Production indexing requires a multi-layered architecture comprising a relational ground truth for ACL enforcement, a keyword index for lexical recall, separate vector indexes for natural language vs. code, and a knowledge graph for structural lineage.
• Multi-tenancy demands physical isolation (per-tenant schemas and indices) to prevent security failures, with search indexes serving only as permissive prefilters before final permission checks in the database.

Why It Matters

This article provides critical architectural guidance for AI practitioners moving beyond prototype RAG systems into enterprise deployment. It highlights that the primary challenge in building a "company brain" is not the LLM itself but the distributed systems engineering required to maintain fresh, secure, and semantically accurate context from heterogeneous data sources at scale. Understanding the distinction between ingestion loops, indexing strategies, and tenancy models is essential for building reliable, production-grade AI applications.

Technical Details

• Continuous Ingestion Loop: The system treats ingestion as an endless reconciliation process rather than a batch job, utilizing a durable workflow orchestrator to manage coverage queries, refresh cadences, and deletion handling across six distinct channels (e.g., source confirmation, parent manifest drop, cursor expiration).
• Declarative Datamaps: Instead of writing custom sync scripts for each source, the system describes data hierarchies (e.g., connection → database → schema → table) as typed data structures. This allows a single mining engine to serve multiple vendors idempotently, deriving item identities from (tenant, type, path) to survive crashes and retries.
• Multi-Index Architecture: A robust context layer employs four distinct indexes: 1) Relational ground truth (Postgres) for authoritative writes and ACL enforcement; 2) Keyword index (BM25) for exact schema name matching; 3) Vector index split into two collections (natural language vs. code/SQL) using different embedding models; and 4) Knowledge graph for structural relationships like table joins and dashboard lineage.
• Embedding Pipelines as ETL: Enrichment occurs before embedding via watermark-based syncs; columns requiring semantic descriptions are processed by LLMs prior to vectorization. Projections lag behind the truth but require explicit cleanup phases and rebuild runbooks to ensure consistency.
• Physical Tenancy Isolation: To avoid security risks from missing filters, the system enforces physical separation per tenant (separate schemas, indices, and collections) rather than logical filtering, treating search indexes as caches that must be reconciled against the relational store's access controls before content reaches the prompt.

Industry Insight

Enterprise AI teams should prioritize building a robust data orchestration layer over simply connecting an LLM to a vector store; the reliability of the "company brain" depends on how well it handles deletion events, identity resolution, and schema evolution. Adopting a multi-index strategy where specialized tools handle specific retrieval needs (e.g., knowledge graphs for lineage, BM25 for schema names) will yield higher precision than relying solely on semantic similarity searches. Furthermore, implementing physical multi-tenancy isolation, while operationally heavier, is a necessary cost to guarantee security compliance and prevent silent data leakage in shared environments.

TL;DR

• 企业级AI应用的核心痛点在于“公司大脑”的构建，即如何让模型掌握企业内部的所有数据（仓库模式、文档、Slack对话等）。
• 简单的周末项目（分块、嵌入、检索）在生产环境中会失效，因为忽略了持续映射、多索引、权限控制等分布式系统难题。
• “上下文层”不仅仅是向量数据库，而是一个包含连续映射、多类型索引、身份解析和十租户隔离的复杂系统。
• 数据摄取不是批处理任务，而是需要处理删除、过期、断连等边缘情况的持续同步循环。
• 单一索引无法满足需求，必须结合关系型事实源、关键词索引、向量和知识图谱，并采用物理隔离的多租户架构。

为什么值得看

这篇文章深刻揭示了从概念验证（PoC）到生产级企业AI应用之间的巨大鸿沟，为从业者提供了构建可靠“公司大脑”的系统性方法论。它强调了在真实场景中处理数据新鲜度、权限控制和多源异构集成的必要性，是指导企业落地LLM应用的必读指南。

技术解析

• 连续映射与摄取：将数据摄取定义为永不终止的重合循环，而非一次性批处理。通过声明式datamap描述源系统的层级结构（如仓库的连接->库->表->列），利用覆盖查询检测缺失或陈旧数据，并配合耐用工作流编排器处理去重、扇出限制和故障恢复。特别指出删除逻辑是最难点，需区分有界集合和无界流的不同删除语义。
• 多模态索引架构：摒弃单一索引方案，建立以Postgres为中心的事实源（Ground Truth），其上构建可重建的投影层。包括BM25关键词索引用于精确匹配（如表名）、分离的向量索引分别处理自然语言和代码内容、以及用于追踪依赖关系的知识图谱（如仪表盘到表的血缘）。
• 身份解析与权限安全：强调在摄取前必须先完成身份解析（Identity Resolution），将外部平台ID映射到内部主体，否则会导致权限泄露或隐藏。主张多租户应采用物理隔离（独立的Schema、Index、Collection），ACL作为最后一道防线作用于关系型存储，而非仅依赖搜索索引的预过滤。
• ETL化的嵌入管道：将Embedding视为复杂的ETL过程而非简单调用。在生成向量前需先进行LLM增强（如自动生成列描述），并通过基于水印的同步机制保证投影相对于事实源的滞后可控，且需具备明确的清理和重跑剧本。

行业启示

• 从“工具思维”转向“系统工程”：企业不应仅关注模型能力或简单的RAG实现，而应投入资源构建具备容错性、可观测性和持续运维能力的底层基础设施（Context Layer），这是决定AI应用生死的关键。
• 数据治理是AI落地的前提：在引入LLM之前，必须先解决数据的标准化、权限管理和生命周期问题。缺乏治理的“野蛮生长”会导致模型产生幻觉或泄露敏感信息，甚至因信任崩塌而被废弃。
• 成本与隔离的权衡：虽然物理多租户增加了运维开销（如资源供应、孤儿监控），但相比逻辑隔离带来的安全风险和调试困难，这种显性的成本是更优解。未来AI平台的设计应将安全性和审计性置于性能优化之上。