How Couchbase built a multi-model AI architecture for Capella iQ with Amazon Bedrock
Couchbase built a multi-model AI architecture for Capella iQ using Amazon Bedrock to support Anthropic’s Claude models, enabling flexible and resilient inference. The solution utilizes an Amazon EKS cluster with specific microservices (cp-api, cp-internal-api, cp-ns) and an Amazon VPC interface endpoint for secure, private connectivity to Bedrock. Cross-Region Inference (CRIS) is leveraged to automatically route traffic across US regions (us-east-1, us-east-2, us-west-2) for high availability an
Analysis
TL;DR
- Couchbase built a multi-model AI architecture for Capella iQ using Amazon Bedrock to support Anthropic’s Claude models, enabling flexible and resilient inference.
- The solution utilizes an Amazon EKS cluster with specific microservices (cp-api, cp-internal-api, cp-ns) and an Amazon VPC interface endpoint for secure, private connectivity to Bedrock.
- Cross-Region Inference (CRIS) is leveraged to automatically route traffic across US regions (us-east-1, us-east-2, us-west-2) for high availability and load distribution without manual intervention.
- Model evaluation followed a BIRD-modeled benchmark suite focusing on functional correctness, determinism, latency, and formatting, with Claude Sonnet 4.5 achieving ~76% accuracy.
- The architecture allows for model upgrades or provider changes via configuration updates at the namespace layer, eliminating the need for code changes or downtime.
Why It Matters
This case study demonstrates a practical, enterprise-grade approach to integrating generative AI into existing software products, specifically highlighting the importance of a model-agnostic inference layer. It provides a blueprint for other organizations seeking to balance rapid AI adoption with operational resilience, security, and cost-efficiency by leveraging managed services like Amazon Bedrock.
Technical Details
- Architecture Components: The system runs on Amazon EKS in us-east-1 and us-west-2. Key pods include
cp-api(request ingestion and orchestration),cp-internal-api(model routing logic), andcp-ns(namespace-level configuration and tenant overrides). - Connectivity & Security: Private connectivity is established via an Amazon VPC interface endpoint, ensuring inference traffic never traverses the public internet. This satisfies strict enterprise data residency and security requirements.
- Resilience Mechanism: Amazon Bedrock’s Cross-Region Inference (CRIS) automatically distributes load and handles failover across us-east-1, us-east-2, and us-west-2, providing seamless scalability during traffic bursts.
- Evaluation Framework: Models were tested against standardized prompt/response sets for workflows like SQL++ generation and index recommendations. Scoring metrics included functional correctness, determinism, latency, and formatting consistency.
- Model Selection: Anthropic’s Claude Sonnet 4.5 was selected as the initial production model after demonstrating ~76% accuracy on internal benchmarks, validating the chosen evaluation methodology.
Industry Insight
- Abstraction is Key: Building a model-agnostic abstraction layer (like the
cp-internal-api) allows organizations to swap underlying foundation models without refactoring core application logic, future-proofing AI investments. - Managed Services Reduce Overhead: Leveraging fully managed inference platforms like Amazon Bedrock eliminates the operational burden of managing GPU infrastructure, allowing teams to focus on application value rather than ML ops.
- Security by Design: Integrating VPC endpoints for AI inference is a critical best practice for enterprise adoption, ensuring that sensitive developer data and prompts remain within the private network boundary.
Disclaimer: The above content is generated by AI and is for reference only.