Show HN: I stopped using an LLM gateway and put rate-limits/fallback in-process
vernLLM is a TypeScript LLM call framework providing resilience, observability, and control for AI API interactions Offers a unified interface across 12+ providers including OpenAI, Anthropic, Gemini, AWS Bedrock, Groq, Mistral, DeepSeek, Cerebras, Together AI, Fireworks AI, and Ollama Built-in reliability features include automatic retries, circuit breaking, fallback chains, rate limiting, request caching, and middleware support All operations run within the caller's own process without introdu
Analysis
TL;DR
- vernLLM is a TypeScript LLM call framework providing resilience, observability, and control for AI API interactions
- Offers a unified interface across 12+ providers including OpenAI, Anthropic, Gemini, AWS Bedrock, Groq, Mistral, DeepSeek, Cerebras, Together AI, Fireworks AI, and Ollama
- Built-in reliability features include automatic retries, circuit breaking, fallback chains, rate limiting, request caching, and middleware support
- All operations run within the caller's own process without introducing additional network hops
- Structured as a pnpm monorepo with the core package at packages/vern-llm and Fumadocs-powered documentation
Why It Matters
vernLLM addresses a critical pain point for AI practitioners: managing the operational complexity of integrating multiple LLM providers while maintaining reliability and observability. As applications increasingly rely on LLM calls as core infrastructure, this framework provides production-grade patterns—circuit breaking, fallbacks, rate limiting—without requiring developers to build these systems from scratch.
Technical Details
- Unified Client Abstraction: Wraps OpenAI-compatible SDKs and provides
fromFetchadapter for any HTTP-reachable provider, enabling a singleVernLLMinterface across all supported LLM services - Resilience Patterns: Implements circuit breaker with configurable thresholds, automatic retry with budget-based rate control (
retryBudgetwith window, min calls, and retry ratio), per-client fallback chains, and configurable timeouts - Rate Limiting & Concurrency: Supports granular rate limiting by requests per minute, tokens per minute, and maximum concurrent connections to prevent provider quota exhaustion
- Configuration-Driven Behavior: Key parameters include
maxRetries,timeoutMs,defaultMaxTokens,defaultReasoningEffort, and structured fallback definitions with optional circuit breaker activation per fallback client - Monorepo Architecture: Built with pnpm workspaces, TypeScript, Oxlint/Oxformatter for code quality, Codecov for test coverage, and Renovate for dependency management
Industry Insight
- The trend toward multi-provider LLM strategies is accelerating, and frameworks like vernLLM that abstract away provider-specific complexity will become essential infrastructure for production AI applications
- Running all logic within the caller's process (vs. a proxy layer) reduces latency and cost—a design choice that aligns with the growing demand for efficient, low-overhead AI integration patterns
- The emphasis on circuit breaking and fallback chains signals that LLM reliability is maturing from an experimental concern to a production-critical requirement, and tools enabling graceful degradation will see strong adoption
Disclaimer: The above content is generated by AI and is for reference only.