Show HN: Made a Free LLM Router
FlexInference is a routing layer that reduces LLM inference costs by up to 48% by leveraging "flex tier" availability without altering model outputs or configurations. The system employs a "flex race" mechanism where requests are attempted on cheaper infrastructure first, escalating to standard tiers only if latency budgets are missed, with zero charge for failed flex attempts. Implementation utilizes Cloudflare Workers for edge routing with minimal overhead (1-5ms) and supports OpenAI, Anthropi
Analysis
TL;DR
- FlexInference is a routing layer that reduces LLM inference costs by up to 48% by leveraging "flex tier" availability without altering model outputs or configurations.
- The system employs a "flex race" mechanism where requests are attempted on cheaper infrastructure first, escalating to standard tiers only if latency budgets are missed, with zero charge for failed flex attempts.
- Implementation utilizes Cloudflare Workers for edge routing with minimal overhead (1-5ms) and supports OpenAI, Anthropic, and Gemini via a simple
start_withinparameter addition. - Security and privacy are maintained through envelope encryption of API keys (AES-256-GCM) and a strict no-storage policy for prompts or responses.
- The service offers agent-friendly error handling with machine-readable codes and fixes, enabling automated debugging without manual intervention.
Why It Matters
This solution addresses the critical pain point of high inference costs in production AI applications by introducing a transparent, low-friction optimization layer that preserves existing client integrations. For developers and enterprises, it demonstrates how intelligent routing and tier arbitrage can significantly improve unit economics without sacrificing reliability or requiring complex architectural changes.
Technical Details
- Routing Architecture: Uses Cloudflare Workers deployed across 300+ cities to route requests, adding only 1-5ms latency on cold starts.
- Cost Optimization Logic: Implements a "flex race" strategy; attempts to fulfill requests via cheaper flex tiers within a user-defined time budget (
start_within). If the flex tier cannot start in time, it escalates to the standard tier. - Security Model: API keys are envelope-encrypted with AES-256-GCM, locked to specific organizations and provider slots, ensuring keys cannot be decrypted outside their intended context. Prompts and replies are passed through without storage or reading.
- Integration Compatibility: Maintains backward compatibility with existing OpenAI, Anthropic, and Gemini clients by requiring only a new
start_withinfield in the request payload, supporting both Python and TypeScript SDKs with strict typing. - Error Handling: Returns standardized error shapes compatible with original SDKs, including machine-readable codes, exact fixes, and documentation URLs to facilitate self-healing agents.
Industry Insight
- Adoption of Middleware Layers: The rise of specialized routing and optimization middleware indicates a market shift towards decoupling inference logic from provider direct access, allowing for greater cost control and flexibility.
- Importance of Latency Budgets: Defining explicit time budgets for inference allows businesses to trade off slight latency increases for substantial cost savings, a viable strategy for non-real-time applications.
- Agent-Centric Design: Features like auto-fixable errors and strict typing suggest that future AI infrastructure must prioritize interoperability with autonomous agents, reducing the operational burden on human developers.
Disclaimer: The above content is generated by AI and is for reference only.