Show HN: A minimalist proxy for your local LLM cluster (~1100 lines) 展示 HN：一个用于本地 LLM 集群的极简代理（约 1100 行）

TL;DR

• smol-llm-proxy is a lightweight, self-hosted API proxy designed to route requests across multiple llama-server instances with per-user token accounting and rate limiting.
• It supports in-memory caching (30s TTL), async SQLite-based usage logging, sliding window RPM/TPM rate limiting, and connection-pooled backend forwarding—all within ~53 MB RAM and under 1100 lines of code.
• The proxy enables multi-model routing via aliases, admin-managed user keys, and external TLS termination, making it ideal for small teams sharing local LLM resources without heavy infrastructure dependencies.

Why It Matters

This project addresses a critical gap in the open-source LLM deployment ecosystem: providing a minimal, efficient, and self-contained gateway for managing shared access to locally hosted llama.cpp servers. For researchers and practitioners running multiple models on limited hardware, smol-llm-proxy offers a low-overhead solution that balances functionality (token tracking, rate limiting, routing) with extreme resource efficiency—making it suitable for edge cases where full-scale gateways like LiteLLM are overkill or impractical.

Technical Details

• Architecture: Built as a FastAPI/Uvicorn service with async I/O, using SQLite as its sole persistent store (no Redis/Postgres required). All state (users, routes, rate limits) is loaded into memory at startup with periodic async flushes to disk every second.
• Routing & Caching: Routes requests based on model name or alias using an in-memory cache refreshed every 30 seconds. Supports model-to-GGUF file mapping via config.yaml or Admin API.
• Rate Limiting: Implements sliding window counters for RPM (requests per minute) and TPM (tokens per minute) per API key, enforced via in-memory reservations reconciled against SQLite after each request. Returns HTTP 429 + Retry-Header on quota exhaustion.
• Usage Tracking: Logs prompt/completion tokens and timing data asynchronously in batches (max 50 items or 1-second interval), retaining records for 90 days before daily purging.
• Deployment Options: Available via Docker Compose, standalone binary, or Python package (pip install). Listens on 0.0.0.0:8000 by default; TLS handled externally (e.g., nginx, Cloudflare).

Industry Insight

As organizations increasingly adopt self-hosted LLMs for cost control and privacy, there’s growing demand for lightweight, modular components that can orchestrate inference workloads without introducing operational bloat. smol-llm-proxy exemplifies this trend—a focused tool that solves one problem exceptionally well compared to monolithic alternatives. Its design suggests a future where specialized, composable proxies become standard building blocks in local AI infrastructures, enabling fine-grained access control and observability even in resource-constrained environments. Teams managing heterogeneous model deployments should consider such tools before reaching for enterprise-grade gateways unless they require broader cloud-provider integration or complex billing systems.

TL;DR

• smol-llm-proxy 是一个轻量级 API 代理，专为自托管 llama.cpp 环境设计，支持多实例路由、用户级密钥管理与 token 使用追踪。
• 采用 SQLite 后端实现零外部依赖，内存缓存 + 异步批处理刷新机制保障低延迟（~0.13ms 代理逻辑）与高并发性能。
• 提供 per-user RPM/TPM 限流、模型别名映射、连接池转发及 90 天日志保留策略，适合小型团队内部共享推理资源。
• 对比 LiteLLM 和 llama-swap，其定位更聚焦于本地多 server + 多用户 + 记账场景，代码量 ≤1100 行，内存占用约 53 MB。
• 支持 Docker、pip、源码等多种部署方式，通过 Admin API 动态管理服务器、别名与密钥，配置变更非破坏性同步。

为什么值得看

对于希望在小规模团队中安全、可追踪地共享多个 llama.cpp 推理服务的开发者而言，smol-llm-proxy 提供了一个极简但功能完备的中间层方案，避免了引入复杂数据库或云网关的成本。它填补了“本地多模型分发”与“企业级 LLM 网关”之间的空白，尤其适用于边缘计算、私有化部署或实验性项目中的资源调度与成本控制。

技术解析

• 架构核心：作为 HTTP 代理层，接收来自用户的请求，根据 API key 验证身份、查询路由表（基于 model name 或 alias），将请求转发至对应的 llama-server 实例，并记录 token 用量与响应时间。
• 速率限制机制：每用户 key 独立维护 RPM（每分钟请求数）与 TPM（每分钟 tokens 数），采用滑动窗口算法，在内存中做快速预留判断，每秒异步批量写入 SQLite 以持久化状态，兼顾性能与准确性。
• 数据持久化与缓存：所有配置（servers, aliases）、密钥、用量统计均存入 SQLite；关键路由信息缓存在内存中 TTL=30s，减少 DB 读取压力；启动时 config.yaml 会 merge 进 DB，但不删除已有条目，确保配置增量更新。
• 通信优化：使用连接池客户端保持到 backend llama-server 的 keepalive 连接，降低握手开销；支持 streaming / non-streaming chat/completion/embeddings 模式，兼容标准 OpenAI API 格式。
• 部署灵活性：可通过 Docker Compose、pip install 或直接运行 Python 模块启动，默认监听 0.0.0.0:8000，TLS 卸载由前端反向代理（如 Nginx/Caddy）处理，符合现代微服务安全实践。

行业启示

• 随着本地大模型部署日益普及，对“轻量级、可审计、易集成”的代理层需求上升，smol-llm-proxy 展示了如何在最小化运维负担的前提下实现基本的多租户隔离与用量管控，值得中小团队参考借鉴。
• 在私有化场景中，结合 SQLite 而非 PostgreSQL/Redis 可显著简化基础设施，同时满足大多数非高并发场景下的数据一致性与恢复能力要求，体现“够用即好”的工程哲学。
• 未来若扩展支持 Web UI、配额预警、自动熔断等功能，该工具有望成为开源社区中面向个人开发者与小企业的标准 LLM 接入网关之一，推动本地化 AI 应用的规模化落地。