FreeToken Just Shipped. Is It Better Than Ollama and llama.cpp?
FreeToken is a new edge-native MoE serving engine that uses a dynamic bandwidth-adaptive caching strategy, keeping all expert weights in CPU RAM and using GPU VRAM as a shared LRU cache for recently used experts In benchmarks on an RTX 3050 laptop (6GB VRAM, 24GB RAM) running gpt-oss-20b MXFP4, Ollama was fastest at ~0.38s TTFT, llama.cpp at ~0.5s, and FreeToken lagged at ~1.17s TTFT and lower throughput llama.cpp and Ollama use a static CPU-GPU split at load time, while FreeToken dynamically de
Analysis
TL;DR
- FreeToken is a new edge-native MoE serving engine that uses a dynamic bandwidth-adaptive caching strategy, keeping all expert weights in CPU RAM and using GPU VRAM as a shared LRU cache for recently used experts
- In benchmarks on an RTX 3050 laptop (6GB VRAM, 24GB RAM) running gpt-oss-20b MXFP4, Ollama was fastest at ~0.38s TTFT, llama.cpp at ~0.5s, and FreeToken lagged at ~1.17s TTFT and lower throughput
- llama.cpp and Ollama use a static CPU-GPU split at load time, while FreeToken dynamically decides per-expert whether to fetch over PCIe or compute on CPU based on measured bandwidth
- The author concludes FreeToken's overhead is unjustified for small static workloads on constrained hardware, but its design may pay off on larger models and agent workloads with changing execution patterns
- FreeToken's own model list showed 35B and 27B models as "Insufficient RAM" on the test machine, limiting the ability to validate its headline 753B claim
Why It Matters
This comparison provides a practical reality check on FreeToken's bold claims by testing it against established engines on real consumer hardware, revealing that architectural complexity doesn't always translate to performance gains at every scale. For AI practitioners, it highlights the importance of matching serving engine design to workload characteristics—small static inference favors simple static splits, while dynamic caching may shine in larger, more variable agent scenarios.
Technical Details
- FreeToken's core innovation is a bandwidth-adaptive execution policy: it maintains all expert weights in CPU RAM as the source of truth, uses GPU VRAM as a shared LRU cache for recently accessed experts, and dynamically decides on cache misses whether to transfer weights over PCIe or compute directly on the CPU based on real-time bandwidth measurements
- llama.cpp and Ollama both use a static split approach where model layers are divided between GPU and CPU at load time and remain fixed during inference; Ollama auto-selects the split based on hardware detection, while llama.cpp requires manual configuration via flags
- Benchmark hardware: RTX 3050 Laptop GPU (6GB VRAM), Intel i5-13450HX (6P+4E cores), 24GB RAM, Windows 11; model: gpt-oss-20b in MXFP4 quantization (~13GB on disk)
- Three evaluation prompts tested: short factual explanation, logical reasoning trick question, and long-form prose generation; each run four times with median results reported
- FreeToken requires more system RAM than the test machine could provide for larger models, and its desktop app performed better than its CLI version under WSL due to memory sharing limitations
Industry Insight
- Dynamic expert caching and bandwidth-adaptive routing are promising directions for edge MoE serving, but the overhead of runtime decision-making may not be justified until model sizes and workload variability exceed a certain threshold—practitioners should evaluate whether their use case actually benefits from this complexity
- The static split approach (llama.cpp/Ollama) remains highly competitive for small-to-medium models on constrained hardware, and the performance gap between manual and automatic configuration is modest enough that ease of use may outweigh marginal throughput gains
- FreeToken's true test will come on workstations with sufficient RAM to load models that cannot fit entirely on GPU; until then, its headline 753B claim remains unvalidated in independent benchmarks, and practitioners should wait for real-world data at that scale before adopting
Disclaimer: The above content is generated by AI and is for reference only.