Liquid AI Releases LFM2.5-Encoder-230M and LFM2.5-Encoder-350M: Bidirectional Encoders That Stay Fast at 8K Context on CPU
Liquid AI released two open-weight bidirectional encoders, LFM2.5-Encoder-230M and LFM2.5-Encoder-350M, built on the LFM2 hybrid backbone with an 8,192-token context window. The models are converted from decoder backbones by replacing causal attention with bidirectional masking, making convolutions non-causal, and training with a 30% masked language modeling objective. LFM2.5-Encoder-350M ranks fourth among 14 models on a 17-task benchmark (mean score 81.02), outperforming larger models like XLM
Analysis
TL;DR
- Liquid AI released two open-weight bidirectional encoders, LFM2.5-Encoder-230M and LFM2.5-Encoder-350M, built on the LFM2 hybrid backbone with an 8,192-token context window.
- The models are converted from decoder backbones by replacing causal attention with bidirectional masking, making convolutions non-causal, and training with a 30% masked language modeling objective.
- LFM2.5-Encoder-350M ranks fourth among 14 models on a 17-task benchmark (mean score 81.02), outperforming larger models like XLM-R XL and ModernBERT-large in efficiency-to-performance ratio.
- Both encoders are optimized for CPU-only deployment, achieving significantly faster inference than competitors—e.g., ~28s vs. >90s for ModernBERT-base at 8K tokens on CPU.
- Designed for edge devices, regulated industries (finance, healthcare, legal), and high-volume pipelines requiring low-latency, long-context encoding without GPU reliance.
Why It Matters
These encoders address a critical gap in production AI systems: efficient, accurate, long-context processing on CPUs where GPUs are unavailable or cost-prohibitive. Their performance relative to size makes them ideal for real-time applications such as document analysis, compliance filtering, and intent routing in constrained environments. As organizations increasingly deploy NLP models at scale under strict latency and hardware budgets, Liquid AI’s approach offers a practical path forward without sacrificing accuracy or context length.
Technical Details
- Architecture: Based on LFM2 hybrid backbone combining gated short-convolution blocks with grouped-query attention; hidden size = 1024, vocabulary size = 65,536, supports 15 languages.
- Conversion Method: Transformed from LFM2.5 decoders via three key changes: (1) causal → bidirectional attention mask, (2) short convolutions made non-causal using symmetric center padding, (3) trained with 30% MLM rate (vs. BERT’s 15%).
- Training Pipeline: Two-stage process—first stage uses 1K-token context on large web corpus for general competence; second stage extends to 8K tokens across full data mix to enhance factual, legal, and multilingual understanding.
- Benchmarking: Evaluated on 17 tasks from GLUE, SuperGLUE, and multilingual classification; all models fine-tuned per task using identical AdamW recipe from EuroBERT card, with learning rates selected across 10 values over 3 seeds, final scores averaged over 5 fresh seeds.
- Deployment Optimization: Designed for CPU inference; demonstrated ~28s forward pass at 8K tokens for 230M model vs. >90s for ModernBERT-base; supports Hugging Face
transformersintegration withtrust_remote_code=True.
Industry Insight
The release signals a growing industry shift toward lightweight, long-context encoders tailored for edge and regulated deployments where cloud dependency is unacceptable or too costly. Companies building safety filters, PII detectors, or document classifiers can now adopt models that balance speed, context length, and accuracy without requiring expensive GPU infrastructure. Additionally, the open-sourced evaluation methodology sets a new standard for fair, reproducible benchmarking of encoder architectures—encouraging transparency and accelerating innovation in efficient NLP components.
Disclaimer: The above content is generated by AI and is for reference only.