Mistral AI Releases Leanstral 1.5: An Apache-2.0 Lean 4 Code Agent Model Solving 587 of 672 PutnamBench Problems
Mistral AI released Leanstral 1.5, an open-weight (Apache 2.0) code agent model based on the Mistral Small 4 family, specifically designed for Lean 4 automated theorem proving. The model utilizes a Mixture-of-Experts (MoE) architecture with 119B total parameters and 6.5B active parameters per token, supporting a 256k context window and multimodal input. It achieves state-of-the-art performance, saturating miniF2F (100%) and significantly outperforming larger open-source models and proprietary co
Analysis
TL;DR
- Mistral AI released Leanstral 1.5, an open-weight (Apache 2.0) code agent model based on the Mistral Small 4 family, specifically designed for Lean 4 automated theorem proving.
- The model utilizes a Mixture-of-Experts (MoE) architecture with 119B total parameters and 6.5B active parameters per token, supporting a 256k context window and multimodal input.
- It achieves state-of-the-art performance, saturating miniF2F (100%) and significantly outperforming larger open-source models and proprietary competitors like Opus 4.6 on FLTEval benchmarks.
- Training involves a three-stage process including Reinforcement Learning with CISPO, utilizing multiturn and code agent environments to refine proofs through compiler feedback and filesystem interaction.
- Beyond pure mathematics, the model demonstrates practical utility in software engineering by verifying code complexity (e.g., AVL trees) and detecting genuine bugs in open-source Rust repositories.
Why It Matters
This release marks a significant milestone in the intersection of formal verification and large language models, demonstrating that efficient, open-weight models can rival or exceed expensive proprietary systems in complex reasoning tasks. For AI practitioners and researchers, it provides a highly accessible, cost-effective tool for automated theorem proving and formal method applications, lowering the barrier to entry for integrating LLMs into critical software verification pipelines.
Technical Details
- Architecture: Mixture-of-Experts (MoE) with 128 experts, activating 4 per token. Total size is 119B parameters, with 6.5B activated. It supports 256k token context length and accepts multimodal input (text and images).
- Training Methodology: Utilizes a three-stage pipeline: mid-training, supervised fine-tuning, and reinforcement learning with CISPO. Two specific RL environments were employed: a multiturn environment for iterative proof refinement based on compiler feedback, and a code agent environment interacting with a raw filesystem and Lean language server.
- Verification & Scaling: Correctness is verified using Mistral’s fork of SafeVerify. The model exhibits strong test-time scaling, where increasing the token budget per attempt drastically improves success rates on PutnamBench (from 44 solved at 50k tokens to 587 at 4M tokens).
- Performance Metrics: Achieved 100% on miniF2F, 587/672 on PutnamBench, and new SOTA on FATE-H (87%) and FATE-X (34%). On FLTEval, it achieved a pass@1 of 28.9 and pass@8 of 43.2, surpassing Opus 4.6 at approximately one-seventh the cost.
Industry Insight
- Cost-Efficiency in Formal Verification: Leanstral 1.5 proves that high-performance formal reasoning does not require exorbitant computational budgets. Its ability to outperform larger models at a fraction of the cost makes it viable for widespread adoption in industries requiring rigorous code verification, such as finance and aerospace.
- Integration of Agentic Workflows: The success of the code agent environment highlights the importance of tools that allow LLMs to interact directly with compilers, file systems, and language servers. Developers should prioritize building agentic workflows that leverage real-time feedback loops rather than relying solely on static generation.
- Open-Source Advantage in Niche Domains: The release underscores the competitive strength of open-weight models in specialized domains like mathematical proof and formal methods. Organizations can now deploy these models locally or via affordable APIs without licensing constraints, fostering innovation in secure and verified software development.
Disclaimer: The above content is generated by AI and is for reference only.