In the Next Stage Topic of Large Models, We Found a Chinese Neo Lab Doing Continuous Learning
Mind Lab's Macaron-V1 model achieves SOTA results on 6 benchmarks by using a Mixture-of-LoRA (MoL) approach, adding only 4B parameters to a base model. The company has developed infrastructure for managing millions of LoRA models and enabling efficient training and inference with minimal parameter updates. Continuous learning via LoRA adaptation is emerging as a key direction for next-generation AI systems, aligning with insights from Richard Sutton and OpenAI's former CTO.
Analysis
TL;DR
- Mind Lab's Macaron-V1 model achieves SOTA results on 6 benchmarks by using a Mixture-of-LoRA (MoL) approach, adding only 4B parameters to a base model.
- The company has developed infrastructure for managing millions of LoRA models and enabling efficient training and inference with minimal parameter updates.
- Continuous learning via LoRA adaptation is emerging as a key direction for next-generation AI systems, aligning with insights from Richard Sutton and OpenAI's former CTO.
Why It Matters
This article highlights an innovative approach to post-training large language models that could significantly reduce the cost and complexity of adapting models for specific tasks or user preferences. By focusing on continuous learning through lightweight LoRA modules, Mind Lab demonstrates a practical path toward personalized AI agents that evolve over time without requiring full retraining. This methodology addresses critical challenges in scalability, efficiency, and personalization that are central to advancing AI applications across industries.
Technical Details
- Model Architecture: Macaron-V1 consists of two versions: Venti (748B total parameters, based on GLM-5.2 with 4B added via four ~1B LoRAs for Chat, Agent, Coding, UI generation) and Tall (50B parameters, based on Qwen3.6). Both support 200K token context.
- Mixture-of-LoRA (MoL): Dynamic switching between specialized LoRA experts based on task type; accumulates user-specific LoRAs over time for personalized adaptation.
- Infrastructure Platform MinT: Manages up to million-scale LoRA models, enabling fast loading/deployment during training, evaluation, and rollback processes—accelerating real-time load speeds by nearly tenfold.
- Efficient Reinforcement Learning: Achieved trillion-parameter RL training using only 64 H800 GPUs (~10% resource usage compared to full-parameter methods), leveraging mixed parallelism strategies including tensor, pipeline, expert, and sequence parallelism along with truncated importance sampling to mitigate distribution shifts.
Industry Insight
The success of Mind Lab’s MoL-based strategy suggests future LLM development will increasingly favor modular, incremental updates rather than monolithic retraining cycles—a shift likely driven by economic pressures around compute costs and demand for hyper-personalization. Companies investing now in flexible infrastructures capable of handling thousands of concurrent adapter variants may gain significant competitive advantages as enterprise clients seek tailored solutions without prohibitive overheads. Additionally, partnerships between hardware vendors (e.g., smart device manufacturers) offering unique behavioral datasets could become pivotal enablers of effective continuous learning loops at scale.
Disclaimer: The above content is generated by AI and is for reference only.