Semalith v1.4: A Calibrated 184M Safety Classifier Achieving State-of-the-Art Prompt-Injection Detection at 44x Fewer Parameters than Llama-Guard-3-8B
Semalith v1.4 is a 184M-parameter safety classifier that simultaneously handles prompt injection, general harm, and financial-services regulatory compliance in a single forward pass. It outperforms Llama-Guard-3-8B on prompt-injection evaluations (7/7) and achieves zero false positive rate on benign agentic prompts, while using 44x fewer parameters. The model is trained on a deduplicated corpus of 76,204 rows from 49 public sources, with 21 of 22 benchmarks at zero contamination. Deployment guid
Analysis
TL;DR
- Semalith v1.4 is a 184M-parameter safety classifier that simultaneously handles prompt injection, general harm, and financial-services regulatory compliance in a single forward pass.
- It outperforms Llama-Guard-3-8B on prompt-injection evaluations (7/7) and achieves zero false positive rate on benign agentic prompts, while using 44x fewer parameters.
- The model is trained on a deduplicated corpus of 76,204 rows from 49 public sources, with 21 of 22 benchmarks at zero contamination.
- Deployment guidance recommends v1.4 for BFSI label coverage or zero-FPR priorities, while v1.3 suits conversational moderation.
- Weak spots are documented, and performance complements Llama-Guard-3 on general-harm benchmarks.
Why It Matters
This work addresses a critical gap in deploying large language models in high-stakes environments like finance and agentic systems by providing a lightweight, multi-axis safety classifier that reduces computational overhead without compromising security. Its ability to detect prompt injections with zero false positives on benign inputs makes it particularly valuable for real-world applications where trust and accuracy are paramount.
Technical Details
- Model Architecture: Based on DeBERTa-v3-base with a 22-class head (BENIGN, nine prompt-injection sub-types, general-harm, eleven BFSI labels) and a 4-class auxiliary super-category head under jointly weighted loss.
- Training Data: Corpus of 76,204 rows mined from 49 public sources, with SHA-1 deduplication against every held-out evaluation set to ensure minimal contamination (max 0.22%).
- Performance Metrics: Achieves FPR = 0.000 on 208 benign agentic prompts compared to 0.063 for Llama-Guard-3-8B; wins all prompt-injection evaluations but lags slightly on general-harm benchmarks like WildGuardMix and HarmBench.
- Benchmark Results: Outperforms Llama-Guard-3-8B on 11 of 18 benchmarks overall while using significantly fewer parameters, demonstrating efficiency gains without sacrificing effectiveness.
Industry Insight
The development of compact yet highly effective safety classifiers like Semalith v1.4 signals a shift toward specialized, domain-aware guardrails tailored for specific use cases such as financial services or autonomous agents. This trend suggests future investments will focus not just on scaling up model size but also on optimizing architectures for targeted tasks—enabling safer deployment across industries where regulatory compliance and user trust are non-negotiable. Practitioners should consider integrating these types of lightweight solutions into their pipelines to balance performance, cost, and risk mitigation effectively.
Disclaimer: The above content is generated by AI and is for reference only.