NeuronFuzz: Safety Neuron Guided Fuzzing for LLM Safety Evaluation
NeuronFuzz introduces a white-box fuzzing framework that uses internal safety neuron activations as continuous feedback for LLM safety evaluation, eliminating the need for full response generation during the fuzzing loop A SafetyOracle converts safety-neuron activations into a differentiable continuous safety alarm score, obtained during the prefill phase, enabling efficient gradient-based optimization The framework identifies a compact set of safety neurons through template-invariant harmful/be
Analysis
TL;DR
- NeuronFuzz introduces a white-box fuzzing framework that uses internal safety neuron activations as continuous feedback for LLM safety evaluation, eliminating the need for full response generation during the fuzzing loop
- A SafetyOracle converts safety-neuron activations into a differentiable continuous safety alarm score, obtained during the prefill phase, enabling efficient gradient-based optimization
- The framework identifies a compact set of safety neurons through template-invariant harmful/benign inputs and stability-aware selection to capture harmful-intent recognition
- NeuronFuzz leverages gradients of the safety alarm score to pinpoint safety-sensitive template positions and uses a masked language model to generate fluent, context-compatible mutations while preserving harmful payloads
- Evaluated across 21 text and multimodal models, achieving 76-100% jailbreak discovery rate on five white-box source models, outperforming baselines by up to 48 percentage points, with strong zero-shot transfer to proprietary models
Why It Matters
This work addresses a critical bottleneck in LLM safety evaluation: existing automated testing methods are expensive and provide sparse feedback, especially against strongly aligned models where most attack candidates fail identically. By shifting from response-level to neuron-level feedback, NeuronFuzz dramatically reduces computational cost while providing denser, more informative guidance for generating effective jailbreak prompts.
Technical Details
- SafetyOracle Construction: Uses template-invariant harmful and benign inputs combined with stability-aware selection to identify a compact set of safety neurons whose activations reliably capture harmful-intent recognition, forming a continuous safety alarm score
- Differentiable Fuzzing Loop: The safety alarm score is differentiable, enabling gradient-based identification of safety-sensitive template positions without requiring full model response generation
- Mutation Generation: Employs a masked language model to produce fluent, context-compatible prompt mutations that preserve the original harmful payload while avoiding additional optimization variables
- Evaluation Scope: Tested across 21 text and multimodal models, with zero-shot transfer evaluation to open-weight and six proprietary target models
- Performance Metrics: Achieved average ASR of 69.6%/92.6% and top-5 ensemble ASR of 44.1%/60.0% across transferred templates
Industry Insight
- The shift from black-box response feedback to white-box neuron-level feedback represents a paradigm change in LLM red-teaming, suggesting that internal model representations can be leveraged for more efficient and effective safety evaluation
- The strong zero-shot transferability of optimized templates to proprietary models highlights the growing capability of automated jailbreak generation, raising urgency for more robust alignment techniques
- Safety neuron identification through stability-aware selection could inform the development of more interpretable and auditable safety mechanisms in future LLM architectures
Disclaimer: The above content is generated by AI and is for reference only.