GAPS: Dimension-Level Gates for Conditional Activation Steering
GAPS introduces dimension-level conditioning to activation steering, adding a second axis of selectivity beyond token-level decisions to determine which neurons to intervene on Two training-free gates are combined: a static separability gate using AUROC to restrict steering to neurons with reliable concept information, and a dynamic posterior gate using a Gaussian model to steer only when activations better explain the undesired concept GAPS plugs into existing conditional methods (CAST, DSAS) w
Analysis
TL;DR
- GAPS introduces dimension-level conditioning to activation steering, adding a second axis of selectivity beyond token-level decisions to determine which neurons to intervene on
- Two training-free gates are combined: a static separability gate using AUROC to restrict steering to neurons with reliable concept information, and a dynamic posterior gate using a Gaussian model to steer only when activations better explain the undesired concept
- GAPS plugs into existing conditional methods (CAST, DSAS) with only O(D) overhead per token
- On toxicity mitigation and concept removal benchmarks with Gemma-3 (4B) and Qwen-3 (1.7B), GAPS consistently matches or improves the Pareto front of token-level counterparts
- DSAS+GAPS reduces Gemma-3's toxicity rate from 6.52% to 0.48% under a fixed capability budget, versus 3.52% for DSAS alone, with ablations attributing most gains to the posterior gate
Why It Matters
This work addresses a key limitation in activation steering—once a conditional method decides to intervene at a token, it applies the full dense steering vector uniformly across all hidden dimensions, wasting capacity on neurons that either lack concept-relevant information or are already in the desired regime. By introducing dimension-level selectivity, GAPS significantly improves the behavior-capability trade-off without requiring additional training, making it a practical upgrade for existing steering pipelines used in safety-critical LLM deployments.
Technical Details
- Static separability gate: Uses AUROC computed on hidden dimension activations to identify neurons with statistically reliable concept information, filtering out dimensions that do not meaningfully encode the target concept
- Dynamic posterior gate: Models each neuron's activation as a Gaussian distribution under competing concepts and steers only when the current activation is better explained by the undesired concept, enabling per-token, per-dimension decisions
- Plug-and-play design: The gates add O(D) overhead per token (where D is the hidden dimension size) and integrate seamlessly into existing conditional steering methods like CAST and DSAS without retraining
- Evaluation: Tested on RealToxicityPrompts (toxicity mitigation) and OneSeC (concept removal) across Gemma-3 (4B) and Qwen-3 (1.7B), demonstrating consistent Pareto front improvements over token-level-only methods
- Ablation findings: The posterior gate accounts for the majority of performance gains, suggesting that dynamic, activation-aware conditioning is more critical than static separability filtering
Industry Insight
- Training-free dimension-level gating offers a low-cost, high-impact upgrade path for existing activation steering systems—teams already using CAST or DSAS can adopt GAPS without retraining or architectural changes
- The Gaussian posterior approach generalizes beyond toxicity to any binary concept steering task, making it broadly applicable for alignment and safety interventions across model families
- As LLMs scale, the O(D) overhead becomes increasingly relevant; future work should explore sparse or low-rank approximations of the gates to maintain efficiency on larger models
Disclaimer: The above content is generated by AI and is for reference only.