DecodeShare: Tracing the Shared Subspace of LLM Decode-Time Decisions
DecodeShare identifies a low-dimensional subspace in LLM hidden states that is consistently shared across different tasks specifically during the decode phase of KV-cached inference. Causal intervention experiments demonstrate that disturbing this shared subspace degrades decision performance significantly more than interfering with prefill-derived or random subspaces. The study reveals that common activation steering directions often overlap with this task-general decode channel, allowing for c
Analysis
TL;DR
- DecodeShare identifies a low-dimensional subspace in LLM hidden states that is consistently shared across different tasks specifically during the decode phase of KV-cached inference.
- Causal intervention experiments demonstrate that disturbing this shared subspace degrades decision performance significantly more than interfering with prefill-derived or random subspaces.
- The study reveals that common activation steering directions often overlap with this task-general decode channel, allowing for clearer separation of functional roles by projecting out the shared subspace.
- Evaluating steering vectors directly at decode-time provides more reliable signals for downstream deployment compared to using prefill-based proxies.
Why It Matters
This research clarifies the internal mechanics of LLM inference by distinguishing between task-specific processing during prefill and task-general processing during decoding. For AI practitioners, it offers a new lever for improving model control and interpretability through targeted activation steering, potentially leading to more robust and efficient fine-tuning or alignment strategies.
Technical Details
- Protocol: The authors propose "DecodeShare," a method to trace and isolate a shared subspace within the hidden states of LLMs during the autoregressive decoding step.
- Causal Analysis: The causal role of this subspace is validated by removing it only during the decode phase, showing superior impact on performance degradation compared to other intervention targets.
- Activation Steering Implications: The work highlights that standard steering vectors may inadvertently target this general decode channel; projecting out the shared subspace helps disentangle specific task behaviors from general decoding dynamics.
- Benchmarking: Performance is measured by comparing the effects of interventions on the discovered shared subspace against random subspaces and those derived from the prefill stage, under equal intervention budgets.
Industry Insight
- Developers should consider evaluating activation steering vectors at the decode time rather than relying solely on prefill metrics to ensure more accurate control over model behavior.
- Understanding the distinction between prefill and decode subspaces can lead to more efficient model compression or quantization techniques that preserve critical general decoding capabilities.
- Future alignment efforts might benefit from explicitly separating task-specific adaptations from the general decode channel to prevent interference and improve stability.
Disclaimer: The above content is generated by AI and is for reference only.