Reward-Informed Sparse Autoencoders and the Solution-Completeness Confound
Reward-informed SAEs (RI-SAEs) trained on GRPO trajectories from Llama-3.1-8B can separate "good" vs "bad" reasoning continuations with a silhouette score of 0.79 on selected features, but this separation largely reflects solution completeness rather than genuine reasoning quality Control experiments reveal that simple TF-IDF text classification (AUC 0.75–0.83) and three structural cues alone—response length, closed reasoning block, and boxed answer format (AUC 0.70)—achieve nearly equivalent cl
Analysis
TL;DR
- Reward-informed SAEs (RI-SAEs) trained on GRPO trajectories from Llama-3.1-8B can separate "good" vs "bad" reasoning continuations with a silhouette score of 0.79 on selected features, but this separation largely reflects solution completeness rather than genuine reasoning quality
- Control experiments reveal that simple TF-IDF text classification (AUC 0.75–0.83) and three structural cues alone—response length, closed reasoning block, and boxed answer format (AUC 0.70)—achieve nearly equivalent class separation
- A reward-blind generic SAE fails to separate the classes entirely (silhouette 0.01), confirming the RI-SAE's discriminative power is in-sample fitting of curated reward signals rather than discovery of intrinsic model structure
- Only two features—symbolic mathematics and procedural/evaluative language—survive as meaningfully interpretable discriminative signals, though the authors treat these as illustrative rather than definitive evidence of reasoning capture
- The paper introduces both the RI-SAE recipe and a control battery as a reusable framework for validating whether interpretability methods actually measure the intended cognitive phenomenon
Why It Matters
This work delivers a critical reality check for the growing effort to use reinforcement learning rewards as a proxy signal for steering sparse autoencoder interpretability toward reasoning. It demonstrates that reward-based data curation, while cheap and label-free, risks conflating superficial completion artifacts with genuine reasoning structure—a confound that could mislead researchers into believing they've isolated reasoning features when they've merely captured formatting patterns.
Technical Details
- RI-SAE construction: The authors split GRPO (Group Relative Policy Optimization) trajectories into high-reward ("good") and low-reward ("bad") reasoning continuations, then train a standard JumpReLU sparse autoencoder with 16,384 features on Llama-3.1-8B activations from these curated datasets
- Separation metrics: A sparse subset of features achieves silhouette score 0.79 for good/bad class separation, compared to 0.005 using the full feature code, suggesting strong but potentially spurious discriminative power
- Control battery: Three validation approaches were deployed—a TF-IDF text classifier (AUC 0.75–0.83), a structural-cue-only classifier using length, closed reasoning block, and boxed answer (AUC 0.70, with 99% of good vs. 69% of bad completions containing boxed answers), and a reward-blind generic SAE (silhouette 0.01, no discriminative features)
- Interpretable features: Despite the confound, two feature categories remained readable—symbolic mathematics representations and procedural/evaluative language patterns—though the authors caution against overinterpreting these as isolated reasoning signals
Industry Insight
- Researchers pursuing reward-informed interpretability should adopt control batteries as a standard validation step; without them, there is a significant risk of mistaking surface-level completion statistics for mechanistic insight into model reasoning
- The 99% vs. 69% boxed-answer rate between good and bad completions reveals a systemic artifact in RL-trained models: reward signals heavily correlate with output formatting conventions, which could propagate into interpretability pipelines and bias feature attribution toward stylistic rather than cognitive signals
- For practitioners building SAE-based interpretability tooling, the finding that reward-blind dictionaries fail to recover discriminative structure suggests that data curation strategy matters more than architecture choice—investing in better signal design and control validation will yield more reliable interpretability than simply scaling feature counts
Disclaimer: The above content is generated by AI and is for reference only.