LLMs Can Annotate Attribution Graphs
Automates the manual step of circuit tracing by using LLMs to group individual features or MLP neurons into supernodes based on feature descriptions Supernodes generated by the pipeline match human annotators in interpretability according to automated interpretability metrics Achieves 97% accuracy in recovering intermediate-hop supernodes on a two-hop Capitals task across 100 prompts Demonstrates open-ended exploration at scale by automatically annotating 1000 attribution graphs from Wikipedia c
Analysis
TL;DR
- Automates the manual step of circuit tracing by using LLMs to group individual features or MLP neurons into supernodes based on feature descriptions
- Supernodes generated by the pipeline match human annotators in interpretability according to automated interpretability metrics
- Achieves 97% accuracy in recovering intermediate-hop supernodes on a two-hop Capitals task across 100 prompts
- Demonstrates open-ended exploration at scale by automatically annotating 1000 attribution graphs from Wikipedia completions and using an LLM judge to flag interesting cases for human review
- Shows that simple automation can produce meaningful attribution graph annotations, motivating further research into automated circuit tracing
Why It Matters
This work addresses a major bottleneck in mechanistic interpretability: the labor-intensive process of manually grouping features into supernodes during circuit tracing. By demonstrating that LLMs can perform this task at human-level quality, it opens the door to scaling interpretability analysis far beyond what is feasible with manual annotation alone. This is particularly significant as the field moves toward analyzing increasingly complex models and larger circuits.
Technical Details
- Pipeline design: Feature descriptions are directly fed to an LLM, which groups them into supernodes without requiring complex fine-tuning or additional training data
- Evaluation on two-hop Capitals task: The pipeline successfully recovers the intermediate-hop supernode in 97 out of 100 prompts, demonstrating strong performance on structured reasoning tasks
- Automated interpretability metrics: Used to quantitatively compare LLM-generated supernodes against human-annotated ones, confirming parity in interpretability quality
- Large-scale proof of concept: Annotated 1000 attribution graphs from Wikipedia prompt completions and deployed an LLM judge to triage and flag interesting graphs for subsequent human review
Industry Insight
- Automated circuit tracing could dramatically accelerate mechanistic interpretability research, enabling teams to analyze more models and more circuits with fewer human resources
- The LLM-as-judge approach for triaging attribution graphs offers a practical template for scaling other interpretability workflows that currently rely on manual inspection
- As model complexity grows, simple automation pipelines like this one will become essential infrastructure for maintaining visibility into model internals, making investable in interpretability tooling a strategic priority for organizations focused on AI safety and reliability
Disclaimer: The above content is generated by AI and is for reference only.