Towards Verifiable Transformers: Solver-Checkable Circuit Explanations

Background

Current mechanistic interpretability focuses on identifying and explaining circuits (subnetworks) within Transformer models. However, validating these explanations typically relies on manual reasoning, examples, and ablation studies. This creates a significant gap: a circuit explanation can be plausible without being proven. The paper argues for a shift toward formal verification to bridge this gap, proposing that explanations should be stated as verifiable properties within a bounded task domain.

Key Methods and Contributions

1. The Verifiable Transformer Framework: The core contribution is a systematic process that begins with a behavior, a finite task domain, and a candidate projection. It then:

   • Extracts a task circuit.
   • Defines verifiable properties such as projected functional equivalence, content invariance, edge necessity, and final-residual robustness.
   • Verifies these properties using an SMT solver, which can produce either a proof of correctness or a counterexample.

2. Two Verification Pathways:

   • Direct Verification: For models using operators that are exactly encodable as SMT constraints. The paper instantiates this with a GPT-style architecture using Signed L1 BandNorm, sparsemax attention, and LeakyReLU. On small symbolic tasks (e.g., quote closing, bracket tracking), they train this SMT-representable Transformer and exhaustively verify properties of the extracted circuits.
   • Surrogate-Mediated Verification: For circuits with intractably encodable components (e.g., standard softmax attention). This method fits an SMT-encodable surrogate model to the extracted circuit's behavior over the task domain. The surrogate is first validated against the original circuit, and then formal verification is performed on the surrogate. This is demonstrated on circuits from GPT-2 scale models, yielding both verified explanations and solver-generated counterexamples.

3. Evaluation and Scope:

   • Proof of Concept: On small, controlled symbolic tasks, direct verification works comprehensively, proving properties like projected functional equivalence (the circuit computes the same function as the model for the task) and edge necessity (removing an edge breaks the function).
   • GPT-2 Scale Exploration: The custom operator stack (Signed L1 BandNorm, etc.) trains stably on OpenWebText at GPT-2 scale, though direct SMT verification for such large models remains intractable. Surrogate-mediated verification is presented as the feasible path for larger models.
   • Explicit Limitation: The authors state the goal is not full-model verification, but rather verifying explanations of task-localized circuits.

Significance and Implications

This work represents a paradigm shift in interpretability evaluation, moving from heuristic validation to formal, mathematical verification. Its significance lies in:

• Rigor: It transforms circuit explanations from "plausible stories" into falsifiable propositions. A solver can definitively confirm an explanation or produce a counterexample that refutes it.
• Methodological Advancement: The framework provides a concrete, reusable pipeline for the community to stress-test their circuit findings.
• Handling Complexity: The two-pronged approach (direct + surrogate) addresses the practical reality that most useful Transformers are not directly encodable in SMT solvers. Surrogate-mediated verification makes the approach applicable to real-world model scales.
• Foundation for Trust: By enabling formal proof of a circuit's properties, this methodology could build greater confidence in model understanding for high-stakes applications, moving beyond correlation toward causation in explanations.