Feature Geometry of LoRA Adapters: A Sparse Autoencoder Analysis of Representational Divergence in Fine-Tuned Language Models

The discovery that LoRA induces geometrically distinct feature structures is a significant methodological insight that reframes how we should view parameter-efficient fine-tuning. For years, the prevailing mental model treated LoRA as a gentle steering wheel for the pretrained model's latent space—a low-rank update that nudged the existing representation toward a new task while largely preserving its geometry. This paper provides strong empirical evidence that this view is incomplete. What we're seeing instead is the construction of parallel, partially disconnected representational highways within the residual stream.

The technical choice to use a "delta activation framework" is clever and revealing. By isolating the adapter's contribution to the residual stream, the researchers effectively filtered out the noise of the base model's persistent activations. The result—that adapter-specific Sparse Autoencoders reconstruct these deltas far better than pretrained SAEs—is the smoking gun. It suggests the LoRA updates aren't just amplifying or suppressing pre-existing features; they are generating patterns that the pretrained model's feature dictionary never had to encode. The pretrained SAE, which learned to decompose the base model's world, is looking for concepts that simply aren't the dominant ones in the delta space. It's as if you tried to describe the architecture of a new building using only the vocabulary developed to describe the old one on the same plot—you'd lack the precise terms.

The implications here ripple across several critical areas. First, for the field of mechanistic interpretability, this is a sobering reality check. Tools like SAEs are often viewed as universal feature finders for a given model architecture. This work demonstrates their domain specificity. An SAE trained on a base model may be a poor guide for understanding the internal reasoning of its fine-tuned descendant, especially for tasks that require substantial capability adjustment. If we want to audit or interpret a LoRA-adapted model, we likely need to train interpretability tools specifically on its adapted activations, treating it as a distinct entity from its base. This complicates but also clarifies the path forward: interpretability cannot be a one-time cost borne only on pretraining; it must be repeated for meaningful fine-tuning.

Second, the finding that feature density increases with LoRA rank and layer depth, while geometric divergence remains stable, is a nuanced engineering clue. It suggests that higher-rank adaptations don't just create a "bigger" adjustment in the same direction; they create a "busier" one, with more discrete features contributing to the delta. Yet the fundamental representational misalignment with the pretrained feature set is present from the lowest rank studied. This has practical implications for choosing LoRA rank: beyond a certain point, increasing rank may yield diminishing returns if the goal is to leverage the pretrained model's understanding, as you're increasingly populating a separate representational subspace.

Finally, and perhaps most critically, this has direct consequences for safety and alignment auditing. The current playbook for evaluating fine-tuned models often involves probing them for known harmful capabilities or biases established in the base model. If LoRA can create novel representational structures, it could potentially harbor novel failure modes or capabilities that are invisible to probes designed for the base architecture. A malicious or careless fine-tune could, in theory, develop dangerous knowledge or reasoning pathways in this delta space that standard safety evaluations wouldn't detect, precisely because the tools aren't looking in the right representational coordinates. The paper's call for "safety auditing of fine-tuned language models" is therefore not just an academic note but a pressing practical warning.

In essence, this research shifts our understanding of LoRA from a lightweight adjuster to a co-creator of model internals. The fine-tuned model is not just the original model with a new set of biases; it is a hybrid system where the adapter's contribution, though small in parameter count, can carve out a distinct representational niche. This forces a more honest reckoning with the complexity of post-training modification and underscores that the tools we build to understand and control AI systems must evolve as rapidly as the systems themselves.