The Changing Geometry of Grammar: Dimensionality and Neighborhood Reorganization across Transformer Layers
Transformer representations concentrate on lower-dimensional sub-manifolds as tokens incorporate relational context across layers, a compression quantified by Intrinsic Dimensionality (ID) Closed-class tokens (function words) expand in ID earlier and collapse sooner than open-class tokens (content words), revealing a distinct layer-wise geometric trajectory tied to grammatical role The expansion and contraction of ID is driven by changes in neighborhood structure, reflecting how inter-word relat
Analysis
TL;DR
- Transformer representations concentrate on lower-dimensional sub-manifolds as tokens incorporate relational context across layers, a compression quantified by Intrinsic Dimensionality (ID)
- Closed-class tokens (function words) expand in ID earlier and collapse sooner than open-class tokens (content words), revealing a distinct layer-wise geometric trajectory tied to grammatical role
- The expansion and contraction of ID is driven by changes in neighborhood structure, reflecting how inter-word relations reorganize within sentences across layers
- Encoder and decoder architectures evolve differently across layers, consistent with their distinct context-integration strategies (bidirectional vs. autoregressive)
- Geometric features alone can recover a token's part-of-speech tag, enabling interpretation of how semantic content for each grammatical category evolves through the network
Why It Matters
This work bridges geometric analysis and linguistic structure in Transformers, offering practitioners a new lens to understand how models internally organize grammatical information. For researchers, it provides empirical evidence that layer-wise dimensionality dynamics are not arbitrary but systematically tied to syntactic processing, which can inform model interpretability efforts and architecture design decisions.
Technical Details
- Intrinsic Dimensionality (ID) tracking: The authors measure ID layer-by-layer across Transformer models, quantifying the minimum number of independent variables needed to represent token embeddings without significant information loss, revealing how representations compress through the network
- Closed vs. open-class divergence: Closed-class items (prepositions, determiners, conjunctions) show earlier ID expansion and sooner collapse compared to open-class items (nouns, verbs, adjectives), suggesting function words are resolved earlier in the processing pipeline
- Neighborhood structure analysis: The expansion and contraction dynamics are explained by reorganization of local neighborhoods in representation space, directly linking geometric changes to relational word dependencies within sentences
- Cross-architecture comparison: Encoders (ModernBERT, bigbird-roberta-large) and decoders (gemma-2-2B, Llama-3.2-3B) exhibit systematically different layer-wise ID trajectories, reflecting their fundamentally different context integration mechanisms
- PoS recovery from geometry: A downstream classification task demonstrates that geometric features alone can recover token grammatical roles, and the authors use this to trace how semantic content for each PoS category evolves across layers
Industry Insight
- Model interpretability tooling should incorporate geometric diagnostics like ID tracking to diagnose how different architectures process syntactic versus semantic information, enabling more informed model selection for grammar-sensitive applications
- The finding that closed-class items resolve earlier suggests potential optimization opportunities: pruning or compressing early layers for function-word processing could yield efficiency gains without significant accuracy loss in syntactically dependent tasks
- The encoder-decoder divergence in geometric evolution patterns implies that hybrid or unified architectures should be evaluated not just on performance metrics but on whether their internal geometric trajectories align with desired linguistic processing behaviors
Disclaimer: The above content is generated by AI and is for reference only.