Representation of syntax in LLMs through the lens of linear distance and similarity-aware entropy 通过线性距离和相似度感知熵的视角看LLM中的句法表征

TL;DR

• The paper disaggregates the undirected unlabeled attachment score (UAS) into UASL (UAS by label), revealing significant variability in syntactic relation reconstruction accuracy across different linguistic categories
• Two primary factors predict most UASL variability: (i) the mean and dispersion of linear distance (log-scaled) between syntactically related words, and (ii) the similarity-aware entropy (diversity) of the syntactic relation's head
• These findings hold consistently across a range of model sizes and architectures, suggesting generalizable geometric principles underlying syntactic representation
• The work connects structural probe methodology with information-theoretic and geometric properties of embedding spaces
• Results illuminate the degree of abstraction in how LLMs represent syntax and its dependence on spatial properties of the latent representation space

Why It Matters

This research provides practitioners and researchers with a finer-grained diagnostic tool (UASL) for evaluating how well language models capture syntactic structure, moving beyond aggregate scores that mask important per-relation differences. The identification of linear distance and head entropy as key predictors offers actionable insights for interpreting model internals and potentially guiding architectural improvements. For the broader AI community, it bridges computational linguistics, information theory, and representation learning, offering a principled framework for understanding syntax encoding in neural models.

Technical Details

• Structural Probes Extension: Builds on Hewitt and Manning's structural probe methodology, which reconstructs syntactic trees from latent representations. The key innovation is the disaggregation of the standard UAS metric into UASL (undirected attachment score by label), enabling per-relation accuracy analysis rather than aggregate evaluation.
• Linear Distance Analysis: Measures the mean and dispersion (on a log scale) of linear distances between words connected by syntactic relations in the embedding space. Shorter and more consistent distances correlate with higher reconstruction accuracy for specific relation types.
• Similarity-Aware Entropy: Introduces an information-theoretic measure quantifying the diversity of a syntactic relation's head words. Relations with heads exhibiting higher similarity-aware entropy show systematically different reconstruction performance, suggesting that head word distributional properties influence syntactic representational fidelity.
• Cross-Architecture Validation: The two predictive factors are demonstrated to hold across varying model sizes and architectures, indicating that the observed geometric-information-theoretic relationships are robust properties of LLM syntactic representations rather than artifacts of specific model designs.

Industry Insight

• Model evaluation pipelines should adopt per-relation diagnostics like UASL to identify specific syntactic weaknesses that aggregate metrics obscure, enabling more targeted model improvement efforts.
• The geometric interpretability of syntactic representations suggests that embedding space regularization techniques (e.g., distance-preserving constraints or entropy-aware losses) could be leveraged to strengthen syntactic encoding without additional supervision.
• As LLMs are deployed in linguistically sensitive applications (legal, medical, technical domains), understanding which syntactic relations are robustly represented versus fragile can inform risk assessment and guide the selection of models for syntax-critical tasks.

摘要

本文将无向无标签依存得分（UAS）细分为 UASL（按标签的 UAS），揭示了不同语言类别中句法关系重建准确率的显著差异
两个主要因素可预测大部分 UASL 差异：（i）句法相关词之间线性距离（对数缩放）的均值与离散度，以及（ii）句法关系中心词的相似度感知熵（多样性）
这些发现在不同模型规模和架构中保持一致，暗示句法表征背后存在可泛化的几何原理
该工作将结构探针方法与嵌入空间的信息论和几何特性联系起来
研究结果阐明了大语言模型在表征句法时的抽象程度，及其对潜在表征空间空间属性的依赖

深度分析

一句话总结

• 本文将无向无标签依存得分（UAS）细分为 UASL（按标签的 UAS），揭示了不同语言类别中句法关系重建准确率的显著差异
• 两个主要因素可预测大部分 UASL 差异：（i）句法相关词之间线性距离（对数缩放）的均值与离散度，以及（ii）句法关系中心词的相似度感知熵（多样性）
• 这些发现在不同模型规模和架构中保持一致，暗示句法表征背后存在可泛化的几何原理
• 该工作将结构探针方法与嵌入空间的信息论和几何特性联系起来
• 研究结果阐明了大语言模型在表征句法时的抽象程度，及其对潜在表征空间空间属性的依赖

研究意义

本研究为从业者和研究者提供了一种更细粒度的诊断工具（UASL），用于评估语言模型捕捉句法结构的程度，超越了掩盖重要逐关系差异的聚合得分。将线性距离和中心词熵识别为关键预测因子，为解读模型内部机制提供了可操作的见解，并可能指导架构改进。对于更广泛的 AI 社区而言，该工作弥合了计算语言学、信息论与表征学习之间的鸿沟，为理解神经网络模型中的句法编码提供了一个原则性框架。

技术细节

• 结构探针