A scaling law of contextual persistence in human language 人类语言中上下文持久性的缩放定律

TL;DR

• The study identifies a scaling law for contextual persistence in human language, showing that word arrangement follows a predictable decay pattern.
• Contextual persistence function P(d) decays approximately as 1/d across diverse corpora, with an exponent α = 1.04 and high consistency (median r² = 0.96).
• This effect is specific to natural language and does not appear in scrambled text, synthetic data, or biological sequences like DNA or proteins.
• Large language models serve as effective probabilistic probes to measure contextual influence beyond simple co-occurrence statistics.
• The near-unity exponent suggests uniform distribution of contextual influence across logarithmic timescales, implying efficient information encoding in human communication.

Why It Matters

This finding reveals a fundamental structural property of human language that may inform the design of more robust and interpretable NLP systems. Understanding how context persists over distance can guide improvements in long-range dependency modeling, prompting mechanisms, and architectural choices in transformers and other sequence models. It also provides a benchmark for evaluating whether artificial language systems replicate key properties of natural linguistic structure.

Technical Details

• Contextual persistence P(d) was defined as the reduction in target perplexity provided by prior context at distance d, relative to scrambled versions of the same word sequence.
• Ten corpora from six language families (including written and spoken modalities) were analyzed using multiple large language models as independent probes.
• The decay of P(d) followed a power-law relationship: P(d) ∝ d⁻α, with mean α = 1.04 and median R² = 0.96 across datasets.
• Control experiments confirmed the effect disappeared when words were randomly shuffled or replaced with synthetic sequences, and no similar pattern emerged in genomic or protein sequences under domain-specific models.
• The analysis relied on perplexity-based metrics derived from pre-trained LMs, avoiding direct parameter inspection while capturing emergent contextual behavior.

Industry Insight

Developers building conversational agents or document-processing systems should consider incorporating mechanisms that emulate this natural decay profile—perhaps via adaptive attention windows or hierarchical memory structures—to better handle long-range dependencies without excessive computational cost. Researchers exploring biomimetic AI architectures might investigate whether neural networks trained on tasks requiring sustained context naturally converge toward similar scaling laws, offering insights into optimal designs for real-world language understanding applications.

摘要

本研究确定了人类语言中上下文持久性的缩放定律，表明词语排列遵循可预测的衰减模式。
上下文持久性函数 P(d) 在多种语料库中大约按 1/d 衰减，指数 α = 1.04，一致性高（中位数 r² = 0.96）。
该效应仅存在于自然语言中，不会出现在打乱文本、合成数据或生物序列（如 DNA 或蛋白质）中。
大型语言模型作为有效的概率探针，可用于测量超越简单共现统计量的上下文影响。
接近 1 的指数表明上下文影响在对数时间尺度上均匀分布，暗示人类通信中的信息编码高效。

深度分析

总结

• 本研究确定了人类语言中上下文持久性的缩放定律，表明词语排列遵循可预测的衰减模式。
• 上下文持久性函数 P(d) 在多种语料库中大约按 1/d 衰减，指数 α = 1.04，一致性高（中位数 r² = 0.96）。
• 该效应仅存在于自然语言中，不会出现在打乱文本、合成数据或生物序列（如 DNA 或蛋白质）中。
• 大型语言模型作为有效的概率探针，可用于测量超越简单共现统计量的上下文影响。
• 接近 1 的指数表明上下文影响在对数时间尺度上均匀分布，暗示人类通信中的信息编码高效。

意义

这一发现揭示了人类语言的一种基本结构特性，可能为设计更稳健和可解释的自然语言处理系统提供指导。理解上下文如何在距离上持续可以指导长程依赖建模、提示机制以及变换器和其他序列模型架构选择的改进。它还为评估人工语言系统是否复制了自然语言结构的关键属性提供了基准。

技术细节

• 上下文持久性 P(d) 定义为在距离 d 处由先前上下文提供的目标困惑度降低量，相对于相同单词序列的打乱版本。
• 分析了来自六个语言家族的十个语料库（包括书面和口语模式），使用多个大型语言模型作为独立探针。
• P(d) 的衰减遵循幂律关系：P(d) ∝ d⁻α，平均 α =