Disentangling Statistical Preemption from Entrenchment in Language Models' Avoidance of Overgeneralization
The paper investigates how language models avoid overgeneralizations (e.g., "Tom laughed me") without explicit negative evidence, comparing two constructionist hypotheses: preemption and entrenchment Controlled rearing experiments on LMs trained on child-caregiver conversations systematically removed preemptive vs. non-preemptive evidence to disentangle the two mechanisms LMs avoid overgeneralizations but do NOT show verb-specific preemption; instead, they exhibit weak but non-zero evidence of a
Analysis
TL;DR
- The paper investigates how language models avoid overgeneralizations (e.g., "Tom laughed me") without explicit negative evidence, comparing two constructionist hypotheses: preemption and entrenchment
- Controlled rearing experiments on LMs trained on child-caregiver conversations systematically removed preemptive vs. non-preemptive evidence to disentangle the two mechanisms
- LMs avoid overgeneralizations but do NOT show verb-specific preemption; instead, they exhibit weak but non-zero evidence of abstract preemption
- Training dynamics analysis reveals LMs treat competing structures as indirect positive evidence rather than indirect negative evidence in verb-specific conditions
- Results suggest neural network learners need sensitivities to indirect negative evidence, and motivate new human experiments testing abstract preemption
Why It Matters
This work bridges computational linguistics and developmental psychology by testing whether language models acquire grammatical knowledge through mechanisms similar to human children. For AI practitioners, it reveals a critical gap: current LMs lack true sensitivity to indirect negative evidence, which has implications for building more robust and human-like language learners. For researchers, it provides a controlled experimental framework for disentangling competing linguistic hypotheses using neural network models.
Technical Details
- Methodology: Controlled rearing experiments where LMs are trained on child-caregiver conversation data with systematic removal of either preemptive evidence (near-synonymous constructions like "she made him laugh") or non-preemptive evidence (general verb usage like "He laughed")
- Hypotheses tested: Preemption (learning is driven by exposure to competing constructions) vs. Entrenchment (learning is driven by all exposures to a verb's grammatical usages)
- Key finding on training dynamics: LMs interpret competing structures as indirect positive evidence rather than indirect negative evidence in verb-specific conditions, suggesting a fundamental difference from human language acquisition
- Abstract preemption: While verb-specific preemption was not observed, weak but non-zero evidence of abstract preemption was detected, indicating LMs may generalize preemption-like behavior at a more structural level
- Subject area: Computation and Language (cs.CL), arXiv:2609.01794
Industry Insight
- The finding that LMs treat competing structures as positive rather than negative evidence suggests current architectures may need explicit mechanisms for indirect negative evidence sensitivity to achieve human-like language acquisition patterns
- Abstract preemption effects in LMs could inform the design of more efficient training regimes that leverage structural competition rather than relying solely on verb-specific exposure
- Researchers should consider designing human experiments specifically targeting abstract preemption, as the paper's results suggest this may be the more plausible route for human learners, creating a productive feedback loop between computational and cognitive science
Disclaimer: The above content is generated by AI and is for reference only.