The Query Knows What to Forget: A Second Erase Direction for Linear Attention
Linear attention models like GDN-2 derive erase vectors solely from token keys, creating a fundamental mismatch since retrieval interference is measured through queries The authors introduce Query-derived Erase Direction (QED), adding a second erase vector derived from the query and orthogonal to the key direction In the fast-weight view, key-directed delta edits cannot modify the key-orthogonal component of reads, leaving query-measured interference unaddressed QED uses the editable state porti
Analysis
TL;DR
- Linear attention models like GDN-2 derive erase vectors solely from token keys, creating a fundamental mismatch since retrieval interference is measured through queries
- The authors introduce Query-derived Erase Direction (QED), adding a second erase vector derived from the query and orthogonal to the key direction
- In the fast-weight view, key-directed delta edits cannot modify the key-orthogonal component of reads, leaving query-measured interference unaddressed
- QED uses the editable state portion to actively cancel old-state content as measured along the query direction
- The approach approximately doubles usable context length on S-NIAH-1 and improves retrieval at every length beyond the training window
Why It Matters
This work addresses a critical bottleneck in linear attention architectures: state interference at long context lengths. By introducing a query-derived erase direction, it closes a theoretical gap in how delta-rule models manage information retention and forgetting, directly impacting the scalability of efficient long-context models.
Technical Details
- Problem: Gated DeltaNet-2 and prior delta-rule models derive erase vectors exclusively from keys, but interference during read operations is query-measured, creating an unreachable interference component
- Solution (QED): Introduces a second erase direction derived from the query and orthogonal to the key, enabling the model to cancel old-state content along the query axis
- Fast-weight analysis: Demonstrates that key-directed delta edits are structurally incapable of changing the key-orthogonal part of reads, justifying the need for an additional erase direction
- Benchmark: Evaluated on S-NIAH-1, showing approximately 2x improvement in usable context length with consistent gains at all lengths beyond the training window
Industry Insight
- Linear attention architectures are approaching practical long-context deployment; query-aware erase mechanisms could become a standard component in next-generation efficient transformers
- The theoretical insight about the key-query mismatch in delta rules may inspire similar architectural refinements across other state-space and recurrent attention models
- Researchers building long-context systems should consider QED-like mechanisms when designing or fine-tuning linear attention variants for retrieval-heavy workloads
Disclaimer: The above content is generated by AI and is for reference only.