Latent Cache Flow: Model-to-Model Communication Without Text

Background

The current method of LLM communication through text incurs substantial latency due to autoregressive decoding processes. Recent advancements, such as Cache-to-Cache (C2C), propose using adapters to translate key-value (KV) caches between models, aiming to reduce the dependency on autoregressive decoding and thus speed up communication. However, C2C adapters are large and costly to train, and they require the target context to match the source context precisely, which is impractical for LLMs with varying contexts.

Key Points

1. Joint Translation and Compression:

• Observation: The authors note that keys and values can be jointly translated and compressed.
• Implementation: This approach reduces the adapter's size from 956 MB in C2C to about 4% of its original size, i.e., around 38 MB for LCF.

2. Context Agnostic Summary Transmission:

• Design Principle: The adapters used in LCF are designed to transmit a summary of new information that the target model does not have.
• Advantage: This makes the adapter context-agnostic, meaning it can be effectively used regardless of whether the contexts match.

Significance

1. Efficiency and Accuracy:

• In shared-context settings, LCF adapters are more accurate than C2C adapters with a smaller size (13 MB vs. 956 MB).
• For different contexts, LCF shows a 23% increase in accuracy and is 8.5 times faster than text-based communication.

2. Broader Applicability:

• The context-agnostic nature of LCF allows it to handle varying contexts more flexibly, enhancing its practical utility.
• This method could lead to significant improvements in the speed and efficiency of LLM communication across different applications and scenarios.

Discussion

The introduction of LCF addresses critical limitations of existing methods by optimizing both size and context adaptability. The ability to compress information while maintaining or improving accuracy makes LCF a promising approach for enhancing LLM communications, particularly in environments where multiple models with differing contexts need to exchange information efficiently.