MemArena: An Ego-Centric Benchmark for On-Device Agentic Personal Memory Assistants at Scale
MemArena introduces the first ego-centric, single-world conversational benchmark for on-device personal memory assistants, addressing gaps in activity-dense interaction, ego-centric perspective, and coherent multi-session worlds Built with the MASim agent simulator, it generates 10.3M dialog-text tokens across 50 agents over 15 days, with co-generated ground truth spanning six recall, reasoning, and trustworthiness dimensions Memory-backend architecture choice has a larger impact on content accu
Analysis
TL;DR
- MemArena introduces the first ego-centric, single-world conversational benchmark for on-device personal memory assistants, addressing gaps in activity-dense interaction, ego-centric perspective, and coherent multi-session worlds
- Built with the MASim agent simulator, it generates 10.3M dialog-text tokens across 50 agents over 15 days, with co-generated ground truth spanning six recall, reasoning, and trustworthiness dimensions
- Memory-backend architecture choice has a larger impact on content accuracy than reader model scaling: switching from Memobase to MemSearch on Qwen3-0.6B yields +32.5/+19.2 percentage points versus only +10.6/+6.8 pp from scaling the reader
- Permission-aware access fails universally across all evaluated backends—Oracle retrieval leaks heavily while other backends are overly restrictive in disclosure
- Search latency on edge hardware (Spark GB10) is moderate and fixed (7–87 ms depending on backend), constituting only a small fraction of total time-to-first-token for most reader-backend combinations
Why It Matters
This benchmark directly addresses a critical gap in on-device AI: evaluating personal memory assistants that must handle private conversations with open-weight models in realistic, ego-centric settings. For practitioners building edge-deployed agents, the findings reveal that backend architecture decisions outweigh model scaling choices—a counterintuitive result with significant cost-performance implications. The universal failure of permission-aware access also highlights an unresolved privacy-utility tradeoff that the industry must solve before deploying personal memory assistants at scale.
Technical Details
- Benchmark construction: MemArena uses the MASim (Memory Agent Simulator) to generate a coherent single-world conversational dataset with 50 virtual agents interacting over 15 days, producing 10.3M dialog-text tokens and 24.1K ego-observed tokens per agent per day
- Evaluation dimensions: Six ground-truth dimensions covering recall, reasoning, and trustworthiness are co-generated alongside interaction histories, enabling multi-faceted assessment beyond simple retrieval accuracy
- Evaluated backends: Five configurations tested—Vanilla context (no memory), BM25-RAG, Oracle retrieval (upper-bound), Memobase, and MemSearch—paired with open-weight reader models including Qwen3-0.6B
- Edge performance profiling: Latency measured on Spark GB10 edge hardware, reporting fixed per-search overhead of 87 ms (BM25-RAG), 7 ms (Memobase), and 48 ms (MemSearch)
- Open release: Code, MASim simulator, and MemArena-L benchmark will be released upon acceptance
Industry Insight
- Backend over scaling: For resource-constrained edge deployments, investing in memory retrieval architecture (e.g., MemSearch over Memobase) delivers substantially better accuracy gains than scaling up the reader model—prioritize the memory layer when optimizing on-device personal assistants
- Privacy-utility deadlock: The universal failure of permission-aware access—either leaking too much (Oracle) or disclosing too little (other backends)—signals that current approaches cannot simultaneously satisfy privacy guarantees and utility requirements; new access-control mechanisms are needed before production deployment
- Edge latency is manageable: Memory search overhead is modest and predictable on edge hardware, meaning latency should not be the primary bottleneck; engineering efforts should focus on retrieval quality and privacy compliance rather than search speed optimization
Disclaimer: The above content is generated by AI and is for reference only.