Memory Tax Paper Finds Dedicated Agent Memory Systems Underperform Naive RAG

A paper titled "Memory Tax: Dissecting the Net Value of Agent Memory" by Xuan He and ten co-authors evaluates dedicated memory systems against naive retrieval over raw history across LongMemEval-s, PersonaMem-v2 and MINT-Eval and multiple model backbones, finding many memory systems score below naive RAG while costing more. The authors split net value into memory benefit, questions memory solves that RAG misses, and memory tax, questions RAG solves that memory loses, attributing the tax to dropped or distorted facts and reduced accessibility. A proof-of-concept that also retrieves from source history and reranks it jointly with memory reduced the tax.
Event version 1.
Key insights
Memory systems trail naive RAG
Across LongMemEval-s, PersonaMem-v2 and MINT-Eval and several model backbones, many dedicated memory systems underperform naive RAG over raw history despite substantially higher costs.
Net value split into benefit and tax
Per-query comparison with RAG divides net value into memory benefit, queries memory solves that RAG misses, and memory tax, queries RAG solves that memory loses.
Sources of the tax
Memory processing can drop or distort information and make preserved information harder to retrieve, and results are more sensitive to backbone model choice than RAG.
Joint retrieval reduces the tax
A proof of concept that also retrieves from the source history and reranks it jointly with memory reduced the tax, supporting a path back to raw history.