Current benchmarks rely on conversational recall, failing to measure if memory actually improves task execution for tool-using agents. The new MERIT benchmark addresses this by evaluating episodic tool-use tasks with explicit cost accounting and automated leak checks. It tracks token and dollar usage to determine the marginal utility of remembering facts across different difficulty levels.
- Replaces dialogue-centric benchmarks with tool-use task evaluation for agents
- Measures marginal utility of memory against explicit token and dollar costs
- Includes automated leak checks to verify true reliance on prior facts
- Tests updated-fact recall and controlled memory corruption scenarios