we are beginning work on an open source memory benchmark test at
@sibyl_labs_
after months of research we have concluded that the primary gauges the field uses for memory do not necessarily test the actual retrieval itself.
agentic environments play a large role in determining how models interact with "memory" or recalling facts.
a users agent.md could simply say "gregs hat is blue" which would remove all need for retrieval from the memory.
abstention rules are important for agents, as it teaches them that a non-answer is better than a confident hallucination.
but this has no affect on memory, or quality of the retrieval.
it is simply the agentic framework doing it's job, in combination with the memory.
so we have multiple moving pieces in an agents environment:
-persona/identity
-operating rules
-memory/retrieval
the way an agent presents the data from the retrieval process should be of no concern to people who want to gauge the performance of the memory. they are separate things.
this means the majority of "memory" benchmarks are in truth, framework benchmarks, or agentic operating systems benchmarks.
if you would like to contribute to this project, and help push the agentic memory, and agentic framework fields into the right direction with us, please feel free to reach out.
our only goal is to give the space an unbiased, objective benchmark format that allows memory companies to be properly scored and judged for metrics grounded in reality, instead of agent.md files.
we hope you want the same✊