Arun
UX Case Studies05 / 05

UX case study · Memory for AI agents

Greymemory

A self-hosted memory layer for AI agents. It pulls atomic memories out of natural conversation, notices when they contradict, supersedes the stale ones while keeping their history, and brings the right one back into context when it's needed. I designed and built it end to end — including the benchmark harness.

TypeOpen-source side project
LicenceApache 2.0
RetrievalBM25 + vector RAG
Year2026
80.0%LongMemEval overall
96%of a funded cloud competitor's score
~$0.013per session, self-hosted
Greymemory website
Fig. 01Greymemory — memory that knows when facts change
“
LLMs are stateless by design. Memory is what makes an agent feel like it knows you.

01Why memory is a design problem

  • An agent that forgets what you told it last week, or answers with a fact that changed two months ago, doesn't read as "buggy" — it reads as not very smart.
  • A wrong answer you can catch and correct. But agents don't just answer any more — they act, and an action taken on missing or stale context isn't a bad sentence, it's a wrong move someone has to notice and undo. The more an agent does on its own, the more the right action depends on the right context — and that context is what memory holds between sessions.
  • Perceived intelligence is mostly memory: does it know me, does it keep up, does it admit when it doesn't know.

02What good memory has to do

Five failure modes

Five abilities — each one a real failure mode.

01

Information Extraction

Recall one specific detail from months of conversation.
March session: "Can you help me plan nut-free lunches for the week? Mrs. Alvarez just banned nuts in my daughter's class and I'm out of ideas." Three months later —
QWho's my daughter's teacher?
AMrs. Alvarez.
Why it's hard: the name surfaced inside a lunch-planning request, never as "her teacher is Mrs. Alvarez." Store only explicit statements and it was never kept — so retrieval has nothing to find.
02

Multi-Session Reasoning

Assemble one answer scattered across many sessions.
Four separate sessions — Korean BBQ in February, a Thai place in March, an Ethiopian restaurant in May, a ramen bar in June.
QHow many new restaurants have I tried this year?
AFour.
Why it's hard: four unrelated dinners joined only by the abstract idea of "a new restaurant I tried." No single query surfaces all four; the system has to gather them, then count.
03

Knowledge Updates

Notice a fact changed, and answer with the current one — not both.
January: "Reviewing every PR myself is still manageable with four engineers." April: "Onboarding is brutal now that we're six."
QHow many engineers do I lead now?
ASix — not "four and six."
Why it's hard: the old number isn't wrong, it's stale — ask what the team looked like in January and four is still true. But "now" wants the current one. Without supersession the system keeps both alive and hands you the ambiguity.
04

Temporal Reasoning

Reason about when things happened, not just what.
Feb 3: a 40-minute outage. May 20: a full outage, DB failover didn't fire. June 10: a near-miss caught before it took prod down.
QHow long was it between our last two real outages?
AAbout 15 weeks.
Why it's hard: that number is written down nowhere — it's Feb 3 → May 20, computed from two dates the system had to store as the event dates, not the dates they happened to be mentioned. And "real outages" has to rule out the June near-miss.
05

Abstention

Know what it doesn't know, and say so instead of confabulating.
June: "My sister's wedding is eating every weekend this month — I'm maid of honor."
QWhat did I think of my brother's wedding?
A"You haven't mentioned a brother."
Why it's hard: the system knows about a wedding and a sibling, so the helpful reflex is to answer about the sister's. Resisting that — and naming the false premise instead of inventing an opinion — is the skill. For most models, a confident wrong answer beats an honest "I don't know."

03The knowledge graph as an interface

  • Everything the agent holds, as one graph — facts, preferences, episodes, and the updates, extends and supersedes edges between them.
  • Type a question and it lights up what it would pull back. This is "memory is an interface" made literal: a person can look at what the agent knows about them, not just hope it's right.
Greymemory knowledge graph — 272 memories, 37 docs, 63 links; nodes coloured by type
Fig. 02The live graph — 272 memories, 37 docs, 63 links; nodes by type, edges for updates / extends / supersedes / source

04How Greymemory handles them

  • Atomic extraction into a typed graph. fact / preference / episode, each with a confidence and a source — so a teacher's name buried in a lunch request still gets kept.
  • Supersession, not overwrite. "I work at Barbell Cartel" → later "I now work at Stripe": the second is classified as an update, the first is superseded but its history stays queryable.
  • Temporal grounding. Real event dates extracted, a timeline injected at read time — this alone moved temporal accuracy 60% → 73.3%.
  • forget() — soft-delete by meaning; gone from every query the moment you call it, kept in the database.
memory.ts9 lines
// a fact, then one that contradicts it
await memory.add([{ role: 'user', content: 'I work at Barbell Cartel as a product designer' }])
await memory.add([{ role: 'user', content: 'I now work at Stripe as a PM' }])

await memory.search('where does Arun work')
// → 'Arun works at Stripe as a PM'  — old fact superseded, not gone

await memory.getHistory('where has Arun worked')  // full chain, newest first
await memory.forget('my old apartment address')   // soft-delete by meaning

05The problem, measured

LongMemEval accuracy
  • LongMemEval (arXiv 2410.10813) breaks long-term memory into those five abilities and tests them across ~500 hand-built questions.
  • The baseline everyone quotes: GPT-4o reads the whole transcript and still scores 60.6% — it gets lost in the middle. ChatGPT's own long-term memory scores 57.7%.
  • Greymemory: 80.0% overall — 96% of a funded cloud competitor's score, self-hosted, at ~$0.013 a session.
Information extraction81.4% → 93.3%
Multi-session44.3% → 66.7%
Knowledge update78.2% → 80.0%
Temporal reasoning45.1% → 73.3%
Preference20.0% → 66.7%
Overall60.6% → 80.0%
Raw context (GPT-4o, full transcript)Greymemory

06The design argument

Memory is an interface. If a person can't see or correct what an agent has stored about them, they can't calibrate how much to trust it — which is why the graph, a readable version history and an instant forget() are part of the product, not an admin afterthought. The same store serves the person reading it and the agent acting on it.

Next case study

01 / 05