# Greymemory

**Designing memory for agents · a side project, open source · 2026 · Arunkumar Alagarsamy**

> LLMs are stateless by design. Memory is what makes an agent feel like it knows you.

A self-hosted memory layer for AI agents. It pulls atomic memories out of natural
conversation, notices when they contradict, supersedes the stale ones while keeping
their history, and brings the right one back into context when it's needed. I
designed and built it end to end — including the benchmark harness.

## Project links

- Website: https://greymemory.vercel.app/
- npm: https://www.npmjs.com/package/greymemory
- GitHub: https://github.com/arun-dev-des/greymemory
- Interactive knowledge graph: https://greymemory.vercel.app/#/viz
- Licence: Apache 2.0 · BM25 + vector RAG

## Why memory is a design problem

- An agent that forgets what you told it last week, or answers with a fact that
  changed two months ago, doesn't read as "buggy" — it reads as **not very smart**.
- A wrong answer you can catch and correct. But agents don't just answer any more —
  they **act**, and an action taken on missing or stale context isn't a bad sentence,
  it's a wrong move someone has to notice and undo. The more an agent does on its own,
  the more the right action depends on the right context — and that context is what
  memory holds between sessions.
- Perceived intelligence is mostly memory: does it know me, does it keep up, does it
  admit when it doesn't know.

## What good memory has to do — five abilities, each a real failure

### 01 · Information Extraction
*Recall one specific detail from months of conversation.*

March session: "Can you help me plan nut-free lunches for the week? Mrs. Alvarez just
banned nuts in my daughter's class and I'm out of ideas." Three months later:
*Q → Who's my daughter's teacher?* *A → Mrs. Alvarez.*

**Why it's hard:** the name surfaced inside a lunch-planning request, never as "her
teacher is Mrs. Alvarez." Store only explicit statements and it was never kept — so
retrieval has nothing to find.

### 02 · Multi-Session Reasoning
*Assemble one answer scattered across many sessions.*

Four separate sessions — Korean BBQ in February, a Thai place in March, an Ethiopian
restaurant in May, a ramen bar in June. *Q → How many new restaurants have I tried
this year?* *A → Four.*

**Why it's hard:** four unrelated dinners joined only by the abstract idea of "a new
restaurant I tried." No single query surfaces all four; the system has to gather
them, then count.

### 03 · Knowledge Updates
*Notice a fact changed, and answer with the current one — not both.*

January: "Reviewing every PR myself is still manageable with four engineers." April:
"Onboarding is brutal now that we're six." *Q → How many engineers do I lead now?*
*A → Six — not "four and six."*

**Why it's hard:** the old number isn't wrong, it's stale — ask what the team looked
like in January and four is still true. But "now" wants the current one. Without
supersession the system keeps both alive and hands you the ambiguity.

### 04 · Temporal Reasoning
*Reason about when things happened, not just what.*

Feb 3: a 40-minute outage. May 20: a full outage, DB failover didn't fire. June 10: a
near-miss caught before it took prod down. *Q → How long was it between our last two
real outages?* *A → About 15 weeks.*

**Why it's hard:** that number is written down nowhere — it's Feb 3 → May 20, computed
from two dates the system had to store as the event dates, not the dates they happened
to be mentioned. And "real outages" has to rule out the June near-miss.

### 05 · Abstention
*Know what it doesn't know, and say so instead of confabulating.*

June: "My sister's wedding is eating every weekend this month — I'm maid of honor."
*Q → What did I think of my brother's wedding?* *A → "You haven't mentioned a brother."*

**Why it's hard:** the system knows about a wedding and a sibling, so the helpful
reflex is to answer about the sister's. Resisting that — and naming the false premise
instead of inventing an opinion — is the skill. For most models, a confident wrong
answer beats an honest "I don't know."

## The knowledge graph as an interface

- Everything the agent holds, as one graph — facts, preferences, episodes, and the
  `updates`, `extends` and `supersedes` edges between them.
- Type a question and it lights up what it would pull back. This is "memory is an
  interface" made literal: a person can look at what the agent knows about them, not
  just hope it's right.
- The live graph shown on the site: 272 memories, 37 docs, 63 links; nodes coloured by
  type (fact, preference, episode, raw chunk), edges for updates / extends /
  supersedes / source.

## How Greymemory handles them

- **Atomic extraction into a typed graph.** fact / preference / episode, each with a
  confidence and a source — so a teacher's name buried in a lunch request still gets kept.
- **Supersession, not overwrite.** "I work at Barbell Cartel" → later "I now work at
  Stripe": the second is classified as an update, the first is superseded but its
  history stays queryable.
- **Temporal grounding.** Real event dates extracted, a timeline injected at read time —
  this alone moved temporal accuracy 60% → 73.3%.
- **`forget()`** — soft-delete by meaning; gone from every query the moment you call
  it, kept in the database.

```ts
// a fact, then one that contradicts it
await memory.add([{ role: 'user', content: 'I work at Barbell Cartel as a product designer' }])
await memory.add([{ role: 'user', content: 'I now work at Stripe as a PM' }])

await memory.search('where does Arun work')
// → 'Arun works at Stripe as a PM'  — old fact superseded, not gone

await memory.getHistory('where has Arun worked')  // full chain, newest first
await memory.forget('my old apartment address')   // soft-delete by meaning
```

## The problem, measured

- **LongMemEval** (arXiv 2410.10813) breaks long-term memory into those five abilities
  and tests them across ~500 hand-built questions.
- The baseline everyone quotes: GPT-4o reads the whole transcript and still scores
  60.6% — it gets lost in the middle. ChatGPT's own long-term memory scores 57.7%.
- Greymemory: **80.0% overall** — 96% of a funded cloud competitor's score,
  self-hosted, at ~$0.013 a session.

### LongMemEval — accuracy by ability

| Ability | Raw context | Greymemory |
|---|---|---|
| Information extraction | 81.4% | 93.3% |
| Multi-session | 44.3% | 66.7% |
| Knowledge update | 78.2% | 80.0% |
| Temporal reasoning | 45.1% | 73.3% |
| Preference | 20.0% | 66.7% |
| **Overall** | **60.6%** | **80.0%** |

## The design argument

Memory is an interface. If a person can't see or correct what an agent has stored
about them, they can't calibrate how much to trust it — which is why the graph, a
readable version history and an instant `forget()` are part of the product, not an
admin afterthought. The same store serves the person reading it and the agent acting
on it.