How I Built a Local Memory Network for My AI Agents

July 21, 2026

Your AI agent forgets everything between sessions. So it re-derives the same decisions, and you pay for it every time.

I got tired of paying. Tired of re-explaining the same project every morning, watching the agent rediscover a gotcha I hit three weeks ago, re-litigating a decision I already made.

So I stopped relying on the agent’s memory and built the memory into the projects themselves.

I’ve been running this system across everything I build for months now: a SaaS, a couple of games, personal tools. This post is the full internals. How the files are structured, how they link to each other, and how the whole thing became searchable in about 0.2 seconds without a single byte leaving my machine.

The whole idea in one sentence

The repository itself is the memory.

Every durable fact gets written to exactly one file, committed where the work happens. Decisions, conventions, gotchas, what I’m working on right now. A fresh agent session doesn’t recall any of it. It reads it.

That sounds obvious. Maybe it is. But the difference between “we have some docs” and “the repo is the memory” comes down to three things most doc folders never get:

  • A map, so an agent can find the one file it needs without scanning the tree
  • Links, so facts have one home and everything else points at it
  • Search, for when you don’t know which file to open

Let me walk through each layer.

The part I didn’t plan

I built this to stop coding agents from forgetting technical context. But once the network held decisions, current work, past mistakes, meeting notes, and my own writing preferences, it turned out to be useful outside the code.

It now drafts my documentation, release notes, blog posts, and social posts from the actual history of the work. I review, correct, and approve everything, but I no longer have to reconstruct what happened before I can explain it. This post was drafted from the same network it describes, and the site it lives on is updated through that system too.

The shape: three tiers

Not everything belongs in a repo. A convention about how I write tests belongs in that project. A rule about how I write emails belongs everywhere. My own working-style preferences belong to me, not to the codebase.

So the network has three tiers.

Diagram of three stacked tiers: repo networks committed with the code, wide memory for cross-project facts, and personal per-project memory
Fig 1. One home per fact. A top-level index file maps all the networks so any session can locate the right one.

The routing rule is the load-bearing part. The first time a rule shows up that’s wider than the current project, it moves up a tier immediately, in the same session. Otherwise you end up with four copies that slowly drift apart, and then you can’t trust any of them.

Inside a repo network

Here’s the anatomy of a single repo’s memory, using my biggest project as the example.

Diagram of one repo's read path: an always-loaded CLAUDE.md at the top, a context map below it, and four lazy-loaded doc types underneath: topic docs, ADRs, OPEN.md, and living notes
Fig 2. The read path of a fresh agent session. The always-loaded file stays lean on purpose; everything heavy is lazy-loaded through the map.

The always-loaded manual holds only the rules that prevent damage. Everything else is behind the map, and the map entry for each file includes one line saying when to read it. That’s what keeps the token cost sane. The agent doesn’t front-load forty documents. It locates, then opens one.

That sounds wrong if you’re used to “more context is better.” But every file the agent loads costs tokens, and most of them wouldn’t have changed the answer.

Three kinds of links hold the network together, and none of them are clever.

Every memory note is one fact in one file, with a little frontmatter and a slug. Other notes reference it with [[that-slug]]. A link to a note that doesn’t exist yet is allowed on purpose. It’s a marker that something is worth writing, not an error.

---
name: verify-before-you-report
description: Never claim a result you didn't read from the tool this turn
type: feedback
---

Confirm the exit code, re-run the check on the committed tree, then say it.
Related: [[done-means-verified]], [[never-invent-a-sha]]

2. Index files as hubs

Every tier has a small index: one line per note with a pointer. The indexes are what get loaded into context; the notes are what get opened on demand. Same shape at every level, from a single repo’s context map up to the top-level file that maps all the networks on the machine.

3. A health check that keeps the graph honest

Links rot. So a script runs at session start and reports three numbers: dangling links (a [[slug]] pointing at nothing), orphans (notes no index points to), and missing file links. This morning it reported 0 dangling and 8 orphans. Those orphans are real debt, but now I can see it instead of finding out later.

A network graph of notes: a few large hub nodes with many connections, many small leaf nodes, and one disconnected orphan node drawn with a dashed outline
Fig 3. I also generate an actual interactive version of this from the real files: a small script walks the folder, parses the wikilinks, and emits a self-contained force-graph HTML page. Zero dependencies, nothing hosted.

The pattern underneath all three: one home, many pointers. A fact lives in one file. Everything else links to it and never copies it. Two copies of the same fact will disagree eventually, and then your memory is lying to you.

How it’s searchable (without sending anything anywhere)

Links work when you know roughly where something lives. Search is for when you don’t.

The search layer is one SQLite file, built by a small indexer that walks the whole network: wide memory, every project’s notes, and the docs folders of the repos I care about. It chunks each file into pieces of about 220 words, respecting markdown sections, and indexes every chunk two ways:

  • Keyword (BM25 via SQLite’s FTS5). Catches exact rare tokens: an env var name, a commit SHA, a number like 0.62. Embeddings genuinely miss these.
  • Semantic (local embeddings via Ollama). Catches paraphrase, so a query about “pricing” finds the note that says “monetization.” Nothing leaves the machine, which matters because this corpus holds financials and unreleased work.

At query time, both searches run and the results get merged by Reciprocal Rank Fusion. Not by mixing scores. A BM25 score and a cosine similarity aren’t on the same scale, so you fuse by rank instead, and it needs no tuning. Answers come back in about 0.2 seconds over roughly eight thousand chunks.

Flow diagram: a query splits into a keyword BM25 branch and a semantic local-embeddings branch, both feed rank fusion, which returns top chunks pointing to one file
Fig 4. If the local embedding model isn't running, it falls back to keyword-only instead of failing. The search has to always answer something, or I'd stop reaching for it.

Two design choices here matter more than the retrieval math.

The index is disposable. Markdown is the source of truth. The database is derived from it, and I can delete it and rebuild from scratch in about a minute. There’s no product sitting between me and my own notes, and nothing to migrate away from later. The notes are still just files on my machine.

Indexing is incremental by content hash. A run with no edits is a no-op that takes milliseconds. That’s what makes it cheap enough to just always be current.

The agent gets this as a one-line query tool, and the standing rule is to orient first: before any real task, ask the network what’s in flight on that topic. The answer is either “here’s the context” or “nothing relevant,” and both are useful.

The habits that keep it alive

Honestly, this is the part that decides whether the whole thing works. The files and the search are just plumbing. What actually kills a system like this is nobody writing anything down.

Three habits, and the mechanisms that enforce them:

Capture in the same pass. When a decision gets made or a gotcha gets hit, it goes to its one home in that same session. Not a follow-up chore, part of “done.” The agent does the writing; my job is to not skip it.

Hooks that surface memory instead of trusting recall. A session-start hook prints the open items and the current focus before any work begins. Another hook fires right before a long conversation gets compacted and re-surfaces the capture rules, because compaction is exactly the moment a learning would otherwise be summarized away.

One-word verbs for the expensive moments. Ending a session well used to mean writing a handoff by hand, so I mostly didn’t. The data made that embarrassing to look at:

Bar chart of session endings: compacted the session to keep going 106 times, cleared and started fresh 65 times, wrote an actual handoff 4 times
Fig 5. 171 session endings, 4 handoffs. I was stretching sessions instead of ending them, because ending them well was work.

The fix was to make the whole close-out cost one word. I type handoff and the agent writes the summary, files the open questions, sweeps for uncaptured gotchas, and stores it where the next session’s hook will surface it. I type pick up and the next session starts exactly where the last one stopped. Same idea for idea (log a passing thought without derailing the session) and queue (one canonical answer to “what’s first today,” computed by a script instead of vibes).

One word removes the friction. The friction was the whole problem.

What this is not

It isn’t a hosted product or a service I’m asking you to buy. It’s also not magic memory for the AI. The agent still forgets everything between sessions. The system just makes forgetting cheap, because everything worth knowing is written down where the next session will look.

And the AI doesn’t get this right on its own. It captures the wrong things, files duplicates, and lets links rot, unless the rules are written down and enforced by the same network it maintains. The judgment stays with me. The system just means I only have to make each call once.

If you want to start

You don’t need the search index or the graph or the hooks on day one. I didn’t have them either. The minimum that pays for itself immediately:

  • One short always-loaded file: what the project is, how to run it, the rules that matter
  • docs/now.md: what’s active right now, a few dated lines
  • docs/decisions.md: one dated entry per real decision, with the why
  • docs/notes.md: gotchas worth not hitting twice
  • The habit: write it down in the same session it happens

Everything else in this post grew out of that seed, one annoyance at a time. The linking showed up when facts started needing two homes. The search showed up when I couldn’t remember which file held a decision. The verbs showed up when I noticed I was avoiding my own system because it cost too much to use.

The public repo, Build With a Memory, includes the seed version: a single prompt you paste into an agent to install the basic structure in an existing project. The tools from this post are in there too, under tools/: the search indexer, the query tool, the graph generator, and the link-health check, generalized to run on any folder of notes.

Writing it down costs less than re-explaining it.

What are you building into your repos?

I’d honestly love to hear how other people are handling this. If you’ve got a memory setup for your agents, even a messy one, what’s the piece that actually gets used? And what died from neglect?

Credits

The network is my own build, but the way of working it grew out of is not all mine. A talk by Matt Pocock flipped committed project memory from something I toyed with into how I actually work. The method repo keeps the full credit record in CREDITS.md. If this post uses an idea I have not credited there, tell me and I will fix it.

Related

I send a short letter when there's something worth saying: what I built, what broke, one thing I learned. No gates, no popups.