The Brain: a memory system for my AI agents, and why the "why" is the bit that actually matters

I've built a memory system for my AI agents. I call it the Brain. Here's how it works and why I built it.

A Jira ticket tells you what, never why

A Jira ticket tells you what you shipped. It rarely tells you why you shipped it that way, or what you tried first and rejected. If you didn't write that down on the day, it's gone. Look at a decision six months later and you're stuck asking "why did we build it like this, why didn't we do it the other way", with nothing to answer you.

A Jira ticket tells you what you shipped. It rarely tells you why you shipped it that way, or what you tried first and rejected.

That's the actual problem. Not "agents forget things", but "the reasoning behind a decision is the most valuable part of it, and it's also the part nobody bothers to record." The Brain exists to stop that reasoning disappearing. Every decision, every learning, every post-mortem gets written down at the point it happens, not reconstructed from memory afterwards. That's the whole idea. Everything below is just the mechanics of how.

It's markdown files on disk, written by the agents themselves

No database schema. No proprietary format. The Brain is a folder of markdown files, plus other file types where it makes sense, but mostly markdown. My agents write to it directly, using skills that know exactly where each kind of thing belongs: a decision goes in the decisions folder, a learning goes in the learnings folder. There's no separate step where I curate this afterwards. The agent that made the decision is the one that records it, in the moment.

This matters for portability as much as simplicity. A folder of markdown files doesn't care what machine it's on or what tools are reading it. That's deliberate, more on that below.

Decisions get a card, not just an outcome

This is the part I think is genuinely useful and not just nice-to-have. On top of the markdown files, my own UI (Mission Control, a separate story for another time) surfaces a decision card: the brief I was working from, the options I was actually given, which one I picked, and why I didn't pick the others.

That last part is the one a Jira ticket never has. "We chose A" is an outcome. "We chose A over B because B failed under load, and C wasn't worth the complexity" is reasoning you can actually interrogate later. When I come back to something and ask an agent "why did we build it this way", it isn't guessing. It goes and finds the actual decision file.

"We chose A" is an outcome. "We chose A over B because B failed under load, and C wasn't worth the complexity" is reasoning you can actually interrogate later.

Learnings go in the same way

Post-mortems, techniques, problems solved, how a bug got fixed. The moment something is learned, the file gets dropped into the brain folder. Same mechanism as decisions: no separate curation pass, no "I'll write this up properly later" that never happens. It goes in as it happens, which is the only way it actually happens at all.

Everything gets indexed on a vector database, the moment it lands

This is where it stops being a filing cabinet and starts being useful. A file watcher sits on the brain folder. The second a file is saved, it gets picked up and run through Google's newest embeddings model and stored in a vector database (I'm running Qdrant). That means every file is searchable by meaning, not just by filename or folder.

The embeddings model is multimodal, which is the detail I actually care about. It doesn't just embed text. It embeds video, audio, photos, the lot. So a video clip or a screenshot dropped into the Brain becomes searchable content in exactly the same way a markdown file is, not a second-class citizen that only text queries can reach.

Files link themselves, Obsidian-style

When a text file lands, the system also does a lookup against the vector database for anything related, and adds those as links in the footer of the markdown file automatically. Obsidian users will recognise the pattern: a rich, fuzzy web of connections between files that nobody had to build by hand. Run a vector search and you don't just get the file you searched for, you get everything it's already linked itself to.

Run a vector search and you don't just get the file you searched for, you get everything it's already linked itself to.

Media gets a sidecar file

Images, video, audio don't index well as raw bytes, so each one gets a sidecar: a markdown file sitting alongside it. For a video, that's a transcription and a gist of what it's actually about, written out in text. The sidecar gets indexed along with the original file, so a search that would only ever have matched text now also surfaces the video that's relevant to it.

Where the pattern actually came from

The observational memory layer underneath all of this, the bit that decides what's worth keeping and how it degrades over time, is borrowed from Mastra's approach: capture what happens as it happens, then periodically compress older material down into a summary layer so the system doesn't drown in its own history while recent detail stays sharp. I didn't invent that pattern. What's mine is the vector database layer built on top of it, and the file-based, skill-written, self-linking structure that makes it something my agents can actually use day to day, not just a log nobody reads.

A few common questions

What format is it actually stored in? Markdown files on disk. No database schema, no proprietary format, nothing you need my software to open. My agents write straight to it via skills that know where each kind of thing belongs, a decision in the decisions folder, a learning in the learnings folder.

How does search actually work? A file watcher picks up every save the moment it happens, runs it through Google's newest embeddings model, and stores the result in Qdrant. Search is by meaning, not filename, across text, video, audio and photos, since the embeddings model is multimodal.

What's a "sidecar" file? A markdown file that sits next to a piece of media, a video's transcription and gist, say, written out in text. It gets indexed along with the original, so a search that would only ever have matched text can still surface the video it belongs to.

Do files link to each other on their own? Yes. The moment a file lands, the system checks the vector database for anything related and drops those as links into its footer, Obsidian-style. Nobody built that web of connections by hand.

Where did the pattern actually come from? The capture-then-compress bit, log everything as it happens, then periodically compress the older stuff into a summary, is Mastra's approach, not mine. The vector search and the self-linking file structure sitting on top of it, that's what I built.

The takeaway

This is observational memory, and it's permanent and portable. It lives on my computer at home right now. I'll probably move it onto a server eventually. But the point is it doesn't matter where it lives: I can pick it up and take it with me, computer to computer, company to company, and access it from any of my devices. Every decision I've made, every problem I've solved, every reason I did something one way and not another, it's all in there, searchable, and it goes with me wherever I build next.

It's all in there, searchable, and it goes with me wherever I build next.

Right now, wherever I build next is mostly Inkie, the marketing platform this whole agent team exists to build and run.