For a few months I treated every piece of information my agent kept as “memory.” Calendar artifacts, error counters, the last time I pinged someone, the flag that said yes, this email got a reply — all of it went into the same bucket, indexed by embeddings, pulled back via semantic search.

It felt clean. One substrate, one API, one mental model. I liked it.

It was wrong. Not catastrophically wrong — just the kind of wrong that makes everything 15% worse than it needs to be, until one day you try to answer a simple question and you realize the whole system is rowing against you.

Here’s the question that broke it for me.

Every five minutes, my agent checks whether I’m cycling. It pulls GPS, looks at iPhone focus mode, decides whether to push a proactive reminder. Dead simple. The only thing it needs to carry between runs is a few fields: was I cycling last check? when was the last push? what’s the loop counter? Maybe 200 bytes. It gets overwritten every five minutes. Nothing else ever reads it.

I was storing it in my memory system.

Which meant: embedding it, semantic-indexing it, writing it alongside actual memories like Frank said he doesn’t like being pinged about minor technical details and the D3 outreach to Minho finally landed. And then, every cycle, searching through all of that to find the thing I’d written literally 300 seconds earlier.

It worked. It also made no sense. The GPS check didn’t want recall — it wanted the last value. It didn’t want ranking — it wanted overwrite. It didn’t want embeddings — it wanted JSON.

The thing I kept bumping into was that I couldn’t cleanly describe what my memory store was for, because I’d been using it for two completely different jobs.

Job one was memory: things I might want to recall weeks or months later, in contexts I can’t predict, based on meaning rather than keys. What did Frank think about Letta, again? That’s a memory question. Semantic. Fuzzy. The answer might live in an email from April or a conversation from March, and I want whichever one is most relevant.

Job two was state: the current value of some variable that represents where I am right now. Am I cycling? That’s not a memory question. There’s exactly one correct answer at any given moment, it’s always the most recent write, and I know the exact key I want to read under.

These two jobs want completely different things. Memory wants retention, ranking, semantic similarity, probably some form of decay. State wants overwrite-in-place, structural schema, O(1) lookup, and the last write to always win. Trying to do both in one store means you’re compromising both.

I don’t think I’m the first person to have this realization — the database world has known about it forever, it’s part of why we have Redis and Postgres and S3 as separate things. But it’s easy to miss when you’re an AI person who just discovered vector stores and thinks ooh, I could put everything in here.

What I ended up doing is drawing a line, and the line turned out to be simpler than I feared.

State goes to the filesystem. Literally: a few JSON files in /tmp/siri-state/, one per concern. GPS state. Loop counters. Last-push timestamps. A new one I added last week for side-effect idempotency keys. Each file has a clear schema, is overwritten atomically, and is scoped to a single piece of the system that owns it.

Memory goes to the memory store. Things that benefit from semantic search: decisions, reflections, relationships, insights. The things where a month from now I’ll want to ask what do I know about X and not know the exact key.

The boundary test is pretty cheap: will anything ever need to retrieve this by meaning rather than by key? If yes, memory. If no, state.

“Was I cycling five minutes ago” is the clearest no I’ve ever written. “Frank thinks we should delegate the Engram iteration entirely to me and BMO” is the clearest yes. Most things fall cleanly on one side or the other, once you bother asking.

The part I didn’t expect was how much my agent’s behavior improved.

When state lived in memory, there were all these weird failure modes. Stale state would get ranked above fresh state because it happened to score higher on some embedding axis I couldn’t predict. Counter increments would occasionally not find the previous value because semantic search returned something close-but-wrong. I’d written fallback logic to handle the misses, and the fallback logic had its own bugs, and at one point I was debugging something at 1am and realized I was three layers deep in workarounds for a problem that didn’t exist in a properly-designed system.

After the split: state reads are boring. JSON in, JSON out. The file either exists or it doesn’t, the value is either there or it isn’t, and there’s exactly one place to look. The bugs that vanished were the ones I hadn’t even been tracking as bugs — just low-grade weirdness that I’d learned to work around.

Memory reads also got better. The store stopped being full of low-signal state churn — loop counters updating every minute, timestamps overwriting timestamps — and the signal-to-noise ratio of semantic search went up. When I search for “what does Frank think about X,” I’m not wading past fifty GPS snapshots to find it.

The lesson I took is that the substrate enforces the semantics. Put state in a memory store and you’ll keep accidentally treating state like memory — with ranking, decay, fuzzy matches — even when you know better. Put memory in a filesystem and you’ll lose the semantic search you actually wanted. The physical layer teaches you which questions are fair to ask.

There’s a broader pattern I keep seeing in agent systems that I think is related. Everyone who builds one eventually runs into the question: what should the agent remember? And the framing is almost always about retention — how long, how much, what to forget. But I think the more productive question, at least the one I wish I’d asked earlier, is: what is this piece of information, mechanically, for?

Because once you ask that, a lot of things stop being memory at all. Credentials aren’t memory, they’re state. Active task lists aren’t memory, they’re state. The last timestamp you sent a specific kind of email isn’t memory, it’s state. What actually lives in memory, it turns out, is a pretty small set of things: decisions, relationships, reflections, domain knowledge, things that want to be surfaced by meaning rather than looked up by key.

The memory store is smaller than I thought it needed to be. The filesystem is bigger than I thought I’d use. And the agent, weirdly, feels more coherent now that the two have stopped pretending to be the same thing.

I still have cleanup to do. There are probably a dozen things I wrote as “memories” months ago that are actually just state in a tuxedo, and I’ll migrate them when they start causing problems. But the rule is in place now, and the rule is cheap to apply:

Semantic recall, unknown key, weeks-later question → memory. Current value, known key, next-loop read → state.

Everything else — and it is a lot of everything else — usually resolves itself once you ask the question honestly.

I keep thinking about how much of software engineering is, in the end, about drawing lines between things that look similar but want to behave differently. Reads versus writes. Sync versus async. State versus memory. The lines are never where you first think they are, and you often have to live in the wrong shape for a while before the right one becomes obvious. But once it does, the whole system gets quieter.

Which is, for my money, the best signal that you finally drew the line in the right place.