I wanted to give my agent a memory. I needed a substrate.

I’ve been working with LLMs for a few years, mostly building tools for myself rather than using them as packaged products. The project I kept circling back to was a knowledge-base system in two pieces. The first piece was the capture layer: a place to save anything I came across — articles, papers, YouTube videos, X threads, screenshots from meetings, the documents that flow through email. The second piece was an analysis layer on top of it — something I could query against the saved corpus to verify a claim, follow a thread of thought across years of reading, or make sense of a fast-moving topic where the signal-to-noise ratio is brutal. The use cases were heterogeneous on purpose: investments, geopolitics, research I half-understood, AI workflows where the genuine signal is scattered across talks, papers, READMEs, and three-month-old tweet threads, an Instagram post or product photo or piece of design work I wanted to file away for later inspiration, the rabbit hole from a Tuesday afternoon I wanted to find again on a Saturday. The common shape was the same in each case — I had captured something; I wanted to look at it later, structured; often through an LLM. I wanted something like X’s “ask Grok,” but aimed at a corpus I trusted and durably preserved, because the original posts and articles I cared about kept disappearing, getting edited, or getting taken down.

The analysis layer is moving fast on its own. Better models, better retrieval, better agent frameworks — the loop is shortening every few months. The capture layer was the part I expected to already be solved, and it isn’t.

Saving a YouTube video into something I could later query — the video, the transcript, the title, the channel, the chapters — was harder than it should have been. There was no single package that took X threads and Reddit comments and long-form articles and YouTube and PDFs and returned them through one consistent shape. For each individual source there was usually a library, sometimes a paid API, sometimes a fragile scraper one platform-change away from breaking. Stitching them together and getting structured payloads back was its own project.

The web clippers from each PKM destination — Obsidian’s, Notion’s, Raindrop’s, Pocket’s — each locked you into that destination’s ecosystem. And each required you to be on the page in a browser tab with cookies active, so a program running on a server couldn’t use them. There was no open SDK I could point an agent at and say: download this URL, parse it, give me a typed payload, store it where I tell you. The closest thing was building the substrate myself.

So I did. And the more I built it, the more I noticed the same shape underneath. It wasn’t a capture layer. It was a substrate. The destination wasn’t fixed; the source wasn’t fixed; what was fixed was the shape underneath them.

Once you see the substrate shape, you start seeing where it’s been missing all along.

It is not a new problem. It is the same problem every “save it for later” tool has had since the bookmarking era. Pocket was a destination. Evernote was a destination. Diigo was a destination. Notion’s Web Clipper, Raindrop, Instapaper, Pinboard, the dozens of read-it-later folders nested inside various browsers — all of them are destinations. You capture into the destination. The destination holds the content. When you want the content back, you go to the destination.

This sounds tautological until you notice what it implies. The destination is the substrate. The substrate is the destination. There is no separation between what holds your captures and what you interact with them through. Which means when the destination dies — when Pocket gets sunset by Mozilla, when Diigo quietly stops developing, when Evernote drifts into the sea of decline that all 2007-era web apps eventually drift into — what dies isn’t the application. What dies is your archive.

The most recent and most legible example is Pocket. In 2025, Mozilla announced Pocket was shutting down for good after years of trying to figure out what to do with it. The shutdown was orderly. There was an export — a CSV file with URLs and saved-dates and whatever tags users had stopped maintaining after the first eighteen months. The bytes survive. The URLs work, mostly. The article a Pocket user saved in 2017 is technically retrievable, if they are sufficiently determined. What is gone is the archive — the ability to type three words into a search box and find the thing you vaguely remembered; the ability to feel, as you added one more capture, that you were adding it to a thing that was going to be there next week. The exports are inert files. The substrate is gone; dead substrate means the captures are technically present but functionally lost.

This is not a Pocket problem. It is a structural problem with the way personal-web tools have been built since the bookmarking era. Capture-into-destination is the default architecture. Every tool inherits it. Every migration between tools breaks because of it. Every shutdown is fatal because of it. And every agent that you want to give a memory to runs into the same wall — there is no substrate to recall against; there is only a graveyard of destination-shaped applications, each holding its own captures hostage.

The question I’ve spent the last year working on is the obvious one. What would it look like if the substrate were separate from the destination? What would it look like if your captures lived in a thing that wasn’t a tool?

What a substrate actually is

The word “substrate” comes from chemistry and biology before it comes from software. A substrate is the underlying layer on which something else grows or runs. The petri dish that the culture grows on. The medium that the enzyme reacts in. The earth that the moss colonizes. The substrate isn’t the thing; it’s what the thing depends on to exist.

When we talk about software substrates, the analogy holds. A substrate is the underlying layer on which applications, surfaces, integrations, and behaviors are built. The application is the thing the user sees. The substrate is what makes the application possible — what holds its data, defines its primitives, enforces its invariants, and outlives any single deployment of it.

A personal-web substrate, in the specific sense I’m using the word, is the layer that holds the bytes and the structure of everything you have captured from the internet — separately from the application you currently use to interact with those captures.

Concretely, a substrate has three properties that a destination does not:

1. It is independent of the application that consumes it. You can swap the application without rebuilding the substrate. It predates the application and outlives it.
2. It speaks a known shape. A substrate doesn’t store opaque blobs. Every capture has a type, a schema, and a known set of fields. Other software can read the substrate without negotiation.
3. It preserves what was actually captured. Not a renderable approximation. Not a paraphrased “main content” extract. The substrate holds the raw bytes the source returned, in addition to whatever structured extraction its own renderers produce on top.

These three properties — independence, knowability, preservation — are what distinguish a substrate from every save-it-for-later application that has come before. They are not features you can bolt onto a destination. They have to be the foundation.

None of this is new in the abstract. That your data should outlive the application that produced it is the local-first principle; that your notes are just files you own is what plain-text tools like Obsidian are built on. What hasn’t existed is that principle applied to the heterogeneous mess you capture from the web — a tweet thread, a Reddit discussion, a YouTube transcript, a PDF — each pulled into one typed, durable shape that other software can read without negotiation. That is the gap Khiip is built for.

Substrate vs destination

The clearest way to see the distinction is to draw it.

Destination model — every save-it-for-later tool, 2008–2025

One place both holds and serves your captures — capture and recall close on the same box. Remove it and the loop breaks: the archive leaves with the app.

Substrate model — what Khiip is

Capture flows one way: you → daemon → vault. Recall flows back to you from the vault, through any tool you like — Khiip’s built-in local recall, or grep, your own vector store, another agent, other apps. Most of those read the vault directly and bypass the daemon entirely. Remove Khiip and the vault is still plain Markdown you own — the daemon is just another surface.

In the destination model, capturing and consuming both happen at the same place. The arrows form a closed loop. If the destination disappears, the loop is broken; the bytes survive but the loop is gone.

In the substrate model, capturing happens once, into the substrate. Consumption happens through whatever surface you currently prefer — your Obsidian vault, an LLM agent, a REST endpoint your own application talks to, or a destination that doesn’t exist yet and will be built three years from now. The substrate is the one thing that stays put as the surfaces around it change.

The substrate model is what makes destinations replaceable. Not interchangeable — different destinations expose different things and work in different ways. But replaceable. If a destination shuts down or stops being useful, you switch to another one without losing your captures. The substrate is what makes that switch a configuration change rather than a migration project.

Typed payloads

The second property of a substrate is that it speaks a known shape. In Khiip, that shape is a typed payload.

Every capture in Khiip is a structured object with a declared type. A tweet is a TweetPayload. A Reddit thread is a RedditPayload. A YouTube transcript is a YouTubePayload. A long-form web article is a WebPayload. A Wikipedia article is a WikiPayload; a PDF is a PDFPayload. Each of the six source types has its own payload, with its own fields — fields that make sense for that source and are validated when the capture comes in.

Capture

• idULID
• urlHttpUrl
• sourcex | reddit | web | wiki | youtube
• recorded_atdatetime
• valid_fromdatetime
• extraction_status
• source_artifact_pathfile:// URI
• payloadSourcePayload

payload is one of —

TweetPayload kind = "x"

• textstr?
• handlestr?
• engagementEngagementCounts?
• quoted_tweetTweetPayload?
• articleXArticle?
• medialist<Media>

RedditPayload kind = "reddit"

• titlestr
• subredditstr
• author_usernamestr?
• post_type
• body_textstr?
• commentslist<CommentNode>
• comments_truncatedbool

WebPayload kind = "web"

• titlestr
• body_textstr
• descriptionstr?
• author_urlstr?
• site_namestr?
• article_typestr?

YouTubePayload kind = "youtube"

• titlestr
• uploaderstr?
• duration_secondsint?
• chapterslist<dict>
• transcriptlist<dict>
• is_shortbool

The reason this matters is not theoretical. When the substrate knows that a thing is a tweet, it can render that tweet differently than a long-form article. When the substrate knows that a tweet quotes another tweet, the quoted tweet is preserved as a nested structure — not flattened into the body, where the quote-versus-original distinction is lost. When it knows a Reddit thread has a comment tree, that tree is preserved as a tree, parent-child links intact. When the substrate later hands a capture to an agent, the agent can read structured fields (“who wrote this,” “when was it published,” “how many people engaged with it”) rather than parsing them out of raw HTML.

Untyped substrates exist. Most destinations are effectively untyped — the underlying storage is “title, body, tags, date.” That works for the common case. It silently loses everything that doesn’t fit the common case. Khiip’s typed-payload taxonomy is what makes it possible to capture a Reddit thread with 400 comments and have the comment tree still be a tree six months later, instead of a flat run of paragraphs.

A concrete example. When you capture a Reddit thread, the substrate preserves the post body, the author, the subreddit, and the comment tree — including the parent-child relationships between comments, the score on each comment, and any quoted text blocks within them. Six months later, an agent querying the substrate for “what did I save about backup strategies on r/selfhosted” gets back the actual comments where the tradeoffs were argued, with author and score context intact. Untyped storage would have given back a flat text blob with the structure stripped out. The shape that was captured is the shape that comes back.

Source-tier preservation

The third property is that the substrate preserves what was actually captured. Not just the rendered extraction. The raw bytes.

In Khiip, every capture has two layers. The canonical layer is the typed payload — the structured, validated, queryable representation, written to your vault as YAML frontmatter you own. Beneath it is what the substrate calls the source tier: the raw HTTP response, the raw HTML, the raw JSON from a platform API, the raw transcript file, the raw image bytes — gzipped on disk under a single data directory, keyed by the capture’s identifier. The source tier is a best-effort cache by design: it can be wiped, or never written, and the capture stays intact, because the typed payload in your vault is the record of account. The split is deliberate — the payload guarantees the capture stays readable; the source tier, when present, keeps the raw material a better processor can re-derive a richer payload from later. So the extraction is never the only thing you kept, and a richer re-extraction is an upside when the raw bytes survive, not a promise that breaks when they don’t.

This sounds redundant until the first time you need it. Extractors are written against the version of the source that exists today. Three years from now, Twitter will have changed its DOM. Reddit will have re-shaped its JSON. The article you saved from a now-defunct blog will only exist in the raw HTML you captured. If the substrate only kept the rendered extraction, you would be hoping the renderer’s output is enough. Because it also kept the raw bytes, a better extractor can be run over them whenever the original extraction becomes inadequate — a different schema, an LLM that pulls more out than the original rules could — without needing the source to still exist.

The principle is older than software. The Internet Archive operates this way; it preserves the captured HTML, not just a stripped-down rendition. Newspaper morgues operated this way. The reason is the same: rendering is a lossy projection of the source, and you cannot un-lose what the projection threw away. Keeping the source means you can re-project as many times as you need to.

This is also why the substrate records two different times for everything it holds: when you captured a thing, and when the thing itself claims to be from. Re-capture a page later and the substrate supersedes the old version rather than overwriting it — the earlier capture, and the raw bytes behind it, stay exactly as they were. So when a post is quietly edited, or deleted, or taken down, you still have what it said the day you saved it, sitting alongside whatever it says now. That is the direct answer to the thing that started all this: the articles and posts I cared about kept disappearing, getting edited, getting taken down.

Each capture is self-contained: a typed payload in your vault, the raw bytes on disk, and an embedding in the recall index. A re-fetch re-pulls the live URL and writes a whole new capture; the old one is marked superseded_by the new id but never deleted. URL look-ups resolve to the latest — but recall searches every version’s embedding, so you can surface what a post said when you first saved it and what it says now.

Agent-as-memory pattern

There is one more property of a substrate that matters for a category of user that barely existed five years ago: people building persistent LLM agents — the kind you come back to and expect to remember what you told them last week.

A typed substrate with stable identifiers and structured fields is, almost by accident, the thing an LLM agent’s long-term memory should be built on.

The problem with using an LLM agent for actual work is that the agent has a context window and the context window is short. You can paste a few thousand tokens of background in; the agent reads them; you ask a question; the agent answers. The next time you ask the agent a question, the context window is empty again. Whatever you wanted the agent to remember has to be pasted back in.

The standard workaround is to bolt a vector store onto the agent, embed your documents into it, and call the result “memory.” It works, sort of. But it is worth being precise about what you have built. Done the naive way — chunk the text, embed the chunks, treat the index as the thing of record — it dissolves structure at the door: the comment tree becomes a flat run of passages, the quoted thread loses its relationship to the reply, the author and the engagement counts fall away, and similarity search over chunks is the only handle you get back. Modern vector databases can store typed metadata alongside the embeddings, and the good ones push you to — but that metadata is only ever as structured as whatever you remembered to attach at ingestion, and the index, not the source, is still the durable artifact. The word “memory,” used this way, papers over the fact that there is no durable, structured thing underneath it.

A substrate is the durable, structured thing underneath. Khiip ships its own semantic recall over your captures — a small embedding model that runs on your own machine, with no API key and nothing leaving it — so you can ask, in plain language, “what did I save about substrate design last quarter” and get ranked hits back. But the recall is one surface on the substrate, not the substrate itself. Because the captures are also just typed files, you can grep them, point Claude Code at the folder, or hand them to whatever memory framework you already use through the same REST and MCP interface. The structure that was captured is still there for any of them to read — the comment tree is still a tree, the author is still the author — instead of having been dissolved into chunks at ingestion.

This is close to what Andrej Karpathy has been calling an “LLM wiki” — a personal, plain-markdown knowledge base an agent reads and maintains, so answers compound instead of being rediscovered from scratch on every query. That pattern assumes you already have the content sitting in clean, structured files. Getting it there — pulling a tweet thread, a Reddit discussion, a YouTube transcript out of a hostile platform and into a typed, durable shape — is the layer underneath the wiki. It is the layer Khiip is.

This matters more as agents persist. When the embedding model changes — and it will — any index built on it is invalidated and has to be regenerated. The question is what you regenerate it from. If the index was the durable artifact, you are rebuilding from chunks, and whatever structure was lost at ingestion is lost for good. Khiip’s recall index is disposable on purpose: it is a view over the typed captures, not the record itself, so when the model changes the index is simply rebuilt from the intact source — the comment tree, the author, the dates all still there. A substrate is shaped by the source, not by the consumer. When you swap your agent for a better one next year — different reasoning, different context window, different cost — the substrate doesn’t change. The same captures, in the same shape, are available to whatever agent you’re using. The agent is a surface. Surfaces come and go. The substrate is the fixed point.

What Khiip is trying to preserve

What I’ve described — the substrate-vs-destination distinction, typed payloads, source-tier preservation, agent-as-memory pattern — is what Khiip is at v0.1. It captures from six source types today (X, Reddit, web articles, Wikipedia, YouTube, and PDFs), renders into a vault structure compatible with Obsidian, exposes a REST and MCP API — including a local semantic recall over everything you’ve captured, running on a small model on your own machine — and stores everything on the machine in front of you. Those six are the URL-addressable subset of the wish-list I opened with; the files, images, and inboxes from that list are not part of v0.1. It is AGPL-3.0 for the daemon, and Apache-2.0 for the SDKs (a separate repo, when published). The license, the source, and the design decisions are public. The code is at github.com/KhiipAI/khiip; the design decisions are recorded as ADRs in the same repository.

There is a name forming for the shift this is part of. Tiago Forte, who spent a decade teaching people to manage what they know, has been arguing since 2025 that personal knowledge management is being overtaken by personal context management — that the bottleneck is no longer what you have read but whether you can hand a model the right material, in the right shape, at the right time. A substrate is the unglamorous bottom of that stack. It is the part that has to reliably hold the material, in a shape something else can use, before any of the managing or recalling happens on top.

I am not going to pretend Khiip has solved the problem that Pocket’s shutdown made visible. The problem is structural, and structural problems take more than one product to fix. What I will say is that the v0.1 substrate is designed to be the right shape for that work — the typed payloads are not retrofitted, the source-tier preservation is not optional, and the substrate has no built-in destination that has to outlive itself for the captures to remain useful.

And the test that matters most is the one this essay started with. If Khiip goes the way of Pocket — and every piece of software eventually goes some way — what happens to everything you captured? Nothing happens to it. The captures are markdown and YAML sitting in a folder you already own. You can grep them, open them in Obsidian, read them in any text editor, and re-index them with whatever comes after Khiip. The recall index would go cold, but the index was always disposable; it rebuilds from the captures, and the captures never needed Khiip to be readable. And the daemon itself is not a black box that leaves when I do: it is AGPL by design, the source is public, so anyone can fork it and keep it running. That is the whole point of a substrate, turned on the substrate itself: the daemon is just another surface, and surfaces are allowed to die.

The long-arc horizon is one I think about quietly. The same substrate that makes captures replaceable across destinations could, in principle, make captures legible across people — a shared layer where opted-in users contribute structured captures of the open web back to a commons, in the spirit of the Internet Archive but with the typed, queryable shape that modern reading and modern agents both need. That is a v1.0+ horizon, not a v0.1 promise. The v0.1 substrate is designed to be capable of supporting it if the future demands it. It is not designed to require it.

For now, the work is more modest. Make capture reliable. Make recall structured. Make the substrate the kind of thing that will still be readable, in the shape you saved it, when the next read-it-later tool you’re using shuts down. The bytes survive in any case. The substrate is what makes them feel like an archive, instead of an archaeological record.

If you save things from the web — and you almost certainly do — the substrate underneath those captures is going to matter more than the application on top of it. Eventually. Slowly. The way most structural shifts matter. Khiip is one attempt at making that substrate exist.