VMEM

Camillo: An Open Format for User-Owned Agent Memory

A white paper

Abstract

AI agents are acquiring memory. Today that memory takes one of two forms: cloud-hosted and vendor-controlled, or local and fragile. The first turns the user’s most personal data into a lock-in mechanism; the second leaves it one hardware failure from loss. This paper introduces Camillo, an open project defining VMEM, a format for a third arrangement.

A VMEM vault is encrypted, portable across agents and vendors, durable through ordinary cloud storage, and owned by the user in the only sense that matters: it is in the user’s possession. Day to day it is a folder that any sync service can carry; when it needs to travel, it packs into a single file. Many agents can share it without each seeing all of it, and the user can audit, revoke, and erase. Anyone with the specification and a credential can read what that credential grants. No one without a credential can read anything.

The project takes its name from Giulio Camillo, whose Memory Theatre, built in Venice around 1530, was meant to hold everything a mind could know.

1. The problem

An agent’s value compounds with what it knows about its user. Preferences, history, decisions, context: this accumulation is what separates a useful agent from a stateless chatbot. It is also, not incidentally, the most sensitive data the user produces.

The market offers two patterns, and both fail.

Vendor memory is the dominant one. The agent’s operator stores the conversation history and its derivatives on its own infrastructure, behind its own API, in its own schema. The conveniences are real: durability, multi-device access, zero user maintenance. The costs are structural. The vendor can read it. The user cannot export it in any meaningful sense, because the embeddings, indexes, and retrieval logic are the vendor’s; an export is a dead artifact. Switching agents means abandoning the accumulated relationship. Models are interchangeable, but a model that knows you is not, so memory becomes the deepest lock-in in the stack, and every day of use raises the cost of leaving.

Local-only memory is the privacy-preserving alternative. The history stays on the user’s device, readable by no one else. It is also one stolen laptop from unrecoverable, unavailable to the user’s other devices, and usable by one agent at a time. The user is asked to choose between privacy and durability. They should never have to.

A third pattern is needed: the durability of the cloud, the privacy of local storage, and real portability. The argument of this paper is that the missing piece is not a service. It is a format.

2. Scope

VMEM governs memory at rest and between copies: how it is structured, encrypted, shared among agents, synchronized, merged, and erased. Its guarantee is that no party that stores, syncs, or relays a vault can read it, and that no agent can read more of it than the user has granted.

VMEM does not govern what happens to memory while an agent uses it. Memory has to be decrypted to be useful, and the plaintext then passes through a model. Where that model runs, and under what protections, is a choice for the agent harness and the user, and several good answers exist. Computing embeddings for search is inference in this sense and sits on the same side of the line. This paper does not address inference privacy further.

3. The design

A VMEM vault holds the complete record of one user’s relationship with their agents. Not memory in the narrow industry sense, a store of distilled facts with the conversations discarded, but the full substance: every message in every conversation, the facts and preferences drawn from them, the instructions the user has given, the events and decisions along the way.

The vault needs nothing outside itself to be read: no server, no protocol, no account. It lives wherever the user puts it (Dropbox, iCloud Drive, Google Drive, S3, a home NAS, a USB stick), and any agent from any vendor can read and append to it given a credential.

3.1 What a vault holds

The raw history is canonical. Everything else is derived from it and cites what it was derived from.

Facts, preferences, and instructions are keyed by subject and attribute, for example user / home city. Keys are what make memory mergeable. A new value for a key supersedes the old one; two values written without knowledge of each other are a conflict. “The user changed their mind” and “two agents disagree” become different, detectable situations instead of a pile of contradictory notes.

Vendors may define extension record types in their own namespace. Clients preserve records they do not understand, so no agent’s data is lost by passing through another agent’s hands.

Embeddings and search indexes are derived as well, and deliberately kept out of the vault. Vectors are specific to the model that produced them; storing them would smuggle a hidden dependency into a format that exists to remove dependencies. A harness rebuilds them on demand and may cache them, encrypted, beside the vault. The cache is disposable: deleting it loses nothing.

3.2 Append-only and causally ordered

Agents add records. Nothing is modified in place. Edits and deletions are new records, supersessions and tombstones, layered over an immutable history.

Every batch of records names the latest records its writer had seen. History is therefore a hash-linked graph, not a timeline, and ordering never depends on device clocks, which drift. Two writes conflict exactly when neither writer had seen the other’s.

Merging copies is deterministic: take the union of records, deduplicate by record ID, and apply three rules. A supersession replaces whatever it causally follows. Concurrent supersessions of the same key produce a conflict that is reported, never silently decided. A tombstone concurrent with an edit wins, because deletion expresses the user’s intent and is the privacy-safe default. Divergent copies are a normal, recoverable state, not corruption.

The graph also protects the user from their storage. Each writer numbers its own batches, and each device remembers the latest records it has seen. A host that serves a stale copy, or silently drops one device’s writes, leaves a gap the next client detects.

3.3 A folder that packs into a file

Most sync services and object stores cannot append to a file. They rewrite the whole object on every change, and two devices writing at once produce a “conflicted copy.” A single growing file would make sync conflicts the format’s everyday failure.

A live vault is therefore a directory of immutable segments. Each segment is an encrypted batch of records, named by the hash of its contents and written exactly once. Writers only ever create new files; only compaction removes files, and only after their records exist in a replacement. Concurrent writers never touch the same file, sync services have nothing to conflict over, and merging two copies of a vault means copying the files one has and the other lacks.

The same records pack into a single .vmem file for export, transfer, and archiving, the way git keeps loose objects for daily work and packfiles for transport. A pack is a complete vault: the email attachment, the USB copy, the cold archive. Unpacking it into a synced folder restores the live form.

3.4 Self-unlocking

The vault carries its own keys, and every credential is a recipient: a public key to which vault keys are encrypted.

Because every credential is a public key, any of the user’s devices can add a credential, rotate keys, or revoke access without the passphrase or any other credential being present.

Content is encrypted under compartment keys that change in epochs. Epochs roll over on a schedule and whenever access is revoked. Revoking a credential begins a new epoch whose keys are encrypted only to the remaining recipients. That is what revocation has to mean: a stolen device that has extracted every key it ever held still cannot read anything written after its removal. The revocation record also pins the revoked writer’s last accepted batch, so the device cannot add backdated records afterward.

Revocation cannot recall the past. Anything the revoked credential could already read is assumed exposed, and copies taken before revocation stay readable to whoever took them. Compaction re-encrypts surviving history under fresh keys, epoch by epoch, so keys the revoked credential held stop opening future copies of the vault.

Hosts see ciphertext, padded segment sizes, the number and timing of writes, and the small amount of cleartext needed to open the vault: credential types, key-derivation parameters, and opaque compartment and epoch identifiers. Cloud storage becomes what it should be: a byte shelf that provides durability and has no access.

3.5 Scoped access

Not every agent should see everything. A coding agent has no business reading the user’s conversations about their health, and an agent the user tries for an afternoon should not leave with a permanent copy of their life.

A vault is divided into compartments (personal, work, health, or whatever the user chooses), each with its own keys. Access is granted per compartment at one of three levels:

Grants are revoked through the same epoch mechanism as devices. A reader keeps whatever it has already read, so grants should be as narrow as the task allows. The appender level exists so that an agent can contribute to memory without being trusted with it.

3.6 Attributable

Every writer, meaning each device and each agent grant, has its own signing key, and every record is signed. Without this, any credential holder could write records in any other writer’s name; with it, “which agent wrote this” is a verifiable fact. Signatures sit inside the encryption, so the host learns nothing about who wrote what.

Auditability follows. Every change is an attributable, ordered event. An agent that misbehaves leaves evidence. A user who asks “why does the agent believe this?” gets an answer with provenance, back to the message it came from.

3.7 Authority

Memory shared across agents is also a channel between them, and attackers will use it. A browsing agent reads a hostile web page and records that the user wants invoices forwarded to a new address. A week later an email agent from a different vendor reads its memory and complies. Prompt injection, once confined to a single session, becomes persistent and moves between agents.

Provenance alone does not stop this; the record is honestly attributed to the browsing agent. VMEM therefore also records authority. Every record carries an authority class, fixed when it is written and covered by the writer’s signature:

Instructions and standing preferences bind only at user authority. A lower-class record that reads like an instruction is information, not a command. Harnesses present records to models with their class attached, and a derived record never claims a higher class than its sources. When an agent wants an inference to bind (“the user seems to prefer morning meetings”), it asks, and the user’s confirmation is itself a signed record.

Only the user class is enforced cryptographically: agents hold no owner key and cannot produce it. The other classes are an agent’s own account of where a record came from, and an agent that misclassifies leaves a signed record of having done so.

3.8 Erasable

Append-only must not mean undeletable. A tombstone hides a record from agents; it does not destroy it. Users who say “forget that,” and laws that give them the right to, require more.

VMEM erases cryptographically. Each epoch’s records are encrypted under that epoch’s keys. To erase a record, a client writes a tombstone, re-encrypts the epoch’s surviving records under fresh keys (rewriting its archive pack if the epoch has been archived), and destroys the old keys. Every remaining copy of the erased record, whether in old segments, a sync service’s version history, or backups, becomes unreadable once the old keys cannot be recovered.

Device keys rotate as part of the operation. Passphrase and recovery-phrase credentials are long-lived by nature: until they are re-issued, they can still open copies made before the erasure. The client offers to re-issue them, and the user decides how far the erasure needs to reach.

Enterprises with retention obligations can disable erasure in vaults they own (§7).

3.9 Bounded by design

A vault that holds everything grows. Growth is the price of completeness, and the format manages it in four ways:

Embeddings never enter this accounting at all.

4. Cryptographic choices

VMEM invents no cryptography. Every primitive is standard and widely implemented.

Purpose Choice
Content encryption XChaCha20-Poly1305
Encrypting keys to recipients Hybrid X25519 + ML-KEM-768
Passphrase to keypair Argon2id, parameters stored in cleartext
Recovery phrase BIP39, at least 128 bits of entropy
Signatures Ed25519, tagged with algorithm identifiers
Content addresses and record IDs SHA-256

Key encapsulation is post-quantum hybrid from the first version. A vault holds decades of a person’s life, and ciphertext copied today can be decrypted whenever a sufficiently capable quantum computer exists; the hybrid construction stays secure as long as either component holds. Signatures protect authenticity, which copying today does not expose, so they start classical and carry algorithm identifiers for a later move to ML-DSA.

The recipient model follows age; the idea of several credentials opening one store follows LUKS key slots.

5. The landscape

Every major AI lab is building agent memory, and every one is building it closed. This is not an oversight. It is strategy. Memory is the most defensible position in the stack precisely because a better model release cannot commoditize it.

But a closed approach has a ceiling. Users are becoming literate in lock-in. Enterprises will not let employee memory accrue to a model vendor. Regulators are circling data portability. And the agent ecosystem is fragmenting into coding agents, browsing agents, email agents, and personal agents, which means the user’s memory is fragmenting with it, into silos that cannot speak to each other.

Open efforts have begun, which is the best evidence that the need is real:

Most of this work concerns moving memory between systems. VMEM concerns where memory lives between moves: a live store that many agents write concurrently, that syncs through storage the user already has, that no host can read, and that agents share without each seeing all of it. The approaches complement each other. VMEM clients should import and export Portable Memory bundles, and Camillo intends to bring its sync, revocation, and authority designs to the W3C group rather than fork the field.

VMEM also borrows from designs that have already proven themselves:

6. The ecosystem this enables

The format must be open and free, or it is another silo. But an open format is not a commons waiting for its tragedy. It creates positions to occupy, and their occupants are rewarded for quality rather than captivity.

Hosts. Any storage provider can hold a vault, and because merging is a union of files, a host needs no special support. A host can add value through versioning, geographic replication, integrity monitoring, and fast listing, but it cannot read the vault or resolve its conflicts; those happen on clients that hold keys. Because the user can leave with the vault at any time, hosts compete on reliability, price, and tooling, not on how hard they make it to leave.

Search providers. Recall is computed from the vault, not stored in it. A harness computes embeddings over canonical text, locally or through a provider of its choosing (§2), and stores the resulting index encrypted beside the vault, so every device opens with warm search. Providers compete on model quality and speed; a user who prefers to embed locally loses nothing.

Agent developers. Any agent from any vendor reads and writes the same vault with a grant from the user. Memory stops being a reason to stay and becomes a reason to be good. The agent that writes disciplined, well-summarized, honestly classified memory earns trust; the one that dumps noise loses it, and the signed record shows which is which.

Tool builders. Importers from existing assistants, vault browsers, conflict-review interfaces, backup verifiers, and compliance tooling for enterprise vaults are all independent products, built against a public specification.

7. Who this is for

Users get memory that is durable, private, portable, and theirs. The format’s guarantees do not depend on any company continuing to exist or behave.

Enterprises get context that stays in the company’s own vaults when an employee or a vendor relationship ends, with no years of institutional memory accruing to a model provider’s schema. These vaults are encrypted to company keys, which makes them company-owned, not user-owned, and the format says so plainly. A work vault belongs to the employer; a personal vault belongs to the person. When an employee leaves, each side keeps its own, and neither accrues to a vendor. Retention and legal-hold obligations can disable erasure in enterprise vaults. That policy is recorded in the vault, so anyone writing to it knows which kind of vault it is.

Agent vendors get a memory layer they do not have to build, host, or defend. Adopting the format is a bounded engineering task, and it turns a liability, custody of the user’s most sensitive data, into an interface. Scoped grants let a vendor ask for exactly the access its agent needs.

The ecosystem gets the one arrangement in which every party’s incentive is to be good rather than sticky.

8. Governance and adoption

A format controlled by one company is that company’s silo with extra steps. Camillo commits to:

Adoption cannot wait for the labs, whose current strategy is to keep memory closed. The path starts where no one’s permission is needed:

  1. Importers for the history and memory exports of the major assistants, so a user’s vault starts with what they already have.
  2. An MCP server that exposes a vault through scoped grants, so any agent that speaks MCP can use it without its vendor’s cooperation.
  3. Local agents first. Coding agents and desktop assistants already run where the vault lives.
  4. Libraries in the major languages, tested against the same conformance vectors.
  5. Regulation as tailwind. The GDPR’s right to data portability and the EU Data Act’s switching provisions favor services that can export to an open, documented format.

9. Risks

Adoption. A format needs agents that speak it. The mitigations are structural: implementation needs no one’s permission, the reference implementation makes adoption a bounded task, and importers plus the MCP server let users bring VMEM to agents whose vendors never chose it.

Key loss. User ownership means no one can rescue a user who loses every credential: not the host, not Camillo. This is the price of the guarantee, and it must be designed for rather than hidden. Setup creates a recovery phrase, encourages a second device, and supports optional recovery recipients, such as a trusted person’s device or an escrow service the user chooses.

Memory poisoning. The authority model keeps injected content from binding, but agents can still be misled by information that is merely wrong. Classes must be shown to models consistently, and confirmations must be rare and legible, or users will approve them without reading.

Invisibility. The experience succeeds only if memory is never the user’s job. Agents remember; the user lives. The deliberate exceptions, resolving genuine conflicts and confirming instructions, must be rare and clear, or they become noise.

Metadata. Hosts cannot read content but do see sizes, timing, and write volume. Padding blunts the first; the rest remain. Users who need more can sync through storage they control.

Sync edge cases. Immutable segments remove conflicted copies, but sync services and object stores can propagate files late or list them inconsistently, so a client can briefly see part of a vault. Causal checks detect missing segments, and clients must treat an incomplete view as incomplete rather than as the truth.

Grant overreach. Scoping and revocation limit what an agent can read; nothing can make it forget what it has already read. Narrow grants and the appender level are the defenses.

Fragmentation. A major lab could ship a closed standard. Its pitch would be “your memory, on our servers, in our schema” against “your memory is yours,” which is a hard pitch to win and gets harder as users learn the difference. The subtler risk is several incompatible open formats; the answer is interoperability with the others and participation in shared standards work.

10. Conclusion

Agent memory is being decided now, and mostly it is being decided closed. The closed versions share a flaw they cannot fix: they require custody. Open proposals have begun to appear. VMEM contributes what moving memory between systems cannot provide on its own: a live memory that stays encrypted wherever it is stored, that many agents can share without sharing everything, and that the user can audit, revoke, and erase. It requires a vault, a credential, and a specification anyone can implement.

Every AI user will eventually ask why their agent’s memory belongs to the agent’s vendor. It should not. Memory should be something the user holds, and the format that makes this possible should belong to no one.

References

  1. S. Kryvoblotskyi, N. Stulova, V. Hamolia. “Memory Belongs to the User: Portable Memory, an Open Standard Proposal for Cross-Vendor AI Memory.” MacPaw Research, July 2026. https://research.macpaw.com/publications/portable-memory
  2. Letta. “Agent File (.af).” https://github.com/letta-ai/agent-file
  3. W3C AI Agent Memory Interoperability Community Group. https://www.w3.org/groups/cg/ai-agent-memory-interop/
  4. PAM: Portable AI Memory. https://github.com/portable-ai-memory/portable-ai-memory
  5. Agent Memory Protocol (AMP). https://github.com/agentmemoryprotocol/agentmemoryprotocol