Large language models do not remember. Each request starts from whatever text is placed in front of the model, and when the request ends, nothing is kept. Everything that feels like memory in an AI assistant, from recalling your name to knowing last week's decision, is built around the model by a separate component. That component is the memory layer, and it sits underneath most products sold as a "company brain."
This explainer is for founders and operators deciding how much to rely on one. It defines an AI memory layer, explains why models need it, describes the kinds of memory researchers distinguish, and walks through how a memory layer writes, stores, retrieves and forgets. It then sets out plainly what a memory layer can do for a company and what it cannot, with a checklist for evaluating one.
Key takeaways
- A memory layer keeps information outside the model and selects what to bring back for each task.
- It is needed because context windows are finite and models use long contexts unevenly.
- It gives continuity: facts, preferences and history no longer have to be re-explained.
- It does not judge relevance to company goals, resolve authority between conflicting facts, own work or verify outcomes.
What is an AI memory layer?
An AI memory layer is the part of an AI system that saves information from past interactions and sources, keeps it outside the model, and retrieves the relevant parts into the model's context when a new task needs them. The model's context window acts as short-term working space. The memory layer is the long-term store, plus the rules for what goes in, what comes out and what gets updated or dropped.
In a company setting, a memory layer might hold customer preferences, past decisions, meeting outcomes, product facts and the history of a deal. When someone asks an assistant about the deal, or an agent drafts a follow-up, the memory layer decides which of those facts the model sees. That choice shapes the answer more than most buyers realise.
Why do AI models need a memory layer?
The obvious reason is that context windows are finite. Even very long windows cannot hold a company's full history, and every token costs time and money. The less obvious reason is that models do not use long contexts evenly.
The 2023 paper Lost in the Middle by Nelson Liu and colleagues found that model performance "is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts." In July 2025, Chroma's technical report Context Rot evaluated 18 models and found that "models do not use their context uniformly; instead, their performance grows increasingly unreliable as input length grows."
Anthropic's engineering team draws the practical conclusion in its guide to effective context engineering: models have an "attention budget," every new token depletes it, and good practice means "finding the smallest possible set of high-signal tokens that maximize the likelihood of some desired outcome." A memory layer exists to make that selection. Pouring everything into the window is not a substitute.
What are the four kinds of AI memory?
Researchers borrow terms from cognitive science to describe memory in AI agents. The 2023 paper Cognitive Architectures for Language Agents (CoALA), by Theodore Sumers, Shunyu Yao, Karthik Narasimhan and Thomas Griffiths, describes short-term working memory, which "maintains active and readily available information" for the current decision, and several long-term memories: episodic, semantic and procedural. Here is what each means for a company.
| Kind | What it holds | Company example |
|---|---|---|
| Working memory | What is relevant to the task in hand | The open quote, the buyer's last email and today's date |
| Semantic memory | Facts about the world | Bata buys S3 safety shoes. Standard payment terms are 30 days. |
| Episodic memory | What happened, in sequence | On Monday Harsh promised a revised quote on a call. |
| Procedural memory | How things are done | Quotes above a set value need finance sign-off before sending. |
Most commercial memory layers focus on semantic and episodic memory: facts and history. Procedural memory, the company's rules and approval paths, is often left in people's heads or in documents the memory layer treats as just more text. That gap matters later, when we look at what memory cannot do.
How does a memory layer work?
Every memory layer runs four operations, whatever its design.
- Write. Decide what to keep from a conversation or source, often by extracting facts or summarising.
- Store. Keep it in a form that can be searched later, such as text with embeddings, structured records or a graph of entities and relationships.
- Retrieve. When a task arrives, choose which memories to load into the context window.
- Update and forget. Merge duplicates, replace outdated facts and drop what no longer matters.
Two research systems show the range of designs. MemGPT, a 2023 paper by Charles Packer and colleagues, proposed "virtual context management," drawing on how operating systems move data "between fast and slow memory" to give the appearance of a much larger memory. The Stanford Generative Agents paper by Joon Sung Park and colleagues kept a memory stream of each agent's experiences and used a retrieval model that "combines relevance, recency, and importance." It also added reflection, which "synthesizes memories into higher-level inferences over time."
Note how importance was scored in that paper: by asking the language model to rate how poignant a memory is on a scale of 1 to 10. That works for a simulated town. In a company, importance depends on goals, deadlines and who is waiting, which a generic score cannot know.
What an AI memory layer can do
- End repetition. Users stop re-explaining who the customer is, what was agreed and how they like things written.
- Carry history across sessions. An agent working on a multi-day task can pick up where it left off. Anthropic describes this as structured note-taking, "a technique where the agent regularly writes notes persisted to memory outside of the context window."
- Personalise. Preferences about tone, format and recurring contacts are applied without being stated each time.
- Ground answers in company facts. Retrieval reduces guessing by giving the model the company's actual data.
- Keep knowledge when people move. Episodic records of decisions survive staff changes better than inboxes do.
These are real gains, and they address a common complaint. The 2025 MIT NANDA report quoted one user, as reported by Virtualization Review, saying a general-purpose assistant "doesn't retain knowledge of client preferences or learn from previous edits" and "requires extensive context input for each session." Memory fixes the first half of that sentence.
What an AI memory layer cannot do
The same user went on: "For high-stakes work, I need a system that accumulates knowledge and improves over time." Improving over time is where memory alone runs out. A memory layer cannot, by itself:
- Judge relevance to your goals. It retrieves what is similar to the query or recent. Whether a fact matters to this quarter's priorities is a separate judgement.
- Decide which fact is current and authoritative. If one email says 30-day terms and a later one says 60, the memory may hold both. Which one governs depends on who had the authority to agree it.
- Notice what nobody asked about. Retrieval is triggered by a query or a task. A broken commitment with no query attached stays in storage.
- Own work. It records that a promise was made. It does not assign it to a person with a date.
- Act or verify. It does not prepare the action, get approval or confirm that the email went out.
- Learn from outcomes. Unless something writes back what happened after each decision, memory grows larger without getting wiser.
These limits are not bugs in a particular product. They follow from what memory is for. Our article on why a company brain is not enough explains the operational cost of stopping at memory.
Memory layer vs RAG vs context engineering vs intelligence
Four terms are often used as if they were one. They are related but distinct.
| Term | What it is | What it answers |
|---|---|---|
| Memory layer | Long-term store with write, retrieve, update and forget rules | What do we know and what happened? |
| Retrieval-augmented generation (RAG) | A technique that fetches documents into the prompt at query time | Which documents relate to this question? |
| Context engineering | The practice of choosing everything the model sees for a given step | What should the model see right now? |
| Company intelligence | Judgement over memory and context, tied to goals, owners and proof | What should we do, who does it, by when, and did it happen? |
RAG is one way to retrieve from memory. Context engineering decides how retrieved memory, instructions and tools are combined. Intelligence decides what to do with the result. See why RAG is not enough, what is context engineering and memory vs context vs intelligence for each in depth.
How to evaluate an AI memory layer
- Test retrieval on your own questions.
Write twenty questions your team actually asks, with known answers. Check how many come back right, with the correct source.
- Test updates.
Change a fact, such as a price, and check whether the old value still appears in answers a day later.
- Test conflicts.
Plant two contradictory facts. See whether the system flags the conflict or silently picks one.
- Test permissions.
Confirm a user cannot retrieve what they could not open in the source system.
- Test forgetting.
Ask for something to be deleted and confirm it no longer appears anywhere, including in summaries.
- Ask what happens after the answer.
If nothing writes back outcomes, expect memory to grow without learning.
Common mistakes with memory layers
- Assuming bigger is better. More stored text raises the chance of retrieving something stale or irrelevant.
- No forgetting policy. Old prices and superseded decisions keep resurfacing with confident citations.
- Treating summaries as facts. A summary written by a model can drop the one condition that mattered.
- Ignoring procedural memory. Approval rules and limits stored as loose text are easy to miss at retrieval time.
- Expecting memory to act. A memory layer makes an assistant better informed. It does not make work get done.
How GeniOS approaches the AI memory layer
GeniOS reads where a company's work happens, such as email, calendar, chat and the business tools it already uses, read-only to start. That reading is its memory of the company. It also forms the company's goals from its own emails, decks and meetings, which gives it a basis for relevance that a memory store alone lacks.
It checks every team's pending work against those goals. What reaches a person is one situation, what to do, why, one owner and a date. The owner answers Delegate, Done, Later or Not relevant, and Delegate hands work to a person or an AI agent, with nothing sent until a human approves.
An item closes only on proof, such as the email that went out, and GeniOS learns from each outcome. That write-back is the part a memory layer usually lacks: a record of what happened after the knowledge was used.
Worked example: what memory recalls and what it misses
Names, dates and numbers in this example are made up to explain the idea. Vaultex, a safety-footwear maker in Greater Noida, gives its sales assistant a memory layer. Harsh in sales asks it to draft a cover note for the Bata quote: 12,000 pairs of S3 safety shoes at $12 a pair, about $144,000, ahead of Friday's vendor review.
The memory layer does its job well. It recalls that Bata prefers short notes, that the last order shipped in two lots, and that Bata's procurement lead asked for 60-day payment terms. The draft is accurate and in the right tone. Harsh no longer needs to explain the account.
Here is what it misses. The supplier's email saying steel toe caps are up 8% is stored, but nothing linked it to the quote, so the draft uses the old price. The 60-day terms request is recalled as a fact, but nobody with authority over terms has agreed it. Line 2 is booked for Reebok samples, which Anushka in production knows and the memory holds, but the draft promises a delivery date that assumes the line is free. The Adidas vendor audit is also pending, and its IS 15298 test reports are stored in old threads with no owner gathering them.
Each fact was in memory. What was missing was the layer that links a supplier change to an open quote, routes terms to the right owner, checks capacity against a promised date and closes the item only when the corrected quote is sent. Memory made the note better written. Intelligence would have made it correct.
Conclusion
An AI memory layer is a necessary part of any system that works with company information. It gives assistants and agents continuity, ends repetition and grounds answers in real facts. Its limits are just as clear: it retrieves, it does not judge, own, act or verify. Evaluate memory on retrieval, updates, conflicts, permissions and forgetting, and expect the judgement about what to do to come from a layer above it. To see memory and that layer working together on your own data, see pricing or book a call.
Sources
- arXiv, Lost in the Middle: How Language Models Use Long Contexts (Liu et al.), July 2023
- Chroma, Context Rot: How Increasing Input Tokens Impacts LLM Performance, 14 July 2025
- Anthropic, Effective context engineering for AI agents, 29 September 2025
- arXiv, Cognitive Architectures for Language Agents (Sumers et al.), September 2023
- arXiv, MemGPT: Towards LLMs as Operating Systems (Packer et al.), October 2023
- arXiv, Generative Agents: Interactive Simulacra of Human Behavior (Park et al.), April 2023
- Virtualization Review, MIT Report Finds Most AI Business Investments Fail, Reveals GenAI Divide, 19 August 2025