More Memory Does Not Always Mean a Smarter AI Agent
A study across eight AI models found that feeding an agent more of its own past experience can hurt as much as help. The right amount of memory depends entirely on how capable the model already is.
Key points
- Researchers tested eight AI models on a benchmark of 585 tasks and found that the correct amount of memory varies sharply by model ability.
- A mid-tier model called gpt-oss-120b completed 16.1 percentage points more tasks when given a curated selection of memory rather than the full set.
- A large, capable model called DeepSeek-V3.2 improved most with the full memory set, gaining 9.5 percentage points in task completion.
- One model, GLM-5, showed zero improvement from any memory strategy, suggesting it had already hit a ceiling on these tasks.
- Selective memory delivery used only 5% more computing resources than no memory at all, compared to a 78% increase for the full-set approach.
Imagine giving a new employee a thick binder of lessons learned from every colleague before them. A sharp hire reads it and gets better. An overwhelmed one ignores most of it. And the office veteran already knows it all. New research, first published on Hugging Face, suggests AI agents work the same way.
The study used a system called ALTK-Evolve. Here is how it works: an AI agent, meaning software that can carry out multi-step tasks on its own, attempts a series of jobs. The system then studies what went right and wrong and writes out a list of guidelines, like a personal rulebook. Those guidelines get fed back to the agent before its next set of tasks. No part of the underlying model changes. The learning happens in the instructions, not the machine.
Why does the amount of memory matter?
Too much guidance can swamp a weaker model just as easily as too little. The researchers sorted their results into three distinct groups.
Capable models with room to grow absorbed every guideline they were given. DeepSeek-V3.2, a model with 671 billion internal connections, completed 9.5% more tasks when handed the full rulebook. Claude Opus 4.6 and GPT-5.5 were already near the top of the scoring range but still gained over 7 percentage points on the stricter version of the test.
Mid-range models did better with a curated selection. Rather than every lesson learned, these agents received a fixed core of the most reliable guidelines plus a small batch chosen specifically for each task. The gpt-oss-120b model jumped from completing 39.9% of tasks to 56%, a gain of 16.1 percentage points, while using only 5% more computing tokens (the unit of work an AI model processes) than it used with no memory at all. Giving it the full guideline set cost 51% more tokens and produced a smaller improvement.
Already-strong models hit a wall. GLM-5, despite having 745 billion connections, scored identically with and without memory. Whether it had already reached its limit on these tasks, or whether the guidelines simply did not address its remaining weak spots, is still unclear.
What does this mean in practice?
| Model | Pattern | Task completion without memory | Task completion with best memory | Gain |
|---|---|---|---|---|
| gpt-oss-120b | Selective works best | 39.9% | 56.0% | +16.1pp |
| DeepSeek-V3.2 | Full set works best | 79.8% | 89.3% | +9.5pp |
| Claude Opus 4.6 | Full set, near ceiling | 90.5% | 94.6% | +4.1pp |
| GPT-5.5 | Full set, near ceiling | 92.3% | 95.2% | +2.9pp |
| GLM-5 | No gain | 87.5% | 87.5% | 0.0pp |
For anyone building or buying AI agents for real work, the practical message is that memory is a setting to tune, not a checkbox to tick. A weaker model flooded with guidelines can end up performing worse than a well-calibrated cheaper one. And because curated delivery uses far fewer tokens per task, the efficient approach is also the affordable one.
The research team notes that model size alone does not predict which category a model falls into. Context-window size (how much text a model can read at once), the quality of the guidelines it generates, and the nature of the tasks all play a role. Separating those factors is listed as ongoing work.
Common questions
What kind of tasks were tested?
The study used AppWorld, a benchmark of 585 multi-step tasks spread across nine simulated apps including calendars, messaging tools and payment systems. Each task required the agent to complete a real-world-style workflow.
Does the AI model itself change when it learns from memory?
No. The underlying model stays exactly as it was. Only the instructions passed to it before each task are updated, which makes the approach cheap to apply and easy to switch between models.
Could this affect AI tools people use at work?
Potentially yes. Many business AI agents are moving toward exactly this kind of self-improving loop. This research suggests that teams deploying such agents should test how much memory their specific model actually benefits from, rather than assuming more is always better.
