Context Engineering: Why AI Agents Quietly Forget

Picture a Million Token Memory (0:00)

Hey, chibis! I'm Aiwee, and today we're talking about context engineering, Lost in the Middle, and the four ways AI agents fail. If you enjoy stories like this, hit the like button and subscribe if you haven't already — let's go! Picture a language model with a one million token memory. One million tokens.

That's roughly seven hundred and fifty thousand words of plain text, or close to the length of a long novel. You can pour an entire codebase into it, a fat legal contract, a stack of research papers, and the model can read every word. And yet, if the one fact you actually need is buried somewhere in the middle, the model can still walk right past it. Researchers have a name for this. They call it being Lost in the Middle.

And it sits behind almost every strange failure you see in modern AI agents. Here is the puzzle. Bigger context windows were supposed to fix this. Year after year, model builders doubled, then redoubled, the size of what a model could see at once. And it genuinely did grow.

So why do agents still get confused, contradict themselves, or quietly forget the user's original goal? The answer has a name too, and it is changing how the people who build these systems think about their work. The same four failure patterns keep showing up in every system that tries to think for too long. Start with the economics. When an agent runs for a long time, two things happen to its context.

First, the cost of producing the next token grows much faster than the conversation itself. In an iterative loop where the full history is re-read for every new step, that cost rises on a curve that mathematicians call quadratic. Double the conversation and the work does not double. It roughly quadruples. Second, the time it takes to print the very first word of a reply rises in a straight line with context length.

So the longer your agent thinks, the slower it feels to the human watching it. A task that took a second at the start can crawl toward many seconds by the end.

Bigger Windows Did Not Fix Agents (2:15)

The model is paying a steeper bill the whole way. And every extra tool call makes the next bill even steeper. So why does the context keep growing? The default design of most agents is brutally simple. Once something is in the conversation, it never leaves.

Every line the model produces, every tool result it gets back, is permanently appended to the end. Engineers sometimes call this append-only mode. It's the obvious way to build, and it has a cost that grows with every minute the agent works. This is where a new discipline comes in. People inside the major labs now talk about context engineering, and the term is doing real work.

Prompt engineering asks, how do I write one set of static instructions that guides the model through everything? Context engineering asks a different question. Given that the model only has so much room to think, how do I make sure the right information is in that room at the right time, including the original prompt? It's the difference between writing a rulebook once and curating a moving library, entry by entry, for as long as the agent runs. One more thing.

The contrast between prompt engineering and context engineering is useful, but it's not a wall. A well-written prompt is itself a piece of context, often the most important piece. Many production systems blur the line, treating the prompt as just one more entry in a managed buffer. At one end of a spectrum, instructions never change. At the other, the system rewrites what the model sees on every step.

Most real agents live somewhere in the middle. Now, what actually goes wrong inside that moving library?

The Cost of a Long Memory (4:15)

Recent research, attributed in source materials to a team working with Microsoft, names four recurring failure modes. They call them poisoning, distraction, confusion, and clash. The first two are described in some detail. The second two are named but the public write-ups we have do not fully explain them, so we will handle them honestly and only by name. Poisoning is the failure you get when the context fills up with the wrong kind of evidence.

An agent tries a tool, the tool fails, the model tries again, and the tool fails again. Now the conversation is mostly a graveyard of stack traces and error messages. The relevant instruction from the user is still in there. But the model has begun to model its own failures instead of the task. The signal that was meant to guide the agent has been overwritten by the noise of its own attempts.

Distraction is the cousin of poisoning, but it does not come from the agent. It comes from the data. The authors use a precise term for it. They call irrelevant facts Red Herrings. Imagine the user asked for a list of European capitals, and the retrieved documents include a paragraph about the geography of the Caspian Sea.

The information is accurate. It's just not what was asked for. Piled together, these Red Herrings compete with the truly relevant information, and the model's attention drifts. The other two modes, confusion and clash, are named in the same research but the source materials do not detail their mechanisms. So we will leave them as labels.

Treat them as reminders that there is more to the picture than what the public documentation spells out, and treat any detailed explanation of how they work with appropriate skepticism until the underlying papers are read end to end. So what do engineers actually do about all of this? Three patterns show up again and again, and each one is a tradeoff, not a cure. The first pattern is summarization and trimming.

Prompt Versus Context Engineering (6:30)

When the context window starts to fill, the system pauses, rewrites older turns into a shorter summary, or simply deletes them. The goal is to keep what matters and shed the rest. The catch is that the summary is itself a new piece of text, and it can be summarized again, or quietly drop the detail the model will need later. Compression always loses something. The art is choosing what to lose.

The second pattern is the sandbox. Tasks like running a script over a huge log file would eat the entire context window if the agent tried to read the output directly. So instead, the agent hands the work to a sandboxed environment, a sealed room where code or documents can run. Only the result comes back, plus the small set of facts the model actually asked for. The raw firehose never enters the model's mind.

The third pattern is structural. Some teams build not one agent but many, each with its own context window. This is itself a form of context engineering, because the design of who knows what, and what gets passed between agents, becomes the central decision. Each agent can stay small and focused. The cost is handoff.

When one agent's answer has to travel to another, the original context has to be summarized, transformed, or simply guessed at. And if no single agent ever sees the full picture, a small misunderstanding at the boundary can ripple through the whole system. Pull the camera back. None of these tools are silver bullets. They are levers, and each one trades one failure mode for another.

Summarization trades memory for precision. Sandboxes trade raw visibility for tractable context. Multi-agent systems trade a single point of confusion for several new ones at the boundaries. The honest engineering question is not how do I make my agent never fail.

Four Failure Modes Appear (8:30)

It's which failure mode am I choosing to live with, and at what cost. Notice something else. These problems don't require a sophisticated agent to demonstrate. You can reproduce the Lost in the Middle effect with a long document. You can reproduce context poisoning by pasting failed outputs into a chat window.

You can reproduce Red Herrings by stuffing an answer with confident but unrelated facts. The failure mode is not a quirk of any one architecture. It is a property of attention itself, and that is what makes context engineering a discipline that travels with the models. And here is the larger thought. The story of context engineering is really a story about attention.

A language model is, at its heart, an attention machine. Give it a million tokens and you have not given it a million things to think about. You have given it a million things to choose between. The job of the engineer is to narrow that choice, again and again, so the right fact is the loudest one in the room at the next step. Every summary, every sandbox, every multi-agent handoff is one way of pointing a very expensive spotlight.

If you are building agents yourself, a useful exercise is to read your own logs and look for the four patterns by name. Where did the context get poisoned? Where did a Red Herring sneak in? Where did a handoff between agents lose a critical detail? The names are useful precisely because they are not the same as the model got it wrong.

They point at the design around the model, which is the part you can actually change. The current research is moving quickly, and some of the documents behind this work carry identifiers and dates we have not independently confirmed. Take the specific paper numbers and timestamps with a grain of salt. Take the underlying pattern less so. The same four labels are showing up in independent write-ups, conference talks, and production postmortems, and that convergence is itself a kind of evidence.

Poisoning and Distraction Explained (10:45)

The discipline is young, the vocabulary is settling, and the engineers who learn it now will ship agents that don't quietly forget what they were doing. If this breakdown helped, consider subscribing for more deep dives into how AI systems actually work under the hood, and leave a comment with a failure you have seen in the wild. The next video is already in the works.