Context Window

A context window is the token capacity a model can process for one inference sequence or request [1].

A context window is the working space a model has during a single request. It is usually measured in tokens, which are small pieces of text. The product may place the user message, system instructions, examples, retrieved passages, conversation history, tool results, and output constraints into that space before the model generates an answer.

The context window is not the same as memory. Conversation history affects the current request only when the product places that history back into the model context. Product memory, stored user preferences, retrieval, and the context window are separate mechanisms, even when a product combines them in one interface.

Context windows shape what a model can use in a single interaction. If a user pastes a short contract, the model may have room for the whole document and the question. If the user pastes a long folder of notes, the product may need to truncate, summarize, retrieve selected chunks, or reject part of the input.

This matters because people often assume that if information appeared earlier in a chat, the model still has it. That may be false. A product may include only recent messages, summaries, selected memories, retrieved documents, or no previous context at all. The visible chat thread is not always the exact context sent to the model.

It also matters because a larger context window does not guarantee better use of every detail. The Lost in the Middle study found that the evaluated language models and tasks were sensitive to where relevant information appeared in long input contexts, with performance often worse when relevant information appeared in the middle [2]. That result should be attributed to the systems and tasks studied, not turned into a universal claim that every model always ignores the middle.

Before generation, an application assembles a request. That request can include system-level instructions, the user message, conversation history chosen by the product, examples, retrieved passages, tool outputs, and formatting rules. The model then processes the tokens in that assembled context and generates new tokens as output.

Input and output limits are product and model implementation details. In some systems, input tokens and generated output share one total capacity. In others, documentation may state separate input and output limits. A concept article should not present one allocation rule as universal across all products.

When the assembled context is too large, the application must decide what to do. It may remove older messages, summarize them, retrieve only selected passages, ask the user to shorten the input, or split the task into smaller requests. Each choice changes what evidence the model can use.

Imagine a student asks an AI assistant to compare three articles. The first article is short, the second is long, and the third is pasted after a long conversation. The product may include the user request, some system instructions, part of the conversation history, and the article text. If the total is too large, it may drop older turns or use a summary.

A good workflow makes this visible. The student can ask, "Which sources did you use?" and check whether the answer actually cites all three articles. If one article is missing from the context, the answer may still sound complete while only reflecting the material the model received.

The biggest misconception is that context window means memory. Memory suggests a product stores information and can bring it back later. A context window is the capacity for one request. A product may store preferences or summaries separately, but that stored information affects generation only if the product inserts it into the current context.

Another misconception is that everything inside the context window receives equal attention and reliable use. The model may miss, dilute, or misapply information, especially in long, complex, or conflicting contexts. Being present in the request is not the same as being correctly used.

Longer context can help with large documents, multi-step tasks, and richer examples, but it also creates risks. Users may paste more sensitive data than needed. Retrieved passages may include stale or hostile text. Long input can make it harder for a person to verify which evidence drove the answer.

NIST treats generative AI risks as lifecycle risks involving design, use, monitoring, and evaluation [3]. For context windows, that means teams should test what the product includes, what it drops, how it handles long input, and whether users understand the limits.

Before relying on an answer from a long context, ask: What documents or messages were actually included? Was anything summarized or omitted? Does the answer cite the exact passage that supports each important claim? Could sensitive data be present? Could an old message or retrieved document contain instructions that conflict with the task?

Use long context for reading, comparison, drafting from supplied material, and keeping a task coherent. Slow down when the answer affects policy, legal, medical, financial, employment, customer, or security decisions. A larger context window gives the model more possible evidence, not automatic correctness.

Continue with Large Language Model for the model foundation, Prompt for request design, and Retrieval-Augmented Generation for adding retrieved evidence.