Concept

In‑context Learning

In‑context learning refers to the phenomenon whereby a language model can acquire the behavior required for a new task simply by being presented with one or more examples of that task embedded in its input prompt, without any weight updates or external training. The model treats the provided examples as part of the context and extrapolates from them, generating outputs that follow the demonstrated pattern. This ability arises from the way large transformer models encode statistical relationships across many tokens and can be triggered by carefully structuring prompts.

The significance of in‑context learning lies in its practical efficiency and flexibility: users can adapt a single, pretrained model to a wide variety of downstream tasks on the fly, saving computational resources that would otherwise be spent fine‑tuning separate models. It also offers insights into how neural networks internalize procedural knowledge, shedding light on the limits of memorization versus abstraction in modern language systems.

You will encounter in‑context learning whenever prompts contain demonstrations—such as few‑shot question answering, code synthesis from example snippets, or style transfer guided by sample sentences. It underpins many interactive AI applications where a user supplies a handful of examples and expects the model to continue the pattern immediately, making it a cornerstone technique for rapid prototyping and deployment of language‑model‑based tools.

2 reads touch this

The AI That Reads Your Assumptions at the Last Moment
"Prepend a handful of example datasets drawn from your target prior — a prefix — in front of the inference task, and train the transformer to condition on both at once."
Some Ais Can Be Talked into Anything. One Training Step Is the Difference.
“The trick was unsettling in its simplicity. Show a safety‑trained language model a handful of examples … and it grows more pliable… The standard explanation was a story about disposition.” (body, first paragraph) – refers to learning from in‑context demonstrations.