Few-Shot Prompting: A Practical Guide to In-Context Examples

Few-shot prompting means putting worked examples in the context window and asking the model to imitate them. You are not fine-tuning anything. You are showing the model what finished looks like so it can pattern-match instead of guessing. The whole practice collapses into one question: which examples, and how many. Get that wrong and you pay for every token while teaching the wrong lesson.

What few-shot prompting is actually doing

Examples do three jobs at once, and it helps to separate them. First, they fix the output format: once the model has seen three JSON objects, it stops adding preamble. Second, they define the decision boundary for a fuzzy label. Third, they set the house style, tone, and length in a way that adjectives in an instruction never manage. When a prompt feels like it is "almost right" but stylistically off, you usually have an instruction problem, not a wording problem, and examples are the faster fix.

How many examples: the short answer

Start with three. Most tasks are already close to solved at three examples, and the curve flattens fast after that. The gains from example five, six, and seven are real but small, and each one costs context window that your task might need for actual input. The exception is tasks with several distinct modes, where one example per mode genuinely helps because the model needs to see that the modes exist.

Hard examples beat more examples

This is the highest-leverage habit in the whole practice, and almost everyone gets it backwards. The instinct is to show three textbook-perfect cases so the model has an easy time. The result is a model that performs beautifully on your demo and falls apart on the cases you did not demo. Examples that the model already gets right teach it almost nothing, because there is no gradient to learn. The examples that carry information are the ones it gets wrong, or gets right for the wrong reason.

Practically: keep a slice of your evaluation set as the example set, and deliberately choose from the boundary. Include the case where two labels are genuinely close and your label is the less obvious one. Include the case that broke your first version. If every example in your prompt is one you would have written yourself with a hundred percent of confidence, you have demonstrated your taste instead of teaching the boundary.

Selection, diversity, and ordering

• Cover the input space, not your favorite input. If customers write three paragraphs, do not demonstrate three one-liners.
• Include the failure you are defending against. One example of the wrong answer you refuse to give is worth three of the right answer.
• Vary length and formatting between examples so the model learns the rule rather than the template surface.
• Keep the label distribution honest. If one class is 80 percent of real traffic but 20 percent of your examples, the model will under-predict it.
• Order examples from simple to complex when the model has a tendency to anchor on the first item.
• Put the most representative example closest to the instruction, since attention to the nearest tokens is usually strongest.

TASK: Classify the support ticket into exactly one billing category.

ALLOWED CATEGORIES: refund_request, payment_failed, plan_change, other

EXAMPLE 1
Input: "I was charged twice this month"
Output: {"category": "payment_failed", "confidence": "high"}

EXAMPLE 2
Input: "can I switch to annual before the trial ends"
Output: {"category": "plan_change", "confidence": "high"}

EXAMPLE 3
Input: "you took my money and gave me nothing, I want it back"
Output: {"category": "refund_request", "confidence": "high"}

EXAMPLE 4
Input: "how do i add more seats"
Output: {"category": "other", "confidence": "low"}

Now classify. Respond with JSON only.

Input: "{{ticket_text}}"
Output:

Note what example 4 does. It is the weakest case in the set, it is the one most people would leave out, and it teaches two things at once: that "other" exists as a real answer, and what "low confidence" looks like in this format. Example 3 teaches the boundary between a refund and a failed payment, which is exactly the confusion a model makes without it.

Building the example set: a repeatable procedure

The selection process is mechanical enough to script, which is what stops it from becoming taste. Collect a larger pool of real inputs than you think you need, run your zero-shot prompt against all of them, and sort by disagreement. Inputs where the model was confidently wrong, where it was right for the wrong reason, and where two of your own annotators disagreed are your example candidates. Everything the model already handles perfectly is, by definition, not teaching you anything.

• Assemble 30 to 50 real inputs, wider than the set you will actually ship as examples.
• Run the zero-shot version and record every output plus any error or hedge.
• Sort into three buckets: handled, wrong, and right-but-wrong-reason.
• Draw examples from the last two buckets, roughly two from each per mode of the task.
• Trim to the smallest set that still moves the frozen evaluation score, then stop.

This procedure has a useful side effect: it tells you where the model is weak before you start choosing examples, which is information you want regardless. A category where every input is wrong is not an example-selection problem, it is a knowledge or retrieval problem, and no number of demonstrations will fix it.

What still goes wrong

Four failure modes account for most disappointing few-shot results. Demonstration leakage, where an example shares identifying details with a test case and the model memorizes rather than generalizes. Style capture, where the examples share a surface form and the model copies it even when the input does not call for it. Label imbalance, where the example set misrepresents the real class distribution and skews predictions. And order sensitivity, where the model anchors on whichever example came first.

Formatting that does not waste effort

Use a delimiter that cannot appear in the data, keep every example in identical structure, and separate instruction, examples, and the live input with explicit markers. Structure beats cleverness here: a rigid template with clear delimiters is easier for you to maintain and easier for the model to parse than a prose example. The moment your examples drift into slightly different shapes, the model treats the differences as meaningful and starts varying its output to match them.

The goal of an example set is not to look thorough. It is to place a small number of decision boundaries exactly where the model is currently uncertain.

When zero-shot is the better choice

Examples are not free. They consume context, they add maintenance, and they create a second place to update when behavior changes. Skip them when the task is well described by instructions alone, when the model already handles the format natively, or when you have not yet confirmed the zero-shot version fails for a reason examples would fix. Adding examples to a working prompt is a guess. Adding them to a measured failure is an engineering decision. The wider discipline is covered in our prompt engineering practices overview.