Temperature in LLMs

What temperature actually does

When an LLM generates a response, it produces a probability distribution over possible next tokens at each step. Temperature scales that distribution before sampling. A temperature of 0 makes the model always pick the highest-probability token, producing deterministic output. A temperature of 1 samples proportionally to the model probability. A temperature above 1 flattens the distribution and increases randomness.

The name comes from thermodynamics, where higher temperature means higher entropy in a physical system. The analogy is direct: low temperature collapses the distribution toward its peak, high temperature spreads probability mass across many options. What feels like creativity from a high-temperature model is the statistical consequence of sampling from a flatter distribution, including tokens the model considers only moderately likely.

This is why the setting matters in production. It is not a style preference. It is a fundamental parameter controlling the variance of your system's output, and the right value depends entirely on the task.

Zero does not mean identical output every time

Temperature 0 makes the model deterministic in the mathematical sense, but identical prompts may still produce slightly different outputs across separate API calls. The reason is floating-point arithmetic differences across hardware and batching. When the same computation runs on different GPU configurations or in different batch sizes, rounding differences accumulate, and token probabilities that appear equal may break ties differently.

For applications requiring truly identical output, this matters. A compliance system that must produce exactly the same classification for the same input cannot rely on temperature 0 alone. It needs deterministic routing logic on top, or output normalization after generation.

For most production applications, temperature 0 produces sufficiently consistent output. The practical implication is: do not write architecture that depends on mathematical equality of outputs at temperature 0. Write architecture that tolerates minor variation in phrasing while enforcing consistency at the structural level.

How to choose a temperature for your use case

The right temperature depends on whether variance in the output is a feature or a bug.

Extraction and classification tasks benefit from low temperature, typically 0 to 0.3. You want the model to consistently identify the entity, label the sentiment, or extract the field, not explore creative alternatives. The answer is either correct or incorrect, and higher temperature increases the rate at which the model samples incorrect tokens.

Factual question answering benefits from low to medium temperature, 0 to 0.5. Summaries and structured outputs benefit from 0 to 0.4. In both cases, the model needs to be accurate and consistent more than it needs to sound varied.

Open-ended generation such as marketing copy, brainstorming, or creative writing benefits from higher temperature, 0.7 to 1.0. Here variance is the point. A model that always produces the same headline for the same brief is less useful than one that surfaces different framings across runs.

Customer support responses benefit from 0.3 to 0.5. Enough variance to vary the phrasing naturally across interactions, keeping the system from sounding robotic, while keeping the substance consistent. A support response that gives different advice depending on the sampling run is a reliability problem.

Temperature and hallucination

Higher temperature increases creativity but also increases the rate of factual errors. The mechanism is the same: the model samples from less probable tokens, and some of those tokens are plausible-sounding but incorrect information. A model at temperature 0.9 is more likely to generate a confident but wrong date, name, or figure than the same model at temperature 0.1.

For applications where factual accuracy matters, low temperature is not a sufficient defense. It reduces the rate at which the model wanders into incorrect territory, but it does not eliminate hallucination. The model can be fully deterministic and still confidently produce wrong information, because the most probable token at each step reflects training distribution, not ground truth.

The correct approach is low temperature plus retrieval of ground-truth sources plus output evaluation. We covered the full picture in hallucination in production. Temperature is one variable in a multi-variable problem, and treating it as the primary defense against hallucination will give you false confidence.

Other sampling parameters

Temperature is not the only lever. Top-p, also called nucleus sampling, sets a probability mass threshold and samples only from the top tokens that cumulatively reach that threshold. If top-p is 0.9, the model samples from the smallest set of tokens whose cumulative probability reaches 90 percent. Top-k restricts sampling to the k most probable tokens regardless of probability mass. A top-k of 40 means only the 40 most likely next tokens are candidates.

Most production use cases need only temperature and can leave top-p and top-k at defaults. The defaults on major model APIs are set to produce sensible behavior across a wide range of tasks. Setting multiple parameters simultaneously without a clear reason for each produces unpredictable interactions and makes debugging harder. If your outputs are wrong, you want to know which parameter to adjust. If you have changed three parameters at once, you have made that diagnosis substantially harder.

The one case where top-p adjustment is clearly useful is when you want to bound the tail of the distribution: prevent the model from sampling very low-probability tokens even at moderately high temperature. But start with temperature alone and add top-p only when you have a measured reason.

The evaluation implication

If you evaluate your system at temperature 0 and deploy at temperature 0.7, your evaluation tells you little about production behavior. The variance introduced by higher temperature means you need more evaluation samples to get a stable estimate of accuracy. A system evaluated on twenty examples at temperature 0.7 has high uncertainty in that estimate, because the same prompt will produce meaningfully different outputs across runs.

As a rough guide: if your target accuracy estimate needs to be accurate within plus or minus 5 percentage points, you need at least 400 evaluation samples at temperature 0, and considerably more at temperature 0.7 where output variance adds noise on top of example variance.

Run your evaluations at the exact temperature you will deploy, and use enough samples that the confidence interval on your accuracy metric is narrow enough to be useful for decision-making. We covered evaluation methodology in LLM evals.

What to set it to if you do not know

For any task that requires factual accuracy, structured output, or consistent formatting: start at 0.2. Run your evaluation set at that temperature, then try 0.0 and 0.4 and compare. The differences in accuracy and consistency will tell you whether the task is sensitive to temperature in your specific setup.

For any task that requires natural language variety, empathy in tone, or creative output: start at 0.7. Run enough evaluation samples that the estimate is stable, then try 0.5 and 0.9 and compare on whichever quality dimension matters most for the use case.

The right answer is the one your evaluation confirms, not the one that sounds intuitive. Teams that set temperature by intuition and skip the measurement tend to either over-constrain creative tasks or under-constrain factual ones. Both are failures that show up in production before they show up in a review.