NVIDIA TensorRT-LLM PR #19390: Why Top-k Requests Got a Greedy Temperature

TensorRT-LLM PR #19390 restores temperature 1.0 for non-greedy, one-model speculative-decoding requests that leave temperature unset.

Previously, a request such as SamplingParams(top_k=50) received 1e-5, the internal temperature reserved for greedy rows. The request was correctly classified as non-greedy, but its temperature could make the probability distribution collapse toward the highest-scoring token.

The patch changes one normalization expression in tensorrt_llm/_torch/speculative/interface.py. It leaves greedy requests and explicitly supplied temperatures unchanged.

The merged implementation addresses the mismatch. Its exact triggering case still needs a focused regression assertion.

An omitted temperature does not determine the sampling mode

TensorRT-LLM resolves the sampling mode from several parameters together.

Leaving temperature unset can mean different things:

Request parameters	Mode selected by the reviewed predicate	Correct internal temperature
All three controls unset	Greedy	Greedy sentinel
`top_k=50`	Non-greedy	`1.0`
`top_p=0.9`	Non-greedy	`1.0`
`temperature=0.7`	Non-greedy	`0.7`
`temperature=0, top_k=50`	Greedy	Greedy sentinel
`top_k=1`	Greedy	Greedy sentinel

The three controls here are temperature, top-k, and top-p, as passed by the reviewed one-model helper.

NVIDIA's public SamplingParams documentation describes the same distinction: an active top-k or top-p request can use temperature 1.0 without explicitly setting it. When all those controls are absent, the default is greedy.

The regular sampler implements this order directly. resolve_sampling_strategy() first checks whether the request is greedy. Only after excluding that case does it resolve:

temperature = temperature or 1.0

Here, unset means “resolve the default for the selected mode,” not “choose greedy decoding.”

The speculative path classified the request correctly, then encoded it incorrectly

SpecMetadata._scan_one_model_sampling() contains a nested helper named _normalize_request_sampling_params().

That helper first calls the shared greedy predicate. For top_k=50 with no temperature, it obtains:

is_greedy = False
temperature = None

It also retains the active top-k value.

The old temperature expression then overrode the meaning of that classification:

normalized_temperature = (
    DISABLE_TEMP_VAL
    if is_greedy or temperature is None or temperature == 0
    else temperature
)

The temperature is None condition was enough to select the greedy sentinel even when is_greedy was false.

The sentinel is derived from another constant:

GREEDY_TEMPERATURE_THRESHOLD = 1e-4
DISABLE_TEMP_VAL = threshold / 10 = 1e-5

The source reserves this value for greedy rows and keeps it nonzero to avoid division by zero. It also documents why the value is tiny: even if a row reaches probability-based sampling, dividing logits by it produces an effectively one-hot distribution for ordinarily separated scores.

The inconsistent result was therefore:

request classification: non-greedy
top-k setting:           50
normalized temperature: greedy sentinel

The top-k value survived normalization. The temperature undermined the distribution over which sampling would operate.

A tiny temperature can overwhelm the filter

Consider an illustrative three-token logit vector:

[2.0, 1.9, 1.8]

At temperature 1.0, softmax gives approximately:

[0.367165, 0.332225, 0.300610]

With top-k set to two, the surviving tokens have probabilities of approximately 0.525 and 0.475.

At temperature 1e-5, the gap between the two largest scaled logits becomes:

(2.0 - 1.9) / 1e-5 = 10,000

The second token's probability relative to the first is exp(-10,000). A stable Python floating-point calculation produces:

[1.0, 0.0, 0.0]

Keeping two candidates cannot restore probability mass that temperature scaling has already eliminated numerically.

This calculation illustrates the mechanism; it is not a TensorRT-LLM execution or a measurement from a model.

There is also an important boundary to the explanation. Tiny positive temperature does not mathematically guarantee a unique winner for every possible logit vector. Exact ties and sufficiently small gaps matter. The verified defect is that a non-greedy request received the wrong temperature, whether or not every resulting sample becomes deterministic.

The fix keeps the sentinel behind the greedy decision

The replacement expression is:

normalized_temperature = (
    DISABLE_TEMP_VAL if is_greedy else (temperature or 1.0)
)

Now the same request produces:

is_greedy = False
temperature = None
normalized temperature = 1.0

A genuine greedy request still receives the sentinel. An explicit nonzero temperature still passes through unchanged.

The use of or 1.0 needs to be read together with the earlier predicate. That predicate classifies an explicit temperature of zero as greedy. Such a request takes the first branch and never reaches the fallback.

Consequently, the patch does not reinterpret temperature=0 as temperature=1.0.

Simply deleting the old temperature is None clause would leave another problem. A non-greedy request with no temperature would then carry None into later construction of a floating-point tensor. The correction must both stop assigning the sentinel and supply a numeric default.

The changed value reaches both request and token buffers

The scan returns normalized tuples containing temperature, top-k, top-p, and a token count.

populate_sampling_params_for_one_model() uses them to populate per-request buffers and expanded per-token buffers. The former provide one set of sampling controls for each request; the latter repeat those controls over the request's relevant token rows.

For a batch containing a non-greedy request, the corrected 1.0 therefore replaces the old sentinel in the data sent to the sampler.

The batch's is_all_greedy_sample flag is computed from request classification. That classification is unchanged by this patch, so the fix does not work by moving top-k requests into a different greedy-versus-advanced graph category. They already belonged in the non-greedy category.

The sampling-buffer cache also includes temperature in its signature:

per-request signature:
(temperature, top_k, top_p) for each request

expanded signature:
those values plus the per-request token counts

The signature logic already distinguishes 1e-5 from 1.0. No additional cache-key change is needed to represent the corrected value.

The existing tests miss the newly corrected default

The merged PR changes one production file and no tests.

Its description points to test_capture_sampling_params.py. That file tests a related CUDA-graph-capture behavior: synthetic capture parameters must not leak into later serving requests.

The reviewed cases exercise explicit capture-time values, completely unspecified requests, and a later serving request with an explicit temperature of 1.0.

Those cases do not test the distinguishing input for this fix:

temperature = None
top_k = 50, or top_p = 0.9

Even the parameterless case checks greedy classification and that the temperature differs from the capture temperature; it does not assert the exact greedy sentinel.

A focused regression should establish both parts of the corrected result:

request remains non-greedy
normalized temperature equals 1.0

Controls should preserve an explicitly greedy request and a request with an explicit nonzero temperature.

An automated review requested this missing coverage. The public record also contains a human approval, but the merged diff still contains no test change. Approval and a regression that distinguishes the old implementation from the new one are different forms of evidence.

As a separate review sanity check, an independently transcribed Python version of the three-parameter logic was evaluated for 12 cases. It confirmed that the changed temperature values were confined to non-greedy cases with no explicit temperature, while the top-k, top-p, and greedy-classification outputs stayed the same. This was a small logic check, not the project's unit suite or GPU validation.

What this change establishes

The patch aligns one-model speculative temperature normalization with the regular sampler's existing default-resolution rule.

Its scope is specific. It does not change all-unspecified requests to stochastic sampling, override a user's explicit temperature, add min-p support to this one-model path, or establish distributional equivalence for every speculative algorithm.

At the reviewed commit, the one-model normalizer explicitly leaves min-p out and documents a separate admission restriction. The PR's discussion of a related min-p feature is background, not evidence that this patch implements that feature.

No throughput benchmark or model-level before-and-after generation result was provided in the reviewed PR. This review did not build TensorRT-LLM or run its GPU sampler, and it does not identify a released package containing the fix.

The source-level conclusion is clear: a row selected for non-greedy sampling must receive the resolved non-greedy temperature. An omitted optional value cannot be translated into a greedy control signal merely because both cases lack an ordinary temperature number.