OpenXLA #35602 — Why Zero Inputs Could Make a Wrong GPU Kernel Look Correct

GPU autotuning, shared reference buffers, and the missing connection between check_buffers and should_init_buffers

Suppose an autotuner compares these two implementations:

Reference:

y = a + b

Broken candidate:

y = a

Now test both with:

a = 0

b = 0

The reference returns:

The broken candidate also returns:

The output comparison passes.

Nothing about that result proves the second implementation used b.

Change the input:

a = 5

b = 7

and the difference becomes visible:

Reference:
12

Broken candidate:
5

This is the boundary OpenXLA PR #35602 repaired.

XLA’s GPU autotuner already had a correctness mode that:

compiled several candidate implementations;
ran a trusted candidate to obtain a reference output;
ran every other candidate with the same inputs;
compared each candidate output with the reference;
rejected candidates that produced different values.

But enabling that comparison did not also enable random initialization of the profile inputs.

The profiler’s input-initialization option remained at its default:

should_init_buffers = false

So a supposedly stronger correctness check could operate on uninitialized or allocator-provided contents. The PR’s stated concern was the particularly dangerous case where those contents were zero: a backend producing incorrect results could still match the reference because the test data made the implementations indistinguishable.

PR #35602 connected the two policies:

check candidate outputs

→ also initialize the input buffers

The production change was one assignment:

profile_options.should_init_buffers =
    autotune_config.check_buffers;

A correctness checker needs more than a trusted reference. It also needs inputs on which incorrect implementations are likely to disagree with that reference.

What the GPU autotuner was trying to prove

An autotuner does not compile one GPU kernel and accept it.

It may produce several candidates using different:

backend implementations;
tiling parameters;
launch configurations;
library algorithms;
fusion strategies;
or scratch-memory choices.

Conceptually:

One HLO operation

        ↓

Candidate A
Candidate B
Candidate C
Candidate D

The simplest autotuning policy would run every candidate, measure its execution time, and choose the fastest one.

That is insufficient.

A kernel can be fast because it performs less work than required.

It can also be fast because it:

skips an operand;
reads the wrong stride;
mishandles a boundary tile;
writes only part of the output;
uses an incompatible layout;
or applies an incorrect epilogue.

XLA therefore supports a higher autotuning level that includes output checking.

At the affected revision:

autotune_config.check_buffers =
    debug_options.xla_gpu_autotune_level() >= 4;

At level four or above, the autotuner attempts to detect both memory-boundary violations and numerical disagreement between candidate outputs.

How the reference comparison works

The autotuner first creates one set of input buffers.

Those buffers are reused for all candidate implementations.

shared profile inputs
        ├── reference backend
        ├── candidate A
        ├── candidate B
        └── candidate C

Using the same inputs is necessary.

Two implementations cannot be compared meaningfully if each one receives unrelated data.

The autotuner then searches for a reference candidate.

It skips backends that declare:

CanProduceWrongResults()

and uses the output of a candidate that is expected to provide a suitable reference.

Every profiled candidate is then checked against that reference output.

candidate output
        ↓
BufferComparator
        ↓
reference output

A mismatch becomes:

FailureKind::kWrongResults

and a failed candidate is excluded from selection.

The final configuration is chosen only from candidates without recorded failures.

That architecture is reasonable.

The missing piece was the data supplied to it.

A comparison can be correct and still prove almost nothing

The buffer comparator can perform its job perfectly.

Suppose it checks every output element and confirms:

candidate output
==
reference output

That statement is valid for the tested input.

It does not automatically establish:

candidate implementation
==
reference implementation

over the full input domain.

Every dynamic test samples only part of that domain.

The quality of the sample determines which implementation differences become observable.

Degenerate input

a = 0

b = 0

Correct and broken addition may agree.

Distinguishing input

a = 5

b = 7

The missing operand becomes visible.

The checker’s strength therefore depends on two separate contracts:

Comparison contract

→ Are outputs compared correctly?

Input contract

→ Are the chosen values capable of exposing
   the classes of failure being checked?

PR #35602 repaired the connection between them.

Why zero is unusually good at hiding GPU mistakes

Zero is not merely one ordinary test value.

Many GPU computations have zero as an absorbing or neutral value.

Multiplication

x × 0
=
0

A kernel that reads the wrong x can still return the correct result when the other operand is zero.

Matrix multiplication

A × B
=
0

if either complete input matrix is zero.

A candidate that incorrectly returns a zero matrix may therefore agree with a correct GEMM on zero inputs.

Missing addition operand

a + b

and:

agree when:

b = 0

Wrong indexing

Two different addresses can contain the same value.

If every element is zero:

correct address
→ 0

wrong address
→ 0

The address bug remains invisible.

Incorrect transpose or layout

Permuting a tensor of identical zeros produces the same tensor.

correct layout
→ all zeros

wrong layout
→ all zeros

Partial output write

A candidate that leaves part of a zero-initialized output untouched may still appear correct when the expected values in that region are also zero.

The same problem can occur with any overly uniform input, not only zero.

But zero collapses especially many numerical distinctions at once.

The profiler had a separate initialization option

The profiling interface already represented the required policy.

ProfileOptions contained:

bool should_init_buffers = false;

Its documented meaning was:

true
→ initialize buffers with random data

false
→ leave them uninitialized

The option was therefore available before #35602.

The missing connection was in the pass that constructed the GPU profiler.

The old code supplied redzone padding:

profile_options.redzone_padding_bytes =
    debug_options.xla_gpu_redzone_padding_bytes();

but left:

should_init_buffers

at its default value.

Correctness comparison and input initialization were configured independently even though the former depended on the latter.

What happened when the flag was false

The GPU profiler created its input storage through RedzoneBuffers.

That layer forwarded:

should_init_buffers

to the redzone allocator.

The allocator performed explicit initialization only when the flag was true:

if (initialize_buffers) {
    InitializeBuffer(
        stream(),
        shape.element_type(),
        &rng_state,
        buffer
    );
}

When false, the allocation was returned without that random initialization step.

The source should be described precisely here.

It did not formally guarantee:

every pre-fix buffer was zero

The buffers were left without the explicit random initialization used for correctness checking.

Their actual contents could depend on:

allocator behavior;
reused device memory;
earlier writes;
platform details;
or buffer state.

The PR title identifies zero-initialized values as the failure mode that motivated the change.

The code-level defect was broader:

correctness checking requested

but

distinguishing profile data not requested

Why reused garbage would not be a good test strategy either

One might argue that uninitialized memory is unlikely to contain only zeros.

Perhaps unpredictable old values would expose more bugs than zeros.

That is not a valid correctness contract.

Uninitialized contents are not controlled test data.

They may vary across:

runs;
devices;
allocators;
process histories;
memory-pressure states;
or candidate ordering.

A correctness checker needs deliberate initialization for two reasons.

Discriminating values

The values should make different implementations likely to produce different outputs.

Reproducibility

The same profiling run should compare all candidates under the same defined input state.

Random initialization in this context does not mean:

uncontrolled or unknowable

It means the profiler intentionally populates the buffers through XLA’s initialization mechanism rather than accepting whatever bytes the allocator happens to return.

The fix tied initialization to the existing correctness gate

PR #35602 changed the profile-options builder from:

ProfileOptions GetProfileOptions(
    const DebugOptions& debug_options
)

to:

ProfileOptions GetProfileOptions(
    const DebugOptions& debug_options,
    const AutotuneConfig& autotune_config
)

It then added:

profile_options.should_init_buffers =
    autotune_config.check_buffers;

And the profiler was constructed with that completed configuration:

GpuProfiler::Create(
    stream_executor,
    GetProfileOptions(
        debug_options,
        autotune_config
    ),
    allocator
);

The resulting policy was:

Autotuning state	Compare outputs	Initialize profile inputs
Buffer checking disabled	No	No
Buffer checking enabled	Yes	Yes

The change does not randomize every profiling buffer unconditionally.

It initializes them specifically when the autotuner intends to use their outputs as correctness evidence.

Why the initialization was not enabled for timing-only runs

Random buffer initialization has a cost.

It requires:

launching or scheduling initialization work;
writing the input allocations;
and completing enough of that work before profiling begins.

When the autotuner is only measuring candidate latency and will not compare their outputs, that cost does not strengthen the selection result.

The patch therefore avoids this policy:

Every autotuning run
→ initialize every input

It uses the narrower rule:

Correctness comparison enabled
→ initialize the inputs being compared

That preserves the previous timing-only behavior while repairing the evidence path.

Redzones and random values detect different failures

The word:

check_buffers

can make these mechanisms sound interchangeable.

They are not.

Redzone checking

Padding is placed around an allocation.

After execution, the checker asks:

Did the kernel write beyond the permitted buffer?

This can detect some out-of-bounds writes.

Output comparison

The candidate result is compared with a reference.

The checker asks:

Did the candidate compute the same values?

This can detect semantic wrong results.

Random input initialization

The operands receive deliberate non-degenerate values.

The initializer asks:

Will common implementation differences
become observable in the outputs?

A random input does not replace a redzone.

A redzone does not make zero-valued arithmetic discriminating.

Output comparison requires both a valid reference and useful inputs.

The reference and candidates must see the same random values

The fix does not independently randomize each candidate.

That would make comparison impossible.

The autotuner creates the input buffers once:

CreateInputBuffers()

It then profiles the reference and every candidate using those same buffers.

Randomly initialized input set I

Reference(I)
Candidate A(I)
Candidate B(I)
Candidate C(I)

This preserves the necessary equality:

same input

while improving the chance that an incorrect implementation produces:

different output

Randomness changes the test point.

It does not change between candidates within that comparison.

One nonzero sample still cannot prove a kernel correct

The patch makes the checker stronger.

It does not make it complete.

A candidate can still be wrong only for:

a particular boundary tile;
a rare stride;
one sign combination;
NaN or infinity;
extreme magnitudes;
one alignment;
one dynamic dimension;
or another state not represented by the generated profile values.

Two different functions can also agree on one randomly selected input by coincidence.

candidate(I)
=
reference(I)

does not mathematically prove:

candidate(x)
=
reference(x)

for every x

Autotuning correctness checks are practical filters, not formal equivalence proofs.

PR #35602 closes an avoidable blind spot.

It does not eliminate every possible false negative.

Some operations need more than generic random values

Certain GPU operations accept inputs that are partly data and partly structural metadata.

Examples can include:

group sizes;
sequence lengths;
indices;
sparse offsets;
or descriptor-like arrays.

Arbitrary random values may violate those inputs’ own contracts.

Current OpenXLA code includes operation-specific initialization for some such cases—for example, grouped GEMM sizes—after the generic profile buffers are created. The general should_init_buffers path remains the starting point, while operation-aware logic corrects inputs that require stronger structure.

This reinforces the larger principle:

Useful correctness input

≠

arbitrary bytes in every position

The test data must be:

valid for the operation

and

discriminating among implementations

The current implementation retains the same contract

The original AutotunerPass has since been reorganized into the current configuration-assignment pipeline.

The equivalent policy remains visible in OpenXLA main.

Current code calculates:

bool check_buffers =
    is_buffer_check_supported
    &&
    debug_options.xla_gpu_autotune_level() >= 4;

It then sets:

profile_options.should_init_buffers =
    check_buffers;

The output-checking option is enabled under the same condition.

correctness comparison

and

input initialization

still share one gate.

The current profiler continues to pass that flag into RedzoneBuffers, and the current allocator continues to invoke InitializeBuffer() only when initialization is requested.

The PR did not publish a named wrong backend

PR #35602 is unusually small.

It changed one production file:

xla/service/gpu/autotuning/autotuner_pass.cc

The public description states the correctness problem:

zero-initialized values can prevent the buffer checker
from detecting backends with wrong results

But it does not identify:

one named backend algorithm that passed incorrectly;
one production model affected by the selection;
one numerical before-and-after output;
one hardware configuration;
or one measured performance consequence.

The PR also did not add a dedicated regression file in its own changed-file set.

The strongest direct public evidence is therefore:

merged configuration defect

+

explicit maintainer-stated failure mode

+

current retention of the repaired policy

—not a published end-to-end wrong-kernel incident.

That boundary should remain visible.

What the merged evidence confirms

The available source confirms that:

XLA can compare autotuned candidate outputs with a reference output;
candidates that disagree can be marked as wrong results and excluded;
buffer checking was enabled at GPU autotune level four or above;
ProfileOptions::should_init_buffers controlled random profile-buffer initialization;
the option defaulted to false;
before #35602, the profile-options builder did not derive it from check_buffers;
the PR sets should_init_buffers equal to the correctness-check state;
input buffers are shared across the reference and candidate runs;
explicit random initialization occurs only when the flag is true;
and the same policy remains present after the current autotuner refactor.

What the evidence does not establish

The public record does not establish that:

every pre-fix profile buffer contained zero;
one named backend was selected despite producing wrong production results;
a user model was publicly shown to fail because of this exact omission;
random initialization detects every incorrect implementation;
one sampled input proves candidate equivalence;
every operand can safely receive unconstrained random values;
the change altered the selected kernel in a published benchmark;
or the initialization cost was measured.

The supported conclusion is narrower:

OpenXLA’s autotuner could request output correctness checks without requesting deliberate input initialization. That made the checker vulnerable to degenerate data—especially zeros—on which incorrect implementations could agree with the reference.

The contract that remains

Output equality is evidence about:

one candidate

on one input

Its value depends on the input carrying enough information to distinguish the implementation being tested.

Reference output exists
+
comparison is exact
+
input erases every difference

→ weak evidence

The fix did not change how XLA compares outputs.

It changed whether those outputs were worth comparing.

A correctness test must choose inputs that give incorrect implementations something to get wrong.

That is the autotuning contract repaired by OpenXLA #35602.

Related material

Patch status: Merged into OpenXLA main on December 22, 2025, as 6fc1fe6a; the equivalent correctness-check/input-initialization linkage remains in current main.

Search This Blog

Resonetic Lab | AI Systems Code Review