ONNX Runtime PR #32121: Why a CUDA Graph Replayed an Overwritten Pointer Array

Why a CUDA Graph Replayed an Overwritten Pointer Array

An asynchronous copy can be finished for the current run and still depend on its source buffer during a future CUDA Graph replay.

That distinction caused a subtle lifetime bug in ONNX Runtime’s CUDA Execution Provider.

CudaAsyncBuffer placed data such as a kernel’s pointer array in pinned host memory, scheduled a host-to-device copy, and then returned the host buffer through the normal deferred-release path.

That was valid during ordinary execution.

It was not valid when the stream was being captured.

The graph remembered an address, not a private copy

Outside CUDA Graph capture, ONNX Runtime could treat the pinned buffer as temporary.

The runtime scheduled cudaMemcpyAsync, kept the buffer alive until the stream consumed it, and then allowed the allocator to reuse the memory.

The lifetime was tied to one queued copy.

During stream capture, the operation had a different future.

The copy became a node in the captured graph. When that graph was launched again, the node read from the host address recorded during capture.

The bytes at that address therefore needed to remain valid across later graph replays.

Returning the buffer to the allocator did not immediately make the stored address disappear. It made the address available for another allocation.

That created the failure.

Why a decode-only loop could hide the bug

Suppose a decode graph was captured while its pointer array occupied a pinned host allocation.

The graph recorded a host-to-device copy from that address.

ONNX Runtime then returned the host allocation through its usual deferred-release mechanism.

If the application only replayed the same decode workload, nothing might immediately request and overwrite that block. The stale lifetime assumption could remain invisible.

Now insert a differently shaped prefill run.

That run can request a compatible pinned allocation, receive the recycled block, and write a different pointer array into it.

The next decode replay still reads from the original host address.

But the address now contains the prefill run’s bytes.

The graph copies those bytes into device memory, and a kernel such as Concat interprets them as its own array of input pointers.

The graph itself has not changed. The meaning of the memory behind its recorded address has.

That is why the failure can appear only after another inference runs between capture and replay.

The fix changes ownership during capture

ONNX Runtime PR #32121 makes CudaAsyncBuffer::CopyToGpu inspect the stream’s capture status after scheduling the copy.

When the stream is not being captured, the previous behavior remains:

AddDeferredReleaseCPUPtr(...);

The host buffer stays alive until the ordinary asynchronous copy has consumed it and can then return to the allocator.

When capture is active, the buffer follows another path:

RetainBufferForGraphCapture(...);

Ownership moves into provider-scoped storage instead of returning to the allocator.

The captured graph can continue reading the same host address during replay because another run cannot reclaim and overwrite that allocation.

The fix is conservative. The buffer is retained until the provider is destroyed rather than being released at the exact end of an individual graph’s lifetime.

That may retain the allocation longer than strictly necessary, but it restores the required address stability without introducing a new graph-lifetime ownership system.

The plugin path needed the same repair

The first version of the patch covered the built-in CUDA Execution Provider.

ONNX Runtime also has a plugin CUDA EP path with a parallel CudaAsyncBuffer implementation. Under the plugin build configuration, the public CUDA kernel header selects that adapter instead.

The plugin implementation still used the original deferred-release behavior.

That meant the same operators could remain exposed to the same stale host-address problem when built through the plugin boundary.

Review caught the missing parallel path.

The final patch added capture detection and provider-scoped buffer retention to both implementations.

This was not an optional consistency cleanup. Both paths supported the behavior that created the lifetime requirement.

Fixing only one would have made correctness depend on how the CUDA EP was packaged.

What the regression evidence proves

The PR reports a compute-sanitizer failure before the fix in a workload that interleaved prefill with graph-replayed decode.

The failure appeared as an invalid, misaligned global read in a Concat kernel. After the lifetime change, the same reported workload ran cleanly.

The checked-in test covers the new retention container more narrowly. It verifies that the container keeps ownership of a buffer and releases it when the container itself is destroyed.

That proves the added ownership mechanism.

It does not reproduce the complete CUDA sequence:

capture
→ allocator reuse
→ overwrite
→ replay
→ incorrect access

The implementation and reported sanitizer result support the fix. The automated regression boundary remains less complete than the original failure path.

What the patch deliberately leaves unchanged

The change protects the pinned host buffer used as the source of the captured copy.

It does not redesign every allocation involved in CUDA Graph replay.

The PR specifically leaves the destination scratch allocation under ONNX Runtime’s existing graph address-stability assumptions. It also does not claim a measured performance improvement.

The supported conclusion is narrower:

A captured copy continued to depend on a host address after ONNX Runtime had allowed the allocator to reuse it. The fix preserves that host allocation whenever the copy is captured into a graph.

The broader lesson is not limited to CUDA.

A resource’s lifetime belongs to its last future reader, not merely to the call that first appeared to consume it.

During ordinary execution, the last reader may be one queued operation.

During graph capture, the last reader may be a replay that has not happened yet.

Sources

https://github.com/microsoft/onnxruntime/pull/32121

https://github.com/microsoft/onnxruntime/commit/cd13bdd819dcbdb9e0be2e8c38d2698ae2e56097

https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/core/providers/cuda/cuda_kernel.h

https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/core/providers/cuda/plugin/cuda_kernel_adapter.h

https://github.com/microsoft/onnxruntime/blob/main/include/onnxruntime/core/framework/allocator.h

Popular posts from this blog

AMD Is Trying to Turn Kernel Diversity Into One Software Asset

The first broad integration was rolled back, but the direction of the software stack is difficult to ignore The GPU software race is often described as a contest over who can write the fastest kernel. That is only part of the problem. A company may have several teams capable of producing high-performance matrix-multiplication kernels. Some may work through template libraries. Others may use compiler-generated code, handwritten assembly, or model-specific optimization stacks. The existence of those kernels does not automatically create one coherent platform. They still need to be: discovered validated matched to the correct problem given the correct arguments launched through a stable runtime and delivered through a reliable software package Recent public code changes inside AMD’s software ecosystem showed an attempt to connect these layers more directly. The central idea was to allow hipBLASLt’s GEMM selection system to choose kernels created by multiple internal kernel technologi...
Read more

PyTorch #188031 — Why Did Forward Use 64-Bit Indexing While Backward Still Used uint32_t?

PyTorch Spatial Softmax, Part 1 of 3 — An asymmetric indexing contract and the patch that aligned it A recent PyTorch change fixed a 64-bit indexing problem in the spatial softmax backward path. The bug was not in the softmax formula itself. It was in the width of the integer used to calculate memory locations. For the same large tensor, the forward path already selected either 32-bit or 64-bit indexing based on the input size. One backward path, however, continued to calculate sizes, strides, and offsets using uint32_t . That created a possible split in behavior: Forward → 64-bit address calculation → accesses the intended elements Backward → 32-bit address calculation → offset wraparound → gradients read from or written to different locations A tensor could therefore complete the forward pass correctly while the corresponding backward kernel could no longer represent every address in that tensor. The changes described in PyTorch #188031 were later landed in PyTorch’s main branch. S...
Read more

NVIDIA CUTLASS #3017 — Why load() and store() Mapped the Same Tile Coordinate to Different Addresses

CUTLASS #3017, Part 1 of 3 — An asymmetry in address calculation and the patch that fixed it In February 2026, while reading through NVIDIA CUTLASS’s RegularTileIterator<layout::PitchLinear> implementation, I noticed an odd asymmetry. load() and store() accepted the same tile coordinate, yet they calculated the corresponding memory address differently. The difference was a single line. load() divided the contiguous coordinate by kElementsPerAccess . store() did not. When each memory access handled only one element, the difference remained invisible. But in vectorized paths, where a single access handles multiple elements, the same logical tile coordinate could resolve to two different physical addresses. I reported the issue as CUTLASS Issue #3017. The missing division was later added to store() in PR #3049. CUTLASS and tile iterators CUTLASS is NVIDIA’s open-source library for high-performance CUDA linear algebra. It provides C++ template abstractions for composing GPU ope...
Read more