ONNX Runtime PR #32410: Why an Unused Pinned Buffer Stalled CUDA Decode
ONNX Runtime’s CUDA Split and Concat operators already had a fast path for a common case: equal-sized partitions with no more than 32 input or output pointers.
That path passed the pointers directly to the CUDA kernel as a small fixed-capacity value. It did not need a host-to-device copy of a pointer table.
The surrounding code still created a CudaAsyncBuffer before it decided which path to run.
Creating that object immediately allocated pinned host memory. When the inline path won, the buffer was never copied to the GPU and never read by the kernel. It was nevertheless allocated and released on every Run().
ONNX Runtime PR #32410 fixes the problem by moving the fast-path decision ahead of the pinned-buffer construction.
The pointer buffer was unused by the selected kernel, but its construction still performed real CUDA memory work.
The inline path already carried its pointers by value
The equal-size implementations of CUDA Concat and Split support two forms of pointer metadata.
For a small number of partitions, the caller can build a TArray:
TArray<const void*, 32> input_ptr_array(input_count);for Concat, or:
TArray<void*, 32> output_ptr_array(num_outputs);for Split.
TArray is a fixed-capacity wrapper intended for small pieces of CUDA launch metadata. ONNX Runtime’s own comment explains the purpose: passing a small array by value is preferable to performing another CUDA memory copy just to place that metadata on the device.
The specialized kernels are templated on the pointer container. They can receive either:
TArray<..., 32>as an ordinary kernel argument, or:
void**pointing to an array stored in device memory.
For 32 or fewer equal-sized inputs or outputs, the first form already contains everything the kernel needs.
The kernel did not need a device-side pointer table. The host code created one anyway.
The cost appeared before the branch
The old structure in both operators was effectively:
prepare tensor shapes
↓
construct CudaAsyncBuffer
↓
allocate pinned host pointer storage
↓
fill the pointer storage
↓
check whether the inline TArray path applies
↓
launch the inline kernel
↓
destroy the unused CudaAsyncBufferThe important detail is the constructor.
A CudaAsyncBuffer created with a count calls AllocCpuPtr(count). That function obtains memory from ONNX Runtime’s CPU-pinned allocator immediately. Allocation is not delayed until CopyToGpu().
When a path really needs a device pointer table, CopyToGpu() also allocates GPU scratch memory, issues cudaMemcpyAsync, and arranges for the pinned source buffer to remain alive until the asynchronous copy is safe.
The inline path never calls CopyToGpu().
Its TArray is populated separately and passed directly into the kernel launch. The earlier CudaAsyncBuffer exists only because it was declared before the code knew that the inline path had been selected.
This is a common performance trap in resource-owning C++ code. A local object can look like harmless preparation even when its constructor has an expensive external side effect.
Pinned host memory is not ordinary temporary storage
Pinned, or page-locked, host memory is useful because it supports genuinely asynchronous transfers between host memory and a CUDA device.
That property also makes its lifecycle more consequential than a normal heap allocation.
CUDA documentation lists page-locked host allocation among operations that can introduce implicit synchronization between otherwise independent work. NVIDIA’s CUDA Graph guidance also treats pinned host allocation and release as operations that cannot be performed normally during stream capture.
The PR describes the visible result in a Qwen 3.8 decode trace.
After removing the unnecessary allocations from the small equal-size paths:
pinned allocation/free pairs:
1,635 → 1,355
time attributed to cudaFreeHost:
4.90 seconds → 11.3 millisecondsThose numbers do not mean the complete model became hundreds of times faster. They are profiler figures for pinned allocation and release activity within the reported trace.
They do show why a tiny pointer table can have a cost unrelated to its byte size. The expensive part was not moving a few pointer values. It was entering the pinned-memory lifecycle in a latency-sensitive decode loop.
The patch makes the branch own its setup
The new Concat flow first gathers the sizes of the input tensors and decides whether every concatenation dimension has the same length.
It then checks the inline-pointer boundary:
if (same_concat_size && input_count <= kMaxInlinePointerCount) {
TArray<const void*, kMaxInlinePointerCount> input_ptr_array(input_count);
for (int i = 0; i < input_count; ++i) {
input_ptr_array[i] = p.inputs[i].tensor->DataRaw();
}
return ConcatSameConcatDimImpl(..., input_ptr_array, ...);
}The function returns from the fast path before any CudaAsyncBuffer exists.
Split now follows the same order. It verifies equal split sizes, creates the output tensors, places their addresses in a TArray<void*, 32>, and returns through SplitSameSplitDimImpl.
Only code that misses the inline path constructs a pinned pointer buffer.
The resulting resource decision is:
equal sizes and no more than 32 pointers
→ pass pointer metadata by value
→ no pinned pointer buffer
equal sizes and more than 32 pointers
→ copy pointer table to device memory
→ use equal-size specialized kernel
unequal sizes
→ copy pointer table and size/mapping metadata
→ use general kernelThe patch does not make every Split or Concat avoid pinned memory. It removes it only from the branch that never needed it.
Why the 32-pointer boundary remains
The number 32 is the capacity of the TArray specialization already instantiated by the CUDA kernels.
TArray enforces that its runtime size does not exceed its compile-time capacity. The final patch names the boundary explicitly:
constexpr int kMaxInlinePointerCount = 32;and uses the same constant for both the dispatch condition and the TArray capacity within each operator.
More than 32 pointers still take the device-array route. That is not an accidental fallback left behind by the optimization. The kernel has no larger by-value specialization available in this path.
The final patch adds CUDA tests with exactly 33 equal-sized inputs for Concat and 33 equal-sized outputs for Split.
Those cases sit one position beyond the inline boundary. They verify that the refactoring did not break the equal-size specialized kernel when its pointer list still has to be copied to device memory.
Review removed another unnecessary allocation
The first version of the refactor still built axis_dimension_input_output_mapping before the equal-size fallback branch.
That mapping is used only by the general unequal-size kernel. It tells the kernel which input or output owns each position along the operated axis.
An equal-size kernel calculates that ownership directly from the common partition size. It does not read the mapping.
For equal-size operations with more than 32 pointers, the early draft therefore avoided the original small-path pinned allocation problem but still performed an unrelated heap allocation and an O(axis dimension) fill loop on every run.
Review identified the mismatch in both Concat and Split.
The merged version creates and fills the mapping only inside the unequal-size branch. It also hoists the loop-invariant equal split dimension assignment, adds direct <algorithm> and <vector> includes, uses the named 32-pointer constant, and leaves comments explaining why the inline dispatch must remain above CudaAsyncBuffer construction.
That comment matters. In this patch, source order expresses resource policy. Moving the declaration upward again would compile and could preserve numerical output while silently restoring the performance problem.
Functional behavior stays on the same kernels
This change does not introduce a new CUDA data-movement algorithm.
For the small equal-size case, it calls the same ConcatSameConcatDimImpl and SplitSameSplitDimImpl overloads that accept a TArray.
For the larger equal-size case, it still calls those specialized implementations with a device pointer array.
For unequal partitions, it still calls the general implementations with pointer, size, range, and axis-mapping buffers.
The refactor changes when metadata storage is allocated. It does not change how tensor elements are assigned to inputs or outputs.
The pull request reports 53 passing Concat and Split tests across four suites, along with a Release ARM64 CUDA build using CUDA 13.4 for SM120 and SM121.
The two new 33-way cases protect the changed fallback boundary. Existing smaller equal-size tests continue to cover the inline behavior.
A successful functional test does not by itself prove an allocation disappeared; that conclusion comes from the code path. The tests establish that removing the allocation did not change the observable tensor results at the affected dispatch boundaries.
The CUDA Graph effect is useful but bounded
Both new fast-path blocks include a comment that avoiding pinned memory also keeps the path CUDA Graph capturable.
That conclusion is appropriately narrow.
The old code performed a pinned allocation even though the selected kernel consumed only by-value pointer metadata. Pinned allocation and release are incompatible with ordinary CUDA stream capture, so removing those operations eliminates that obstacle from this path.
It does not prove that every surrounding model, allocator action, or Split and Concat configuration is capturable. Larger pointer lists and unequal partitions still use CudaAsyncBuffer, and graph capture has additional address-stability and lifecycle requirements.
The supported claim is that the small equal-size path no longer introduces an unnecessary pinned-memory operation of its own.
What the performance evidence does and does not show
The reported trace is strong evidence that the removed lifecycle was expensive in the tested workload.
It is not a universal benchmark for:
all CUDA devices
all model architectures
all Split and Concat shapes
or complete end-to-end inferenceThe PR does not provide a tokens-per-second comparison or a full decode-latency result. It also does not state enough trace context to attribute the same reduction to every system.
The implementation argument does not depend on those numbers.
If a branch passes its pointer list by value, then allocating a pinned host pointer table for that branch is unnecessary. Removing an unnecessary resource operation is correct even on a device where its measured cost is smaller.
The profiler data explains the urgency. The code establishes the boundary.
The contract restored by the change
A fast path should pay only for the resources it consumes.
The corrected sequence is:
inspect the operation
↓
select the execution path
↓
construct only that path’s metadata
↓
launch the kernelThe previous sequence constructed fallback resources first and treated their cost as though it disappeared when the fast path was selected.
It did not disappear. C++ object lifetime made the allocation and release real.
A fast path is not fast if its slow-path setup runs before the branch.
ONNX Runtime PR #32410 fixes that ordering for the common small, equal-size CUDA Split and Concat paths.
Sources
https://github.com/microsoft/onnxruntime/pull/32410
https://github.com/microsoft/onnxruntime/commit/2af39c97088669851e00bd704cbe90944e3593e1
https://docs.nvidia.com/cuda/cuda-programming-guide/03-advanced/advanced-host-programming.html
https://docs.nvidia.com/dl-cuda-graph/cuda-graph-basics/constraints.html