ONNX Runtime #29631 — Why cuDNN AveragePool Could Not Represent ONNX’s End Padding or Dilation

CUDA pooling dispatch, descriptor capability mismatch, and the custom kernel that restored the missing window semantics

The ONNX operation could describe this:

begin padding

end padding

dilation

The cuDNN pooling descriptor could describe this:

one symmetric padding value per axis

no dilation

That difference was enough to produce a valid CUDA execution with the wrong averages.

The old ONNX Runtime path calculated the output shape using the complete ONNX attributes.

It then configured cuDNN by copying only:

pads[0 .. spatial_rank)

—the begin-padding half of the ONNX pad array.

The end-padding half was not represented.

There was nowhere to put it.

Dilation had an even simpler failure mode:

cuDNN pooling descriptor

→ no dilation field

So a dilated ONNX AveragePool could reach cuDNN as an ordinary non-dilated pool.

No CUDA error was required.

The descriptor was valid.

It simply described a different operation from the one in the ONNX graph.

PR #29631 added a capability boundary:

Symmetric padding
+
dilation = 1
→ keep cuDNN

Asymmetric padding
or
dilation > 1
→ use ONNX Runtime’s custom CUDA kernel

A backend call is not correct merely because the library accepts it. The backend must first prove that the library can represent the source operation without losing an attribute.

One ONNX pad array contained two sides

For a one-dimensional pool, ONNX stores padding as:

[pad_begin, pad_end]

For two dimensions:

[
    height_begin,
    width_begin,
    height_end,
    width_end
]

The two sides do not need to match.

A valid request can use:

pad_begin = 0

pad_end = 3

This affects more than output shape when:

count_include_pad = 1

Padded positions participate in the divisor even though they contribute zero to the sum.

The old CUDA path calculated the output dimensions with both pad values, but its cuDNN descriptor received only:

pad_begin = 0

cuDNN therefore saw no padding on either side.

ONNX output shape
→ includes the tail-padded windows

cuDNN pooling arithmetic
→ does not know the tail padding exists

The tensor could have the expected number of outputs while those outputs used the wrong divisors.

The minimum 1D example

The merged regression uses:

input:
[1, 2, 3, 4, 5, 6, 7, 8, 9]

kernel:
7

stride:
3

pads:
[0, 3]

ceil_mode:
1

count_include_pad:
1

The expected output is:

[4.0, 5.5714283, 4.0]

The old cuDNN path produced:

[4.0, 6.5, 8.0]

The first output does not expose the defect.

Its window contains seven ordinary input values:

1 + 2 + 3 + 4 + 5 + 6 + 7
=
28

28 / 7
=
4

Both implementations agree.

The difference begins at the right boundary.

The second window needed one padded position

The second window begins at input index three.

Its in-bounds values are:

4, 5, 6, 7, 8, 9

Their sum is:

ONNX’s tail padding contributes one legal padded position to this window.

With:

count_include_pad = 1

the divisor is therefore:

The correct result is:

39 / 7

=

5.571428...

cuDNN had received:

padding = 0

It knew only about the six in-bounds values.

Its result became:

39 / 6

=

6.5

The data values were not corrupted.

The window contract was incomplete.

The final window exposed the largest difference

The final window begins at index six.

Its real values are:

7, 8, 9

Their sum is:

The ONNX padded extent allows three tail-padding positions after those values.

A seventh nominal position produced by ceil_mode lies beyond the declared input-plus-tail-pad extent and is not included in the divisor.

The correct divisor is therefore:

3 real positions

+

3 declared padding positions

=

6

The correct result is:

The old cuDNN descriptor still described no padding.

It averaged only the three real values:

That produces the recorded split:

ONNX:
4.0

old CUDA:
8.0

This was not ordinary floating-point tolerance.

The two paths were dividing by different windows.

Two dimensions magnified the same mistake

The corresponding 2D regression used:

pads:
[0, 0, 3, 3]

kernel:
[7, 7]

stride:
[3, 3]

At the bottom-right output position, the old CUDA path returned:

71.0

The correct result was:

17.75

Why such a large gap?

The real bottom-right input region had an average of 71.

But with:

count_include_pad = 1

that sum had to be divided across the larger declared padded window.

The missing tail padding affected both axes.

one-dimensional divisor error

→ multiplied across height and width

The value 71 was plausible.

It was also the result of a different pooling request.

Asymmetric padding was most dangerous when pad cells affected the divisor

There is an important boundary here.

For:

count_include_pad = 0

the average divides only by in-bounds cells.

In the same 1D example, the correct exclude-pad result is:

[4.0, 6.5, 8.0]

That happens to match what the old cuDNN route produced when it did not represent the tail pad.

The merged dispatch guard still sends all asymmetric AveragePool requests to the custom kernel:

if (
    asymmetric_pads
    ||
    !default_dilations
) {
    use custom CUDA kernel;
}

That is a conservative rule.

It is wider than the minimum correctness boundary for some exclude-pad cases.

The review record explicitly noted that this can route an asymmetric exclude-pad request away from cuDNN even when the numerical result would have matched.

The optimization was left for a separate change.

Therefore the accurate claim is:

Asymmetric include-pad pooling
→ confirmed correctness failure in the old route

All asymmetric pooling
→ conservatively routed to the custom kernel after the patch

No performance comparison for that broader routing was published.

`SAME_UPPER` and `SAME_LOWER` could create the same problem

A model does not need to contain an explicit pad array such as:

[0, 3]

ONNX Runtime can derive asymmetric pads from:

auto_pad = SAME_UPPER

or:

auto_pad = SAME_LOWER

For some input, stride, and kernel combinations, the total padding is odd.

One side receives the extra position.

For example:

SAME_UPPER
→ [0, 1]

or:

SAME_LOWER
→ [1, 0]

The dispatch check runs after ONNX Runtime resolves the final pad values.

It compares:

pads[i]

with:

pads[spatial_rank + i]

Therefore naturally asymmetric SAME_* padding reaches the custom kernel even though the model did not author separate numeric pads directly.

`ceil_mode` alone was not the routing condition

It would be easy to summarize the failure as:

ceil_mode was broken on CUDA

That would be too broad.

A symmetric explicit-padding case with:

ceil_mode = 1

count_include_pad = 1

remains on cuDNN.

The pre-existing CUDA test for that configuration already validated the clamped ONNX divisor.

The custom route is selected when the resolved request contains:

asymmetric pads

or:

non-default dilation

ceil_mode can help create boundary windows and can interact with SAME_* padding.

It is not independently sufficient to trigger the new kernel.

Dilation was a separate representability failure

Padding was not the only lost attribute.

ONNX AveragePool can specify:

dilations > 1

A three-element kernel with dilation two samples:

position 0

position 2

position 4

rather than:

position 0

position 1

position 2

Consider the input:

[1, 2, 3, 4, 5, 6, 7, 8, 9]

with:

kernel = 3

dilation = 2

pads = [2, 2]

stride = 1

At a central output position, the intended taps can be:

3, 5, 7

The average is:

Ignore dilation, and the taps become adjacent:

3, 4, 5

The average is:

The pads are symmetric.

The padding guard alone cannot detect this case.

That is why the merged condition also checks:

!pool_attrs_.default_dilations

A non-default dilation always leaves the cuDNN pooling route, regardless of pad symmetry.

The descriptor could not be repaired by passing different numbers

This was not a simple argument-order bug.

For asymmetric padding, the destination interface contained only one padding value per axis.

For dilation, it contained no corresponding field.

There was no correct combination of existing cuDNN descriptor arguments that could preserve the complete ONNX request.

The backend had three broad choices:

1. Reject the unsupported operation.

2. Transform the input and operation into a representation cuDNN can express.

3. Execute the operation through another kernel.

PR #29631 chose the third.

The ordinary cuDNN route remains available where its descriptor is sufficient.

The unsupported semantic region receives a dedicated implementation.

The custom kernel keeps both sides of every window

The new CUDA kernel receives separate values for:

pad_head

pad_tail

dilation

for each spatial axis.

For one dimension, its window begins at:

start
=
output_index × stride
-
pad_begin

Its upper bound is:

end
=
min(
    start + kernel_size × dilation,
    input_size + pad_end
)

The loop advances by:

dilation

and adds only in-bounds input elements.

out-of-bounds tap

→ no input read

→ zero contribution

The kernel separately records:

counted
=
number of in-bounds taps

That allows it to implement both divisor policies.

The two divisor rules remain distinct

Exclude padding

For:

count_include_pad = 0

the divisor is:

counted

Only values actually read from the input contribute.

Include padding

For:

count_include_pad = 1

the divisor is derived from the number of legal tap positions inside the declared padded extent.

For each axis:

1
+
(end - start - 1) / dilation

The multi-dimensional divisor is the product across the active axes.

This preserves three separate boundaries:

real input

declared padding

ceil-generated phantom extent beyond declared padding

Real and declared padded positions count.

Phantom positions beyond:

input_size + pad_end

do not.

That is why the last 1D example divides by six rather than three or seven.

Half-precision input accumulates in float

The custom implementation instantiates multiple CUDA data types.

For:

FP16

BF16

the accumulator type is:

float

The final result is converted back to the requested output type.

The checked-in regression includes a CUDA-only FP16 case for the asymmetric include-pad path.

The source also instantiates BF16 support, although the test and schema coverage varies by operator version.

This patch was not intended to change numerical precision policy beyond matching the existing pooling semantics.

The common path remains on cuDNN

The custom kernel is not used for every AveragePool.

The dispatch can be summarized as:

AveragePool request	CUDA path
GlobalAveragePool	cuDNN
Symmetric pads, dilation 1	cuDNN
Explicit asymmetric pads	Custom CUDA kernel
Resolved asymmetric `SAME_UPPER` or `SAME_LOWER`	Custom CUDA kernel
Any dilation greater than 1	Custom CUDA kernel

This preserves cuDNN for the ordinary case.

The code path for symmetric, non-dilated pooling is not rewritten by the patch.

That is a source-level routing fact.

The PR did not publish a performance benchmark comparing the two kernels, so the article should not claim a measured zero-cost result across all affected shapes.

MaxPool did not share the same divisor failure

The patch is limited to:

AveragePool

MaxPool does not divide a sum by a window size.

Padding cells do not change its result in the same way because they are ignored rather than included in an average divisor.

The test suite includes an asymmetric MaxPool control and keeps it on its existing path.

Dilated MaxPool already had a separate custom CUDA route when non-default dilation was present.

The presence of similar attributes does not mean both pooling operators share the same representability failure.

The regressions test capability boundaries, not only one output

The final suite covers:

explicit asymmetric tail padding

count_include_pad = 1

count_include_pad = 0

SAME_UPPER

SAME_LOWER

symmetric padding with dilation > 1

symmetric-padding control

FP16

NCHW and CUDA-NHWC 1D/2D paths

NCHW 3D asymmetric padding

The float tests use the CPU reference implementation as the oracle while leaving CUDA enabled as the target under test.

The FP16 case runs CUDA-only because the available CPU configurations could not provide a valid matching FP16 oracle for that boundary.

The PR author reported the full PoolTest run on A100 as:

53 passed

2 DML-skipped

0 failures

The retained tests establish NCHW 3D coverage and CUDA-NHWC coverage for the included 1D/2D cases.

An explicit 3D NDHWC regression was not part of the merged test set.

What the merged evidence confirms

The source and test record confirm that:

ONNX stores separate begin and end padding values;
the old cuDNN descriptor setup copied only the begin-padding half;
the cuDNN pooling descriptor had no dilation field;
asymmetric include-pad AveragePool could therefore use the wrong divisor on CUDA;
dilated AveragePool could execute as a non-dilated pool;
the recorded 1D result changed from [4, 6.5, 8] to the correct [4, 5.5714283, 4];
the recorded 2D boundary value changed from 71.0 to 17.75;
PR #29631 adds a custom CUDA AveragePool kernel;
asymmetric or non-default-dilation requests are routed to it;
symmetric, non-dilated and global pooling remain on cuDNN;
the custom kernel retains per-side pads, dilation, and both divisor modes;
and the dispatch guard, kernel, and regressions remain in current main.

What the evidence does not establish

The public record does not establish that:

every asymmetric exclude-pad request was numerically wrong before the patch;
every cuDNN pooling configuration was affected;
MaxPool shared this exact defect;
every execution provider had the same behavior;
every tensor layout and rank received identical runtime coverage;
the custom kernel is faster than cuDNN;
the conservative asymmetric routing has no cost;
or a released ONNX Runtime package containing the fix was validated across every CUDA and cuDNN version.

The supported conclusion is narrower:

ONNX Runtime’s CUDA AveragePool path sent requests to a cuDNN descriptor that could not represent separate end padding or dilation. The merged fix routes that unsupported semantic region to a custom CUDA kernel while retaining cuDNN for representable pooling configurations.

The complete boundary

ONNX AveragePool request
        ↓
output shape computed with full ONNX attributes
        ↓
cuDNN descriptor created
        ↓
only begin padding is copied
        ↓
dilation has no destination field
        ↓
descriptor remains valid
        ↓
CUDA executes a different pooling window
        ↓
output shape can look correct
while boundary averages are wrong

The repaired path is:

Resolve final ONNX pads
        ↓
check representability
        ├── symmetric + dilation 1
        │   → cuDNN
        │
        └── asymmetric or dilated
            → custom CUDA kernel
            → preserve both pads and dilation

The contract that remains

A backend adapter sits between two operator languages.

The source language may express more than the target library.

When that happens, copying the fields that happen to exist is not a valid lowering.

Source operation
→ richer contract

Library descriptor
→ narrower contract

The missing attributes do not disappear.

They become unrepresented semantics.

Before dispatching to a vendor library, the backend must prove that the destination descriptor can encode every source attribute that changes the result.

ONNX Runtime #29631 repaired AveragePool by making that proof explicit.

Related material

Patch status: Merged into ONNX Runtime main on July 10, 2026, as f79df447; the dispatch guard, custom kernel, and parity regressions remain in current main.

Search This Blog

Resonetic Lab | AI Systems Code Review