ONNX Runtime PR #32301: Why an FP16 CPU Kernel Caused a 3,556× Gemm Regression
An FP16 model does not automatically run faster because its values use fewer bits.
ONNX Runtime issue #32255 reported the opposite result after upgrading from version 1.28.0 to 1.29.0 on Windows x64. A single FP16 Gemm node went from a warmed-up mean of 0.121 milliseconds to 430.339 milliseconds. That is approximately a 3,556× regression for the reported shape and environment.
The slowdown did not come from converting the model to a larger data type. It appeared after ONNX Runtime stopped converting the operation to FP32.
PR #32301 fixes that routing decision by registering native CPU FP16 Gemm and MatMul kernels only when an accelerated HalfGemm backend is actually available.
The faster path was an FP32 fallback
The minimal reproduction used an FP16 Gemm with these dimensions:
A: [14, 1024]
B: [4096, 1024], transB=1
Y: [14, 4096]In ONNX Runtime 1.28.0, the CPU Execution Provider did not have a matching FP16 Gemm kernel for this path.
That absence mattered. InsertCastTransformer could promote the FP16 operation to an FP32 graph, run the matrix multiplication through the optimized FP32 CPU implementation, and cast the result back when required.
The model contained FP16 values, but the chosen computation used a mature, hardware-optimized FP32 path.
ONNX Runtime 1.29 added CPU FP16 kernels for operations including Gemm and MatMul. Kernel lookup could now find a native FP16 implementation, so the graph transformer no longer had a reason to insert the FP32 fallback.
On a machine without accelerated half-precision matrix multiplication, execution then reached the portable FP16 implementation.
The path looked more direct:
FP16 input
↓
native FP16 kernel
↓
FP16 outputIt was also dramatically slower in the reported Windows x64 case.
The earlier path performed more visible conversions:
FP16 input
↓
cast to FP32
↓
accelerated FP32 Gemm
↓
cast as requiredThose conversions cost less than running the matrix multiplication through an unaccelerated half-precision kernel.
Kernel registration changes graph optimization
A kernel registry does more than answer whether an operator can run.
It also influences which transformations remain available.
When the CPU provider advertises an FP16 Gemm kernel, ONNX Runtime can keep the node in FP16 and assign it to that provider. The existence of the registered kernel prevents the graph from taking the previous FP32-promotion route.
Checking acceleration only inside Gemm::Compute() would therefore be too late to restore the fast path. By that point, the graph has already retained the FP16 node and selected the CPU kernel.
The registration decision needs to express a stronger statement:
This build and machine have
an accelerated implementation
for FP16 matrix multiplication.A merely functional scalar or portable implementation is not enough when registering it suppresses a much faster fallback.
CPU FP16 capability was not the right question
An early version of the fix used MlasFp16AccelerationSupported() as the registration gate.
Review found that this predicate answered a related but different question: whether the CPU reported FP16 acceleration capability.
The actual requirement was whether MLAS could select an accelerated HalfGemm backend for this build, machine, and backend configuration.
That distinction exposed two opposite errors.
ARM64EC could report capability without the required kernel
An ARM64EC machine could report FP16 capability, while the Windows MLAS build did not compile the Neon HalfGemm kernel used by the native ARM64 path.
A capability-only gate could register FP16 Gemm and MatMul even though execution still fell through to the portable implementation.
That would preserve the regression the PR was intended to remove.
KleidiAI could provide acceleration outside the simple capability test
KleidiAI can install an optimized HalfGemm override on supported ARM systems.
A gate based only on the basic FP16 capability query could miss a configuration where that override supplied the accelerated implementation.
That would remove a useful native FP16 kernel and force an unnecessary FP32 fallback.
The final patch centralizes the decision in:
MlasHalfGemmAccelerationSupported(...)The same helper is used by kernel registration and by the FP16 Gemm and MatMul execution paths.
The registration policy and the runtime dispatch policy now ask the same question.
The fix has a compile-time and runtime boundary
The patch adds a compile-time macro:
MLAS_HALF_GEMM_ACCELERATION_POSSIBLEIt identifies builds in which an accelerated HalfGemm backend can exist, including supported native ARM64, KleidiAI-enabled ARM64, and configured RISC-V RVV/Zvfh paths.
Only those builds compile the conditional FP16 Gemm and MatMul registration table.
At runtime, ONNX Runtime evaluates:
MlasHalfGemmAccelerationSupported(nullptr)for the process-wide CPU kernel registry.
If the default backend configuration has an accelerated HalfGemm implementation, the FP16 kernels are registered.
Otherwise, they are absent from the registry and the graph can return to FP32 promotion.
The resulting decision is:
Accelerated HalfGemm available
→ register FP16 Gemm and MatMul
→ keep eligible nodes in FP16
Accelerated HalfGemm unavailable
→ do not register those FP16 kernels
→ allow FP32 promotionThis is narrower than claiming that the CPU supports FP16 values or instructions in some form. It asks whether the operation that matters can run through an accelerated matrix-multiplication backend.
The reported regression was larger than one microbenchmark
Issue #32255 also described a Qwen3-ASR FP16 encoder containing 146 Gemm nodes.
The report compared ONNX Runtime 1.28.0 with 1.29.0:
146 Gemm nodes, total:
24.018 ms → 19,338.133 ms
Median Gemm node:
0.098 ms → 98.738 ms
Encoder for a 5.69-second input:
201 ms → 120,620 ms
End-to-end inference:
2,649 ms → 139,762 msThe issue attributed about 99.4% of the 1.29 encoder execution time to FP16 Gemm.
These measurements establish a severe regression in the reported Windows x64 environment. They do not establish the same ratio for every CPU, shape, model, operating system, or ONNX Runtime build.
The important design result is independent of the exact multiplier: selecting an available kernel can be a performance regression when that kernel is not the best executable path.
The tests check registration and execution
The added registry test checks FP16 Gemm and MatMul across several operator-version ranges:
Gemm:
opsets 8, 10, 12, 13
MatMul:
opsets 8, 10, 12, 13It compares their presence with MlasHalfGemmAccelerationSupported(nullptr).
The same test includes FP32 positive controls. That prevents an empty or broken registry from making all FP16 checks pass vacuously on a machine where the expected FP16 result is false.
Two end-to-end tests build small FP16 MatMul and Gemm models and run them on the CPU provider.
On an accelerated machine, the native FP16 kernel can execute them.
On an unaccelerated machine, the absence of that kernel allows ONNX Runtime to produce a runnable fallback path rather than selecting the slow portable FP16 matrix multiplication.
The tests verify that the routing decision still produces correct results. They do not reproduce the 3,556× benchmark or measure performance in CI.
One session-level exception remains
CPU kernel registration is process-wide and uses the default backend-selector configuration.
A session can later disable KleidiAI with mlas.disable_kleidiai=1.
On a build where KleidiAI is the only accelerated HalfGemm implementation, the process may already have registered FP16 Gemm and MatMul. Disabling KleidiAI for one session does not rebuild the process-wide registry.
That session can therefore retain the FP16 kernel while its accelerated backend is disabled, allowing execution to reach the non-accelerated fallback inside the kernel.
The PR documents this residual asymmetry and leaves a behavioral fix for follow-up work.
It does not affect the main Windows x64 repair, where the absence of an accelerated HalfGemm backend prevents the FP16 kernels from being registered in the first place.
The restored contract
Supporting a data type and accelerating an operation are separate capabilities.
An FP16 kernel may be numerically valid while still being the wrong route for a CPU that can execute FP32 matrix multiplication far more efficiently.
The registration contract restored by PR #32301 is:
Register native FP16 Gemm or MatMul
only when the selected runtime can reach
an accelerated HalfGemm backend.A lower-precision representation is not a performance guarantee. The useful question is which implementation the runtime will actually execute.
Sources
https://github.com/microsoft/onnxruntime/pull/32301
https://github.com/microsoft/onnxruntime/issues/32255
https://github.com/microsoft/onnxruntime/commit/d97b5498a1d608c2fdffb3faf7081233a6373178