NVIDIA CUTLASS kElementsPerAccess — Why Vectorization Exposed a Hidden Addressing Error

 

CUTLASS #3017, Part 2 of 3 — How dividing by one concealed the asymmetry between load() and store()

In Part 1, I examined a problem in NVIDIA CUTLASS’s RegularTileIterator<layout::PitchLinear> implementation.

load() and store() received the same tile coordinate, yet they could calculate different memory addresses.

The difference was a single term.

load() divided the contiguous coordinate by ThreadMap::kElementsPerAccess.

Before the fix, store() did not.

But the mismatch did not appear in every configuration.

When kElementsPerAccess was 1, the division changed nothing.

The bug was not absent.

Dividing by one was simply hiding it.

What does kElementsPerAccess represent?

A CUTLASS tile iterator divides a large matrix or tensor into smaller tiles and calculates which memory locations each GPU thread should read from or write to.

Under the Tile Iterator model, load() reads a tile from memory into thread-level fragments, while store() writes those fragments back to memory.

Both paths can adjust their access position by applying a logical offset to an internal pointer.

A thread, however, does not always access memory one element at a time.

Several adjacent elements may be grouped into a single access unit.

ThreadMap::kElementsPerAccess represents the number of elements contained in one such access.

For example:

kElementsPerAccess = 1
One element is handled per access.

kElementsPerAccess = 2
Two adjacent elements are handled per access.

kElementsPerAccess = 4
Four adjacent elements are handled per access.

This value is therefore more than a performance-tuning parameter.

When the access width changes, the mapping from logical element coordinates to access-unit coordinates must change with it.

Element coordinates and access-unit coordinates are not the same

Suppose an iterator needs to move 64 elements along the contiguous dimension.

If each access handles one element, the distance corresponds to 64 access units:

64 elements ÷ 1 element per access
= 64 access units

If each access handles four elements, the same logical distance corresponds to only 16 access units:

64 elements ÷ 4 elements per access
= 16 access units

The logical displacement is still 64 elements.

But once four elements are grouped into each access, its coordinate in access units is 16 rather than 64.

The affected CUTLASS load() path accounted for this conversion:

tile_offset.contiguous()
    * Shape::kContiguous
    / ThreadMap::kElementsPerAccess

Before the patch, store() omitted the final division:

tile_offset.contiguous()
    * Shape::kContiguous

Both paths received the same tile coordinate.

One converted the logical element distance into access units.

The other did not.

That mismatch could produce different base addresses whenever kElementsPerAccess was greater than 1.

Why did nothing appear wrong when kElementsPerAccess = 1?

The formulas can be reduced to a simple model.

Let:

  • C be the contiguous coordinate

  • K be the number of elements per access

Then:

load offset  = C / K
store offset = C

When K = 1:

load offset  = C / 1 = C
store offset = C

The results are identical.

The source code still contains an asymmetry, but the runtime values do not expose it.

For example, if C = 64:

load  = 64 / 1 = 64
store = 64

A test limited to this configuration would not reveal the missing division in store().

The formulas had not become equivalent.

The value 1 had merely eliminated their numerical difference.

CUTLASS Issue #3017 noted that many ordinary SIMT configurations could use kElementsPerAccess = 1, turning the division into a no-op and allowing the mismatch to remain dormant.

When the value becomes 4, the addresses separate by a factor of four

Now keep the same logical coordinate and change only kElementsPerAccess to 4.

Assume:

Contiguous coordinate C = 64
Element type            = float
Size of float           = 4 bytes
kElementsPerAccess K    = 4

load() accounts for the access width:

64 / 4 = 16
16 × 4 bytes = 64 bytes

Before the fix, store() did not:

64 × 4 bytes = 256 bytes

For the same logical tile coordinate:

load  → 64-byte offset
store → 256-byte offset

The store() offset becomes four times larger than the load() offset.

If kElementsPerAccess were 2, the difference could be a factor of two.

If it were 8, the difference could be a factor of eight.

As the vector width increased, the pre-existing asymmetry expanded with it.

Vectorization did not introduce a new addressing bug.

A value greater than one turned an existing formula mismatch into an observable address mismatch.

Testing only the origin can hide the same problem

kElementsPerAccess = 1 was not the only condition capable of concealing the bug.

If the tile offset itself is zero, both paths also produce the same result:

C = 0

load  = 0 / K = 0
store = 0

A test suite can therefore miss the asymmetry if it repeatedly exercises only:

  • kElementsPerAccess = 1

  • tile_offset.contiguous() = 0

Reading and writing the first tile with scalar-width accesses may pass every time.

The mismatch becomes visible only when both of the following are true:

The access width is greater than 1
+
The iterator moves to a nonzero tile coordinate

This is why the issue is best understood as a boundary-condition failure.

Correct behavior under ordinary inputs does not prove that the address-mapping contract remains consistent across the full parameter space.

What should a vectorized test have covered?

A test designed specifically for this failure should do more than verify the final output of a kernel.

It should also test the address invariant directly.

For the same logical tile coordinate, the test space should include combinations such as:

kElementsPerAccess = 1
kElementsPerAccess = 2
kElementsPerAccess = 4

tile_offset.contiguous() = 0
tile_offset.contiguous() > 0

The central invariant is straightforward:

Given the same logical tile coordinate, load() and store() must calculate the same base address.

Conceptually, the test could compare:

load_base_address(tile)
==
store_base_address(tile)

Another option would be to write a fragment at one tile coordinate and then read from the same coordinate, verifying that the original data is recovered.

The important point is not to test only a single default configuration.

A parameter used inside the address formula must be exercised at values that actually change the formula’s behavior.

When a performance parameter becomes a correctness parameter

At first glance, kElementsPerAccess looks like a performance setting.

It controls how many adjacent elements are handled in one memory access, so it naturally appears related to throughput and memory efficiency.

In this case, however, the value also affected the coordinate system used to calculate the address.

Changing the access width changes the transformation:

Logical number of elements
→ Number of access units
→ Final byte address

load() applied that transformation.

Before the patch, store() did not.

The same value was treated as structurally meaningful in one path and effectively ignored in the other.

At that point, kElementsPerAccess was no longer only a performance parameter.

It had become part of the correctness contract that determines which memory location the iterator addresses.

A passing default configuration does not prove the contract is correct

Default values can conceal many classes of bugs.

Zero removes offset differences.

One removes differences between multiplication and division.

An empty array may avoid entering the loop that contains the defect.

A single device cannot expose a distributed synchronization failure.

These are all legitimate test inputs.

But they are also degenerate cases in which different implementations may accidentally produce the same result.

In CUTLASS #3017, kElementsPerAccess = 1 made the different load() and store() formulas numerically identical.

A passing default path therefore proved only this:

The implementation produced the same result under the default condition.

It did not prove this:

The address-mapping contract remains consistent for every supported access width.

Those are different claims.

Vectorization did not create the bug. It tested the boundary.

Calling this only a “vectorization bug” can obscure the actual cause.

Vectorization itself was a valid feature.

It simply configured the iterator to handle several elements in one access.

The failure appeared because load() and store() did not apply the same coordinate transformation when that configuration was active.

The causal sequence was not:

Vectorization
→ A new bug is created

It was closer to:

Existing formula asymmetry
+
kElementsPerAccess > 1
→ The hidden address mismatch becomes observable

Vectorization was not the source of the defect.

It was the condition that finally exercised the existing contract.

The bug had been present all along. The default value was hiding it.

The next question is the unit of the offset

The address formulas used by load() and store() were later aligned.

But CUTLASS Issue #3017 raised another question.

What does the number passed to add_pointer_offset() represent?

Is it a number of elements?

Or a number of bytes?

The higher-level Tile Iterator contract reads as though the offset is expressed in elements.

The affected PitchLinear implementation, however, added the value directly to a uint8_t*, which makes the operation behave as a byte offset.

The current path can still work if both the caller and the implementation implicitly agree on the same unit.

But if that unit is not declared clearly at the interface boundary, a future caller can interpret the same number differently.

Part 3 examines this ambiguity between Element-based and Byte-based offsets, and why internally working code is not necessarily the same thing as a clearly defined API contract.

Previous article

NVIDIA CUTLASS #3017 — Why load() and store() Mapped the Same Tile Coordinate to Different Addresses

https://resoneticlab.blogspot.com/2026/08/nvidia-cutlass-addpointeroffset-where.html

Related material

This is Part 2 of a three-part series on CUTLASS #3017.

Part 3 examines whether the value passed to add_pointer_offset() represents Elements or Bytes, and why code that works internally may still expose an ambiguous interface contract.

#NVIDIA #CUTLASS #CUDA #GPUProgramming #Vectorization #CodeAnalysis #SoftwareArchitecture


Popular posts from this blog

AMD Is Trying to Turn Kernel Diversity Into One Software Asset

The first broad integration was rolled back, but the direction of the software stack is difficult to ignore The GPU software race is often described as a contest over who can write the fastest kernel. That is only part of the problem. A company may have several teams capable of producing high-performance matrix-multiplication kernels. Some may work through template libraries. Others may use compiler-generated code, handwritten assembly, or model-specific optimization stacks. The existence of those kernels does not automatically create one coherent platform. They still need to be: discovered validated matched to the correct problem given the correct arguments launched through a stable runtime and delivered through a reliable software package Recent public code changes inside AMD’s software ecosystem showed an attempt to connect these layers more directly. The central idea was to allow hipBLASLt’s GEMM selection system to choose kernels created by multiple internal kernel technologi...
Read more

PyTorch #188031 — Why Did Forward Use 64-Bit Indexing While Backward Still Used uint32_t?

PyTorch Spatial Softmax, Part 1 of 3 — An asymmetric indexing contract and the patch that aligned it A recent PyTorch change fixed a 64-bit indexing problem in the spatial softmax backward path. The bug was not in the softmax formula itself. It was in the width of the integer used to calculate memory locations. For the same large tensor, the forward path already selected either 32-bit or 64-bit indexing based on the input size. One backward path, however, continued to calculate sizes, strides, and offsets using uint32_t . That created a possible split in behavior: Forward → 64-bit address calculation → accesses the intended elements Backward → 32-bit address calculation → offset wraparound → gradients read from or written to different locations A tensor could therefore complete the forward pass correctly while the corresponding backward kernel could no longer represent every address in that tensor. The changes described in PyTorch #188031 were later landed in PyTorch’s main branch. S...
Read more

NVIDIA CUTLASS #3017 — Why load() and store() Mapped the Same Tile Coordinate to Different Addresses

CUTLASS #3017, Part 1 of 3 — An asymmetry in address calculation and the patch that fixed it In February 2026, while reading through NVIDIA CUTLASS’s RegularTileIterator<layout::PitchLinear> implementation, I noticed an odd asymmetry. load() and store() accepted the same tile coordinate, yet they calculated the corresponding memory address differently. The difference was a single line. load() divided the contiguous coordinate by kElementsPerAccess . store() did not. When each memory access handled only one element, the difference remained invisible. But in vectorized paths, where a single access handles multiple elements, the same logical tile coordinate could resolve to two different physical addresses. I reported the issue as CUTLASS Issue #3017. The missing division was later added to store() in PR #3049. CUTLASS and tile iterators CUTLASS is NVIDIA’s open-source library for high-performance CUDA linear algebra. It provides C++ template abstractions for composing GPU ope...
Read more