NVIDIA CUTLASS add_pointer_offset() — Where the Contract Between Elements and Bytes Became Ambiguous

CUTLASS #3017, Part 3 of 3 — Why the internal call path worked even though the interface contract was unclear

A number in code does not explain what it measures.

The value 4 might mean four elements.

It might also mean four bytes.

That distinction becomes critical when the number is used to move a memory address.

In C++, adding 4 to a float* advances the pointer by four float elements — 16 bytes.

Adding 4 to a uint8_t*, however, advances it by exactly 4 bytes.

The number is identical.

The unit implied by the pointer type is not.

The second concern raised in NVIDIA CUTLASS Issue #3017 was located at precisely this boundary.

The name add_pointer_offset() and the higher-level Tile Iterator interface appeared to describe an offset measured in elements.

But the affected RegularTileIterator<layout::PitchLinear> implementation treated the value as a byte offset.

The unit promised by the CUTLASS interface

The CUTLASS Tile Iterator concept describes add_pointer_offset() as adding a linear offset to the internal pointer in units of elements.

Reduced to its basic form, the interface looks like this:

void add_pointer_offset(Index pointer_offset);

A caller following that contract would naturally interpret:

add_pointer_offset(4)
=
advance by four elements

If the element type is float, the expected physical displacement is:

4 elements × 4 bytes
= 16 bytes

This unit is not merely an implementation note.

Generic code relies on a common interface so that multiple Tile Iterator implementations can be used without knowing the type of pointer stored inside each one.

The caller should be able to provide a number of elements and trust the iterator to convert that value into the correct physical address.

The affected PitchLinear implementation moved in bytes

The implementation recorded in CUTLASS Issue #3017 had the following form:

void add_pointer_offset(LongIndex pointer_offset) {
  pointer_ += pointer_offset;
}

The important detail is that pointer_ was not an Element*.

It was a uint8_t*.

Because one uint8_t occupies one byte, adding a number to this pointer advances it by that exact number of bytes:

pointer_ += 4
=
advance by 4 bytes

Suppose the element type is float, and a caller trusts the element-based contract:

add_pointer_offset(4);

The caller expects a 16-byte movement.

The implementation moves only 4 bytes.

Expected:
4 float elements
= 16 bytes

Actual:
uint8_t pointer + 4
= 4 bytes

The iterator advances by the size of one float, not four.

The function name stayed the same.

The meaning of the argument changed.

Why did the internal path still work?

This did not mean that every existing call immediately produced an incorrect address.

Inside the affected PitchLinear path, add_tile_offset() converted the logical displacement into bytes before calling add_pointer_offset().

The calculation recorded in the issue was effectively:

int offset =
    sizeof_bits<Element>::value
    * logical_offset
    / 8;

add_pointer_offset(offset);

Suppose the element type is float, and the iterator needs to move four logical elements:

float = 32 bits
logical offset = 4 elements

The caller first calculates:

32 bits × 4 / 8
= 16 bytes

It then calls:

add_pointer_offset(16);

Because the internal pointer is a uint8_t*, the implementation advances by 16 bytes:

uint8_t pointer + 16
= 16 bytes

The final address is correct.

Both the internal caller and the implementation were using byte units.

The problem was not that the internal path could never work.

The internal path agreed on bytes, while the visible interface appeared to promise elements.

Internally consistent code is not the same as a consistent contract

This distinction matters.

Suppose every current caller knows the following undocumented rule:

Convert the element count into bytes before calling add_pointer_offset().

The existing internal path can continue to work.

But that rule is not expressed by the function name, and it conflicts with the higher-level Tile Iterator concept.

A new developer could reasonably follow the documented interface and write:

iterator.add_pointer_offset(element_count);

That developer would be following the apparent contract.

If one specialization expects bytes while another expects elements, the same generic call can produce different results:

Implementation A:
4 means 4 elements

PitchLinear implementation:
4 means 4 bytes

At that point, add_pointer_offset() is no longer one common interface.

It is two different contracts sharing the same name.

Why can a unit mismatch remain hidden for so long?

Unit errors do not always cause an immediate crash or an obvious diagnostic.

Under some element types and call paths, the values can accidentally agree.

If every internal caller performs the byte conversion first, existing tests may pass.

An offset of zero also hides the distinction:

0 elements = 0 bytes

If the element type occupies one byte, the numerical values agree as well:

4 elements × 1 byte
= 4 bytes

Under those conditions, different contracts produce the same address.

The mismatch becomes visible when:

  • the element size is greater than one byte, and

  • a caller passes an element count directly rather than converting it into bytes first.

This is not necessarily an arithmetic failure present in every execution path.

It is a boundary failure that appears when callers operating under different unit assumptions finally meet.

Converting the pointer to uint8_t* removes unit information

A typed pointer preserves the unit of pointer arithmetic.

float* ptr;
ptr += 4;

The compiler interprets this as an advance of four float objects.

A byte pointer removes that element-size information:

uint8_t* ptr;
ptr += 4;

Now the value 4 always means 4 bytes.

Byte pointers are useful.

They allow low-level code to handle arbitrary element types, alignment rules, and manually constructed memory layouts. They are common in GPU memory-address calculations for exactly that reason.

But once the type system stops performing the element-to-byte conversion, that responsibility has to move somewhere else.

At least one of the following must be explicit:

  • the function converts elements into bytes internally

  • the function name states that the argument is measured in bytes

  • separate element-based and byte-based APIs are provided

  • the interface documentation clearly declares the exception

If the type information is removed while the old name remains, callers are forced to infer the unit from the implementation.

pointer_offset does not identify the unit

The name pointer_offset tells us that the value represents a displacement.

It does not tell us how that displacement is measured.

These two names communicate different contracts:

add_element_offset(element_offset);
add_byte_offset(byte_offset);

This name does not:

add_pointer_offset(offset);

If the higher-level CUTLASS concept defines the argument in element units, the most natural design is for every implementation to preserve that contract.

A byte-based internal representation could still be handled through a second layer:

void add_pointer_offset(LongIndex element_offset) {
  add_byte_offset(
      element_offset * sizeof_bits<Element>::value / 8);
}

void add_byte_offset(LongIndex byte_offset) {
  pointer_ += byte_offset;
}

This separates the public contract from the internal representation.

If the intended public API is byte-based instead, the name should communicate that directly:

add_byte_offset(byte_offset);

The correct migration strategy depends on compatibility requirements and existing callers.

But whichever design is chosen, the unit should not remain implicit knowledge shared only by the current implementation and its current callers.

Specialization-specific exceptions are especially dangerous in generic abstractions

An abstraction such as RegularTileIterator is useful because multiple memory layouts and element types can be manipulated through a similar interface.

Generic code should not need to inspect every specialization before calling a shared method.

Suppose a generic algorithm is written like this:

template <typename Iterator>
void advance(Iterator& iterator, int element_count) {
  iterator.add_pointer_offset(element_count);
}

The function follows the Tile Iterator concept and passes an element count.

It may work correctly for most iterator implementations.

But a specialization that expects a byte count will move to a different address.

There may be no compilation error.

The function name exists.

The argument type is valid.

Only the resulting address is wrong.

This class of problem is difficult for the type system to detect and may require a specific specialization and a specific element size before it becomes visible.

The more generic the interface, the more strictly its unit contract must be preserved.

This is separate from the problem fixed by PR #3049

CUTLASS Issue #3017 raised two related but distinct concerns.

The first was the load() and store() address-calculation asymmetry discussed in Parts 1 and 2.

The second was the Element-versus-Byte ambiguity in add_pointer_offset() discussed here.

PR #3049 directly fixed the first problem.

It added the missing kElementsPerAccess division to store(), restoring agreement between the base addresses calculated by load() and store() for the same tile coordinate.

But restoring symmetry between two address formulas is not the same as defining the unit of an API argument.

load/store asymmetry
=
the same coordinate could resolve to different addresses

Element/Byte ambiguity
=
the same argument could be interpreted in different units

Fixing the first issue does not automatically resolve the second.

The original issue described separate possible responses:

  • align the coordinate transformations in load() and store()

  • define add_pointer_offset() consistently in either elements or bytes

  • if byte semantics are intentional, expose them through a name such as add_byte_offset()

The concerns appeared in the same issue, but they existed at different abstraction layers.

Units are part of the API, not an implementation detail

Types, function names, and parameter names are sometimes treated as formal details surrounding the “real” behavior.

In a memory-address API, however, the unit is part of the behavior.

These operations are not equivalent:

advance by 4 elements
advance by 4 bytes

They happen to produce the same displacement only when one element occupies one byte.

If the documented unit and the implemented unit differ, the consequence is not merely unclear wording.

The resulting address can change.

Code can continue working while the current caller and implementation share the same undocumented convention.

But a contract does not exist merely to preserve knowledge among the people who wrote the current path.

Its purpose is to let the next caller use the interface correctly without reconstructing its internal assumptions.

If a caller must read deeply into the implementation to discover that an argument must already be converted into bytes, that information has not yet been expressed as part of the interface.

An explainable boundary lasts longer than code that merely works

The unit concern in CUTLASS #3017 is not a simple claim that every internal calculation was unconditionally wrong.

The internal path could remain consistent because both sides used bytes.

The deeper problem was the unexplained gap between the element-based contract described by the higher-level concept and the byte-based behavior expected by the specialization.

That code may remain correct for the callers that exist today.

It becomes fragile when the system is reused, refactored, extended with another specialization, or called through a generic algorithm.

Once the same function name begins to represent different units, a caller cannot construct the correct argument without opening the implementation.

The central question is therefore:

Does the code happen to use the same unit today?
Or does the interface clearly guarantee that unit to every caller?

Those are not the same condition.

Internally consistent code protects the path that already exists.

A clearly defined contract also protects the caller that has not been written yet.

Previous articles

  • NVIDIA CUTLASS #3017 — Why load() and store() Mapped the Same Tile Coordinate to Different Addresses

    https://resoneticlab.blogspot.com/2026/08/nvidia-cutlass-3017-why-load-and-store.html

  • NVIDIA CUTLASS kElementsPerAccess — Why Vectorization Exposed a Hidden Addressing Error

    https://resoneticlab.blogspot.com/2026/08/nvidia-cutlass-kelementsperaccess-why.html


Related material

This is Part 3 and the final article in the CUTLASS #3017 series.

Part 1 examined the actual address mismatch between load() and store() and the patch that fixed it. Part 2 explained why vectorized access exposed a difference hidden by the default value. This final article examined what happens when the same numerical value can mean either elements or bytes at an interface boundary.

#NVIDIA #CUTLASS #CUDA #GPUProgramming #APIContract #CodeAnalysis #SoftwareArchitecture

Popular posts from this blog

AMD Is Trying to Turn Kernel Diversity Into One Software Asset

The first broad integration was rolled back, but the direction of the software stack is difficult to ignore The GPU software race is often described as a contest over who can write the fastest kernel. That is only part of the problem. A company may have several teams capable of producing high-performance matrix-multiplication kernels. Some may work through template libraries. Others may use compiler-generated code, handwritten assembly, or model-specific optimization stacks. The existence of those kernels does not automatically create one coherent platform. They still need to be: discovered validated matched to the correct problem given the correct arguments launched through a stable runtime and delivered through a reliable software package Recent public code changes inside AMD’s software ecosystem showed an attempt to connect these layers more directly. The central idea was to allow hipBLASLt’s GEMM selection system to choose kernels created by multiple internal kernel technologi...
Read more

PyTorch #188031 — Why Did Forward Use 64-Bit Indexing While Backward Still Used uint32_t?

PyTorch Spatial Softmax, Part 1 of 3 — An asymmetric indexing contract and the patch that aligned it A recent PyTorch change fixed a 64-bit indexing problem in the spatial softmax backward path. The bug was not in the softmax formula itself. It was in the width of the integer used to calculate memory locations. For the same large tensor, the forward path already selected either 32-bit or 64-bit indexing based on the input size. One backward path, however, continued to calculate sizes, strides, and offsets using uint32_t . That created a possible split in behavior: Forward → 64-bit address calculation → accesses the intended elements Backward → 32-bit address calculation → offset wraparound → gradients read from or written to different locations A tensor could therefore complete the forward pass correctly while the corresponding backward kernel could no longer represent every address in that tensor. The changes described in PyTorch #188031 were later landed in PyTorch’s main branch. S...
Read more

NVIDIA CUTLASS #3017 — Why load() and store() Mapped the Same Tile Coordinate to Different Addresses

CUTLASS #3017, Part 1 of 3 — An asymmetry in address calculation and the patch that fixed it In February 2026, while reading through NVIDIA CUTLASS’s RegularTileIterator<layout::PitchLinear> implementation, I noticed an odd asymmetry. load() and store() accepted the same tile coordinate, yet they calculated the corresponding memory address differently. The difference was a single line. load() divided the contiguous coordinate by kElementsPerAccess . store() did not. When each memory access handled only one element, the difference remained invisible. But in vectorized paths, where a single access handles multiple elements, the same logical tile coordinate could resolve to two different physical addresses. I reported the issue as CUTLASS Issue #3017. The missing division was later added to store() in PR #3049. CUTLASS and tile iterators CUTLASS is NVIDIA’s open-source library for high-performance CUDA linear algebra. It provides C++ template abstractions for composing GPU ope...
Read more