FP4 Is No Longer Just a Hardware Feature — The NVFP4 Software Stack Is Taking Shape

What packing, traffic accounting, and numerical guardrails appearing across PyTorch, Triton, and oneDNN tell us

The precision race in AI hardware is often described as a sequence of smaller numbers.

FP16 became FP8.

Now FP8 is moving toward FP4.

Reducing the number of bits suggests an obvious set of benefits:

• smaller model storage

• lower memory traffic

• higher arithmetic throughput

• better performance per watt

But a GPU supporting 4-bit operations does not mean an AI model can simply switch to FP4.

A higher-precision tensor must first be converted into an actual FP4 representation.

Two 4-bit values may need to be packed into one byte.

Scales must be calculated and stored so that the original value range can be reconstructed.

The compiler must decide whether those scale calculations and packing operations can be fused with surrounding work.

Profilers must distinguish FP4 arithmetic from the movement of packed values and scale metadata.

The numerical system must also prevent scales from becoming so small or so large that the quantization process becomes unstable.

In other words, FP4 requires more than tensor-core support.

Hardware format
        ↓
Quantization recipe
        ↓
Packing and layout
        ↓
Compiler fusion
        ↓
Cost and traffic accounting
        ↓
Numerical guardrails
        ↓
Model execution and validation

Recent changes across PyTorch, Triton, and oneDNN are filling in different parts of this stack.

They do not prove that the FP4 ecosystem is complete.

But they do point in one clear direction:

FP4 is beginning to move from a hardware specification into an actual software contract.

---

NVFP4 is more than a 4-bit value

NVIDIA introduced NVFP4 as part of the Blackwell generation.

Its underlying value format is E2M1:

1 sign bit
2 exponent bits
1 mantissa bit

An E2M1 value has a very limited numerical range, reaching approximately ±6.

That range is too narrow to represent the full dynamic range of many AI tensors directly.

NVFP4 compensates for this with a hierarchical scaling structure.

In simplified form:

16 consecutive FP4 values
→ share one FP8 E4M3 block scale

The full tensor
→ also uses a tensor-level FP32 scale

A usable tensor value is reconstructed by combining:

the packed E2M1 value
+
its local block scale
+
the tensor-level scale

The real data contract is therefore not merely:

one 4-bit number

It is:

packed FP4 values
+
one block scale per group of 16 values
+
a tensor-level scale
+
a defined layout for those values and scales

NVIDIA’s Transformer Engine documentation describes NVFP4 through this combination of E2M1 payloads and hierarchical scaling.

Two FP4 values may fit into one byte, but matrix multiplication still requires the packing direction, scale grouping, and memory layout to match what the hardware and software kernels expect.

Moving to FP4 is therefore not the same as changing a tensor’s dtype name.

It means introducing a new data path that manages values, scales, packing, and layout together.

---

The first change: PyTorch is bringing FP4 packing into compiler fusion

Converting a higher-precision tensor into NVFP4 generally requires several stages.

The system first calculates the maximum absolute value within a small block.

It derives a scale from that value.

The original values are divided by the scale and rounded into the E2M1 range.

Pairs of 4-bit values are then packed into bytes.

Higher-precision input
        ↓
Block-level amax reduction
        ↓
Scale calculation
        ↓
Divide values by the scale
        ↓
Convert to E2M1
        ↓
Pack two values into one byte

If these operations are executed as separate kernels, much of FP4’s theoretical efficiency can be lost to intermediate memory traffic.

Write amax results to memory
        ↓
Launch another kernel to read them
        ↓
Write scales to memory
        ↓
Read the tensor again
        ↓
Quantize and pack

The data becomes smaller only after the system has already moved and materialized several intermediate results.

A recent PyTorch Inductor change expands reduction-epilogue fusion so that consumers of interleaved value pairs can remain attached to the parent reduction.

One explicit use case is standalone NVFP4 packing.

The test path performs a sequence resembling:

amax reduction
→ scale generation
→ transform even and odd elements
→ pack the pair into FP4 storage

inside one compiler-generated fused path.

The implementation also rejects cases where fusion would be ambiguous or unsafe, including mutated consumers, non-leaf consumers, and incompatible derived layouts.

That distinction matters.

The change does not simply add another FP4 conversion utility.

It treats FP4 packing as an operation the compiler should place, schedule, and fuse within the graph.

FP4 conversion is beginning to move from an external preprocessing step into the compiler’s execution plan.

The current boundary must still be stated carefully.

The public NVFP4 packing tests include CUDA inline assembly and target SM100-or-later hardware.

The evidence supports:

Compiler fusion for an NVFP4 packing path
→ present

It does not yet support:

A fully generalized FP4 lowering for every backend
→ not established

Nor does it show that FP4 packing has become the default path for ordinary PyTorch models.

---

The second change: Triton is beginning to measure FP4 as FP4

A new data format does not become operationally useful merely because a kernel can execute it.

The software stack also needs to understand what that kernel actually did.

Compilers, runtimes, and profilers need measurements that can answer questions such as:

• How many operations were performed?

• How many bytes were read and written?

• How much of the traffic came from payload values?

• How much came from scale metadata?

• Was the kernel compute-bound or memory-bound?

• Is one kernel configuration genuinely better than another?

A recent change in triton_kernels adds FP4-specific matmul launch metadata.

Previously, a packed FP4 operand stored in a uint8 container could be treated too much like an ordinary 8-bit value.

The new path distinguishes an FP4-by-FP4 operation with:

flops4

rather than counting it under:

flops8

It also models more than the physical container size.

The traffic calculation can include:

packed FP4 payload bytes

block-scale bytes

tensor-level scale bytes

logical rows in swizzled storage

scale bytes associated with active tokens or experts
in ragged matmul workloads

This is an important distinction.

A uint8 tensor may physically contain bytes, but those bytes can represent two packed FP4 values rather than one 8-bit numerical value.

Treating the storage format as the arithmetic format can distort both compute and traffic estimates.

Incorrect arithmetic classification
→ misleading FLOP counts

Ignoring scale metadata
→ incomplete traffic accounting

Counting physical padded storage instead of logical data
→ distorted bandwidth estimates

A profiler that does not understand the real structure of the format cannot compare kernels accurately.

This Triton change is therefore less about enabling a new FP4 instruction and more about recognizing FP4 as an independent unit in the software performance model.

FP4 is becoming something the runtime can measure, rather than merely something the hardware can execute.

The current boundary remains limited.

More accurate FP4 metadata
→ confirmed

Metadata directly changing autotuner decisions
→ not yet established by this patch

End-to-end model performance improvement
→ not demonstrated by this patch alone

Still, measurement is not a minor addition.

A format that cannot be measured accurately is difficult to optimize consistently.

---

The third change: oneDNN is defining a safe numerical range for FP4 scales

FP4’s difficulty is not limited to the four-bit payload.

The scales are also stored in finite-precision formats.

Suppose a block’s maximum value produces a very small scale.

After conversion into the scale dtype, that value may become zero or lose too much precision.

If the scale is later used as a divisor or reconstruction factor, the resulting numerical error can become severe.

A scale that is too large creates the opposite problem.

It may exceed the representable range of the scale dtype.

A recent oneDNN change clamps NVFP4 dynamic scales into a type-aware interval:

epsilon of the scale dtype
≤
dynamic scale
≤
maximum value of the scale dtype

The same boundary was incorporated into several layers:

CPU reference matmul

Intel GPU dynamic-scale code

benchdnn reference calculations

The commit described the purpose directly as avoiding an “arithmetic catastrophe.”

This is not evidence merely that oneDNN can recognize an FP4 format.

It shows that the library is defining the numerical conditions under which that format can be used safely and tested consistently.

The legal range of an FP4 scale is becoming part of the software contract.

The presence of this work in oneDNN is also notable.

NVFP4 originated as an NVIDIA Blackwell format, yet its scale semantics and numerical safeguards are now being handled in a library that also serves CPU and Intel GPU execution paths.

The distinction must remain precise:

oneDNN interprets and validates NVFP4 scaling
→ confirmed

Intel GPUs provide Blackwell-equivalent native FP4 acceleration
→ not demonstrated by this change

This is evidence about software-level format semantics.

It is not evidence of hardware-performance parity.

---

The three changes mean more when viewed together

Each patch can appear small in isolation.

PyTorch
→ adds a fusion capability

Triton
→ improves metadata accounting

oneDNN
→ clamps a scale

But the projects are filling in different boundaries of the same data format.

PyTorch Inductor
→ How should FP4 values be created and packed?

Triton
→ How much FP4 arithmetic and traffic actually occurred?

oneDNN
→ Which FP4 scale values are numerically valid?

Placed into one execution path:

Higher-precision tensor
        ↓
Compiler fuses scale reduction and packing
        ↓
FP4 payload and scale tensors are created
        ↓
Runtime executes FP4 matmul
        ↓
Profiler records flops4 and scale traffic
        ↓
Library enforces numerical scale boundaries

This connection is my interpretation of the separate source changes.

FP4 is no longer being treated as one specialized GPU instruction. It is becoming a data format that compilers, runtimes, profilers, and numerical libraries must maintain together.

---

NVIDIA is trying to extend a hardware feature into a format ecosystem

NVIDIA introduced NVFP4 as a Blackwell-supported hardware format.

But hardware support alone does not create a format ecosystem.

The surrounding stack also needs:

a quantization recipe

training and inference APIs

packing and scale layouts

compiler lowering

optimized matrix multiplication

checkpoint formats

profiling and debugging

distributed execution

NVIDIA’s software stack already provides parts of this route.

Transformer Engine exposes NVFP4-oriented training recipes, while TensorRT and related inference tooling are expanding FP4 execution and quantization support.

NVIDIA has also published JAX and MaxText training results using NVFP4 on Blackwell.

In one vendor-reported example, NVIDIA described up to a 1.73× training-speed improvement over an FP8 baseline while maintaining a similar loss curve.

That result should be interpreted carefully.

It reflects workloads and hardware selected by NVIDIA and should not be generalized automatically to every model or system.

The broader signal is more important than one benchmark number.

Independent open-source compiler and library layers are beginning to implement:

• NVFP4 packing

• FP4-specific cost accounting

• scale-layout handling

• scale-stability rules

If that process continues, NVIDIA may be doing more than selling chips that support 4-bit arithmetic.

It may be extending a hardware format into a software convention that other projects need to understand.

This represents a different kind of platform influence from CUDA.

CUDA
→ the execution environment through which software reaches NVIDIA GPUs

NVFP4
→ a contract for how low-precision AI values are represented,
scaled, packed, moved, and reconstructed

If a hardware company influences both execution APIs and tensor representation, its platform boundary becomes wider than the chip itself.

---

Format adoption and hardware exclusivity are not the same thing

The spread of NVFP4 through frameworks and libraries can strengthen NVIDIA’s ecosystem.

Developers become familiar with a scaling and packing structure that was designed around Blackwell hardware.

Tooling, checkpoints, and optimization recipes can begin to assume that structure.

However, wider format support can also increase portability.

An NVFP4 checkpoint exists
        ↓
Another backend learns to read or transform it
        ↓
The format becomes more portable

Two separate hypotheses must therefore be distinguished.

NVFP4 becomes an important ecosystem format
→ NVIDIA’s influence over AI data representation may increase

NVFP4 workloads must run only on NVIDIA hardware
→ not established

The current code signals strengthen the first hypothesis.

They do not prove the second.

A format can become widely used while multiple hardware backends compete to execute or translate it.

---

FP4 does not automatically mean lower total HBM demand

For a fixed workload, FP4 can reduce the storage required for model values compared with FP16 or FP8.

NVIDIA has estimated that NVFP4, including its scaling overhead, can substantially reduce model memory footprint compared with higher-precision formats.

Those figures depend on model structure, quantization coverage, and implementation choices, so they should be treated as vendor estimates rather than universal constants.

The larger infrastructure effect is more complicated.

Fewer bytes per parameter
        ↓
A larger model can fit on the same GPU

More experts can remain resident

Longer context windows become practical

More users can run concurrently

More test-time compute can be applied

Therefore:

less memory per model

does not necessarily imply:

less total HBM demand across the data center

Efficiency can increase usage.

A lower cost per token or per model may encourage systems to process more tokens, serve more users, or deploy larger models.

The code changes discussed here show that the FP4 software route is becoming more concrete.

They do not determine whether aggregate HBM demand will ultimately rise or fall.

---

The real competition is not the 4-bit operation count

FP4’s theoretical throughput matters only if the surrounding overhead can be controlled.

That overhead includes:

scale calculation

amax reductions

packing and swizzling

format conversion

scale-metadata traffic

additional kernel launches

quantized communication

accuracy validation

A tensor core may execute FP4 arithmetic extremely quickly.

But if the system repeatedly materializes intermediate scales, launches separate packing kernels, or transfers excessive metadata, the end-to-end workload may capture only part of the theoretical gain.

The competitive question is therefore shifting.

It is no longer only:

Which accelerator supports FP4?

It is increasingly:

Which software stack can create, move, calculate, measure, and reconstruct FP4 values while eliminating the most overhead?

The PyTorch change addresses packing and fusion overhead.

The Triton change makes the remaining cost more visible.

The oneDNN change protects the format against a numerical failure boundary.

All of these layers are needed before hardware throughput becomes sustained model throughput.

---

The current stage lies between feature existence and a default execution path

The maturity of the NVFP4 stack can be summarized as follows:

NVFP4 hardware format
→ exists

Official training and inference recipes
→ exist

Compiler packing and fusion
→ appearing in selected paths

FP4-specific profiling metadata
→ appearing

Numerical scale guardrails
→ appearing

Default dtype and ordinary framework path
→ still developing

Native acceleration across multiple backends
→ not established

Broad production adoption
→ requires more evidence

The following claims would therefore be premature:

FP4 has replaced FP8
✗

All AI models are moving to FP4
✗

Every backend accelerates NVFP4 equivalently
✗

The source evidence supports a narrower statement:

Software layers required to use NVFP4 in real workloads are becoming more explicit across multiple projects.

---

What I will watch next

Whether inline assembly becomes a first-class compiler operation

Some of the current packing tests rely on CUDA inline assembly and SM100-or-later hardware.

A mature framework path should eventually generate FP4 conversion and packing without requiring users or library authors to manage assembly manually.

specialized manual path
→ ordinary compiler lowering

That transition would be an important sign of maturity.

---

Whether FP4 metadata begins to influence tuning decisions

Recording flops4 and scale bytes is the beginning of measurement.

The next step is whether those measurements influence:

• autotuning

• kernel selection

• scheduling

• fusion

• cost models

A metadata field becomes more strategically important when it changes which implementation the system chooses.

---

Whether every layer shares the same scale contract

A safe reference implementation is not enough if optimized kernels or other backends use different boundaries.

The following layers need to agree:

quantization recipe

compiler transformation

runtime implementation

optimized hardware kernel

reference test

If they interpret zero blocks, very small scales, or maximum scale values differently, backend results can diverge.

---

Whether FP4 beats FP8 in complete models

A faster matrix-multiplication microbenchmark does not automatically produce a faster model.

The complete workload also contains:

• attention

• communication

• KV-cache movement

• memory allocation

• quantization overhead

• accuracy-recovery mechanisms

End-to-end evidence must include throughput, latency, power, memory consumption, and model quality.

---

How non-NVIDIA backends interpret NVFP4

The appearance of NVFP4 semantics in oneDNN suggests that the format is already moving beyond NVIDIA-only software.

The next question is whether the industry converges on a common FP4 representation or splits across several incompatible variants.

NVFP4 becomes a broadly understood format

or:

NVFP4, MXFP4, and vendor-specific variants
remain separate software islands

The answer will affect both portability and platform power.

---

What would weaken this thesis

The FP4 software-stack thesis would weaken if:

• packing and scaling overhead consumes most of the theoretical gain

• training stability requires too many operations to remain in higher precision

• NVFP4, MXFP4, and other FP4 variants require costly conversions

• compiler support remains limited to architecture-specific manual paths

• communication becomes the dominant bottleneck, leaving FP4 compute underutilized

• production deployments remain on FP8 while FP4 stays confined to demonstrations and benchmarks

A format existing in code is not the same as a format becoming an industry standard.

---

Technology changes and stock prices operate on different clocks

The appearance of FP4-related code across multiple repositories does not mean NVIDIA’s share price should respond immediately.

A software format moves through several stages before it becomes a business result.

Hardware support
        ↓
Framework and compiler integration
        ↓
Stable model recipes
        ↓
End-to-end benchmarks
        ↓
Checkpoint and production deployment
        ↓
Expanded hardware purchases and usage
        ↓
Revenue contribution

The current changes sit in the middle of that sequence.

This is not a short-term price forecast.

The long-term question I am recording is:

Can NVIDIA turn Blackwell’s FP4 capability into an AI data-format ecosystem that compilers, libraries, and model developers increasingly build around?

---

The core idea

The important fact about FP4 is not merely that the number of bits is four.

It is that software is beginning to form around those four bits.

How is the scale calculated?

How are two values packed into one byte?

Where are the scales stored?

How much data actually moved?

Which scale values are safe?

Which operations can be fused with the conversion?

Once answers to those questions are fixed in code, FP4 becomes more than a specialized tensor-core feature.

It becomes an execution contract shared by compilers, runtimes, and numerical libraries.

PyTorch is defining where packing can be fused.

Triton is beginning to count the real cost of the format.

oneDNN is defining numerical boundaries that prevent scale failure.

NVIDIA’s own stack is expanding the corresponding training and inference recipes.

Each change is a small fragment.

Together, they point in the same direction:

The 4-bit era of AI may have begun in the tensor core, but its competitive outcome will be determined by the software stack.

The company I am watching at the center of this transition is NVIDIA, ticker NVDA.

NVIDIA may not capture every benefit created by FP4 adoption.

But if a hardware format introduced with Blackwell becomes a software contract understood across open-source frameworks and libraries, NVIDIA’s influence may extend beyond chip performance into data representation and execution conventions.

I do not know when — or how fully — the market will recognize that transition.

That is why I am recording it before the outcome is fully visible in reported results.

---

This article is an independent long-term observation based on publicly available technical, industry, and company information.

Detailed source-discovery paths and research methodology are not disclosed.

This article does not constitute a recommendation to buy or sell any security.

#AIInfrastructure #LowPrecisionAI #FP4 #NVFP4 #NVIDIA #NVDA #PyTorch #Triton #oneDNN

---