OpenVINO #37435 — Why Dropping an ov::Output Port Could Turn TopK Indices into Values

OpenVINO GGUF Frontend, Part 2 of 3 — Multi-output tensor identity, dynamic k, and attention shapes that remain valid under both SDPA and PagedAttention

Part 1 examined how OpenVINO’s GGUF frontend separated model semantics from runtime-state ownership.

The frontend first produces a stateless graph with explicit KV-cache inputs and outputs.

A consumer that wants OpenVINO-managed state registers GGUFMakeStateful, which rewrites the cache path into:

ReadValue
        ↓
Gather by beam_idx
        ↓
Concat new cache rows
        ↓
Assign

PR #37435 also repaired a different class of frontend failure.

These problems were not primarily about which operation should be used.

They were about preserving the exact identity and layout of the tensor produced by that operation.

A frontend does not translate only:

Add
MatMul
TopK
Reshape
Transpose

It must also preserve:

Which output port of the operation is being consumed?

Which axis represents tokens?

Which axis represents heads?

Which dimensions must remain statically known?

Which graph pattern must remain visible to a backend matcher?

Those details can change the compiled result even when the high-level mathematical expression appears unchanged.

The clearest example was TopK.

OpenVINO’s TopK produces two outputs:

output(0)
→ selected values

output(1)
→ selected indices

GGML’s TOP_K and ARGSORT translators need the second one.

But an ov::Output<ov::Node> identifies more than the producing node.

It identifies:

producer node
+
output-port index

If a translator converts that output into only:

std::shared_ptr<ov::Node>

through:

get_node_shared_ptr()

the port identity is no longer carried by the value.

A later operation that consumes “the node” can implicitly resolve it as output 0.

For TopK, that means the graph can silently move from:

indices

to:

values

or fail when a downstream translator expects an integer index tensor but receives the values port instead.

PR #37435 therefore reinforced a broader frontend rule:

A tensor produced by a multi-output operation is not identified by its node alone.
Its output port is part of the tensor’s identity.

The same patch extended that principle into shape and attention translation.

A dimension was not treated as merely a number.

Its axis position, static knowledge, runtime placement, and relationship to backend graph matchers all had to be preserved.

---

A node and one of its outputs are different objects

For a single-output operation, the distinction can appear unimportant.

Consider:

Add
→ one output

Whether code passes:

add_node

or:

add_node->output(0)

the intended tensor is obvious.

A multi-output operation changes the contract.

For TopK:

TopK node
        ├── output 0: values
        └── output 1: indices

The node represents the complete operation.

The output represents one tensor edge leaving it.

In OpenVINO, that tensor edge is represented by:

ov::Output<ov::Node>

It contains the producing node and the selected output index.

Conceptually:

TopK output 0
=
(TopK node, port 0)

TopK output 1
=
(TopK node, port 1)

Those are not interchangeable tensors.

---

get_node_shared_ptr() retains the producer but not the selected edge

Suppose a translator correctly has:

topk->output(1)

This identifies the index tensor.

Now suppose it calls:

topk->output(1).get_node_shared_ptr()

The result identifies:

the TopK node

It no longer independently carries:

port 1

If that node pointer is later passed into an API that accepts a node and implicitly selects its first output, the effective tensor becomes:

TopK output 0

The operation is still TopK.

The tensor is different.

Before conversion

TopK node
+
port 1
→ indices

After reducing it to a node pointer

TopK node
+
implicit port 0
→ values

This is why a source graph can retain the correct producer while losing the correct data edge.

The PR description identifies this exact risk: taking .get_node_shared_ptr() can resolve a multi-output value to output 0, with TopK given as the concrete failure case.

---

TopK makes the mistake visible through type as well as meaning

The two outputs differ semantically.

They may also differ in element type.

A typical configuration is:

input
→ f32

TopK output 0
→ f32 selected values

TopK output 1
→ i32 or i64 selected indices

A downstream routing operation may require:

expert IDs
→ integer tensor

If it receives the values output instead:

floating-point scores
→ supplied where indices were expected

the graph may fail conversion or validation.

In a more permissive path, an accidental conversion might allow execution while preserving the wrong semantics.

The danger is therefore not limited to a clean type error.

Correct node type
+
wrong output edge
→ incorrect tensor identity

---

Shape queries must also preserve the output port

The same boundary appears when a translator asks for a tensor’s shape.

OpenVINO’s GGUF helper has an overload shaped like:

get_dimensions(
    const ov::Output<ov::Node>& output,
    const std::vector<int>& dims
)

It constructs ShapeOf directly from the selected ov::Output.

TopK output 1
        ↓
ShapeOf(output 1)
        ↓
Gather requested dimensions

The helper does not first convert the output into a producer node.

The shape query therefore remains attached to the intended port.

The difference is:

Correct

ShapeOf(TopK output 1)
→ shape of indices

versus:

Incorrect

ShapeOf(TopK node)
→ implicit output 0
→ shape of values

The shapes may often be equal for TopK.

That does not make the port loss harmless.

The graph edge and its element type are still different, and another multi-output operation may have ports with different shapes.

---

The regression test checks the actual input-port index

PR #37435 added a structural test named:

GetDimensionsKeepsOutputPort

The test constructs a TopK, explicitly passes:

topk->output(1)

into get_dimensions(), then inspects the ShapeOf input.

The assertion is not merely:

The resulting shape looks correct.

It checks:

shape_of->input_value(0).get_index()
==
1

In other words:

Did ShapeOf remain connected to TopK port 1?

The test would fail if the helper measured output 0, even in a shape where both outputs happened to have identical dimensions.

This is a stronger regression contract than comparing shape values alone.

When two ports happen to have the same shape, a value-only test can pass while the graph still points to the wrong tensor.

---

Numerical equality cannot always detect identity loss

Suppose the two TopK outputs both have shape:

[2, 2]

A test that asks only:

Did ShapeOf return [2, 2]?

cannot distinguish them.

Values port shape
→ [2, 2]

Indices port shape
→ [2, 2]

Both pass.

The port-index assertion checks the graph relationship directly.

Expected producer edge
→ TopK output 1

not merely:

Expected dimension values
→ [2, 2]

This is an important testing pattern for graph frontends.

Some correctness properties are topological.

They cannot be proven from the final numerical tensor alone.

---

ARGSORT and TOP_K now share one index-construction helper

Both GGML operations can be implemented through OpenVINO TopK.

TOP_K

Select the k largest elements
→ return their indices

ARGSORT

Select the complete axis
→ order ascending or descending
→ return all indices

Before the patch, the two translators built their own TopK nodes separately.

PR #37435 introduced:

make_topk_indices(...)

The helper accepts:

input

k

axis

MIN or MAX mode

index element type

stable-order flag

It constructs the OpenVINO TopK and returns explicitly:

topk->output(1)

The helper’s return type remains:

ov::Output<ov::Node>

That keeps the index port attached to the tensor as it moves between translator utilities.

---

The helper centralizes a semantic choice, not just repeated code

The important shared rule is:

GGML TOP_K and ARGSORT want indices

→ OpenVINO TopK output 1

If the translators construct TopK independently, they can drift.

One might use:

output(1)

while another accidentally returns:

output(0)

One might use the decoder-specified index type.

Another might hard-code i32.

One might derive k dynamically.

Another might require a static dimension.

The shared helper fixes one part of that contract in one location:

index output
→ always port 1

The ARGSORT and TOP_K translators still determine their own k, axis, and mode, but they no longer independently decide which port represents the operation’s result.

---

ARGSORT uses the complete last dimension as k

GGML ARGSORT returns an ordering for every element along its sorting axis.

The translator therefore calculates:

k
=
size of the input’s last axis

It does not require that dimension to be a compile-time C++ integer.

Instead:

input
        ↓
ShapeOf
        ↓
Gather last-axis dimension
        ↓
Squeeze to scalar k
        ↓
TopK

The sort direction selects the TopK mode.

Ascending
→ MIN

Descending
→ MAX

The indices port becomes the translated output.

The current implementation derives the last axis from the static rank when possible and uses the frontend’s rank-4 convention as the fallback when rank information is not fully static.

---

The old TOP_K translator required a static output extent

The original TOP_K path used logic equivalent to:

const int64_t k =
    output_shape[last_axis].get_length();

get_length() requires the dimension to be static.

If the output extent was dynamic:

output last dimension
→ dynamic

the translator could throw during conversion before an executable graph was produced.

The problem was not that OpenVINO TopK requires a compile-time constant.

Its k input is itself a graph input.

The limitation came from the frontend materializing k as a C++ integer too early.

---

The new translator keeps a static fast path and a runtime fallback

The fixed translator first checks whether the relevant output dimension is static.

Static extent

Known output last-axis size
        ↓
create scalar Constant(k)

Dynamic extent

Corresponding runtime input dimension
        ↓
ShapeOf / Gather
        ↓
Squeeze
        ↓
runtime scalar k

The translator can therefore construct the graph without calling get_length() on a dynamic dimension.

This is another form of preserving information.

Unknown at conversion time

does not mean:

unknown at execution time

The shape value can remain represented inside the graph rather than being demanded by the C++ translator.

---

Static and dynamic shape values have different owners

A static k is owned by the frontend at conversion time.

Frontend reads dimension

→ embeds Constant

A dynamic k is owned by runtime shape evaluation.

Frontend builds shape-expression graph

→ runtime supplies dimension

The arithmetic operation is still TopK.

The source of the scalar is different.

A correct frontend must know which facts it may safely collapse into constants and which must remain graph values.

---

The shared helper does not force ARGSORT and TOP_K to have the same k

The two translators share:

TopK construction

index-port selection

index-type handling

They do not share the exact meaning of k.

ARGSORT

k
=
full input dimension

TOP_K

k
=
requested output extent

The common helper accepts k as an ov::Output.

It does not decide where that scalar came from.

Shared mechanism
→ TopK indices

Operation-specific policy
→ how k is derived

This is the same narrow-abstraction principle seen throughout the patch.

Share the contract that is truly common.

Do not merge distinct operation semantics merely because the final OpenVINO primitive is the same.

---

The TOP_K result order is not fully contractual in ggml

The added TOP_K unit test does not compare the returned indices position by position.

The test comments explain that ggml’s kernel deliberately swaps the first two result slots to emphasize that their order is not important.

The contractual property is the selected set.

For example:

Row:
1, 9, 3, 7, 5

k:
3

Expected index set:
{1, 3, 4}

The OpenVINO output may list those indices in a deterministic value order.

The test sorts the returned indices before comparing the set.

This avoids converting an implementation detail of ggml’s output ordering into a stronger frontend contract than the source operation actually promises.

---

A frontend test should not demand semantics the source does not guarantee

An overly strict test might require:

[1, 3, 4]

in exactly that order.

If ggml guarantees only:

the selected indices are {1, 3, 4}

then an order-sensitive assertion would test the wrong contract.

The opposite error is also possible.

A test that checks only:

output type is i32

would not prove that the correct experts were selected.

The regression chooses the supported middle ground:

correct index tensor

correct number of indices

correct selected set

no unsupported ordering guarantee

---

The patch does not add a dedicated dynamic-k execution fixture

The source clearly adds a dynamic graph path for k.

The visible TOP_K runtime test in the merged patch uses a static output extent:

input last dimension:
5

output last dimension:
3

The structural port test also uses a constant k.

I did not find a dedicated newly added runtime test in PR #37435 whose TOP_K output extent itself remains dynamic through conversion.

Therefore, the evidence should be separated.

Dynamic-k source path
→ directly confirmed

Dedicated dynamic-k runtime regression in this PR
→ not confirmed

The full frontend suite reported 137 passing tests, but that total alone does not identify a dynamic-k fixture.

---

Tensor identity also includes axis meaning

The port problem is easy to express:

node
+
port

Attention translation adds another dimension.

A tensor’s shape may contain the same numbers while assigning them to different semantic axes.

Consider two layouts.

Batch-major SDPA layout

[B, L, H, S]

where:

B
→ batch

L
→ token length

H
→ attention heads

S
→ head size

Canonical SDPA compute layout

[B, H, L, S]

Now consider PagedAttention lowering.

The token count may move into the leading dimension:

[L, 1, H, S]

The buffer may contain the same element sequence.

The axis contract has changed.

A reshape that pins the leading dimension to 1 can silently move token identity to another axis.

That can produce a graph that is numerically shaped but semantically wrong.

---

The two GGUF decoder paths deliver attention tensors differently

The frontend supports an existing llama.cpp cgraph decoder and is being prepared for the separate native .gguf builder.

The FLASH_ATTN_EXT translator distinguishes their layouts through op_case.

Cgraph decoder: op_case == 0

The Q, K, and V tensors already arrive in:

[B, H, L, S]

This is the canonical layout expected by OpenVINO SDPA.

The head axis is:

axis 1

Native builder convention: op_case == 100

The tensors arrive in ggml-natural layout:

[B, L, H, S]

The head axis is:

axis 2

The translator first expands K/V heads along that axis, then transposes Q, K, and V into:

[B, H, L, S]

before constructing SDPA.

The same operation name therefore cannot be translated by hard-coding:

heads are always dimension 1

Layout is part of the decoder contract.

---

GQA head tiling must follow the actual head axis

Grouped-query attention can have:

number of query heads
>
number of key/value heads

K and V must be repeated to match the number of query heads.

Conceptually:

Hq = 8

Hkv = 2

repeat factor = 4

But the axis to repeat depends on the incoming layout.

[B, Hkv, L, S]
→ head axis 1

[B, L, Hkv, S]
→ head axis 2

PR #37435 derives:

head_axis

from the decoder path and places the temporary repeat axis immediately after it.

The resulting reshape also preserves leading dimensions through special_zero rather than replacing them with static assumptions.

---

Live stateful tensor shapes may be dynamic even when head facts are static

On a stateful KV-cache path, the OpenVINO node feeding K or V may come from:

ReadValue
+
Concat

Its batch or sequence dimensions can be dynamic.

The number of heads and head size are still model facts known to the GGUF decoder.

The translator therefore uses:

context.get_input_shape(...)

from the decoder for head count and head size instead of relying solely on the live OpenVINO node’s inferred partial shape.

Live OV tensor

batch / sequence
→ dynamic

Decoder-declared ggml shape

head count / head size
→ static

The frontend combines those two forms of knowledge rather than demanding that the runtime state node make every dimension static.

---

A literal 1 can destroy PagedAttention layout

The attention split reshape converts a flat projection into heads.

In batch-major SDPA execution:

input:
[1, 1, T, H*S]

output:
[1, T, H, S]

A reshape target such as:

[1, -1, H, S]

looks correct.

After SDPAToPagedAttention, however, the same logical activation can be token-major:

input:
[T, 1, 1, H*S]

If the reshape still pins the first dimension to 1, the token count can be folded into another dimension.

The expected result is:

[T, 1, H, S]

not:

[1, T, H, S]

The solution is OpenVINO’s special_zero reshape behavior.

0 in target shape
→ copy corresponding input dimension

The pattern becomes:

[0, -1, H, S]

with:

special_zero = true

Now the same reshape works for both layouts.

Input [1, 1, T, H*S]
→ Output [1, T, H, S]

Input [T, 1, 1, H*S]
→ Output [T, 1, H, S]

The leading dimension stays where the active attention backend placed it.

---

The reverse reshape must preserve the token axis too

After attention, the head dimensions are merged again.

Batch-major form

[1, T, H, S]
        ↓
[1, 1, T, H*S]

Token-major form

[T, 1, H, S]
        ↓
[T, 1, 1, H*S]

The patched target again begins with:

0

so the first input dimension is copied.

The translator also retains rank 4.

That rank is important because the next operation may be a residual Add against another rank-4 activation.

OpenVINO broadcasts elementwise operands from the right.

A rank-3 result can right-align its dimensions incorrectly and produce a token-by-token outer broadcast rather than one residual addition per token.

The source explicitly identifies the potential:

T × T

broadcast error when the token dimension appears on an unexpected axis.

---

The regression test executes both physical layouts

The patch adds paired tests for reshape cases 1 and 2.

The test supplies the same flat values under:

batch-major input

and:

token-major input

It then verifies:

correct output shape for each layout

and

identical output element order

For the split-heads case:

Batch-major:
[1, 1, T, H*S]
→ [1, T, H, S]

PagedAttention layout:
[T, 1, 1, H*S]
→ [T, 1, H, S]

For the merge-heads case:

Batch-major:
[1, T, H, S]
→ [1, 1, T, H*S]

PagedAttention layout:
[T, 1, H, S]
→ [T, 1, 1, H*S]

The comments state that these tests fail when the target shape pins a literal leading 1.

---

The same buffer can have more than one valid layout interpretation

The tests emphasize that the data do not need to move.

Same contiguous values

different shape metadata

different axis ownership

The reshape changes how the runtime interprets the buffer.

This makes metadata correctness as important as data correctness.

No copy occurred

does not imply:

No semantic change occurred

A metadata-only reshape can determine which tokens belong to which batch and which values belong to which head.

---

PERMUTE case 4 had to preserve static head dimensions

Another issue appeared when an active sequence count was available as a runtime Parameter.

The original reshape pattern combined:

dynamic n_seq_active

-1

static n_heads

static head_size

through one Concat.

Although the final two values were constants, the combined shape expression could become insufficiently bounded for downstream shape inference.

The head size could reach SDPA as dynamic.

The GPU plugin could then fail to retain the optimized SDPA representation and decompose it into:

Gemm

+

SoftMax

PR #37435 split the metadata transformation into two reshapes.

---

First reshape: resolve the dynamic sequence grouping

The first target is constructed from:

n_seq_active

-1

Conceptually:

flat projection
        ↓
reshape according to active sequence count

This step handles the runtime-dependent part of the layout.

---

Second reshape: restore the static head structure

The second target is a fully constant pattern:

[0, -1, n_heads, head_size]

The leading zero copies the already-established first dimension.

The final two dimensions remain statically visible:

n_heads

head_size

to shape inference and backend pattern matching.

Dynamic sequence metadata
→ handled first

Static head metadata
→ reintroduced through constant pattern

The code comments note that both reshapes are metadata-only, so separating them does not require an additional tensor copy.

---

One reshape was mathematically sufficient but compiler-inferentially weaker

From a pure element-count perspective:

one reshape

and:

two consecutive reshapes

can be equivalent.

The compiler does not consume only element counts.

It also consumes:

which shape values are constant

which are dynamic graph expressions

which dimensions remain provably static

which patterns backend matchers can recognize

The two-step version preserves stronger static knowledge.

Same numerical layout

+
better shape evidence

→ different backend lowering opportunity

This is another reason compiler frontends cannot be judged solely by final mathematical equivalence.

---

The PERMUTE tests pin the split-head semantics

PR #37435 added explicit tests for:

op_case 1
→ plain head/token axis swap

and:

op_case 4
→ split flat projection into heads
→ then swap head and token axes

For case 4:

input:
[1, 1, token=2, heads*size=6]

output:
[1, heads=3, token=2, size=2]

The test fills the input with a known increasing sequence and checks the exact output order.

It also adds a negative case verifying that an unsupported op_case throws rather than silently emitting an approximate graph.

---

Mathematically identical constants can produce different fusion graphs

For the builder-style attention layout, Q, K, and V each require the same transpose order:

[0, 2, 1, 3]

A natural graph construction might create one shared constant:

order = Constant([0, 2, 1, 3])

Transpose(Q, order)

Transpose(K, order)

Transpose(V, order)

The three transposes are mathematically correct.

But the GPU plugin’s TransposeSDPAMatcher requires the order constant to have:

consumers_count == 1

A single shared constant has three consumers.

The matcher can fail to absorb the transpose pattern.

The permutes can remain in the decode path and prevent the broadcast-to-SDPA fusion.

PR #37435 therefore creates a separate but numerically identical order constant for every transpose.

Constant Q
→ one consumer

Constant K
→ one consumer

Constant V
→ one consumer

The values are the same.

The graph topology is different.

---

Common-subexpression sharing is not always optimization-safe

Compiler engineering often encourages deduplication.

Same constant value
→ share one node

This can reduce graph size.

But a pattern matcher may define its contract through node ownership or consumer count.

Shared constant
→ three consumers
→ pattern rejected

Three equivalent constants
→ one consumer each
→ pattern accepted

Neither graph changes the transpose arithmetic.

Only one preserves the backend’s expected pattern identity.

Graph equivalence for mathematics is not automatically graph equivalence for optimization.

---

Backend matchers consume topology as evidence

The matcher does not execute the graph to discover that the transposes are harmless.

It recognizes a structural pattern.

That pattern may require:

specific operation types

specific constant values

specific consumer relationships

no unexpected operations between nodes

A frontend that inserts a mathematically neutral node or shares a constant differently can make the optimized path unreachable.

This can turn:

fused attention

into:

separate transpose

broadcast

matmul

softmax

matmul

without changing the model’s source-level formula.

---

A no-op Convert could also disable PagedAttention

The FLASH_ATTN_EXT translator uses ConvertLike to make K and V match Q’s element type.

After lowering, some of those conversions are no-ops because the tensors already share the same precision.

Numerically:

f16
→ Convert to f16
→ same values

It may seem harmless to leave such a node in the graph.

The state-management matcher expects the KV-cache path to have no Convert between:

cache Concat

and:

SDPA

A same-type conversion can therefore break the pattern and silently disable PagedAttention.

PR #37435 runs:

ConvertConvertLike

then

EliminateConvert

so redundant same-type conversions are removed before backend pattern matching.

---

Numerical no-ops are not necessarily compiler no-ops

A node may satisfy:

output values
=
input values

while still changing:

graph topology

matcher reachability

ownership boundaries

fusion eligibility

Examples in this patch include:

shared versus private transpose constants

same-type Convert versus no Convert

one combined reshape versus two metadata-only reshapes

All can preserve numerical meaning.

They do not preserve the same optimization graph.

---

SDPA and PagedAttention need one layout-polymorphic frontend graph

The frontend cannot assume that the graph will always remain in its initial SDPA form.

A later transformation may convert it into PagedAttention.

The same translator output must therefore preserve enough information for both.

Initial SDPA graph

[B, L, H, S]
or
[B, H, L, S]

PagedAttention-transformed graph

token count may move into leading dimension

If the frontend hard-codes:

batch dimension = 1

inside every reshape, it may be correct before the transformation and wrong afterward.

The patch instead copies the active leading dimension.

shape target begins with 0

special_zero = true

The frontend does not need to know whether that leading dimension currently means:

batch

or:

tokens

It preserves the caller’s current layout contract.

---

Layout polymorphism is not fully dynamic shape support

The patch should not be described as making all GGUF attention tensors arbitrarily dynamic.

It preserves a specific family of layouts.

Batch-major SDPA

and

token-major PagedAttention

Other facts remain static or constrained.

rank
→ often expected to be 4

head count
→ decoder-provided static fact

head size
→ decoder-provided static fact

supported op_case
→ explicitly enumerated

The graph is flexible at the axes that need to move.

It remains strict where backend lowering requires static information.

---

Dynamic does not mean every dimension should become unknown

The PERMUTE fix illustrates this clearly.

The sequence count is dynamic.

The head layout is static.

A single dynamic shape expression could erase both distinctions.

The two-step reshape preserves:

sequence-related dimensions
→ runtime-dependent

head count and head size
→ compile-time constants

This is stronger than labeling the entire tensor “dynamic.”

A good shape system preserves the maximum safe amount of knowledge.

---

Output ports, axes, and matcher topology are the same class of problem

At first, these fixes appear unrelated.

TopK port 1

dynamic k

special_zero reshape

static head size

private transpose constants

redundant Convert elimination

They share one underlying contract.

A frontend must preserve every fact that later compilation stages use to determine meaning.

Port identity

Which tensor left the producer?

Axis identity

Which dimension represents tokens or heads?

Static-shape identity

Which dimensions are proven constants?

Pattern identity

Which exact graph structure can a backend recognize?

Losing any one can produce a different compiled system.

---

A graph edge is more than its producer name

The TopK case shows:

producer node alone
→ insufficient

The complete tensor identity is:

producer
+
port
+
element type
+
shape

The attention cases add:

axis semantics
+
layout generation
+
backend pattern role

The frontend is responsible for transporting that identity from the source model into OpenVINO IR.

---

A shape is more than a list of integers

Consider:

[1, T, H, S]

and:

[T, 1, H, S]

They contain the same dimensions.

They do not assign the same meaning to those dimensions.

Likewise:

[1, -1, H, S]

and:

[0, -1, H, S]
with special_zero

may produce the same result for a batch-major input.

Only the second preserves a token-major leading dimension.

A test using only:

T = 1

could allow both to appear correct.

The added layout tests use:

T = 2

so moving the token count becomes visible.

---

Degenerate dimensions hide layout errors

Dimensions equal to one are especially dangerous.

batch = 1

token count = 1

number of KV heads = 1

can make several distinct layouts numerically indistinguishable.

For example:

[1, 1, H, S]

can be interpreted under multiple axis conventions without changing the shape tuple.

Once the token count becomes greater than one:

T = 2

incorrect broadcasting or axis movement appears.

This is why the new tests use nontrivial token and head counts.

---

A correct frontend must preserve downstream optimization eligibility

It is possible to produce a graph that is numerically correct but operationally much worse.

SDPA pattern recognized
→ fused attention or PagedAttention path

Extra Convert or unmatched transpose topology
→ decomposition into smaller operations

The frontend’s responsibility is not necessarily to guarantee one backend’s ultimate performance.

But when it knowingly emits an equivalent graph, it should avoid destroying a supported optimization pattern without reason.

PR #37435 makes several such patterns explicit through code comments and tests.

---

What PR #37435 directly changed in this area

The merged patch directly changed or added:

A shared make_topk_indices() helper

Explicit return of TopK output(1)

Port-preserving shape queries

A structural output-port regression test

Dynamic graph construction for TOP_K k

Shared TopK-based logic for ARGSORT

Static-head preservation in PERMUTE op_case 4

Layout-polymorphic RESHAPE cases 1 and 2

Separate attention handling for cgraph and builder layouts

Head-axis-dependent GQA tiling

One transpose-order Constant per Q/K/V transpose

Removal of redundant same-type Converts before state-management matching

Tests for TopK index sets

Tests for PERMUTE head splitting

Tests for both batch-major and token-major attention reshapes

The PR summary groups these under translator correctness and layout robustness, and reports 137 passing GGUF frontend tests.

---

What the patch does not prove

The merged evidence does not establish that:

Every multi-output translator in OpenVINO is free of port-loss bugs.

Every possible dynamic TOP_K shape has a dedicated runtime regression.

Every GGUF attention architecture reaches PagedAttention.

The GPU plugin always fuses every graph using these shapes.

Every plugin requires private transpose constants.

All mathematically redundant nodes must always be removed.

The separate native .gguf builder is already merged.

The translator changes provide a published end-to-end latency improvement.

The supported conclusion is narrower:

The GGUF frontend now preserves several graph facts that were previously easy to erase: the selected output port, runtime-derived k, active token-axis placement, static head dimensions, and the topology expected by attention matchers.

---

The native builder remains a separate evidence layer

PR #37435 was merged as groundwork for the existing decoder-based frontend path.

The separate native .gguf builder remains in PR #37421.

Some op_case comments describe how the native builder is expected to present tensors, and the merged translator now understands those conventions.

That does not mean direct .gguf file ingestion itself was merged by #37435.

The distinction remains:

Merged

decoder-based frontend groundwork
+
translator corrections
+
stateful transformation
+
tests

Separate open work

native .gguf container parsing
+
architecture-specific graph construction

This boundary was stated directly in the PR description.

---

The deeper lesson is that frontends preserve contracts, not just operations

A superficial translator can map:

GGML TOP_K
→ OpenVINO TopK

and appear complete.

A correct translator must ask:

Which TopK output?

Which index type?

How is k obtained?

Which axis is sorted?

What result ordering does the source guarantee?

Likewise:

GGML attention
→ OpenVINO SDPA

is not enough.

The translator must preserve:

head axis

token axis

batch or token-major layout

static head size

GQA expansion axis

reshape rank

backend-recognizable topology

The operation name is only the beginning.

A model frontend is correct when the tensor arriving at the next stage retains the same identity, axis meaning, and execution contract as the tensor described by the source graph.

---

A tensor’s full identity extends beyond its values

The final boundary can be written as:

Tensor identity
=
producer operation
+
output port
+
element type
+
shape
+
axis semantics
+
graph-generation role

Two tensors can hold the same numerical values and still be different compiler objects.

Same values
+
different output port
→ different tensor

Same flat buffer
+
different token axis
→ different layout

Same transpose order
+
different constant-sharing topology
→ different fusion result

Same dtype before and after Convert
+
extra node in state path
→ different matcher result

PR #37435 repaired all four kinds of boundary.

---

Why these errors are difficult to detect in small tests

A small test can accidentally erase the difference.

TopK values and indices
→ same shape

token count = 1
→ batch-major and token-major shapes look similar

number of heads = 1
→ head-axis mistakes can broadcast successfully

no backend fusion check
→ decomposed graph still produces correct values

The test passes.

The production graph is still wrong or slower.

The strongest tests therefore choose inputs that make the hidden distinction observable.

Inspect port index directly

Use token count greater than one

Use head count different from token count

Compare both layout forms

Check graph operation structure

Check unsupported cases fail

---

The output-port test and layout tests prove different layers

Port test

Does ShapeOf consume TopK output 1?

This is structural graph evidence.

TopK numerical test

Are the selected index sets correct?

This is result evidence.

Reshape layout tests

Does the same buffer remain valid in batch-major and token-major forms?

This is layout-contract evidence.

Backend-matcher comments and source structure

Does the emitted graph preserve known fusion preconditions?

This is lowering-path evidence.

No single test substitutes for all four.

---

Part 3: why a self-written oracle repeated the same mistaken formula

The final article examines the test boundary behind PR #37435.

Before the patch, activation tests could compare a translator against a NumPy or C++ formula written by the same developers who interpreted the ggml operation.

That proves:

translator
=
test author’s formula

It does not independently prove:

test author’s formula
=
actual ggml implementation

GELU_QUICK demonstrated the failure.

Its correct ggml form is:

x * sigmoid(1.702 * x)

A plausible but different GELU formula could be implemented in both:

translator

and

test oracle

and the test would pass.

PR #37435 added expected output captured from real ggml execution and introduced a coverage gate that compares the registered translator table with the operations actually converted by the test binary.

Part 3 examines:

Why real ggml output is a stronger oracle

Why a test list can drift from an op table

Why the coverage gate runs during global TearDown

Why a filtered test run must disable that gate

And what 137/137 actually proves

---

Previous article

OpenVINO #37435 — Why GGUF Statefulness Became a Consumer Choice Instead of a Decoder Property

Related material

• OpenVINO PR #37435 — GGUF frontend stateful conversion, translator fixes, and test infrastructure

• Merge commit — 57070bcb

• TOP_K translator

• ARGSORT translator

• Shared TopK and output-port helpers

• PERMUTE translator

• RESHAPE translator

• FLASH_ATTN_EXT translator

• Translator and layout regression tests

• Separate native .gguf builder PR #37421

---

Patch status: Merged into OpenVINO master as commit 57070bcb
Multi-output contract: TopK output 0 is values; output 1 is indices
Port-preserving type: ov::Output<ov::Node>
Shared helper: make_topk_indices() returns TopK::output(1)
Dynamic TOP_K behavior: Uses a static constant when possible and a runtime shape-derived scalar otherwise
Attention layouts: Supports cgraph [B,H,L,S] and builder-style [B,L,H,S] inputs
PagedAttention boundary: Leading dimensions are copied with special_zero rather than pinned to literal 1
Static head boundary: Dynamic sequence reshaping is separated from a constant head-layout reshape
Fusion boundary: Q/K/V transposes use separate order constants; redundant same-type Converts are removed
Validation: ov_gguf_frontend_tests reported 137/137
Evidence limitation: No dedicated dynamic-k runtime fixture or published end-to-end performance benchmark was identified in this PR

This is Part 2 of a three-part series on OpenVINO’s GGUF frontend groundwork.

Part 1 examined why KV-cache state became a consumer-selected transformation rather than a decoder property.

This article examined why a multi-output tensor must preserve its port identity, why dynamic dimensions should remain graph values, and why attention reshapes must preserve both runtime token placement and static head information.

Part 3 examines why a translator tested against a formula written by the same author can still be wrong, how real ggml output exposed the GELU_QUICK mismatch, and how the new op-coverage gate prevents registered translators from remaining untested.

#OpenVINO #GGUF #GGML #TopK #PagedAttention #SDPA #GraphCompiler #TensorLayout #ModelFrontend #CodeAnalysis