OpenVINO #37435 — Why GGUF Statefulness Became a Consumer Choice Instead of a Decoder Property

OpenVINO GGUF Frontend, Part 1 of 3 — Stateless KV-cache I/O, SetRows as a neutral boundary, and the GGUFMakeStateful rewrite

A model format does not automatically determine how its runtime state should be owned.

A transformer may require a KV cache.

That does not necessarily mean the frontend reading the model should decide:

This model is stateful.

The same converted graph may need to serve different consumers.

One consumer may want explicit cache tensors:

past key/value cache
→ model inputs

updated key/value cache
→ model outputs

Another may want OpenVINO to own the cache internally:

ReadValue
→ update state
→ Assign

A beam-search runtime may also need to reorder that state between decoding steps.

A stateless integration may not need a beam index at all.

OpenVINO PR #37435 made this distinction explicit in the GGUF frontend.

The frontend’s translation path first describes KV-cache updates through ordinary graph inputs, outputs, and an intermediate SetRows operation.

By default, the resulting model remains stateless.

A consumer that wants OpenVINO-managed state registers:

GGUFMakeStateful

through a:

DecoderTransformationExtension

That transformation runs before the frontend’s built-in stateless lowering.

It consumes the KV-cache SetRows operations, replaces explicit cache I/O with OpenVINO variables, inserts beam-aware cache reordering, and turns the updated cache output into an Assign sink.

The key architectural decision is:

A GGUF decoder describes the operations found in the model.
The consumer decides how those operations should become runtime state.

The merged PR is groundwork for the existing decoder path

PR #37435 did not add direct native .gguf file loading.

Its changes apply to the GGUF frontend path already reachable through a supplied:

GgufDecoder

The existing integration can receive a decoder wrapping a live ggml graph, including the path used by the llama.cpp ggml-openvino backend.

The flow is approximately:

llama.cpp builds a ggml graph
        ↓
GgmlOvDecoder exposes that graph
        ↓
OpenVINO GGUF frontend translates the operations
        ↓
OpenVINO model is produced

The separate native-file builder is being developed in PR #37421.

Its intended path is:

.gguf file
        ↓
OpenVINO reads the container directly
        ↓
architecture-specific graph builder
        ↓
same GGML operation vocabulary
        ↓
same frontend translators

At the time of this article, that native builder remains a separate, open and unmerged PR.

Therefore, PR #37435 should not be described as:

OpenVINO can now load every .gguf file through core.read_model().

The accurate statement is:

The existing GGUF frontend gained a clearer execution-mode boundary, broader translator correctness, and stronger test infrastructure before native .gguf construction is merged.

A decoder describes model semantics, not deployment policy

The decoder interface explicitly avoids adding fields such as:

is_stateful

is_static

use_openvino_cache

use_paged_attention

Those are deployment decisions.

The decoder’s job is to expose facts such as:

which GGML operations exist

which tensors enter those operations

which attributes belong to them

which outputs they produce

which leaves are model inputs or weights

A cache write may appear in the graph as:

GGML_OP_SET_ROWS

That describes an operation.

It does not, by itself, decide whether the converted OpenVINO model should expose the cache as an input and output or store it in an internal variable.

The frontend preserves that distinction.

Decoder
→ describes the model

Transformation extension
→ chooses the execution mode

Why the stateless graph is the baseline

A stateless cache makes the state transition visible at the model boundary.

For one inference step:

Current token inputs
+
past KV cache
        ↓
model execution
        ↓
logits
+
updated KV cache

The caller owns the state.

It can:

keep the cache in its own data structure,
route it between devices,
serialize it,
replace it,
truncate it,
batch it,
or feed it into another runtime.

The graph itself does not retain anything between calls.

A simplified stateless signature may look like:

Parameters

input_ids

position information

past_key_cache

past_value_cache

Results

logits

updated_key_cache

updated_value_cache

This is also a useful neutral representation for a frontend.

The conversion does not need to know which plugin, serving runtime, or generation framework will eventually own the cache.

`SetRows` is the neutral cache-update boundary

The GGUF frontend initially represents relevant row updates through an internal operation named:

SetRows

Conceptually:

new cache rows

+

row or position information

+

existing cache tensor
        ↓
SetRows
        ↓
updated cache tensor

The same intermediate operation can be consumed in different ways.

Stateless lowering

SetRows
        ↓
ScatterUpdate
        ↓
updated cache remains a normal graph output

Stateful lowering

SetRows
        ↓
ReadValue
+
beam-aware Gather
+
Concat
+
Assign

SetRows is therefore not intended to survive final conversion.

It is an intermediate contract that gives normalization passes a chance to choose the appropriate runtime representation.

Without an extension, every `SetRows` receives the stateless lowering

The frontend’s transformation pipeline includes a built-in:

LowerSetRowsStateless

pass.

When no stateful extension is registered:

translated GGML graph
        ↓
SetRows placeholders
        ↓
LowerSetRowsStateless
        ↓
ScatterUpdate operations
        ↓
explicit cache Parameters and Results remain

The resulting graph contains no OpenVINO variables and no state sinks.

A regression test in PR #37435 verifies this baseline.

For a simple cache-write graph, the final model contains:

Variables:
0

Sinks:
0

ScatterUpdate:
1

SetRows:
0

The cache remains part of the model interface.

cache Parameter
→ remains

cache_out Result
→ remains

The model also contains no:

beam_idx

input.

That absence is intentional.

A beam index belongs to stateful cache reordering, not to the original stateless ggml model.

Transformation extensions run before the built-in stateless pass

The order of normalization is the mechanism that makes execution mode configurable.

The relevant structure is:

Translate the GGML graph
        ↓
register caller-provided transformation extensions
        ↓
run those extensions
        ↓
run LowerSetRowsStateless on anything left

A consumer can register:

DecoderTransformationExtension(
    GGUFMakeStateful()
)

The stateful transformation receives the first opportunity to consume qualifying SetRows nodes.

KV-cache SetRows
→ consumed by GGUFMakeStateful

Other SetRows
→ remain in graph
→ later handled by LowerSetRowsStateless

This ordering matters.

If the stateless pass ran first:

SetRows
→ already converted to ScatterUpdate

the stateful pass would lose the neutral operation it was designed to recognize.

Statefulness is chosen during conversion, not after an unrelated cleanup phase

The transformation extensions are part of the GGUF frontend’s own conversion pipeline.

The caller does not need to:

convert the model

then

inspect arbitrary final nodes

then

guess which ScatterUpdates were KV caches

then

rewrite the graph manually

Instead:

register desired state transformation
        ↓
frontend translates
        ↓
state transformation runs before default lowering
        ↓
final model already has the selected execution form

The extension is therefore a first-class policy input to conversion.

`GGUFMakeStateful` does not convert every `SetRows`

A generic SetRows operation is not automatically a KV-cache write.

The GGUF graph may use row updates for other tasks, including routing-related data movement.

The pass applies a narrower rule.

It scans the graph for SetRows nodes whose destination input is a model:

Parameter

Conceptually:

SetRows destination is a model Parameter
→ candidate cache write

SetRows destination is another intermediate tensor
→ leave it alone

The comments explicitly mention that non-cache row writes, such as MoE routing writes, remain available for the built-in stateless lowering.

The pass therefore does not classify by operation name alone.

It also checks the ownership of the destination tensor.

The graph is collected first and rewritten afterward

The pass first gathers all qualifying cache writes into a vector.

Only after the scan finishes does it mutate the graph.

Walk model operations

→ identify cache SetRows nodes

→ store them

→ finish walk

→ rewrite collected nodes

This avoids modifying the operation list while iterating over it.

It also gives the transformation one stable set of cache candidates before it starts replacing nodes and removing model inputs or outputs.

A skipped cache remains stateless

GGUFMakeStateful accepts:

skip_caches

as a set of cache names.

A matching cache is left untouched.

Cache name in skip_caches
→ stateful pass does not consume its SetRows
→ built-in stateless pass handles it later

The source gives a concrete reason for this boundary.

A normal autoregressive KV cache often grows through append semantics.

past tokens

+

new token rows
        ↓
larger cache

A sliding-window cache may behave differently.

oldest rows removed from the front

new rows appended at the back

An append-only OpenVINO variable would not reproduce front eviction correctly.

Therefore:

ordinary append-grown cache
→ suitable for GGUFMakeStateful

sliding-window cache with front eviction
→ may remain stateless through skip_caches

The pass does not pretend that every tensor called a cache has the same state transition.

The stateful rewrite removes cache tensors from model I/O

For each qualifying cache write, the pass replaces the explicit cache interface with internal state.

Before

cache Parameter
        ↓
SetRows
        ↓
updated-cache Result

After

empty initial state
        ↓
ReadValue
        ↓
Gather by beam_idx
        ↓
Concat new rows
        ↓
Assign

The cache Parameter is removed.

The Result that returned the updated cache is removed.

A variable is added to the model.

An Assign is added as a sink.

Cache before rewrite:
caller-owned I/O

Cache after rewrite:
OpenVINO-owned variable

The pass identifies the exact cache Result through graph identity

The transformation does not remove results merely because their names resemble the cache name.

Before replacing the SetRows, it examines the consumers of that specific node’s output.

If a consumer is a:

Result

that result is collected for removal.

This is more robust than relying on a naming convention such as:

Result name == cache name

Names can change.

Graph relationships define which result actually returns the cache update.

The cache becomes an OpenVINO `Variable`

The pass creates an:

ov::op::util::Variable

for every converted cache.

The variable keeps:

cache element type

cache name

cache shape contract

The append axis becomes dynamic because the number of accumulated tokens changes between generation steps.

Other cache dimensions remain tied to the declared cache layout.

For example:

[batch, tokens, kv_heads, head_size]

may become:

[1, ?, 2, 4]

where:

?
→ accumulated token count

The cache’s friendly name becomes the variable identifier.

The token axis can be inferred only under a narrow shape contract

By default, GGUFMakeStateful receives:

append_axis = -1

This means:

infer the append axis

The pass scans the cache shape and expects exactly one dynamic dimension.

one dynamic axis
→ inferred token axis

The remaining dimensions must be static.

This allows a shape such as:

[1, ?, 2, 4]

to identify:

axis 1
→ token axis

But a shape such as:

[?, ?, 2, 4]

is ambiguous.

more than one dynamic axis
→ inference rejected

A fully static preallocated cache is also ambiguous under this rule.

[1, 2048, 2, 4]

no dynamic axis
→ cannot infer which dimension is append-grown

The caller must supply an explicit axis for that case.

The pass fails rather than guessing.

A static rank is required

The cache rank itself must be known.

Rank 4
→ dimensions can be interpreted

Dynamic rank
→ token-axis position cannot be established safely

The pass asserts that the cache rank is static before constructing the variable, reshape pattern, gather, and concat.

This is another example of the transformation making its assumptions explicit rather than treating every dynamic tensor as interchangeable.

Non-token dimensions must remain static

The initial state needs a concrete empty tensor.

The pass creates an initial shape with:

token axis extent = 0

and preserves every other cache dimension.

For:

[1, ?, 2, 4]

the initial tensor becomes:

[1, 0, 2, 4]

To create that constant, the non-token dimensions must have known lengths.

batch
→ static

KV head count
→ static

head size
→ static

If another non-token dimension is dynamic, the pass rejects the graph.

The transformation supports a dynamic history length.

It does not claim to support arbitrary dynamic cache geometry.

The empty initial tensor is required, not cosmetic

The pass creates an empty constant and feeds it into:

ReadValue

The initial state therefore has:

zero past tokens

but still has a real parent edge.

The source comments identify a backend-specific reason.

The CPU MemoryInputSDPA path can abort when a memory input has zero parent edges.

Therefore:

ReadValue with no explicit initial input
→ not sufficient for this path

The zero-length constant ensures:

initial cache contains no tokens

and

the state node still has an explicit initialization edge

This is a small graph detail with runtime significance.

The new cache rows may not already have the cache’s layout

The SetRows placeholder can present incoming rows in a flattened layout.

For example, the cache may logically use:

[batch, tokens, kv_heads, head_size]

while new rows arrive in a form closer to:

[batch, 1, tokens, kv_heads × head_size]

The number of elements agrees.

The axis split does not.

Before appending, the pass constructs a reshape pattern that restores the cache layout.

new rows
        ↓
Reshape according to cache dimensions
        ↓
Concat with past cache

The token dimension is inferred through:

-1

in the reshape pattern.

Static dimensions after it are restored from the cache shape.

Dimensions before it may be copied from the incoming tensor through special-zero reshape behavior.

The stateful transformation therefore does not assume that the placeholder’s flattened view is already the final state layout.

Beam search requires past-state reordering

During beam search, several candidate sequences may exchange their positions between decoding steps.

Suppose the previous batch order was:

Beam A
Beam B
Beam C

The next step may choose:

Beam C
Beam A
Beam A

The past KV cache must follow that new beam ordering.

Otherwise:

new tokens for one beam

would attend to

the history of another beam

The pass handles this by gathering the previous state along the batch axis before appending the new rows.

ReadValue
        ↓
Gather(past, beam_idx, axis=0)
        ↓
Concat(new_rows, token_axis)

The current beam mapping is therefore applied to the history before the new decoding step extends it.

`beam_idx` belongs to OpenVINO state, not to ggml

The original ggml graph does not contain an OpenVINO state variable.

Therefore it also has no input whose purpose is:

reorder an OpenVINO Variable

If a decoder declared beam_idx unconditionally:

stateless conversion
→ beam_idx Parameter exists

but

no stateful Gather consumes it

The result would be a consumer-less input in the stateless graph.

PR #37435 assigns ownership differently.

Decoder
→ does not declare beam_idx

GGUFMakeStateful
→ creates beam_idx when state is actually introduced

This keeps the input tied to the feature that uses it.

The pass reuses an existing `beam_idx` when one is already present

Before creating a new parameter, the pass searches the model for an existing input with the configured beam-index name.

If one exists:

reuse it

If none exists:

create an i32 one-dimensional dynamic Parameter

The default name is:

beam_idx

This supports callers that already introduced a compatible beam-index parameter or a second application of the pass.

`beam_idx` is added only if at least one cache is converted

The pass may construct a local beam-index node before it has finished identifying cache writes.

But if no qualifying cache write exists:

cache_writes.empty()
→ return false

The parameter is never added to the model.

When caches are converted, it is added only after the new Gather nodes have been constructed.

No state conversion
→ no beam_idx input

State conversion performed
→ beam_idx added

The model does not gain an unused state-management input merely because the pass was registered.

Batch size one still retains the Gather

For batch size one:

beam_idx = [0]

and the Gather is numerically an identity.

It may look removable.

The source explains why the operation is retained.

Gather on past state
→ part of the pattern recognized by CPU stateful_sdpa_fusion

It is also required when beam search later uses a nontrivial mapping.

The graph preserves the state-management contract even when the current batch makes that operation mathematically trivial.

The past and new rows are appended through `Concat`

After reordering the previous state, the pass creates:

Concat(
    past,
    new_rows,
    append_axis
)

The result is both:

the cache visible to remaining consumers in the current graph

and:

the value assigned to the state variable for the next call

The original SetRows node is replaced by this concatenated value.

Consumers that previously read updated SetRows output

→ now read the grown state tensor

An:

Assign

sink commits that value to the variable.

The stateful graph separates read, use, and commit

The rewritten graph has a clear lifecycle.

Read

ReadValue
→ obtain cache accumulated by previous calls

Reorder

Gather
→ map previous histories into current beam order

Extend

Concat
→ append this call’s new cache rows

Use

attention and other consumers
→ read the grown cache

Commit

Assign
→ store the grown cache for the next call

State is no longer implicit in an input/output loop owned by the caller.

It is represented directly in the graph.

The stateless and stateful graphs begin from the same translation

This is the central design advantage.

The frontend does not need two complete translation stacks.

Stateless decoder translator

and

stateful decoder translator

would risk diverging across every GGML operation and model architecture.

Instead:

one GGML translation
        ↓
one SetRows intermediate contract
        ↓
two possible lowerings

Default consumer

SetRows
→ ScatterUpdate
→ explicit cache I/O

Stateful consumer

SetRows
→ ReadValue / Gather / Concat / Assign
→ internal cache state

The model semantics are shared.

Only the execution representation changes.

Remaining `SetRows` nodes still receive a valid lowering

GGUFMakeStateful does not need to consume every placeholder.

After caller extensions run, the built-in pass processes whatever remains.

KV cache writes selected for state
→ stateful rewrite

Skipped cache writes
→ stateless lowering

MoE or other row writes
→ stateless lowering

This allows mixed execution modes inside one converted graph.

One cache may remain explicit while another becomes a variable.

The pass’s skip_caches option uses exactly this mechanism.

The regression test compares the two execution modes

PR #37435 includes paired tests built from the same cache-write fixture.

No extension

Expected graph:

Variables:
0

Sinks:
0

ReadValue:
0

Assign:
0

ScatterUpdate:
1

SetRows:
0

cache remains Parameter

cache_out remains Result

beam_idx absent

`GGUFMakeStateful` extension

Expected graph:

Variables:
1

Sinks:
1

ReadValue:
1

Assign:
1

Gather:
1

ScatterUpdate:
0

SetRows:
0

cache Parameter removed

cache Result removed

beam_idx added

The test also verifies the variable contract:

Variable ID:
cache

Element type:
f16

Shape:
[1, ?, 2, 4]

This is stronger than checking only that conversion no longer throws.

It checks the intended graph ownership transition.

A skip-list test verifies the negative boundary

The test suite also registers:

GGUFMakeStateful({"cache"})

for the same fixture.

Because the only cache is skipped:

stateful pass changes nothing

The built-in lowering then produces the same stateless form as if no stateful extension had been registered.

Variables:
0

ScatterUpdate:
1

cache input:
preserved

cache output:
preserved

This protects the rule that registration of the pass does not automatically convert every named cache.

The extension order is itself part of the regression contract

The stateful test verifies that:

no ScatterUpdate exists

after conversion.

That indirectly confirms the ordering.

If LowerSetRowsStateless had run first:

SetRows would already be gone

ScatterUpdate would exist

GGUFMakeStateful would have nothing to consume

The expected ReadValue, Gather, Concat, and Assign structure would not appear.

The final graph therefore tests both the rewrite and its placement in the normalization pipeline.

Why this is useful for multiple consumers

The same frontend can serve callers with different state ownership requirements.

Explicit-state serving runtime

Choose default stateless conversion

Own cache tensors externally

Control truncation, transfer, and storage directly

OpenVINO generation runtime

Register GGUFMakeStateful

Let the model retain cache Variables

Use beam-aware state reordering

Backend-specific implementation

Register another DecoderTransformationExtension

Consume SetRows through a different state representation

The extension point is not hard-coded to one pass.

GGUFMakeStateful is one implementation of the stateful policy.

The frontend does not need to know the final plugin

The translated model may eventually be compiled for:

CPU

GPU

NPU

a serving layer

a GenAI runtime

The frontend itself does not need to encode every plugin’s preferred state-management strategy.

It exposes a neutral operation boundary and lets a consumer register the transformation it needs.

That reduces coupling between:

model ingestion

and

runtime deployment

The pass is append-oriented, not a universal state converter

The stateful rewrite assumes that a qualifying cache grows by appending new rows along one token axis.

past cache

+

new rows
→ Concat

It does not model:

arbitrary overwrites,
ring buffers,
front eviction,
multiple independently dynamic axes,
fully dynamic cache geometry,
or every recurrent-state form.

The separate native-builder PR #37421 already identifies additional recurrent-state rewriting for architectures whose state is not represented as a KV-cache SetRows.

That work is outside the merged #37435 patch.

The accurate boundary is:

GGUFMakeStateful in #37435 converts append-grown KV caches represented by qualifying SetRows writes. It is not a universal transformation for every form of model state.

Direct `.gguf` file ingestion is still a separate layer

The GGUF frontend remains hidden from normal model auto-detection in the merged groundwork.

The existing merged path accepts a supplied:

shared_ptr<GgufDecoder>

It does not make this call resolve automatically:

core.read_model("model.gguf")

The native builder in #37421 is intended to construct the GGML-style graph directly from the container and reuse the same translators.

Until that work lands, the merged stateful pass should be understood as:

runtime and conversion infrastructure

for the existing decoder-based frontend

rather than complete standalone GGUF file support.

PR #37435 also changed more than statefulness

The merged PR includes several other substantial changes:

Preserve exact output ports across translators

Fix multi-output behavior such as TopK

Support dynamic k in TOP_K

Make attention shapes valid for both SDPA and PagedAttention layouts

Add Q2_0 ternary quantization for Bonsai models

Add an op-coverage gate

Compare activation translators with captured real ggml output

Create a dedicated GGUF frontend CI component

Those changes share one theme:

The frontend should preserve the semantics of the original ggml graph
rather than approximating them through implicit assumptions.

Parts 2 and 3 examine those boundaries separately.

What this patch directly changed

PR #37435 directly added or changed:

the public GGUFMakeStateful model pass,
DecoderTransformationExtension support in the GGUF frontend,
ordering caller transformations before LowerSetRowsStateless,
append-grown KV-cache conversion into OpenVINO variables,
automatic or explicit token-axis selection,
beam-index creation and past-cache reordering,
cache Parameter and Result removal,
cache ReadValue, Concat, and Assign construction,
named cache exclusions through skip_caches,
and regression tests for stateless, stateful, and skipped-cache forms.

What this patch does not establish

The merged evidence does not establish that:

native .gguf path loading is already merged,
every model architecture is supported,
every cache can become append-grown state,
sliding-window eviction is implemented by this pass,
arbitrary recurrent model state is handled,
every plugin supports the resulting stateful graph identically,
beam search has been benchmarked across all plugins,
PagedAttention is always selected,
the open native-builder PR will land unchanged,
or all twenty-five checkpoints reported by the stacked builder branch are part of the merged #37435 runtime contract.

The supported conclusion is narrower:

The merged GGUF frontend now has an explicit, tested boundary between stateless translation and consumer-selected OpenVINO KV-cache state.

Statefulness is not a property of the file alone

A GGUF model may describe the computations required to update a cache.

The file does not necessarily decide who owns that cache at runtime.

Same model semantics

→ caller-owned cache tensors

or

→ OpenVINO-owned Variables

Both can be valid deployments.

Encoding one choice permanently inside every decoder would mix:

what the model computes

with

how one runtime wants to operate it

PR #37435 separates those concerns.

The transformation owns the inputs that only statefulness requires

beam_idx is the clearest example.

Stateless graph
→ no OpenVINO state to reorder
→ no beam_idx required

Stateful graph
→ past Variable must follow current beam mapping
→ beam_idx required

Therefore:

decoder does not invent beam_idx

stateful transformation creates beam_idx

The feature that introduces a dependency also introduces the input needed to satisfy it.

That is a precise ownership rule.

The deepest contract is between the neutral operation and its lowerings

SetRows is useful because neither execution mode has to be encoded prematurely.

GGML operation semantics

→ SetRows intermediate

→ consumer-selected lowering

A neutral intermediate is valuable only when its meaning is sufficiently clear for both lowerings.

For KV caches, the stateful pass adds narrower assumptions:

destination is a model Parameter

cache is append-grown

token axis is known

non-token dimensions are static

skipped caches remain outside this policy

The transformation does not hide those assumptions.

It checks them.

A model can be stateless in its interface and stateful in its use

Even a stateless graph can participate in stateful generation.

The caller can feed the output cache back into the next invocation.

Call 1 output cache
        ↓
Call 2 input cache
        ↓
Call 2 output cache
        ↓
Call 3 input cache

The difference is where the loop lives.

Stateless model

state loop
→ outside the graph

Stateful model

state loop
→ represented through Variables inside the graph

PR #37435 lets the consumer choose the location of that loop.

The final lesson is about separating description from ownership

The decoder knows:

A cache tensor exists.

New rows are written.

The updated cache is consumed.

The consumer knows:

I want explicit cache I/O.

or

I want OpenVINO to retain the state.

or

I need another backend-specific representation.

The frontend connects them through a transformation boundary.

A model frontend should preserve what the source graph means before deciding who owns the state created by that meaning.

OpenVINO’s GGUF path now translates one cache-update contract and allows more than one execution policy to consume it.

Part 2: why preserving an output port mattered for `TopK`

The next article examines the translator boundary.

A helper that converted an:

ov::Output<ov::Node>

into:

get_node_shared_ptr()

discarded the original output-port identity and implicitly selected output 0.

That is harmless for single-output operations.

It is not harmless for:

TopK values

TopK indices

Part 2 examines why multi-output tensor identity must include both:

node

and

output port

It also covers dynamic k, ARGSORT/TOP_K index construction, and attention layouts that must remain valid under both SDPA and PagedAttention.

Part 3: why a NumPy oracle repeated the same mistaken formula

The final article examines the new test infrastructure.

Earlier activation references were implemented through NumPy formulas.

That can test:

Does the OpenVINO translator match the formula written in the test?

It cannot independently prove:

Did the test author understand ggml’s real implementation correctly?

PR #37435 replaced those references with output captured from actual ggml execution and added a coverage gate preventing a newly registered translator from shipping without a conversion test.

Part 3 examines how the earlier GELU_QUICK error survived a self-derived oracle and why source-independent reference output is a stronger contract.

Related material

Patch status: Merged into OpenVINO master as commit 57070bcb
Merged ingestion path: Supplied GgufDecoder, including the existing live ggml integration
Default execution form: Stateless cache Parameter/Result I/O
Stateful selection: Consumer registers GGUFMakeStateful through DecoderTransformationExtension
Intermediate boundary: SetRows
Stateful graph: ReadValue → Gather(beam_idx) → Concat → Assign
Cache-I/O change: Cache Parameters and corresponding Results are removed
Append-axis rule: Explicit axis or exactly one inferable dynamic dimension
Sliding-window boundary: Named caches can be left stateless through skip_caches
Validation: ov_gguf_frontend_tests reported 137/137
Native file-builder status: PR #37421 remains separate and open at the time of this article

This is Part 1 of a three-part series on OpenVINO’s GGUF frontend groundwork.

Part 2 examines why losing an ov::Output port silently changed the identity of a multi-output tensor, and how the patch repaired TopK, dynamic k, and attention-layout assumptions.

Part 3 examines why captured output from real ggml provides a stronger oracle than reimplementing the expected formula in NumPy, and how the new test-coverage gate makes translator support auditable.

#OpenVINO #GGUF #GGML #LlamaCPP #KVCache #StatefulModel #BeamSearch #PagedAttention #ModelFrontend #CodeAnalysis