Edge AI Hasn’t Gone Quiet — Big Tech Code Has Shifted from “Adding Support” to Operating On-Device Generative AI

The second wave of edge AI emerging across ExecuTorch, LiteRT, ONNX Runtime, Arm, and Qualcomm

For a while, edge-AI development was easy to spot in open-source repositories.

A new NPU backend was added.

A mobile GPU delegate was connected.

Names such as Core ML, Vulkan, XNNPACK, Qualcomm QNN, and Arm Ethos-U appeared directly in commit titles.

Recently, that activity may seem quieter.

But a closer look at the code suggests that edge AI has not slowed down.

The names and depth of the patches have changed.

The earlier question was relatively simple:

Can this model run on a smartphone or embedded device?

The questions now being addressed are much more specific:

How much of the model can remain on the NPU or GPU?

How can the KV cache be compressed as a conversation becomes longer?

Do dynamic shapes and mutable state retain their meaning when they cross a delegate boundary?

Can a multi-gigabyte model be loaded without copying the entire file into RAM at application startup?

How many graph nodes did the accelerator actually execute, and how many partitions was the model divided into?

Can a WebGPU kernel adapt its subgroup structure to the underlying device?

Edge AI has not gone quiet.

It has moved from:

First wave
→ Make AI run on the device

to:

Second wave
→ Operate generative AI within real product constraints

---

The first wave was about proving that the backend existed

The initial edge-AI stack had a relatively clear goal.

PyTorch or TensorFlow model
        ↓
Convert it into a mobile graph
        ↓
Connect a CPU, GPU, or NPU backend
        ↓
Complete the first on-device inference

At that stage, adding an operator or registering a new delegate was itself a major achievement.

Many target workloads were small CNNs or image-classification models.

If an unsupported operation had to fall back to the CPU, that could still be acceptable as long as the full model eventually ran.

Commit titles reflected that stage clearly:

Add NPU backend

Enable Vulkan delegate

Support Core ML

Add mobile GPU execution

Enable Arm operator

Once the backend exists, however, a harder question appears:

The model runs.

Can it function as a real product?

The gap is especially large for generative AI.

An LLM or VLM does not process one input and return one fixed output.

It generates tokens repeatedly.

It preserves state from previous steps.

It continually updates a KV cache.

It handles dynamic sequence lengths.

If part of the model falls back to the CPU, tensors may cross the accelerator boundary during every generated token.

The existence of a backend is therefore no longer enough.

---

Smartphone runtimes are now trying to keep an entire VLM decoder on the GPU

A recent change to ExecuTorch’s Vulkan backend illustrates this shift.

The patch addressed two boundary failures found while bringing up LFM2.5-VL, a vision-language model, on Android.

The first problem came from pattern matching and delegation.

A matched pattern included placeholder bindings that were not actually computed by the fused operation.

Those feeder nodes were then treated as though the Vulkan partition owned them.

That could bypass the node’s own support check.

One affected operation was aten.index.Tensor, used by rotary positional embedding to look up values in a two-dimensional frequency table.

The Vulkan path had previously assumed a narrower indexing shape.

Once the higher-rank table entered the delegate under that incorrect assumption, the resize logic attempted to replace a rank-two output shape with a rank-one shape and aborted.

The second problem involved the contract between the serialized Vulkan graph and the delegate call.

The backend calculated an output offset using:

output_offset
=
args.size()
-
num_outputs

Because the values were unsigned, a graph declaring more outputs than the delegate call supplied did not produce a negative number.

The subtraction wrapped into a very large positive value.

Later output access could then read through an invalid pointer and surface as a segmentation fault with little indication of the real mismatch.

The patch added an explicit argument-count check before that subtraction.

It also corrected the Vulkan gather path and the delegation boundary for the higher-rank table.

According to the device results recorded with the change, LFM2.5-VL 450M and 1.6B decoders were lowered into a single Vulkan call and produced both text and image responses on a Galaxy S26 Ultra with an Adreno GPU.

The reported decode rates were approximately:

LFM2.5-VL 450M

Vulkan
→ about 150 tokens/s

XNNPACK
→ about 100 tokens/s

and:

LFM2.5-VL 1.6B

Vulkan
→ about 72 tokens/s

XNNPACK
→ about 54 tokens/s

The recorded time to first token also improved.

The important signal is not one performance number.

The patch is no longer asking:

Can Vulkan execute one gather operation?

The real question has become:

Can a smartphone GPU retain a VLM decoder containing:

RoPE

dynamic shapes

mutable state

multiple graph outputs

delegate partitions

The unit of edge deployment has expanded from an individual operator to the decoder itself.

---

After model weights, the next memory bottleneck is the KV cache

On-device LLM memory does not end with model weights.

Weights are loaded before execution and remain relatively fixed.

The KV cache grows as tokens are generated.

Model weights
→ relatively fixed memory

KV cache
→ memory that grows with conversation length

For short prompts and brief answers, model weights may remain the most visible constraint.

As conversations become longer, agents perform multi-step work, and vision or audio context must also be retained, runtime state becomes increasingly important.

A recent ExecuTorch Arm change added support for exporting a statically quantized int8 KV cache for Llama.

The path goes beyond declaring that the cache tensor uses int8.

It introduces a flow resembling:

Calibration before export
        ↓
Determine quantization parameters
        ↓
Create mutable int8 key and value caches
        ↓
Update selected cache positions
        ↓
Dequantize state for the attention path

The change connects this cache representation to Arm-oriented paths including:

• TOSA

• VGF

• Ethos-U

It also adds model-configuration validation, cache-replacement tests, calibration handling, and lowering coverage.

The broader implication is clear.

On-device LLM optimization
≠
weight quantization alone

It increasingly means:

weights
+
KV cache
+
mutable runtime state
+
backend-supported update operations

The patch does not prove that every Arm device now achieves the same speed, accuracy, or long-context stability.

Those questions still require device-specific validation.

The narrower signal is that the memory target has expanded from static model storage to the model’s working memory.

---

A large model must be loaded before it can be accelerated

Kernel execution time is not the only important edge metric.

A multi-gigabyte model first has to be loaded.

Copying an entire model file into a separate in-memory buffer can create several costs:

Original model file
+
copied model buffer
+
additional runtime memory

This can increase both startup latency and peak RAM usage.

On a mobile operating system, high memory pressure can also cause applications to be terminated or prevent them from remaining active in the background.

A recent LiteRT-LM change added a file-backed path for certain NPU AOT models.

When the model contains the required auxiliary section and does not require in-place flatbuffer mutation, the runtime can preserve the file-backed model source rather than copying the entire flatbuffer into a separate buffer.

The conceptual difference is:

Possible previous path

Model file
        ↓
Copy entire model into memory
        ↓
Create runtime model

versus:

NPU AOT model file
        ↓
Preserve file-backed or memory-mapped resources
        ↓
Use required pages during runtime

This patch does not add a new neural-network operator.

It reduces the operational cost of packaging and starting a large on-device model.

That distinction becomes more important as edge AI moves from demos into applications.

Users do not directly experience theoretical NPU throughput.

They experience:

How quickly the application opens

How long the first token takes

Whether the device becomes hot

Whether other applications remain usable

Whether the feature works reliably offline

---

“The NPU was used” is no longer enough

The presence of an NPU delegate does not mean that the full model runs on the NPU.

Only part of the graph may be delegated.

Unsupported operations may remain on the CPU.

Supported nodes may also be broken into several accelerator partitions.

NPU partition 1
        ↓
CPU fallback
        ↓
NPU partition 2
        ↓
CPU fallback
        ↓
NPU partition 3

Every transition can introduce costs such as:

• tensor synchronization

• memory conversion

• command submission

• device-boundary traffic

Each NPU kernel may be fast while the complete model remains slower than expected.

LiteRT and the TFLite benchmarker recently began tracking the work performed by each accelerator separately.

The new metrics include:

Total graph nodes

NPU delegated node count
NPU partition count

GPU delegated node count
GPU partition count

CPU delegated node count
CPU partition count

Remaining undelegated nodes

This indicates a change in how edge acceleration is evaluated.

The old question was:

Was the NPU delegate connected?

The current questions are:

What percentage of the graph remained on the NPU?

How many partitions was it divided into?

Where did CPU fallback occur?

How often did tensors cross accelerator boundaries?

The important concept is no longer backend existence.

It is graph ownership.

---

The browser is becoming a deeper edge runtime

Edge AI does not refer only to native smartphone applications or embedded devices.

The browser is also becoming an edge runtime.

WebGPU can allow local AI execution without requiring a native application or a continuous server connection.

The initial achievement was simply:

GPU compute can execute inside the browser.

But running the same WGSL kernel across different GPU architectures can produce very different performance.

Hardware may differ in:

• subgroup width

• workgroup behavior

• supported shader features

• preferred execution geometry

ONNX Runtime recently added infrastructure that allows a WebGPU compute pipeline to request a fixed subgroup size.

On supported Intel devices, the quantized MatMulNBits path can request:

subgroup size = 32

The requested subgroup size is also included in the program cache identity.

That prevents two pipelines compiled under different subgroup assumptions from being treated as the same cached program.

The path now resembles:

Shared WebGPU API
        ↓
Check device capabilities
        ↓
Select a hardware-sensitive subgroup width
        ↓
Include that configuration in the cache key

This is a deeper stage than merely achieving portability.

The WebGPU runtime is beginning to preserve a common programming surface while managing device-specific kernel contracts underneath it.

---

Qualcomm’s work is expanding into a complete GenAI deployment pipeline

A compiler alone is not enough to deploy on-device generative AI.

The full workflow must become repeatable:

Load the model and tokenizer
        ↓
Prepare a calibration dataset
        ↓
Apply PTQ or QAT
        ↓
Compile for the target SoC
        ↓
Transfer the artifact to a smartphone
        ↓
Execute it on the device
        ↓
Collect outputs and performance results

The Qualcomm path in ExecuTorch is being organized into a GenAI pipeline that separates these stages.

A recent step added adapter interfaces around external systems.

The adapters include:

QuantizerAdapter
→ quantization and calibration

CompilerAdapter
→ ExecuTorch and QNN compilation

DeviceRunnerAdapter
→ push, execute, and pull through ADB

ModelLoaderAdapter
→ Hugging Face model and tokenizer loading

CalibrationDataAdapter
→ PTQ calibration data

TrainingDataAdapter
→ QAT training data

External systems are placed behind injectable protocol interfaces.

This allows the orchestration layer to be tested without every test requiring the real SDK, hardware, or Hugging Face environment.

The current boundary still matters.

The public description identifies this as the fourth step in a larger multi-PR Qualcomm GenAI pipeline plan.

Some compiler behavior remains represented by placeholder implementations while later recipe APIs, compilation strategies, inference strategies, and end-to-end tests are still pending.

The accurate conclusion is:

A complete Qualcomm GenAI deployment pipeline
→ not yet finished

but:

A structured route from model preparation
to real device execution
→ actively being assembled

---

Why did edge AI appear quieter?

The activity did not disappear.

Commit titles became more specialized.

The first wave was easy to recognize:

Add NPU backend

Enable mobile execution

Support Vulkan

Add Core ML delegate

The current wave appears under titles such as:

Fix delegate argument-count mismatch

Accept SymInt across a delegate boundary

Quantize the KV cache

Enable file-backed NPU AOT loading

Track accelerator partition counts

Control WebGPU subgroup size

Add device-runner adapters

The phrase “edge AI” may not appear.

But the work is directly related to operating generative models on local devices.

The development has become less visible and more operational.

Adding support
→ repairing operational boundaries

Small models
→ LLMs and VLMs

Static inputs
→ dynamic shapes and mutable state

One-shot inference
→ repeated token generation

Accelerator presence
→ graph coverage and partition counts

Kernel latency
→ startup, TTFT, memory, and sustained device throughput

---

The second wave of edge AI is centered on four problems

1. Keeping the full generative graph on the accelerator

Supporting one operator matters less than keeping the decoder inside one backend partition.

If RoPE, gather, dynamic shapes, cache updates, or state mutation leave the delegate, every token may pay the boundary cost.

---

2. Managing runtime memory rather than only model weights

Weight quantization remains important.

But long-context and agentic workloads also require control over:

KV cache

mutable state

temporary tensors

scale metadata

The longer the model works, the more important these become.

---

3. Reducing operational cost before and after inference

Product performance includes:

model loading

memory mapping

startup latency

graph compilation

cache reuse

tracing

profiling

These are not separate from the user experience.

They are part of it.

---

4. Measuring the actual amount of acceleration

Registering an NPU or GPU is insufficient.

The meaningful metrics are becoming:

delegated node count

partition count

CPU fallback locations

accelerator transitions

real TTFT

sustained tokens per second

---

Cloud AI and edge AI are unlikely to replace each other completely

The current code direction suggests that edge execution is not necessarily attempting to replace every cloud inference workload.

A more likely outcome is finer workload separation.

This is an inference from the combined software direction rather than a single explicit roadmap.

Edge
→ immediate response
→ privacy-sensitive inputs
→ offline operation
→ small and medium models
→ repeated personalized work

Cloud
→ very large models
→ large-scale compute and memory
→ complex reasoning
→ shared services for many users

Some requests can be completed entirely on the device.

Others can be preprocessed locally, with selected context then sent to the cloud.

A local model may also post-process, personalize, or verify a cloud-generated answer.

Edge-AI growth may therefore represent:

a more precise division of AI work between the cloud and the device

rather than the disappearance of cloud AI.

---

Why it is still too early to choose one company

Several layers of the industry can capture value from this transition.

Mobile SoCs and NPUs
→ Qualcomm, MediaTek, Apple, Google

CPU and NPU architecture IP
→ Arm

On-device runtimes
→ Meta, Google, Microsoft

Mobile GPUs
→ Qualcomm, Arm, Apple, Intel, AMD

Memory and storage
→ LPDDR and NAND supply chains

Model and compiler toolchains
→ multiple open-source ecosystems

The code currently provides stronger evidence about the shared bottlenecks than about where the greatest economic value will accumulate.

The confirmed structural change is:

Deploying generative models on devices
has shifted from operator support
to operational integration.

That does not yet identify one inevitable winner.

This article therefore does not attach the trend to one ticker.

Not every technical development should immediately become a stock recommendation.

---

What I will watch next

How often the full decoder stays on one accelerator

Operator-support lists are no longer enough.

The relevant evidence will include:

NPU or GPU delegation percentage

partition count

CPU fallback locations

backend transitions per token

---

Whether KV-cache compression preserves both quality and speed

An int8 KV cache may reduce memory while introducing costs in:

• long-context accuracy

• attention quality

• calibration generalization

• quantization and dequantization

End-to-end model evidence is required.

---

Whether startup and first-token latency decline materially

File-backed loading and AOT artifacts should eventually be evaluated through:

startup time

peak RAM

model-load latency

time to first token

first run versus repeated runs

---

Whether sustained performance survives power and thermal limits

A short benchmark may be fast while continuous generation triggers thermal throttling.

The relevant comparison is not only:

tokens per second during the first 10 seconds

but also:

tokens per second after several minutes

battery consumption

surface temperature

---

Whether runtime fragmentation declines

ExecuTorch, LiteRT, ONNX Runtime, Core ML, QNN, and OpenVINO can require different model formats and quantization contracts.

If fragmentation remains high, deployment cost may offset some of the advantage of local execution.

The balance between common export paths and backend-specific optimization will matter.

---

Whether the work reaches real product features

The final evidence is not a benchmark alone.

It is repeated deployment in:

operating-system features

smartphone applications

offline assistants

real-time translation

personalized search

vision and audio agents

industrial, automotive, and robotic devices

---

What would weaken this thesis

The second-wave edge-AI thesis would weaken if:

• most models continue to depend heavily on CPU fallback

• smaller local models fail to narrow the quality gap with cloud models

• KV-cache and runtime state still exceed device memory limits

• quantization quality loss exceeds product tolerances

• sustained generation remains constrained by battery and heat

• backend and model-format fragmentation creates excessive development cost

• users show limited demand for offline and local AI features

A code path existing is not the same as mass consumer adoption.

---

The core idea

The first wave of edge AI built a route to the device.

Model
→ mobile runtime
→ CPU, GPU, or NPU

The second wave is about operating generative AI on that route.

LLM or VLM decoder
        ↓
dynamic shapes
        ↓
KV cache and mutable state
        ↓
delegate coverage
        ↓
file-backed loading
        ↓
hardware-specific kernel configuration
        ↓
real-device TTFT, throughput, memory, and power

Recent patches no longer look like dramatic backend announcements.

Instead, they close small boundaries that can prevent a product from working.

One incorrectly delegated node.

One output-count mismatch.

One unsigned wraparound.

One unsupported cache update.

One unnecessary model copy.

One unmeasured partition.

If those problems remain, on-device generative AI stays a demonstration.

As those boundaries are closed, local AI begins to form an execution system designed for the device rather than a smaller copy of cloud inference.

Edge AI has not gone quiet.
It has moved from “the model can run on the device” to “generative AI can be operated as a product on the device.”

That transition is appearing in code before it becomes obvious in product announcements.

---

This article is an independent long-term industry observation based on publicly available technical and industry information.

Detailed source-discovery paths and research methodology are not disclosed.

This article does not constitute a recommendation to buy or sell any company or security.

#EdgeAI #OnDeviceAI #GenerativeAI #MobileAI #ExecuTorch #LiteRT #ONNXRuntime #Vulkan #NPU #KVCache

---