← Writing
AI Infra

LLM Inference on Edge Platforms

Prefill, Decode, and the Economics of KV Cache

Publication Date: 14 March 2026
Author: Ailven LIU 刘翔
All Rights Reserved

Abstract

Large Language Model inference is often described in high-level product terms: prompt in, tokens out. That description is sufficient for a user experience discussion, but it is too coarse for systems engineering. Real inference stacks treat generation as a sequence of distinct computational regimes, each with a different bottleneck, resource profile, and optimisation strategy. The most useful decomposition is into embedding, prefill, and decode.

This article argues that the engineering significance of this decomposition becomes especially clear on edge and on-device platforms. In such environments, the decisive constraints are rarely abstract model quality alone; rather, they are the concrete economics of memory capacity, memory bandwidth, scheduling overhead, and cache persistence. The discussion therefore focuses on four tightly coupled issues: the asymmetry between prefill and decode, the role of the key-value (KV) cache as persistent model state, the practical value and limits of prefix reuse, and the architectural choices required to make these mechanisms viable on bandwidth-constrained hardware.

The central thesis is that KV cache is not merely a convenience layer bolted onto autoregressive decoding. It is the state-management substrate that makes LLM serving possible at all. Yet the same mechanism that saves computation also shifts the dominant systems problem from matrix arithmetic to data movement. On edge hardware, where bandwidth and thermal headroom are scarce, this shift becomes the defining challenge of inference design.

I. Introduction: Why Stage Decomposition Matters

The Transformer architecture does not explicitly divide inference into three named phases. From the perspective of pure model mathematics, a forward pass is simply a forward pass. Nevertheless, modern inference engines almost universally reason in terms of:

tokenisation -> embedding -> prefill -> decode

This decomposition survives because it corresponds to real differences in workload shape.

During embedding, the system maps discrete token IDs into dense vectors. During prefill, it processes the entire prompt in parallel and materialises the initial KV cache. During decode, it generates one token at a time while repeatedly consulting previously stored state. The point is not terminological neatness; the point is that the hardware bottleneck moves from stage to stage. A design that is ideal for prompt ingestion may be poor for token-by-token generation, and a layout that maximises raw throughput in the data centre may still fail on a mobile or embedded device because of memory traffic.

This is why stage decomposition has become a stable abstraction in production systems. It allows engineers to reason separately about:

On edge platforms these distinctions are not academic. They determine whether a model is merely runnable in a benchmark or genuinely deployable in a user-facing application.

II. A Systems Cost Model for LLM Inference

To discuss inference professionally, it helps to separate three kinds of cost: weight access, activation/attention compute, and KV-cache traffic.

Suppose a model has:

The KV cache size for a sequence of length T is roughly:

KV bytes ~= 2 x L x T x N_kv x D x b

The factor of 2 reflects keys and values. This expression is intentionally simple, but it captures the engineering point: cache growth is linear in sequence length and layers, and the constant factors are large enough to dominate memory budgets surprisingly quickly.

Now consider the two main runtime phases:

In asymptotic notation, prefill has the familiar quadratic dependence on prompt length because the attention matrix grows with the prompt. Decode, with KV cache reuse, avoids recomputing the full history; however, each step still requires reading prior KV state. This means the computational cost per step is modest relative to prefill, while the memory-access cost remains tied to context length.

That distinction is crucial. In other words, decode is cheaper in arithmetic than naive recomputation, but not cheap in bytes moved. The engineering bottleneck therefore shifts from FLOPs to bandwidth.

This is also the right place to introduce a practical serving distinction:

An inference stack that optimises only one of these metrics is incomplete. Production systems must usually trade between them.

III. Prefill as a Throughput-Oriented Compute Phase

Prefill is the stage in which the system absorbs the prompt and converts it into reusable model state. Conceptually, it is still a standard forward pass, but operationally it is the most throughput-friendly part of inference.

The reason is straightforward. During prefill, the model processes many prompt tokens together. This gives the runtime substantial parallel work:

For this reason, prefill is often described as compute-bound. That description is directionally correct, especially on server-class GPUs, but it is worth being more precise. Prefill is compute-favourable because it exhibits higher arithmetic intensity than decode. The machine can do a relatively large amount of arithmetic per byte loaded, and the runtime has enough parallel work to keep tensor cores or vector units occupied.

Yet even here, edge systems complicate the picture. On smaller accelerators, limited SRAM, reduced on-chip cache, and lower memory bandwidth can make prompt ingestion less cleanly compute-bound than it appears in data-centre discussions. A long prompt may still be expensive not only because the attention matrix grows, but also because the device cannot keep intermediate tensors and weights resident efficiently. Therefore, the practical statement is:

Prefill is the most compute-friendly phase, but not always purely compute-bound on edge hardware.

This nuance matters for implementation. Systems intended for constrained devices often rely on:

The by-product of prefill is equally important: the runtime materialises the KV cache for every processed token. This cache is not a secondary artifact. It is the persistent state that decode will depend on for the remainder of the session.

IV. Decode as a Stateful, Bandwidth-Dominated Pipeline

Decode is where the user sees tokens appear, but from a systems perspective it is also where inference becomes structurally awkward. The pipeline has little token-level parallelism because each step depends on the previous output. The runtime therefore loses the high-throughput geometry that made prefill efficient.

At each decode step, the model:

  1. embeds the newly chosen token
  2. computes fresh queries, keys, and values for that token
  3. reads previously stored KV state from all earlier positions
  4. produces the next-token logits
  5. appends the new token’s KV entries to the cache

The key observation is that only a tiny amount of new computation is being introduced per step, while a large volume of old state must still be consulted. That is why decode is typically memory-bandwidth dominated.

This statement is sometimes simplified into ‘decode is linear rather than quadratic’, but that formulation can be misleading for engineering work. What matters in practice is not just asymptotic complexity, but the ratio between arithmetic and bytes transferred. Decode often has poor arithmetic intensity: the accelerator performs relatively little compute while repeatedly streaming cache lines, global-memory pages, or unified-memory segments containing KV state.

On edge devices, this situation is even harsher for several reasons:

The result is that decode latency often degrades as the conversation grows, even if the model weights remain constant and the amount of new work per token appears small.

This is why decode should be understood not merely as ‘generation’ but as a stateful streaming pipeline. Its primary challenge is not deep mathematics; it is the repeated movement and management of persistent attention state.

V. KV Cache as the Persistent State Layer of Inference

In many high-level discussions, KV cache is introduced as an optimisation. That is true in a narrow sense, but the label understates its role. In serving systems, KV cache is better understood as the persistent state layer that bridges prefill and decode.

For every layer and every processed token, the runtime stores:

These tensors become the historical memory against which future queries attend. Once this viewpoint is adopted, several engineering issues become clearer.

1. Capacity is a first-class resource

KV cache competes with model weights, temporary workspaces, runtime bookkeeping, and sometimes the application itself for memory budget. On an edge device, where memory is often shared with the operating system and other workloads, cache capacity is not an abstract scaling concern. It is an operational limit.

2. Allocation strategy matters

If cache is allocated as large contiguous regions per request, fragmentation and reallocation become problematic. Long-running sessions, uneven output lengths, and dynamic batching can quickly produce allocator inefficiency.

3. Cache layout affects kernel efficiency

Even when two layouts store the same logical information, their physical arrangement influences coalesced memory access, cache locality, and page reuse. A poor layout can convert an already bandwidth-sensitive decode path into a pathological one.

4. Cache lifetime must be managed explicitly

Inference runtimes need policies for session eviction, truncation, reuse, and sharing. Without such policies, a system may achieve good single-request performance while collapsing under multi-session pressure.

For these reasons, serious inference systems usually adopt block-based or page-like cache organisation rather than naive contiguous tensors.

VI. Prefix Reuse, Block Tables, and the Difference Between Shared Storage and Shared Work

Many serving workloads contain repeated prompt structure. System instructions, safety scaffolding, tool descriptions, and conversation history often form a relatively stable prefix, while only the newest user turn acts as the suffix. This observation leads naturally to prefix reuse.

If the prefix is stable, the runtime can precompute and retain the prefix KV cache. Later requests then only need to process the suffix before entering decode. In abstract form:

prefill(prefix) -> persistent cache

prefill(suffix) -> request-local cache

decode

When the prefix is large and the suffix is small, this eliminates a substantial amount of redundant prefill work. In practice, however, making prefix reuse effective requires more than simply storing a tensor and pointing multiple requests at it.

Modern runtimes therefore use block tables or similar indirection mechanisms. Instead of forcing each request to own one contiguous KV region, the runtime composes a logical sequence out of blocks. A shared prefix can then be represented once and referenced by many requests, while each request appends its own suffix and generated tokens through separate blocks.

This yields several benefits:

However, prefix sharing introduces a subtle but extremely important distinction:

shared storage is not the same as shared compute

Even if the prefix KV exists only once in memory, many decode kernels still reread that prefix separately for each active request. The storage cost has been amortised, but the bandwidth cost may not have been. This is the hidden limit of naive prefix reuse.

That distinction is especially important for edge serving. A mobile or embedded device may have enough capacity to hold the shared prefix exactly once, yet still lack the bandwidth to scan it repeatedly for multiple decode streams. This is why prefix reuse is necessary but not sufficient. It solves redundant prefill computation, not automatically redundant decode traffic.

VII. Why Edge Inference Is Structurally Harder Than Server Inference

It is tempting to treat edge deployment as a smaller version of server deployment. That intuition is usually wrong. Edge inference is harder not just because the device is weaker, but because the shape of the constraints is different.

1. Memory bandwidth is disproportionately scarce

Server GPUs are designed to sustain very high-bandwidth tensor workloads. Edge devices are not. Once decode becomes bandwidth-limited, server-oriented assumptions break down quickly.

2. Memory is often shared and externally contested

An on-device runtime may share memory fabric with the CPU, display pipeline, camera stack, ISP, or other application subsystems. The inference engine is therefore not merely slower; it is less isolated.

3. Thermal limits are part of the serving model

Data-centre accelerators are engineered for sustained throughput. Phones, laptops, and embedded modules are engineered for mixed workloads and power envelopes. A decode loop that looks acceptable for ten seconds may degrade materially over longer sessions.

4. Weight placement and cache placement interact

On smaller devices, it is not enough to ask whether the model fits. One must ask whether weights, KV cache, scratch buffers, and runtime metadata can coexist while still leaving enough bandwidth headroom for generation. This is where quantised weights alone are often insufficient. Compressing the model without addressing cache growth may simply move the bottleneck rather than remove it.

5. Scheduling policy becomes product-critical

In a server environment, one may accept queueing and multiplexing to improve aggregate throughput. On-device, the user usually experiences one foreground interaction at a time, so responsiveness, jitter, and thermal stability matter more than peak batch throughput.

For these reasons, edge inference should not be understood as a truncated form of cloud inference. It is a distinct systems regime with different optimisation priorities.

VIII. Practical Optimisation Directions and Their Trade-Offs

Because the bottlenecks of edge inference are dominated by state size and state movement, the most valuable optimisations are those that either reduce KV footprint, reduce bytes read per step, or improve the usefulness of each memory transfer.

Technique Primary benefit What it does not solve
Weight quantisation Reduces model memory and weight bandwidth Does not stop KV cache from growing with context
KV-cache quantisation Reduces cache capacity and decode traffic May introduce quality or implementation complexity
Grouped-query / multi-query attention Shrinks KV representation by reducing KV-head multiplicity Does not remove long-context rereads entirely
Sliding-window attention Caps effective context length and stabilises decode cost Sacrifices access to older context unless combined with retrieval/summarisation
Prefix reuse Eliminates repeated prefill for shared prompt segments Does not automatically eliminate repeated decode scans
Paged/block allocation Improves memory utilisation and sharing Cannot by itself create missing bandwidth
Speculative decoding Can raise output throughput in some regimes On constrained devices, verifier overhead may erase gains

Several broader lessons follow from this table.

First, there is rarely a single decisive optimisation. Edge inference usually becomes viable through layered reductions across weights, cache size, scheduler overhead, and kernel inefficiency.

Second, techniques that appear equivalent in theory are not equivalent in implementation. For example, reducing logical context length, reducing physical KV precision, and changing the number of KV heads all affect different parts of the bottleneck stack.

Third, optimisations should be evaluated against the actual product objective. If the goal is low TTFT for a short-lived assistant query, chunked prefill and prompt reuse may dominate. If the goal is long interactive sessions on-device, cache management and decode bandwidth become much more important.

IX. A Better Mental Model for Future Work

The most productive way to think about inference on edge platforms is not ‘How do we run a Transformer on a smaller machine?’ but rather:

How do we manage persistent model state under strict bandwidth and thermal constraints while preserving useful output quality?

That framing shifts attention towards the right engineering questions:

These are not merely kernel-level questions. They cut across model architecture, runtime design, scheduler policy, and product constraints.

In that sense, the future of efficient edge inference is unlikely to come from a single miracle kernel. It will more likely come from co-design: models with cache-friendly attention structure, runtimes with more intelligent state sharing, and serving policies that explicitly acknowledge that memory bandwidth, not just parameter count, is the scarce currency.

X. Conclusion

The standard decomposition of LLM inference into embedding, prefill, and decode is not a pedagogical convenience. It is a systems abstraction that exposes three different operational regimes.

Embedding is relatively lightweight and interface-defining. Prefill is the throughput-oriented phase in which prompt tokens are transformed into reusable internal state. Decode is the stateful, low-parallelism phase in which the runtime repeatedly trades memory traffic for avoided recomputation.

KV cache sits at the centre of this design. It makes autoregressive generation practical, but it also redefines the optimisation problem. Once cache is introduced, efficient inference is no longer just a question of arithmetic throughput. It becomes a question of how much state must be stored, how often that state must be reread, how intelligently it can be shared, and whether the hardware can sustain those transfers under real deployment constraints.

For edge platforms, this is the decisive engineering reality. The governing resource is often not abstract model size alone, but the full economics of state: capacity, bandwidth, lifetime, locality, and thermal sustainability. Any serious discussion of professional LLM inference infrastructure therefore has to begin, and eventually end, with KV cache.