Publication Date: 11 September 2026
Author: Ailven LIU 刘翔
All Rights Reserved
Abstract
High-performance Transformer inference has historically relied heavily on expert-written CUDA kernels, operator fusion, and bespoke implementations tuned to particular model shapes. Over the past few years, however, the technical environment has changed materially. Low-level libraries such as cuBLASLt, CUTLASS/CuTe and cuDNN have continued to broaden their coverage; compilers such as TVM Relax and TorchInductor can now perform graph-level pattern matching, backend dispatch, autotuning and operator fusion; and lower-level DSLs such as CuTe DSL and Triton have exposed to compilers forms of hardware specialisation that previously required substantial CUDA C++ template engineering. By 2026, PyTorch Inductor could place CuTeDSL/NVGEMM, ATen and Triton within a single autotuning process, while FlashAttention-4 itself was implemented in CuTe DSL. This sharpens a longstanding question: once general-purpose compilers and high-performance operator libraries have become sufficiently capable, how much value remains in developing specialised HPC kernels for Transformer inference?
There is no meaningful universal answer such as ‘5%’, ‘20%’ or ‘2×’. The reason is that the remaining performance headroom has progressively migrated away from isolated arithmetic kernels towards data layout, quantisation representation, dynamic scheduling, KV-cache access, sparsity, cross-operator dataflow, and communication–computation co-ordination. For standard dense GEMMs with large, regular shapes and common numerical formats, the benefit of reimplementing the main loop from a strong baseline is narrowing. By contrast, in low-effective-batch inference, dynamic attention, MoE, model-specific low-precision formats, cross-operator fusion and multi-GPU execution, specialisation can still alter the amount of data moved, the synchronisation structure and the critical path itself; in such cases the benefit may be far larger than what would traditionally be described as ‘operator optimisation’.
This article re-examines the issue from five perspectives: compiler capability boundaries, hardware utilisation, Roofline-style reasoning, Amdahl’s law, and service-level SLOs. Its central argument is that specialised HPC optimisation has not lost its value, but that its economic value is shifting away from ‘owning more private kernels’ towards ‘identifying workload structure that the general stack does not yet express well, and specialising that structure at the lowest sustainable maintenance cost’.
Keywords: Transformer inference; AI compiler; CUTLASS; CuTe; cuBLASLt; operator fusion; attention; MoE; kernel specialisation; GPU performance engineering
Software status and published experimental results discussed in this article are current to September 2026. Hardware, models, baselines and benchmarking methodology differ across the cited work. Performance figures are therefore used only to illustrate the nature of remaining optimisation headroom; they are not averaged across papers and should not be extrapolated directly to unrelated workloads.
1. Reframing the Problem: From Operator Specialisation to Workload Specialisation
In this article, a specialised HPC operator is not restricted to hand-written CUDA C++. Any implementation may be regarded as specialised if it is tailored to a particular hardware architecture, model structure, shape distribution, numerical format or runtime state, and if its performance depends materially on those assumptions. Such an implementation may be written in CUDA, CuTe DSL, Triton or another low-level DSL; it may also extend beyond a conventional operator into a persistent kernel, mega-kernel, or communication–computation fused execution unit.
Likewise, a ‘modern AI compiler’ can no longer be understood merely as a graph compiler that performs constant folding and pointwise fusion. TVM Relax’s Dataflow Pattern Language can identify subgraphs, fuse them, or dispatch them to external backends such as CUTLASS, cuBLAS and cuDNN, while allowing projects to define their own rewrite patterns. TorchInductor, meanwhile, can place Triton, ATen/cuBLAS, CUTLASS and CuTeDSL/NVGEMM into practical candidate-generation and autotuning flows.[1–4]
CUTLASS itself is also changing in character. CUTLASS 4.8.0, released in August 2026, continued the expansion of CuTe DSL and added support for new FP8/FP4 Tensor Core capabilities on Rubin, alongside enlarged TMEM and shared-memory capabilities. At the same time, the CUTLASS Operator API wraps DSL kernels as discoverable, filterable and instantiable operators, and can use nvMatmulHeuristics to rank large sets of valid candidates.[1][5]
The old framing of ‘compiler versus hand-written kernel’ is therefore no longer accurate. A more representative stack now looks like:
$$ \text{graph compiler} \rightarrow \text{pattern / cost model} \rightarrow \text{kernel family} \rightarrow \text{autotuning} \rightarrow \text{CuTe/Triton/CUDA implementation}. $$
Tile selection, pipeline design, epilogue construction and architecture-specific specialisation – once largely embedded in the tacit knowledge of individual CUDA experts – are increasingly becoming parts of a parameter space that compilers can search and invoke.
This does not eliminate low-level optimisation. It changes who owns it.
From this perspective, the question of whether specialised HPC work remains worthwhile can no longer be reduced to:
Can a particular kernel be made faster than the library implementation?
The more important question is now:
Once the current compiler stack and public kernel libraries have already captured most conventional optimisations, which structures in the real workload remain poorly represented by the public abstractions?
Those structures may concern shape, but they may equally concern layout, precision, dynamic behaviour, sparsity, communication topology, or the execution order of the model as a whole.
2. The Strong Baseline: What Modern Compilers and High-Performance Libraries Already Solve
2.1 The Value of Customisation Depends First on the Baseline
Before deciding whether in-house optimisation is worthwhile, the baseline must be defined correctly.
A common but misleading benchmark compares a hand-tuned kernel against an untuned framework default. The observed performance gap may be real, yet it does not answer whether maintaining a private implementation is justified. A significant part of that gap may disappear simply by enabling the correct library, increasing workspace, caching algorithm-selection results, enabling fusion, or adopting an existing high-performance backend.
A more useful hierarchy is:
$$ \begin{aligned} B_0 &= \text{current production implementation}, \\ B_1 &= \text{best deployable public implementation under production constraints}, \\ B_2 &= \text{implementation with private specialisation added on top of }B_1. \end{aligned} $$
The quantity that matters for an investment decision is $\Delta_{\text{custom}} = B_2-B_1,$ not $B_2-B_0.$
‘Best public implementation’ does not mean adopting experimental software without constraint. It remains bounded by CUDA and driver versions, AOT requirements, model-quality validation cycles, licensing, deployment stability and release cadence. However, if a team has merely failed to enable an already available high-performance backend, it is misleading to attribute the resulting gap to the long-term value of proprietary HPC capability.
2.2 Public Libraries Are No Longer Just Collections of Fixed Kernels
cuBLASLt is a useful example. It is not a single GEMM implementation, but an algorithm space plus a selection interface. Current documentation explicitly describes heuristic selection based on matrix shape and GPU configuration, and notes that a single heuristic query may itself consume tens of microseconds of host time; for stable workloads, results should therefore be cached and reused.[6]
TorchInductor’s CuTeDSL backend follows a similar principle. It can enumerate a large set of valid kernel configurations, use nvMatmulHeuristics to reduce that set, and then benchmark a small number of top-ranked candidates against ATen and Triton implementations on real hardware. This already introduces three layers of optimisation:
- kernel family;
- performance model;
- hardware autotuning.[3][5]
For standard GEMMs, it is therefore still possible that a library or compiler fails to select the absolute fastest implementation, but it is no longer reasonable to assume this as the default case.
More importantly, ‘public capability’ no longer means only vendor libraries. FlashInfer, FlashAttention and DeepGEMM began as highly specialised systems, but once such implementations are integrated into vLLM, SGLang, TensorRT-LLM or PyTorch, they should be treated as part of the public baseline rather than counted as value that an internal HPC team can newly create.
The first principle of modern inference optimisation is therefore not to write a kernel quickly, but to establish accurately where $B_1$ actually lies.
3. Remaining Headroom in Dense Transformers: GEMM, Fusion and Low Batch
3.1 Standard GEMM: The Headroom Has Not Vanished, but the Competition Is Strong
Transformer linear layers reduce largely to GEMM. For common dtypes, standard layouts and regular large matrices, this is also one of the most mature areas of vendor-library optimisation.
CUDA-L2, published in 2025, provides a useful counterexample to the claim that ‘mature GEMM leaves no room for improvement’. On an A100 over 1,000 HGEMM configurations, the paper reports an average 11.4% improvement in offline mode over cuBLASLt results selected by autotuning from as many as 100 candidates; in a server mode intended to simulate intermittent inference calls, it reports 15.9%.[7]
This shows that the statement ‘mature GEMM has no remaining headroom’ is false. It does not, however, imply that ‘custom GEMM typically offers 10–15%’.
There are at least four important constraints:
- the experiments target A100;
- they cover HGEMM only;
- the shape set is defined by the paper;
- the results are kernel-level GEMM benchmarks, not end-to-end Transformer gains.
More fundamentally, local GEMM gains are constrained by the hardware resource that is already acting as the bottleneck.
If a GEMM already achieves a fraction $u$ of the sustainable limit of the relevant bottleneck resource, and a new implementation neither reduces arithmetic work nor data movement nor changes the limiting resource, then the theoretical speed-up available purely from higher utilisation is bounded by $S_{\max}\leq\frac{1}{u}.$
For example, if $u=0.9$, the upper bound is only about $1.11\times.$
The 90% figure here is purely illustrative; it is not a measured claim about cuBLASLt or CUTLASS.
Optimisations that exceed this bound generally do so by changing the problem itself rather than by merely executing the same GEMM more efficiently:
- reducing numerical precision;
- changing layout;
- folding epilogues into GEMM;
- eliminating intermediate tensors;
- keeping data on-chip for subsequent computation;
- adopting a quantisation format that reduces HBM traffic.
This is precisely why the centre of gravity in GEMM optimisation has shifted.
3.2 The Operations Around GEMM Are Becoming the New Optimisation Centre
In a highly optimised Transformer block, matrix multiplication may already be close to hardware limits, while normalisation, activation, residual updates, scaling, quantisation and reduction continue to write intermediate values to global memory only for the next kernel to read them again.
CODA formalised this observation in 2026. It fixes a high-performance GEMM main loop and rewrites normalisation, activation and reduction as GEMM epilogue programs, allowing them to execute while the output tile is still resident on-chip.[8]
This reveals a more important fact than the familiar statement that ‘fusion reduces kernel launches’:
$\text{the main value of fusion} \neq \text{fewer API calls},$ and is better understood as fewer ‘HBM round trips + longer on-chip data lifetime + elimination of unnecessary materialisation’.
The Transformer block can therefore become faster even if the GEMM itself does not.
These optimisations are also being absorbed into compilers. The NVGEMM/CuTeDSL path in PyTorch 2.14 supports unified autotuning across epilogue fusion, scaled GEMM, NVFP4 GEMM and grouped-reduction epilogues.[3]
This suggests an important engineering trend: successful ‘specialised kernel techniques’ do not necessarily remain private implementations for long. Once the pattern is sufficiently general, it tends to be re-expressed as a template, epilogue DSL, pattern rewrite or backend capability, and eventually becomes part of the next public baseline.
3.3 Why Low-Batch Decode Still Encourages Specialisation
Autoregressive LLM decoding has different computational characteristics from large-batch training or prefill.
Consider a linear layer $Y_{M\times N}=X_{M\times K}W_{K\times N},$ it performs approximately $2MNK$ floating-point operations. If weight reads dominate HBM traffic during low-batch decode, and each weight occupies $q$ bytes, a full weight scan costs roughly $qKN$ bytes.
Ignoring other traffic, the arithmetic intensity is then approximately $I \approx \frac{2MNK}{qKN} = \frac{2M}{q}$ FLOP/byte.
For BF16 or FP16 weights, $q=2$, giving $I\approx M.$
At one token or small effective batch, Tensor Core peak FLOPS may therefore be less relevant than weight bandwidth, kernel launch, synchronisation and other memory-bound operations.
This is why batch-1 inference deserves separate treatment. FlashFormer explicitly explores whole-model kernels for low-batch Transformer inference, motivated by the rising relative importance of memory bandwidth and launch overhead in this regime.[9]
That conclusion should not, however, be applied mechanically to every Transformer. An application-level batch size of one does not imply $M=1$ in the underlying GEMM. Vision Transformers, video Transformers and BEV Transformers may still have large $M$ because spatial or temporal token dimensions remain substantial even at a frame batch of one.
The relevant object of analysis is therefore the actual matrix shape and dataflow, not the application’s batch parameter in isolation.
4. Attention: Specialised Optimisation Remains Active, but Is Rapidly Becoming Platform Capability
4.1 New Hardware Continues to Reopen Kernel-Level Headroom
Attention is perhaps the clearest domain in which to observe these changes.
FlashAttention-4 redesigns the pipeline around Blackwell-specific hardware characteristics. Tensor Core throughput on Blackwell has grown faster than several other resources, so the bottleneck structure differs materially from Hopper. FA4 co-designs asynchronous MMA, tile sizes, softmax rescaling, Tensor Memory and 2-CTA MMA accordingly.
In the authors’ reported B200 BF16 experiments, FA4 reaches up to 1.3× the performance of cuDNN 9.13 and up to 2.7× that of the Triton implementation used for comparison.[10]
The result has two implications.
First, even a mature vendor library may leave meaningful optimisation headroom when a new GPU architecture alters the balance between compute, memory and auxiliary units.
Second, this class of ‘expert kernel’ no longer necessarily requires extensive traditional CUDA C++ template engineering. FlashAttention-4 is implemented entirely in CuTe DSL, and its authors also report significantly faster compilation than the conventional C++ template path.[10]
The effect of the modern compiler ecosystem on expert HPC work is therefore not to eliminate expert-level optimisation, but to reduce the cost of expressing and integrating it.
4.2 Non-Standard Attention Is Moving from ‘Rewrite the Kernel’ to ‘Express the Semantics’
PyTorch FlexAttention can now translate user-defined score and mask modifications into CuTe DSL and instantiate FlashAttention-4 through JIT compilation. This allows variants such as ALiBi, sliding-window attention, document masking and soft-capping to reuse the same backend.[11]
A class of attention variants that once required maintaining a separate CUDA implementation may therefore now require only an expression of the semantic difference.
Specialisation has not disappeared. It has moved from reimplementing the attention algorithm towards expressing the model semantics in a form that a high-performance template can consume, and ensuring that the compiler recognises those semantics.
This change is important because it reduces the amount of private implementation without necessarily reducing performance.
4.3 Dynamic KV-Cache and Irregular Workloads Are Still Not Ordinary GEMM Problems
In production LLM serving, the difficult part of attention is often not $QK^\top$ itself.
Requests have different sequence lengths; KV-cache may be paged or block-sparse; GQA and MQA alter key/value sharing; requests enter and leave batches dynamically; and long contexts create strongly uneven amounts of work between queries.
FlashInfer is designed around precisely these concerns. It addresses KV-cache representation, JIT attention customisation, dynamic load balancing and the static execution constraints imposed by CUDA Graphs. In the reported experiments, relative to the compiler backend used for comparison, it achieves:
- 29–69% lower inter-token latency;
- 28–30% lower latency in long-context inference;
- 13–17% speed-up in parallel-generation serving.[12]
These figures should not be interpreted as the expected return from ‘writing a custom attention kernel’.
FlashInfer itself has already been integrated into public serving stacks including vLLM, SGLang and MLC-Engine.[12]
Once such an optimisation enters a public backend, it belongs in the new $B_1$. It should no longer be counted as expected value from newly developing a private implementation.
This is one of the most common valuation errors in high-performance inference: taking the gain reported by an older paper against an older compiler baseline and treating it as the business case for rebuilding the same class of optimisation today.
4.4 Automatic Compilation of Complex Attention Is Also Advancing
If the CuTeDSL backend demonstrates how expert templates can enter a compiler, Nautilus points in a different direction: automatically discovering complex GPU transformations from a higher-level mathematical description.
In 2026, Nautilus reported that its auto-scheduler could start from an algebraic description of attention and discover an implementation structure resembling FlashAttention-3. Across multiple models and configurations on GH200 and RTX 5090, the authors report up to 23% and 42% higher throughput, respectively, than the state-of-the-art compiler used for comparison, while matching or exceeding hand-written cuDNN kernels in many long-sequence cases.[13]
This does not imply that manually engineered kernels are about to disappear. Nautilus remains a research system with explicit bounds on its search space, hardware coverage and operator domain.
It does, however, alter the long-term judgement. Complex fusion, reduction reordering and attention tiling cannot safely be treated as domains that compilers will never reach.
An investment in private kernel engineering therefore carries an additional risk: the half-life of any technical advantage is shrinking.
The more successfully an optimisation can be expressed as a stable pattern, template or scheduler rule, the more likely it is to migrate quickly into the public compiler stack.
5. What General-Purpose Libraries Struggle to Flatten: Representation, Sparsity and MoE
5.1 In Low-Precision Inference, the Hard Problem Is the Entire Data Path
Low-precision inference is a representative case in which model-specific specialisation can retain substantial value.
If only the quantised GEMM kernel is benchmarked, it is easy to report very high Tensor Core utilisation. The complete data path, however, also includes:
- activation quantisation;
- scale computation;
- scale layout;
- transpose;
- packing;
- output conversion.
DeepGEMM’s interface documentation makes this explicit. Certain FP8 paths require scaling factors in a transposed layout compatible with TMA; input transposition and FP8 casting must either be performed separately by the caller or fused into a preceding kernel. The simple PyTorch utility functions supplied by the project are not intended as peak-performance implementations.[14]
A meaningful performance model is therefore not $T=T_{\text{GEMM}},$ but $T=T_{\text{quant}}+T_{\text{layout}}+T_{\text{scale}}+T_{\text{GEMM}}+T_{\text{postprocess}}.$
A ‘faster GEMM’ that requires an additional transpose may lose its entire kernel-level advantage once integrated into the model.
Conversely, this is exactly why model-specific specialisation remains valuable:
- fixed weights can be pre-arranged offline;
- the producer can emit the scale layout required by the downstream GEMM;
- quantisation can be fused into normalisation or another upstream producer.
In such cases, the gain does not come from executing the same instruction sequence faster. It comes from not executing the original transformation at all.
This kind of optimisation is harder for a general-purpose library to cover completely because it depends on the data contract between neighbouring model stages.
5.2 Sparse Attention Derives Its Value Primarily from Doing Less Work
FlashMLA provides a concrete example for DeepSeek-style MLA and token-level sparse attention. Its public implementation includes dense and sparse prefill, dense and sparse decode, and FP8 KV-cache support. Its sparse FP8 KV-cache also uses a model-specific layout in which the NoPE component is quantised while the RoPE component retains higher precision where required.[15]
The distinction between such an implementation and ordinary FlashAttention is no longer merely a question of which tile is better.
It couples model algorithm semantics, sparse indexing, KV-cache representation, quantisation format and hardware access patterns.
If sparse attention accesses only a subset of the full KV-cache, the potential gain comes first from reading less data and performing less computation, not from achieving a higher peak FLOPS rate for the same workload.
Sparsity is not free, however. Index accesses, gather operations, irregular memory transactions, load imbalance and metadata processing all introduce new costs. Whether a specialised implementation is worthwhile can therefore only be decided from the actual sparsity pattern and sequence-length distribution.
A reduction in theoretical FLOPs is not, by itself, a performance prediction.
5.3 MoE Pushes the Optimisation Boundary Beyond the Single Kernel
Mixture-of-Experts appears GEMM-heavy, but its actual critical path includes:
- token routing;
- dispatch;
- expert grouping;
- grouped GEMM;
- activation;
- combine;
- multi-GPU expert-parallel communication.
DeepGEMM’s current Mega MoE path no longer treats these stages as isolated operators. It organises EP dispatch, the first FP8×FP4 linear layer, SwiGLU, the second linear layer and EP combine into a co-operative mega-kernel, while attempting to overlap NVLink communication with Tensor Core computation.[14]
This introduces a problem that a conventional kernel benchmark cannot capture well:
The fastest GEMM does not necessarily produce the fastest model.
Suppose one GEMM occupies all SMs and therefore reaches the highest standalone TFLOPS, but in doing so prevents communication from progressing. Another implementation uses fewer SMs and appears slower in isolation, yet permits communication and computation to overlap. The second implementation may finish the entire MoE layer sooner.
KTransformers, which targets CPU/GPU hybrid MoE inference, illustrates the point even more clearly: overall performance may be constrained by CPU kernels, CPU–GPU synchronisation and asynchronous scheduling rather than by GPU GEMM peak throughput.
In its SOSP 2025 evaluation of large hybrid MoE scenarios, the paper reports, relative to the compared systems:
- 4.62–19.74× prefill speed-up;
- 1.25–4.09× decode speed-up.[16]
These large factors should not be compared directly with GPU-only GEMM tuning. They arise at a different optimisation layer: the system changes data placement and heterogeneous execution.
The broader conclusion is nevertheless clear:
Once individual kernels are already fast, much of the remaining performance headroom shifts into the relationships between kernels.
6. From Operator Optimisation to Execution-Model Optimisation
6.1 Persistent Kernels and Mega-Kernels
Conventional deep-learning runtimes typically follow a kernel-per-operator execution model $K_1\rightarrow K_2\rightarrow K_3\rightarrow\cdots$, each operator has its own grid, synchronisation boundary and global state.
CUDA Graph can substantially reduce host launch overhead, but it does not automatically turn the thread blocks of multiple operators into a single, unified on-device scheduling system.
Mirage Persistent Kernel (MPK) attempts to change this execution model directly. Its compiler transforms a tensor program into an SM-level task graph and schedules that graph at fine granularity inside a persistent mega-kernel. This enables:
- cross-operator software pipelining;
- computation overlap;
- multi-GPU communication overlap;
- finer-grained device-side scheduling.
The paper reports up to roughly 1.7× lower end-to-end latency in its LLM inference experiments.[17]
The implication for the value of specialised HPC work is straightforward.
If ‘specialised operator’ is defined narrowly as a single CUDA function, its remaining value is indeed shrinking.
If it is understood more broadly as a hardware-level execution program specialised to a fixed model graph, there remains substantial research and engineering headroom.
The latter, however, is better described as compiler/runtime co-design than as a traditional operator library.
6.2 Public Inference Frameworks Are Increasingly Specialised-Kernel Integrators
TensorRT-LLM currently exposes specialised paths including:
- GPT attention plugins;
- cuBLASLt GEMM plugins;
- FP8 row-wise GEMM;
- low-latency GEMM;
- GEMM+SwiGLU;
- GEMM+AllReduce;
- MoE plugins;
- norm–quant fusion;
- paged KV-cache.[18]
The boundary between a ‘general framework’ and a ‘specialised kernel stack’ is therefore increasingly blurred.
A modern inference framework derives much of its value from:
$$ \text{recognising the workload} \rightarrow \text{selecting a specialised implementation} \rightarrow \text{composing specialised implementations}. $$
For deployment teams, the relevant question is not whether custom kernels exist, but who owns their maintenance.
If NVIDIA, PyTorch, FlashInfer or DeepSeek already maintain an implementation for the same model pattern at comparable performance, the economic case for a private copy weakens rapidly.
Conversely, when a production model uses a distinctive attention mechanism, quantisation format, fixed-shape regime or unusually strict latency envelope, public libraries may have little incentive to optimise precisely for that narrow workload. Private specialisation becomes more defensible.
6.3 A Custom Operator Can Also Become a New Compiler Boundary
Compiler visibility introduces another cost that is easy to overlook.
In PyTorch, torch.library.custom_op() can expose a user-defined operator to torch.compile while keeping its implementation opaque: the compiler understands the operator schema, but does not inspect or optimise through the implementation. By contrast, torch.library.triton_op() can expose the Triton implementation to the compiler for further analysis.[19]
This creates a perfectly plausible situation in which $T_{\text{custom kernel}} < T_{\text{original kernel}},$ yet $T_{\text{whole graph with custom}} > T_{\text{whole graph before custom}}.$
Possible causes include:
- loss of cross-boundary fusion;
- additional temporary allocations;
- disruption of layout propagation;
- insertion of new synchronisation points.
A 20% improvement in a kernel microbenchmark is therefore not an adequate deployment criterion.
The correct unit of evaluation is the net effect of the custom code on the complete graph.
7. Calculating the Real Engineering Return of Specialised HPC Optimisation
7.1 Published Benchmarks Cannot Be Turned Directly into an Expected Return
By 2026, published results span a very wide range:
| Work | Representative author-reported result | What it actually demonstrates |
|---|---|---|
| CUDA-L2 | A100 HGEMM, average +11.4% over cuBLASLt autotuning in offline mode | Mature GEMM still contains configuration-level search space |
| FlashAttention4 | Up to 1.3× over cuDNN 9.13 on B200 BF16 | New architectures can reopen kernel-level optimisation headroom |
| FlashInfer | 29–69% lower ITL in specific serving benchmarks | KV-cache, dynamic scheduling and the attention runtime retain system-level headroom |
| Nautilus | Up to +23% on GH200 and +42% on RTX 5090 | Compilers are entering complex tiled-kernel optimisation domains once dominated by manual expertise |
| MPK | Up to roughly 1.7× lower LLM inference latency | Cross-operator scheduling and persistent execution can break through the operator-by-operator model |
| KTransformers | 1.25–4.09× decode speed-up in specific hybrid MoE settings | Changing heterogeneous execution and data placement can matter far more than single-GPU kernel tuning |
The relevant sources are [7][10][12][13][16][17].
These values should not be averaged.
They optimise different layers and use different baselines. Some concern a single kernel, some an attention engine, some the entire GPU execution model, and some CPU/GPU placement.
The only robust conclusion is:
Remaining performance headroom is not a constant. It is concentrated in workload structures that the current public stack does not yet express well.
7.2 Use Amdahl’s Law Before Starting the Project
Suppose a candidate optimisation region accounts for a fraction $p$ of the original critical path, the region becomes $s$ times faster, and additional conversion, dispatch or synchronisation introduces overhead $\epsilon$. The total speed-up is approximately $S=\frac{1}{(1-p)+p/s+\epsilon}.$ If a kernel accounts for 10% of total latency, then even an infinitely fast replacement is bounded by $S_{\max} = \frac{1}{0.9} \approx1.111.$
The total model latency can fall by at most 10%.
If the kernel is merely doubled in speed $S= \frac{1}{0.9+0.05} \approx1.053,$ the whole model is only about 5% faster.
This calculation should be performed before kernel development begins, not after.
In asynchronous execution, $p$ must represent contribution to the actual critical path. It cannot be obtained by naively summing all CUDA-kernel durations in a profiler. Two kernels that each execute for 100 $\mu s$ in parallel do not create a 200 $\mu s$ critical path.
This is one reason simple profiler rankings are particularly misleading in MoE, multi-stream and communication-overlap workloads.
7.3 Serving Systems Ultimately Need SLO Goodput, Not TFLOPS
LLM serving has several distinct performance objectives:
- TTFT: time from request submission to first token;
- TPOT: average time per output token after the first;
- ITL: interval between streamed output tokens;
- end-to-end latency;
- request or token goodput under latency constraints.
vLLM’s current benchmarking tools distinguish TTFT, TPOT and ITL explicitly, and can calculate goodput subject to TTFT, TPOT and end-to-end latency SLOs,[20] therefore $\text{kernel throughput} \neq \text{model throughput} \neq \text{SLO goodput}.$ A kernel may increase peak throughput but require more workspace or higher occupancy, degrading concurrent scheduling.
Another optimisation may barely alter single-request latency while reducing KV-cache or temporary memory enough to support more concurrent requests, thereby increasing SLO goodput.
For production deployment, the latter is often economically more important than a higher microbenchmark TFLOPS figure.
7.4 Where Custom-Optimisation Gains Are Most Commonly Overstated
Weak baseline
Default PyTorch eager execution is not the same baseline as fully autotuned Inductor/CuTeDSL. Likewise, a single cuBLAS heuristic result is not the same as profiling a broad candidate set.
If the gap comes from failing to enable an existing backend, it is an integration gain, not a custom-kernel gain.
Format conversion excluded from the timed region
An FP8 GEMM benchmark that excludes quantisation, scale layout or transpose will systematically overstate model-level benefit.
Benchmark shapes differ from production shape distributions
Real serving workloads are distributions of GEMM shapes $(x_i,w_i).$ The correct weighted execution time is $T=\sum_i w_i t_i,$
not the arithmetic mean of per-shape speed-up ratios.
Only the mean is considered
In dynamic batching, MoE routing and long-context workloads, P95 and P99 frequently expose load imbalance that the mean conceals.
Numerical semantics are ignored
Performance gains from FP8, FP4, approximate softmax or altered accumulation order must be compared under equivalent model-quality constraints.
A performance difference between numerical formats cannot be attributed entirely to kernel engineering.
The maintenance half-life is ignored
An architecture-specific trick that works extremely well on Blackwell may need redesign on the next GPU generation. Conversely, it may be absorbed quickly into CUTLASS, cuDNN or Inductor.
A more realistic engineering return model is therefore:
$$ROI= \frac{ \int_0^{T} V(t)\Delta P(t)\,dt}{C_{\text{dev}} + C_{\text{validation}} + C_{\text{maintenance}}},$$
where $\Delta P(t)$ may decline as upstream implementations catch up.
8. Engineering Boundaries: What Is Losing Value, and What Still Deserves Investment
8.1 Work That Is Losing Investment Value
Once a strong public baseline exists, several categories are increasingly difficult to justify as long-lived proprietary advantages.
Reimplementing Standard Large BF16/FP16 GEMMs
If the shape, layout and epilogue all lie within the mainstream coverage of mature libraries, the competitor is now cuBLASLt, CUTLASS/CuTe plus an autotuning system – not a naïve CUDA implementation.
Isolated Hand-Written Optimisation of Common Pointwise Operators
These operations are usually better absorbed by compiler fusion. Introducing an additional operator boundary may even reduce whole-graph optimisation potential.
Standard MHA/GQA Already Fully Covered by Public Attention Backends
If the target semantics can already be expressed efficiently by FlashAttention, FlexAttention, cuDNN, FlashInfer or TensorRT-LLM, maintaining another complete implementation is difficult to justify unless its performance advantage is both material and persistent.
Tile Tuning for a Single Benchmark Shape
If the gain depends on an extremely narrow shape regime while production traffic spans a broader distribution, a local peak is unlikely to translate into system-level value.
The key point is not that these problems cannot be optimised, but that:
the advantage is increasingly difficult to sustain.
8.2 Areas Where Strong Specialised HPC Capability Still Matters
Model-Specific Data Layouts
Where a producer and consumer have a fixed relationship, intermediate representations can be designed directly to eliminate transpose, packing and layout conversion.
New or Irregular Attention
This includes sparse attention, MLA, unusual KV-cache formats, non-standard positional transformations, dynamic top-k patterns and irregular page access.
Low-Batch, Strict-Latency Inference
Where launch cost, HBM traffic and small-GEMM efficiency dominate latency, whole-block or whole-model specialisation retains substantial potential.
MoE
Performance depends simultaneously on token routing, grouped GEMM, load balancing, expert-parallel communication and resource contention. Optimising GEMM alone is rarely sufficient.
Low-Precision Execution Pipelines
The important object is not an isolated FP8 or FP4 GEMM, but the complete quantise–layout–GEMM–dequantise/epilogue path.
Cross-Operator and Cross-GPU Fusion
When the objective is to remove synchronisation, overlap communication or construct persistent execution, the traditional operator abstraction itself may become the limiting factor.
These problems share a common property that the specialised implementation changes the problem being executed, rather than merely executing the same problem faster.
The latter is increasingly easy for the public compiler stack to catch up with. The former still requires joint reasoning about model semantics, runtime behaviour and hardware.
8.3 A More Rational Layering of Performance Engineering
In a modern AI compiler stack, performance engineering can be organised into four layers of increasing cost.
Layer 1: Use the Public Stack Correctly
This includes correct backend dispatch, autotuning, workspace settings, CUDA Graphs, existing fused operators, numerical precision and layout.
There is little reason to create private kernels at this layer.
Layer 2: Specialise the Graph and Dataflow
This includes pattern rewrites, layout propagation, offline weight transformation, producer–consumer fusion and better composition of existing high-performance kernels.
This layer often has a particularly attractive maintenance-to-benefit ratio.
Layer 3: Develop Local Specialised Kernels
Only where profiling and modelling reveal a genuine gap should specialised CuTe, Triton or CUDA implementations be added for attention, quantisation, small GEMM, sparse or grouped workloads, with a public fallback retained.
Layer 4: Change the Execution Model
Persistent kernels, mega-kernels, communication–computation fusion, SM partitioning and heterogeneous scheduling belong here.
This layer may offer the largest system-level gains, but it also carries the highest verification cost and strongest coupling to hardware and runtime details.
A mature team does not need to begin at Layer 4 in order to demonstrate technical sophistication.
A healthier process is to progress from Layer 1 downwards until the expected marginal gain no longer justifies the maintenance burden.
8.4 A Quantitative Answer to ‘How Much Value Is Left?’
If the entire discussion must be compressed into an engineering judgement, the following is defensible.
For standard dense Transformers with common shapes and dtypes, already running on a current, high-quality autotuned backend, it is no longer reasonable to assume that rewriting ordinary GEMMs by hand will yield large double-digit end-to-end gains. Published work shows kernel-level improvements above 10% in specific cases, but after Amdahl scaling, the model-level impact depends entirely on the share of the critical path occupied by that region.[7]
For attention and other complex single operators, new algorithms, GPU architectures or non-standard semantics can still produce differences of tens of percentage points at kernel level. FlashAttention-4 demonstrates that the headroom is real on Blackwell; the evolution of FlexAttention shows that a substantial portion of such specialisation can also be absorbed into general interfaces over time.[10][11]
For dynamic KV-cache access, low batch, sparse execution, specialised quantisation formats and irregular MoE, the remaining headroom often comes not from arithmetic micro-optimisation but from reducing data movement and unnecessary work. In such cases, double-digit end-to-end improvements remain entirely plausible.[12][14][15]
For cross-operator, cross-communication-boundary or execution-model-changing optimisation, published research still reports system-level gaps approaching 1.5× or more. At that point, however, the work is no longer well described as ‘writing a faster operator’. It is compiler/runtime/hardware co-design.[16][17]
What is declining, therefore, is not HPC performance engineering itself, but this development model:
maintaining a long-lived private fork of a standard operator in order to remain a few percentage points ahead of a mature public implementation.
What is becoming more valuable is a different capability:
identifying model structure, dataflow and execution constraints that the current compiler stack does not yet express well, and converting those structures into less data movement, less synchronisation, less redundant computation, or better communication–computation overlap.
The long-term trend can therefore be described more accurately as $\text{micro-kernel specialisation}\;\longrightarrow\;\text{workload specialisation}$
The capability worth accumulating is no longer the ability to maintain as many private CUDA kernels as possible, but the ability to construct a complete performance-reasoning chain:
workload characterisation
-> roofline / critical-path analysis
-> strong public baseline
-> specialisation hypothesis
-> kernel or dataflow implementation
-> end-to-end SLO verification
Within this chain, CuTe, CUTLASS, cuBLASLt and the compiler are not competitors to specialised performance engineering. They are its infrastructure.
The mistake to avoid is equating being able to write a faster kernel with the system ought to own a private kernel. The difference between those two statements is where the real value of modern Transformer inference engineering now lies.
References
[1] NVIDIA. CUTLASS Documentation: CUTLASS 4.8.0 Changelog [EB/OL]. 2026-08-25.
[2] Apache TVM. Dataflow Pattern Language (DPL); Operator Fusion [EB/OL]. Apache TVM Documentation.
[3] PATEL N, LAZOS M, GUESSOUS D, et al. Generating State-of-the-Art GEMMs with TorchInductor’s CuteDSL Backend [EB/OL]. PyTorch Blog, 2026-04-07.
[4] PyTorch Foundation. PyTorch 2.14 Release Blog [EB/OL]. 2026-09-02.
[5] NVIDIA. NVIDIA Matmul Heuristics; CUTLASS Operator API: Heuristics [EB/OL]. NVIDIA Documentation, 2026.
[6] NVIDIA. cuBLAS 13.4 Documentation: cuBLASLt Matmul and Heuristics [EB/OL]. CUDA Toolkit Documentation, 2026.
[7] SU S, SUN X, LI X, et al. CUDA-L2: Surpassing cuBLAS Performance for Matrix Multiplication through Reinforcement Learning [J/OL]. arXiv:2512.02551, 2025.
[8] GUO H, ZHANG J, MENON A, et al. CODA: Rewriting Transformer Blocks as GEMM-Epilogue Programs [J/OL]. arXiv:2605.19269, 2026. DOI: 10.48550/arXiv.2605.19269.
[9] NRUSIMHA A, BRANDON W, MISHRA M, et al. FlashFormer: Whole-Model Kernels for Efficient Low-Batch Inference [J/OL]. arXiv:2505.22758, 2025. DOI: 10.48550/arXiv.2505.22758.
[10] ZADOURI T, HOEHNERBACH M, SHAH J, et al. FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling [J/OL]. arXiv:2603.05451, 2026.
[11] GUESSOUS D, STERN R, HOEHNERBACH M, et al. FlexAttention + FlashAttention-4: Fast and Flexible [EB/OL]. PyTorch Blog, 2026-03-05.
[12] YE Z, CHEN L, LAI R, et al. FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving [J/OL]. arXiv:2501.01005, 2025.
[13] ZHAO Y, YANG Y, BUDIU M, MISAILOVIC S. Nautilus: An Auto-Scheduling Tensor Compiler for Efficient Tiled GPU Kernels [J/OL]. arXiv:2604.14825, 2026.
[14] DeepSeek-AI. DeepGEMM: High-Performance Tensor Core Kernels for Modern LLMs [EB/OL]. GitHub, 2025–2026.
[15] DeepSeek-AI. FlashMLA: Efficient Multi-head Latent Attention Kernels [EB/OL]. GitHub, 2025–2026.
[16] CHEN H, XIE W, ZHANG B, et al. KTransformers: Unleashing the Full Potential of CPU/GPU Hybrid Inference for MoE Models [C]//Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles. ACM, 2025. DOI: 10.1145/3731569.3764843.
[17] CHENG X, ZHANG Z, ZHOU Y, et al. Mirage Persistent Kernel: A Compiler and Runtime for Mega-Kernelizing Tensor Programs [J/OL]. arXiv:2512.22219, 2025.
[18] NVIDIA. TensorRT-LLM Plugin Configuration and Performance Tuning Documentation [EB/OL]. TensorRT-LLM Documentation, 2026.
[19] PyTorch. torch.library; Using User-Defined Triton Kernels with torch.compile [EB/OL]. PyTorch Documentation, 2026.
[20] vLLM Project. Benchmark CLI: TTFT, TPOT, ITL and Goodput [EB/OL]. vLLM Documentation, 2026.