Publication Date: 30 September 2026
Author: Ailven LIU 刘翔
All Rights Reserved
Abstract
An end-to-end autonomous-driving planner – a 12-layer Transformer trunk whose attention subgraphs are compiled into fused Flash Attention kernels – failed in two independent ways once Flash Attention was switched on. It emitted NaN, and it shipped a constant data segment and a device memory footprint far larger than the model's logical requirements. The same sources lowered through batch matmul plus softmax were numerically clean, which placed the numerical fault inside the Flash kernel rather than in the model.
The two failures share a compilation path but not a root cause. The NaN comes from one subtraction in the online softmax rescale step, where -inf minus -inf is an invalid operation under IEEE 754 and yields NaN. The bloat comes from a causal mask that is logically a single 68.9 MB tensor and physically twenty-four device buffers, produced by a chain of stages each of which duplicated what the previous one handed it.
The repair is three changes at three layers of the stack: a compile-time -inf guard in the kernel, a content-level constant deduplication pass in the compiler, and an existing but disabled cross-function constant sharing switch in the VM compiler. Together they remove the NaNs, cut the constant data segment by 63% and the peak device memory by 39%, and leave inference latency unchanged.
1. Two failures, one compilation path
The model is an end-to-end autonomous-driving planner. A 12-layer Transformer trunk consumes a sequence of historical feature frames together with the current token sequence, and produces trajectory deltas and classification logits. Its attention subgraphs are compiled into fused Flash Attention kernels.
Switching Flash Attention on introduced two independent faults. The first is numerical: the model emits NaN, while the same weights and the same sources lowered through batch matmul plus softmax are numerically clean. Only the compilation configuration and the kernel path differ between the two builds, so the fault has to lie inside the Flash kernel's numerical path. The second is volumetric: both the constant data segment of the compiled artefact and the device memory the runtime reserves are far larger than the model's logical requirements, and nothing in the model explains the excess.
| Metric | Before | After |
|---|---|---|
| Numerical correctness | NaN in the outputs | zero NaN; 2,586 kernel outputs byte-identical to an fp32 reference |
| Constant data segment | 1,357.1 MB | 526.65 MB, down 63% |
| Peak device memory | 2,039.68 MB | 1,247.7 MB, down 39% |
| Inference latency | 39.0 ms / 44.8 ms | 38.7 ms / 45.0 ms, unchanged |
The repair is three changes, each at a different layer of the stack: a numerical guard in the kernel's online softmax, content-level constant deduplication at compile time, and cross-function constant sharing at load time. Each addresses a separate root cause, and the last two turn out to be upstream and downstream of each other.
2. What the compiler has to materialise
The trunk has 12 layers, sequence length L = 1467, 16 heads, head dimension 64, and hidden size 1024. Each layer contains two Flash Attention kernel calls with identical input shapes – a Q, a K, a V and one attention mask – whose outputs are concatenated back into the hidden dimension. That is 24 Flash kernels. The two kernels within a layer share a causal mask, and before subgraph partitioning they reference the same bias input.
The mask is where the arithmetic becomes interesting. The model is autoregressive, so token t may only attend to positions before t. That constraint is expressed as an additive attention mask built from a lower-triangular matrix: zeros on and below the diagonal, a blocking value above it. The blocking value is written as an fp16 extreme and is semantically -inf. Added to the logits before softmax, it drives exp(-inf) to zero, removing future positions from the attention weights without disturbing the normalising denominator over past positions.
The mask is stored as [1, 16, 1467, 1467] in fp16. One copy is 16 × 1467 × 1467 × 2 bytes, or 68,866,944 bytes – about 68.9 MB.
The model needs one copy. What it gets is 24 copies in device memory, about 1.65 GB, and 12 copies in the constant data segment, about 826 MB. That expansion from one to twenty-four is the whole of the second problem.
Two details matter for what follows. The mask keeps its head dimension rather than being collapsed to [1, 1, 1467, 1467], because the interface has to accommodate per-head biases; the sixteen heads repeat the same causal pattern, so the mask is redundant along that axis, but the conclusion that one copy is logically sufficient is unaffected. And the compiler folds the softmax scale into the bias, dividing the mask constant by scale = 1/sqrt(64) = 1/8. Numerically that multiplies the mask by eight, and the fp16 extreme on the upper triangle – whether written as the smallest finite fp16 value, -65504, or as a literal -inf – overflows to a literal -inf. The kernel therefore sees a genuine -inf after the bias addition rather than a large negative number, and the NaN analysis that follows has to be carried out under IEEE 754 semantics rather than as a question of magnitude.
The trunk is not the only large operator. A convolutional stage produces a BEV feature map of roughly 201 MB in fp16, and it dominates the non-attention memory and arithmetic. There are KV cache reads and writes, twelve [1, 1467, 2048] outputs, plus rotary embeddings, layer norms, and a large population of small fused graph operators – reshape, strided_slice, take, concatenate – with an argmax at the output. A dump of the 2,586 kernel directories shows the distribution: almost everything is a fused_cast_reshape, fused_layer_norm or fused_strided_slice, alongside the 24 Flash kernels and the convolution kernels. The model is overwhelmingly made of small operators. The only tensors large enough to distort the constant segment are the ones the Flash kernels share.
The weights account for about 577 MB, and several groups within them are byte-identical, layer norm weights shared across layers being the clearest case. The largest single group is duplicated eleven times, so roughly 112 MB is recoverable in principle. Those duplicates have the same origin as the mask duplicates: graph-level copying that the compilation path never recognises or reclaims.
3. How the pipeline makes the copies
The stack is a graph compiler plus operator fusion plus a heterogeneous operator library, in the lineage of TVM and its CUTLASS backend. It runs in eight stages, and the expansion is what those stages produce when each one does something locally reasonable and none of them is responsible for spotting the repetition.

Figure 1 – the compilation path. The tuning stage is where bias-scale folding materialises new constants; content deduplication runs alongside structural CSE after it.
Export and partitioning
PyTorch exports the model to a graph IR, and the export separates computation from data. Operators become graph nodes, constants become ConstantNode objects, weights become parameters.
On the torch side the mask is computed twelve times, once per layer. Export therefore produces twelve mutually independent ConstantNodes holding byte-identical data. Those twelve are fixed by the model's graph structure, and they set the base for everything that follows.
The graph compiler does not emit code per operator. It partitions subgraphs matching a known pattern into standalone global subfunctions and hands each one to the operator library. Flash Attention is one such pattern: QK^T, the mask addition, online softmax and the PV weighting are recognised as a single unit. After partitioning, Flash Attention is a global subfunction and the causal mask is one of its parameters. The two kernels within a layer shared a bias input before partitioning and still reference the same mask constant node afterwards. This stage invents no new duplication; it preserves the one-per-layer structure of twelve.
The tuning rewrite
This stage is where both problems originate. It runs several passes in sequence: a quantisation scale rewrite, type inference, bias-scale folding to push the softmax scale into the bias, and a bias shape rewrite to a broadcast-friendly layout. An operator annotator then labels each subgraph with its op type, shapes and dtypes, and a profiler plus an automatic search selects kernel configurations.
Bias-scale folding is the one that matters. A Flash kernel would rather not multiply by the softmax scale inside the kernel, so the scale is pre-multiplied into the bias. The identity is
$$\text{softmax}\left((QK^{T} + \text{mask}) \cdot s\right) = \text{softmax}\left(QK^{T} \cdot s + \text{mask} \cdot s\right)$$
so dividing the mask by 1/s is equivalent. This is textbook constant folding, and it is necessary: subfunction parameters bind directly to kernel arguments at codegen, so leaving the scale unfolded would either keep a divide operator in the graph – an extra elementwise division over a 68.9 MB tensor, roughly 138 MB of read and write traffic – or change the kernel signature. Folding converts a runtime cost into a compile-time constant.
The defect is in how it folds. The pass folds independently at each call site, and each fold genuinely evaluates divide(mask, scale) and allocates a fresh ConstantNode for the result. Twenty-four call sites produce twenty-four independent ConstantNodes holding byte-identical data. After tuning, the graph carries twenty-four mask constants where it carried twelve. Nothing in the IR layer can tell that the new objects have the same contents.
Why structural CSE cannot see it
The pipeline runs structural common subexpression elimination after tuning. Its criterion is structural equality: two expressions merge only when their operator, their attributes and all their subexpressions are structurally identical.
Consider divide(mask_A, scale_1) and divide(mask_B, scale_2). Even when mask_A and mask_B hold byte-identical data, they are different ConstantNode objects, and the structural comparison of a ConstantNode compares object identity rather than content. The two expressions are therefore not structurally equal, CSE leaves them alone, and the twenty-four folded mask constants survive. Enabling the option that hoists constants into function parameters to make them shareable changes how the constants are parameterised, not the criterion by which they are compared.
The general point is that CSE removes redundant computation – the same expression evaluated twice – and not redundant data. When duplication takes the form of identical content under different structure, every structural tool in the pipeline is blind to it.
Codegen, the constant pool and the runtime
After CSE the IR enters codegen. Each subgraph becomes C++/CUDA source, and nvcc compiles it into a dynamic library that carries all kernel machine code and determines the arithmetic cost and the latency.
The graph functions are then compiled to virtual machine bytecode, and the data behind every ConstantNode is collected into a global constant pool, which is serialised into the data segment. Constant handling does deduplicate, but only within a narrow scope. The first layer is a per-function hash table keyed on the ConstantNode pointer and scoped to the function currently being compiled: a function that references the same ConstantNode twice pools it once. The twenty-four masks live in twenty-four different subfunctions, so across functions the table does nothing at all. The second layer is the global pool itself, which appends every ConstantNode it has not seen before.
Twenty-four functions therefore pool the mask twenty-four times. The data segment ends up with twelve entries rather than twenty-four, and the reason is this. VM compilation walks functions one at a time; the two call sites in a layer share the same folded result object after the rewrite, so compiling the paths through those two call sites hits one ConstantNode, while the twelve layers are twelve distinct objects. Twelve or twenty-four makes no difference to the mechanism, which is that the pool keys on object identity and charges separately for objects whose contents agree.
At runtime the VM loads the dynamic library and the constant pool, with the whole pool entering device memory, and executes the bytecode. The load-constant instruction means "take entry index of the pool, ensure it has a device buffer, point the register at it", and the implementation caches by index: the first execution of a given index allocates the buffer, and later ones reuse it.
Had the twenty-four kernels all referenced one index, device memory would hold one buffer. VM compilation has already assigned them different indices, because it pooled twelve distinct objects, and a cache keyed on index cannot hit across them. Each index gets its own allocation, and the mask's device footprint becomes twenty-four copies.

Figure 2 – the same tensor at each stage. The logical requirement is one copy; the data segment holds twelve entries; device memory holds twenty-four buffers.
Five stages, five copies, and no stage recognises the copy the previous one made. That is the whole of the expansion from one to twenty-four.
4. Where the NaN comes from
Three conditions have to hold at once for the NaN: the kernel takes the non-causal path, the upper triangle of the mask is a literal -inf, and online softmax rescales its running accumulators.
The first is a property of the kernel as compiled. The original Flash kernel does not enable its built-in causality test, so it walks every column block – block size 128, twelve blocks over a sequence of 1467. Causality is supplied entirely by the external mask. The kernel has no idea which positions lie in the future, and reads only the values in the mask.
The second follows from the scale folding described above. After bias-scale folding, the extreme on the upper triangle has overflowed to a literal fp16 -inf. For a query at an early position t, every column block past the causal boundary – every key position greater than t – is uniformly -inf.
The third is the structure of online softmax. Flash Attention maintains a running maximum scores_max and an accumulated output acc_o, block by block, rescaling the accumulated values by the ratio of the new maximum to the old one so that the result matches a global softmax.
The rescale
The fault is in the rescale function, whose signature takes a Check_inf template parameter defaulting to false.
float scores_max_cur = !Check_inf
? scores_max(mi)
: (scores_max(mi) == -INFINITY ? 0.0f : scores_max(mi));
float scores_scale = exp2f((scores_max_prev(mi) - scores_max_cur) * softmax_scale_log2);
scores_sum(mi) *= scores_scale;
// ... acc_o_rowcol(mi, ni) *= scores_scale;
scores_max_prev is the previous round's maximum; scores_max is the maximum over the current column block after reduce_max. The subtraction of scores_max_cur from scores_max_prev is the whole of the bug.
The trace
Take a query row whose causal boundary lies early enough that two consecutive column blocks are entirely in its future, so both are uniformly -inf.

Figure 3 – the trigger sequence. Two consecutive fully masked tiles are required; one alone is harmless.
The first of the two blocks leaves scores_max at -inf, since the maximum of a set of -inf values is still -inf. Entering the second block, the kernel copies the previous maximum into scores_max_prev = -inf, reduces the new block to scores_max = -inf, and with Check_inf false sets scores_max_cur to -inf. The scale is then exp2f((-inf - (-inf)) * scale_log2). Under IEEE 754, -inf minus -inf is an invalid operation and yields NaN; NaN multiplied by anything is NaN; exp2f(NaN) is NaN. scores_sum and acc_o are multiplied by NaN, and the NaN enters the accumulated output.
Why it stays hidden
A single fully masked column block triggers nothing. prev = -inf together with cur = -inf requires two consecutive fully masked blocks, and if prev = -inf while cur is finite then (-inf) minus a finite value is -inf, exp2f(-inf) is zero, and the scale clears the accumulated history, which is mathematically correct. The bug therefore depends on where the causal boundary falls relative to the column-block boundaries. It fires only at particular sequence positions, which is why it never surfaced in a simple unit test and yet appeared reliably on real long sequences.
The upstream code does guard one step of this computation: the exp2 applied to the scores themselves is written as max == -INFINITY ? 0 : exp2(scores). That guard covers the exp2 of the scores, not the rescale of the accumulated output. scores_scale is computed independently inside the rescale function, from an unguarded copy of scores_max_prev, so the protection sits at a different point in the formula from the one where -inf meets -inf.
Propagation
Once NaN is in the accumulated output, the rest is determined. The epilogue normalises by 1/scores_sum, which propagates it, and the result is cast to fp16 and written to global memory as the kernel's output. That output is added back into the residual stream, x = x + attn_out, and the addition contaminates the whole hidden state. Every subsequent layer norm spreads it further, since the mean and variance of a vector containing NaN are themselves NaN, and the MLP and the next attention follow. After twelve layers, every output – logits, trajectory deltas, KV cache – is NaN.
Residual connections and layer norms do not absorb NaN; they broadcast it. That is why one kernel producing NaN at one point takes down the entire output.
5. The bloat, accounted for
The original data segment is 1,357.1 MB, and the causal mask accounts for twelve copies of it: one per layer, 68.9 MB each, about 826 MB, or 61% of the segment. Peak device memory is 2,039.68 MB, of which the mask's buffers account for twenty-four copies – one independent buffer per kernel, about 1,652 MB. One copy would have been 69 MB.
The instructive part is that no single stage is a bug. Each behaves correctly inside its own abstraction.
| Stage | What it did | Why that is defensible |
|---|---|---|
| Export | one independent ConstantNode per layer, twelve in total |
it reflects the model's graph structure faithfully, and has no reason to compare two tensors byte by byte |
| Tuning fold | materialised a new ConstantNode per call site, twenty-four objects |
folding exists to remove runtime work, and materialising per call site is the direct implementation |
| Structural CSE | failed to match the content duplicates | its criterion is structural equivalence, and it carries no data-comparison duty |
| Constant pool | pooled by object pointer, no cross-function reuse | a compiler does not assume that two distinct objects agree |
| Runtime | indices differed, so the cache never hit | the index cache already allocates once per index; it was handed indices that never coincide |
The bloat is what those correct local behaviours add up to. Every stage discharges its own responsibility, and none of them owns the cross-stage question of whether two constants hold the same bytes. Fixing any single stage in isolation would not close the chain.
Twelve entries, twenty-four allocations
There is a second copy, and it is not the same as the first. The data segment is counted in constant pool entries, of which there are twelve. Device memory is counted in allocations, which follow the load instruction indices. One pool entry referenced from several functions under different indices is allocated once per index. Twelve entries, each referenced by two kernels under two different indices, give twenty-four buffers.
Data is therefore duplicated twice, once at serialisation and once at allocation, and repairing only the serialisation stage leaves the runtime duplicating as before. The two stages need separate repairs because they key on different things: the pool on object identity, the runtime cache on index.
6. The three fixes
A compile-time guard in the kernel
The smallest change sits in the kernel. A compile-time constant decides whether -inf protection is enabled, and it is passed to the rescale function's Check_inf parameter.
// An external additive bias (position bias / swin attention mask) may contain
// -inf, and those paths need the same protection in the softmax_rescale stage;
// otherwise two consecutive fully masked key tiles compute
// exp2f(-inf - (-inf)) = NaN.
constexpr bool kCheckInf = Is_causal
|| Kernel_traits::Has_Position_Bias
|| Kernel_traits::Has_Swin_Attn_Mask;
All three rescale call sites pass Check_inf = kCheckInf, and with Check_inf true the evaluation of scores_max_cur folds -inf to zero:
float scores_max_cur = scores_max(mi) == -INFINITY ? 0.0f : scores_max(mi);
Now scores_max_prev (-inf) minus scores_max_cur (0) is -inf, exp2f(-inf) is zero, the scale is zero, and the accumulated history is cleared correctly instead of being multiplied by NaN.
Making the guard a compile-time constant is the right choice because whether a kernel instance carries an external bias is a static fact about the instantiation rather than a runtime condition. A constant lets the compiler eliminate the irrelevant branch during instantiation, so no runtime if is added to kernels that do not need it.
One alternative was rejected. Giving the kernel a causal-mode template parameter, in the manner of standard Flash Attention, would clip the column-block loop to the causal region and never enter two fully masked blocks. The change was line-for-line equivalent to a known-correct fallback at source level, yet measured byte-identical inputs into the first Flash kernel produced outputs correlating at 0.087 with the reference. The root cause was never pinned down; the plausible explanation is that adding and removing template parameters perturbs C++ template instantiation and compiler optimisation – register allocation, shared-memory layout, instruction scheduling – into a numerically different and wrong result. The minimal change was taken instead: one compile-time boolean, no alteration to the traversal, the smallest possible disturbance to instantiation.
Content-level deduplication at compile time
The second change is a new compile-time pass that deduplicates constants by content. Its core is under twenty lines. It removes redundant data – constants whose bytes agree but whose object identities differ – and leaves redundant computation alone.
Three things matter in it. The content fingerprint is a key of (dtype, shape tuple, sha256 of the bytes), a triple that determines a constant's content identity. A size filter drops constants below a threshold, 4096 elements say, which avoids hashing a large population of small scale and bias vectors that would save nothing if merged. And the representative table is shared across functions: an ExprMutator rewrites visit_constant so that a constant whose content key is already registered is replaced by the representative, and one that is not registers itself.
The sharing is the point. The twenty-four Flash kernels are twenty-four independent global functions. With one representative table per function, deduplication can only happen inside a function, and none of the twelve masks can merge with any other. With a single table for the module, the mask encountered in the first function registers as the representative and the remaining twenty-three references are replaced by that same ConstantNode: twenty-four references collapse to one object and one copy of the data. How wide the table is decides the ceiling on what deduplication can recover.
The pass runs after the tuning rewrite, alongside structural CSE, with CSE first. Running it after tuning matters because the duplicate ConstantNodes are exactly what bias-scale folding produced; run before tuning, deduplication sees twelve shared bias parameters after partitioning and finds nothing to merge, and tuning then materialises twenty-four new objects. CSE first matters because it merges the structurally equivalent expressions and leaves content deduplication to handle the structurally different but byte-identical remainder. The two are complementary, and doing CSE first hands content deduplication a smaller graph.
Cross-function constant sharing at runtime
The third change sits at the boundary between VM compilation and runtime: enabling a cross-function constant sharing switch that already exists and is off by default.
The VM compiler handles a constant node through three branches. The first reuses within a function, reusing the register with a single Move when the current function has already handled this ConstantNode. The second reuses across functions, and this is the branch the fix enables: when the underlying data has already been pooled by any earlier function, it emits an instruction loading the existing index instead of appending a new constant. The third is the default path, appending a new constant to the pool and recording both the in-function mapping and the data-to-index mapping. Only the first and third are active by default; the second is behind an environment variable, and the fix turns it on at the compilation entry point.
The second branch is backed by a global map from the underlying data object, an NDArray, to a constant pool index. Its header comment already describes its purpose as sharing constants across VM functions, and it compares data objects by pointer. The third branch registers data against index when it pools, and the second branch looks the data up before pooling.
The switch on its own fixes nothing, because the map compares pointers. If the twenty-four masks are twenty-four distinct data objects, identical bytes notwithstanding, the pointers differ and the lookup never hits. Content deduplication has to unify the twenty-four references onto one ConstantNode and one copy of the data first, and only then do the pointers agree and the lookup succeed. In the other direction, content deduplication without the switch shrinks the data segment, since some paths benefit once the objects are unified, but VM compilation still pools per function through the third branch, the runtime still allocates per index, and the memory saving is discounted.
With both in place, every kernel's load instruction points at the same index, the runtime caches by index, the first allocation is reused by all the rest, and all the kernels share one device buffer. The mask's device footprint falls from twenty-four copies, about 1,652 MB, to one, about 69 MB. Content deduplication unifies object identity; cross-function sharing unifies the index. The first gives the second something to share, and the second makes the sharing real in device memory.
How the three fit together

Figure 4 – where each fix attaches to the path in Figure 1.
The first fix restores numerical correctness, closing the one place where -inf minus -inf leaks a NaN. The second fixes the data segment, collapsing twenty-four identical objects into one. The third fixes device memory, letting different functions share one pool entry and one device buffer.
There is a causal chain between the last two: deduplication creates the pointer identity that sharing depends on, and sharing turns unified objects into shared memory. The first fix is independent of that chain and acts on correctness. Deduplication and cross-function sharing both happen at compile time and load time, so neither adds per-token cost, and the kernel guard is evaluated at compile time and eliminated at instantiation. Latency is unchanged, and the dynamic library grows by nothing measurable.
| Metric | Result |
|---|---|
| Numerical correctness | zero NaN; 2,586 kernel outputs byte-identical to an fp32 reference |
| Constant data segment | down 63%, 830 MB saved |
| Peak device memory | down 39%, 792 MB saved |
| Latency | unchanged, no regression |
| Dynamic library size | essentially unchanged |
7. Two kinds of redundancy
The two failures came from three gaps in one compilation path, and the gaps are not of the same kind. The NaN is a missing guard in the online softmax rescale, at the single operation where -inf meets -inf. The data segment bloat is a missing content-level comparison in the IR, which leaves the constant pool keying on object identity. The device memory bloat is a missing index reuse in the VM compiler, which leaves the runtime cache keyed on indices that never coincide. The last two sit upstream and downstream of each other, and neither works alone.
Diagnosing the NaN meant refusing to stop at the observation that the output contains NaN, and following IEEE 754 semantics line by line until the one operation capable of producing it was isolated. Diagnosing the bloat meant separating structural equivalence from content equivalence. They are different kinds of redundancy: structural CSE tests for the first, and byte-identical data behind distinct objects falls outside its criterion altogether. Those two facts between them explain why the numerical fix belongs in the kernel and the volumetric fix belongs in the compiler's constant handling.