Publication Date: 1 May 2026
Author: Ailven LIU 刘翔
All Rights Reserved
1. Overview
The evolution from Relay to Relax and then to Unity in Apache TVM is not merely a sequence of syntactic replacements. It reflects a deeper shift in the compiler architecture of TVM: from a graph-level optimisation system centred on a high-level neural-network IR, towards a modular, cross-level machine learning compilation infrastructure in which graph programs, tensor programs, runtime objects, external libraries, and hardware-specific code generation can be represented and optimised together.
In concise terms:
NNVM Graph
|
V
Relay
|
V
Relax + TensorIR + TVMScript + MetaSchedule + Runtime / Backend Integration
|
V
TVM Unity Modular Compilation Flow
Relay was TVM’s first mature high-level graph IR. Relax is the next-generation high-level IR designed for dynamic shapes, modern model structures, LLM workloads, Python-first workflows, and cross-level optimisation with TensorIR. Unity is not a single IR, but a broader architectural direction that aims to unify Relax, TensorIR, TVMScript, MetaSchedule, the runtime, libraries, and target backends into a coherent and composable compilation system.
Therefore, the essential transformation can be summarised as follows:
Relay:
high-level graph IR for neural-network compilation
Relax:
symbolic-shape-aware dataflow/program IR integrated with TensorIR
Unity:
modular cross-layer ML compilation architecture
The transition from Relay to Relax should not be interpreted as a superficial renaming. It represents a change in the underlying compilation philosophy: from a largely staged pipeline of high-level graph optimisation followed by low-level tensor-program lowering, towards an IRModule-centred system in which high-level graph programs and low-level tensor programs coexist, interact, and can be transformed together.
2. Before Relay: The NNVM Graph Era
Before Relay became the central graph-level IR in TVM, early TVM workflows relied on NNVM as the high-level graph representation. In that period, the overall compilation pipeline could be roughly described as:
Framework Model
|
V
NNVM Graph
|
V
Operator Lowering
|
V
TVM Tensor Expression / Schedule
|
V
Code Generation
This approach was effective for many early deep learning workloads. Models such as CNNs, MLPs, and relatively regular RNN variants could often be represented as static operator DAGs:
Conv2D -> BatchNorm -> ReLU -> Pooling -> Dense
However, NNVM-style graph IRs had several structural limitations.
First, their expressive power was limited. A conventional computation graph is well suited to representing a DAG of tensor operators, but it is not a natural representation for control flow, subgraphs, higher-order functions, recursion, or dynamic computation.
Second, the boundary between compile-time representation and runtime representation was insufficiently clean. The graph was both an optimisation object and a structure close to runtime execution, which restricted the compiler’s ability to perform deeper program transformations.
Third, the interaction between high-level graph optimisation and low-level tensor-program optimisation was weak. A graph-level pass could identify that a subgraph contained a convolution, a matrix multiplication, or an elementwise operation, but it had limited access to the details that matter at the tensor-program level: tiling, vectorisation, memory layout, shared-memory usage, tensorisation, cache locality, and target-specific scheduling.
These limitations motivated the design of Relay.
3. Relay: TVM’s First Mature High-Level IR
3.1 Historical Role of Relay
Relay was introduced around 2018 as the successor to NNVM. It was designed as a new high-level IR for machine learning frameworks and was often described as NNVM v2.0. Its purpose was to provide a more expressive and principled representation for machine learning programs than a traditional computation graph.
Relay can be characterised as:
statically typed
purely functional
graph-level / high-level
machine-learning-oriented
intermediate representation
Unlike a simple DAG representation, Relay introduced concepts from programming language design: functions, variables, let-bindings, function calls, operator calls, tuples, conditional expressions, and a type system. This made Relay significantly more expressive than NNVM-style graphs.
Relay was therefore not merely a graph container. It was a typed functional IR designed to represent machine learning models as programs.
3.2 What Relay Solved
Relay solved several important problems in the earlier TVM stack.
First, it introduced a stronger type system. Relay could represent tensor types, tuple types, function types, and other structured types, enabling more robust static checking, type inference, and program transformation.
Second, Relay provided functional abstraction. Models no longer had to be represented only as flat operator DAGs. They could include function definitions, let-bindings, control-flow constructs, and nested computation structures.
Third, Relay allowed graph optimisation to be expressed as program transformation. Passes such as constant folding, operator fusion, layout rewriting, dead-code elimination, common subexpression elimination, and quantisation rewriting could be implemented as transformations over Relay programs.
Fourth, Relay became the main graph-level IR in TVM for a substantial period. It served as the central representation for frontend import, high-level graph optimisation, operator fusion, and the initial stages of lowering towards tensor-level computation.
A typical Relay-based compilation flow looked like this:
PyTorch / TensorFlow / ONNX / TFLite / MXNet
|
V Frontend Import
Relay IRModule
|
V Relay Graph-Level Optimisation
Optimised Relay
|
V Operator Lowering / Fusion
TE / TIR
|
V Scheduling / Auto-Tuning / Code Generation
Runtime Module
From an engineering perspective, Relay played four major roles:
model-structure representation
graph-level optimisation
unified frontend import
operator fusion and lowering decision layer
This made Relay one of the most important components in the maturation of TVM as a deep learning compiler.
3.3 Limitations of Relay
Relay was not a failed design. On the contrary, it was a key stage in the evolution of TVM. However, as model architectures, hardware targets, and deployment requirements changed, Relay began to expose structural limitations.
3.3.1 Limited Support for Dynamic Shape Workloads
Relay was designed during a period when many important workloads were still dominated by relatively static shapes:
fixed batch size
fixed image resolution
often fixed sequence length
regular operator structure
Modern workloads are different. Transformers, LLMs, VLMs, diffusion models, dynamic batching, variable sequence lengths, KV cache management, ragged tensors, and prefill/decode separation all require stronger support for symbolic and runtime-dependent shapes.
For example, an LLM inference workload naturally contains two different execution regimes:
Prefill:
input shape = [batch, prompt_length]
Decode:
input shape = [batch, 1]
KV cache shape = [num_layers, batch, num_heads, sequence_length, head_dim]
sequence_length grows token by token
Relay could support certain dynamic-shape cases, but it was not designed around first-class symbolic shape as a central organising principle. As dynamic workloads became increasingly important, this became a fundamental limitation.
3.3.2 A Hard Boundary Between Graph IR and TensorIR
In the Relay-era compilation stack, the relationship between the high-level graph and low-level tensor program was usually staged:
Relay Graph Passes
|
V
Lowering to TE / TIR
|
V
TIR Scheduling / Code Generation
This structure created a relatively hard boundary between high-level and low-level optimisation.
A graph-level pass could decide whether to fuse operators, rewrite layouts, or transform subgraphs, but it often lacked precise information about low-level scheduling consequences. For example, whether a fusion decision is beneficial depends not only on graph topology, but also on:
register pressure
shared-memory usage
tensor-core-friendly layout
memory coalescing
occupancy
vectorisation feasibility
target-specific code-generation behaviour
A purely graph-level view is insufficient for such decisions. Conversely, low-level scheduling may reveal constraints or opportunities that should influence graph-level transformation. Relay’s architecture made this kind of bidirectional feedback difficult.
3.3.3 Limited Modularity of the Pass Pipeline
Relay supported custom passes, external code generation, BYOC, custom operators, and multiple frontend paths. However, many practical workflows still felt like a relatively fixed pipeline:
frontend import
|
V
relay.transform.Sequential([...])
|
V
relay.build(...)
This was acceptable for conventional model deployment, but less suitable for compiler engineers, hardware vendors, and AI infrastructure developers who needed to insert custom transformations, replace lowering paths, integrate external libraries, mix hand-written kernels with generated kernels, and inspect intermediate IR in a disciplined way.
As TensorIR, TVMScript, and MetaSchedule matured, TVM required a more modular, inspectable, and programmable compilation flow.
3.3.4 Insufficient Natural Fit for LLM Workloads
Relay was not originally designed around modern LLM inference. LLM workloads impose a set of requirements that are difficult to express elegantly in a traditional graph-centric compiler flow:
symbolic sequence length
dynamic KV cache
paged attention
prefill/decode separation
shape-dependent memory planning
external kernel integration
quantisation-aware graph rewriting
tensor parallelism
pipeline parallelism
runtime shape dispatch
These are not marginal extensions to ordinary graph execution. They require a representation that can naturally express graph structure, symbolic shapes, runtime objects, external function calls, TIR kernels, device placement, and memory/runtime behaviour within a single composable system.
This requirement led to Relax.
4. TVM Unity: The Architectural Vision Behind Relax
4.1 Unity Is Not a Single IR
TVM Unity should not be understood as a replacement IR. It is a broader architectural direction for TVM.
The central goal of Unity is to break the rigid boundaries between several layers of the machine learning system stack:
computational graph
tensor programme
library / runtime
hardware primitive
Historically, machine learning systems often separated these concerns into relatively isolated layers. Graph compilers optimised graph-level structure; tensor compilers generated kernels; vendor libraries supplied high-performance primitives; runtimes handled execution and memory management. These divisions simplified system design, but they also constrained cross-layer optimisation.
Unity’s central judgement is that future ML compilation cannot be a one-way lowering pipeline from graph to kernel. It must allow graph IR, tensor programs, external libraries, runtime systems, and hardware primitives to interact.
Thus, Unity should be understood as:
not one IR to rule them all
but interoperable IRs in one modular system
In practical terms, Unity consists of several mutually connected components:
Relax:
graph-level and programme-level abstraction
TensorIR:
tensor-program abstraction
TVMScript:
inspectable and writable IR surface
MetaSchedule:
automatic search-based scheduling
DLight and rule-based scheduling:
practical default scheduling automation
Runtime / VM / external codegen:
execution and backend substrate
Unity does not attempt to collapse all levels of abstraction into a single monolithic representation. Instead, it attempts to make multiple representations interoperable within one coherent compilation framework.
5. Relax: Relay Next
5.1 Historical Position of Relax
Relax emerged as ‘Relay Next’ under the TVM Unity vision. Its purpose was not to discard Relay’s achievements, but to evolve the high-level IR in response to new model and hardware requirements.
Relax inherits Relay’s role as the graph-level IR, but redesigns the abstraction around several new requirements:
dynamic shape
symbolic shape
TensorIR integration
cross-level optimisation
runtime object representation
external kernel integration
modular transformation
Python-first inspectability
The result is a high-level IR that is better suited to contemporary machine learning workloads, especially Transformer- and LLM-based systems.
5.2 Core Objectives of Relax
The essential objectives of Relax can be summarised as follows:
G0: bridge graph IR and TensorIR
G1: provide first-class symbolic shape support
G2: represent dynamic workloads naturally
G3: improve customisation and composability
G4: integrate cleanly with TVMScript and IRModule
G5: support emerging workloads such as Transformers, LLMs, and dynamic models
Relax was therefore designed not merely as a cleaner graph IR, but as the high-level program representation required by the Unity architecture.
5.3 Basic Design of Relax
The key design idea of Relax is that high-level graph programs and low-level tensor programs can coexist inside the same IRModule.
A simplified Relax + TensorIR module may look conceptually like this:
@tvm.script.ir_module
class Module:
@T.prim_func
def matmul(...):
...
@R.function
def main(x, w):
y = R.call_tir(matmul, (x, w), ...)
return y
This example illustrates the essential relationship:
Relax functions:
high-level dataflow, graph structure, runtime-level composition
TIR PrimFuncs:
low-level tensor programmes and kernels
IRModule:
shared container that holds both Relax and TIR
R.call_tir:
explicit bridge from Relax-level programme to TIR-level kernel
This is a major change from the older model in which a Relay graph was lowered into tensor-level code through a relatively staged pipeline. In Relax/Unity, the IRModule becomes the central object of transformation.
6. Core Differences Between Relay and Relax
6.1 IR Style: Functional Graph IR versus Dataflow/Binding-Oriented Program IR
Relay is strongly functional in style. It represents programs using expressions such as:
let-binding
function
call
tuple
if-expression
recursion
Relax is more explicitly structured around dataflow blocks, bindings, symbolic shape information, StructInfo, and explicit boundaries between pure dataflow and effectful/runtime operations.
A simplified comparison is:
Relay:
machine learning programme as typed functional expression
Relax:
machine learning programme as structured dataflow with bindings,
symbolic shape, TIR calls, and runtime integration
Relax is closer to a modern SSA-like dataflow program representation while still preserving the high-level graph structure needed for model optimisation.
6.2 Type System: Relay Type versus Relax StructInfo
Relay uses a type system to describe expressions. Examples include tensor types, tuple types, and function types.
Relax introduces the concept of StructInfo. StructInfo describes the structural information of expressions, including:
TensorStructInfo
ShapeStructInfo
TupleStructInfo
FuncStructInfo
ObjectStructInfo
StructInfo is not merely a conventional type system. It is better suited to representing partially known and partially dynamic information:
dtype
number of dimensions
symbolic shape
known and unknown dimensions
runtime object structure
shape expressions
This is crucial for dynamic-shape workloads. For example, in LLM inference:
input_ids:
Tensor[(batch, sequence_length), int32]
kv_cache:
runtime object / tuple / opaque cache structure
logits:
Tensor[(batch, sequence_length, vocab_size), float16]
Here, sequence_length may be symbolic rather than compile-time constant. Relax’s StructInfo model is designed to represent this kind of partial static knowledge cleanly.
6.3 Shape: Auxiliary Type Information versus First-Class Symbolic Shape
In Relay, shape information is largely attached to tensor types. Dynamic-shape support exists, but symbolic shape is not the central organising principle of the IR.
Relax treats symbolic shape as a first-class design concern. This is essential for workloads such as:
Transformer variable sequence length
LLM prefill/decode execution
dynamic batching
ragged input
vision-language models with variable token counts
diffusion models with dynamic resolution
shape-polymorphic library dispatch
From an AI infrastructure perspective, Relax makes it natural to represent partially dynamic programs such as:
n = T.int64()
m = T.int64()
x: R.Tensor((n, m), "float16")
This is fundamentally different from forcing all shapes to be fixed at compile time.
6.4 High-Low Level Relationship: Lowering Boundary versus IRModule-Level Co-Design
In the Relay-era flow:
Relay graph
|
V
lowering to TE / TIR
|
V
TIR scheduling and code generation
In the Relax/Unity flow:
IRModule {
Relax functions
TIR PrimFuncs
external functions
runtime calls
}
|
V
IRModule-to-IRModule transformations
|
V
build
This is a major architectural change.
The compiler can now examine and transform high-level dataflow, low-level TIR kernels, external calls, device information, symbolic shape, and runtime calls within the same module-level representation.
This enables cross-level optimisation: high-level graph transformations can be informed by tensor-program details, and tensor-program generation can be shaped by high-level graph structure.
6.5 Pass Structure: Graph Passes versus Modular IRModule Transformations
Relay passes are often expressed as transformations over Relay functions or Relay expressions.
Relax places stronger emphasis on:
IRModule -> IRModule
This design fits naturally with TVMScript, structural equality, and testable compiler passes. A pass can be developed and tested by comparing:
before IRModule
|
V pass
after IRModule
This improves engineering discipline. It becomes easier to inspect intermediate IR, write unit tests, build custom flows, and reason about transformation correctness.
6.6 Runtime: Graph Executor / Relay VM versus Relax VM and Modular Runtime Integration
Relay workflows commonly used the Graph Executor or Relay VM.
Relax naturally aligns with a more flexible runtime model, especially the Relax VM and runtime objects. This is important because modern ML inference is often not just static graph execution. LLM inference, for example, must manage:
KV cache
paged attention metadata
sampling
token-by-token generation loops
dynamic sequence length
batch scheduling
runtime shape
external fused kernels
Relax is therefore better positioned as an interface between compiler IR and runtime execution.
7. Unity Connection and the Integration of Relax
7.1 External Development Before Mainline Integration
Relax was initially developed partly outside the main TVM tree in order to reduce disruption to existing Relay users and production deployments. This explains why, for a period, users encountered references to:
apache/tvm main branch
apache/tvm unity branch
mlc-ai-related documentation
Relax-related forks and experimental branches
This separation was not accidental. It allowed the community to experiment with the new architecture while preserving the stability of existing Relay-based workflows.
7.2 Relax as the Future Graph IR
Over time, the community direction became clearer. Relax moved from an exploratory ‘Relay Next’ effort towards the intended graph IR for new TVM development. Meanwhile, legacy components such as Relay, TE schedule, AutoTVM, and AutoScheduler remained important for existing users and historical workflows, but the new modular direction increasingly centred on:
Relax as graph/program IR
TensorIR as tensor-program IR
MetaSchedule as the primary automatic scheduling infrastructure
TVMScript as the inspectable IR surface
This does not mean Relay disappears immediately. Relay still has historical importance, existing production users, mature frontend paths, and a large body of accumulated compiler logic. The more accurate interpretation is:
new development:
favour Relax and the Unity modular flow
legacy production:
Relay remains relevant where existing pipelines depend on it
long-term direction:
move from Relay towards Relax as the principal graph IR
8. The Essential Transformation: Relay to Relax to Unity
8.1 From Graph IR to Program IR
Although Relay was more expressive than a traditional computation graph, it largely served as TVM’s graph-level IR.
Relax shifts the abstraction closer to a complete machine learning program IR. It can represent not only high-level dataflow, but also:
shape computation
runtime-dependent structure
tensor computation
runtime objects
external calls
device placement
memory planning boundaries
control and dataflow boundaries
This is increasingly necessary because modern model deployment is no longer just graph execution. LLM inference, for example, is graph computation plus runtime loops, cache management, memory scheduling, sampling, dynamic batching, and external kernel invocation.
Relax therefore represents a move from:
neural network graph IR
towards:
machine learning programme IR
8.2 From Staged Lowering to Cross-Level Co-Optimisation
The Relay-era mental model was often:
perform high-level graph optimisation first
then lower to low-level tensor computation
then schedule and generate kernels
The Unity-era mental model is:
represent high-level and low-level programmes together
perform transformations across abstraction boundaries
allow graph and tensor-level decisions to inform each other
This distinction is critical for performance.
Consider attention optimisation. A graph-level representation may see:
matmul -> softmax -> matmul
A Relax + TensorIR + external-kernel system can represent several alternatives:
call_tir(fused_attention_tir)
call_external("flashinfer.paged_attention")
rewrite pattern into a vendor-specific kernel
preserve symbolic sequence length and dispatch different kernel variants
This makes it significantly easier to integrate FlashAttention, FlashInfer, CUTLASS, cuBLASLt, TensorRT plugins, custom CUDA kernels, NPU backends, and other specialised execution paths.
8.3 From Static Model Compilation to Dynamic Workload Compilation
Early TVM workflows were often optimised for the following scenario:
known model
known input shape
known target
compile once
run many times
Modern AI infrastructure often faces a different scenario:
partially dynamic shape
dynamic batch size
runtime dispatch
hybrid compiler/library execution
model-specific kernels
target-specific graph rewriting
quantisation variants
multi-device execution
Relax’s symbolic shape, StructInfo system, runtime object support, call_tir, and external call mechanisms make it more suitable for these workloads.
8.4 From Operator-Centric Compilation to IRModule-Centric Compilation
Relay-era workflows often felt like:
import model
|
V
optimise graph
|
V
lower operators
Unity-era workflows are more naturally expressed as:
construct IRModule
inspect IRModule
transform IRModule
schedule TIR
replace subgraphs
call external libraries
tune kernels
build runtime module
The IRModule becomes the central object of the compilation process.
This is particularly valuable for AI infrastructure engineers. With TVMScript and Relax, one can inspect and modify the program at each stage:
print(mod.script())
apply transformation pass
print(mod.script())
insert custom PrimFunc
replace Relax call with call_tir
lower to target-specific code
This is far more controllable than a black-box frontend-to-build pipeline.
9. Significance for LLM and MLC-LLM
9.1 LLM Inference Is Not a Conventional Relay Workload
LLM inference contains structures that are awkward to express in a traditional static graph compiler:
prefill/decode separation
KV cache read/write
sequence length that grows dynamically
dynamic attention masks
sampling, top-k, and top-p
paged KV cache
tensor parallelism
pipeline parallelism
quantised weights
activation and weight layout transformations
external fused kernels
Several questions arise naturally:
Should the KV cache be represented as a tensor or a runtime object?
Should the decode loop be inside or outside the graph?
Is sequence length a compile-time constant or runtime value?
How should paged-attention metadata be represented?
How should different kernel variants be dispatched?
Relax provides a more suitable framework for these questions because it can represent graph structure, symbolic shape, TIR kernels, runtime objects, and external calls within the same system.
9.2 Value of Relax for MLC-LLM
MLC-LLM aims to compile LLMs across multiple backends, including CUDA, Metal, Vulkan, WebGPU, and CPU. Such a system requires:
dynamic-shape support
TIR-level kernel optimisation
runtime-level model execution
cross-platform lowering
quantisation rewriting
library and external kernel integration
These requirements align naturally with Relax and Unity.
A typical LLM compilation flow can be understood as:
Relax graph:
model architecture, high-level dataflow, module composition
TIR PrimFunc:
matmul kernel
decode attention kernel
RMSNorm kernel
rotary embedding kernel
sampling kernel
External call:
FlashInfer
cuBLAS / cuBLASLt
vendor library
runtime packed function
Relax makes it possible to represent these elements within one IRModule instead of scattering logic across frontend import, Relay passes, TE schedules, runtime glue code, and external backend hooks.
9.3 Relax and Model-Level Rewriting
Modern LLM deployment often relies on model-level rewriting. For example, a floating-point matrix multiplication may be rewritten into a quantised path.
At a high level, a graph expression such as:
matmul(x, w)
may become:
dequantize(w_q, scale, zero_point)
matmul(x, dequantized_w)
or, more efficiently:
call_tir(quantized_matmul_kernel, x, w_q, scale, ...)
or even:
call_external("cutlass_int4_gemm", ...)
Relax is better suited to these transformations because it can express graph-level patterns and then lower selected subgraphs either into TIR or into external libraries.
10. Significance for Hardware Vendors and AI Engine Engineering
Relax and Unity are also important for hardware vendors and AI Engine engineers.
A hardware backend often needs to express or control:
custom memory hierarchy
custom tensor core or matrix engine
custom DMA behaviour
custom layout
custom vector ISA
custom runtime integration
custom graph partitioning
Relay-era TVM supported many of these through BYOC, external codegen, and custom operators. However, the engineering experience could be heavy and fragmented.
Unity aims to make this integration more natural:
graph partition
|
V
external library or accelerator call
|
V
TIR scheduling or external code generation
|
V
runtime integration
Relax + TensorIR allows hardware vendors to participate simultaneously in:
high-level pattern rewriting
layout propagation
operator fusion
TIR scheduling
tensorisation
runtime calls
device placement
This is particularly relevant for heterogeneous execution across CPU, GPU, and specialised accelerators.
11. Common Misinterpretations
11.1 Relax Is Not Merely a New Name for Relay
Incorrect interpretation:
Relay was renamed Relax.
More accurate interpretation:
Relax inherits Relay's role as the high-level graph IR,
but redesigns the abstraction around symbolic shape,
TensorIR integration, modular transformation, and the Unity architecture.
11.2 Unity Is Not Identical to Relax
Incorrect interpretation:
Unity = Relax.
More accurate interpretation:
Unity:
the next-generation modular compilation architecture of TVM
Relax:
the high-level graph/programme IR within Unity
TensorIR:
the tensor-program IR within Unity
TVMScript:
the inspectable and writable representation layer
MetaSchedule / DLight:
automatic and rule-based scheduling infrastructure
Runtime / VM / external codegen:
execution and backend integration layer
11.3 Relay Does Not Disappear Immediately
Relay has a large amount of historical code, mature frontend infrastructure, many passes, and existing production users.
A realistic view is:
Relay:
still important for historical understanding and existing deployments
Relax:
preferred direction for new graph-level development
Unity:
broader system direction for future TVM compilation workflows
12. Historical Timeline
2017-2018: From NNVM to Relay
TVM’s early graph-level representation was NNVM. Around 2018, Relay was introduced as a more expressive high-level IR, intended to solve the limitations of conventional graph representations by adding typed functional program structure.
2018-2021: Relay as the Core Graph IR
Relay became the main graph-level IR in TVM. It supported frontend import, graph optimisation, operator fusion, layout rewriting, quantisation workflows, and BYOC-style integration.
2021: The TVM Unity Vision
The Unity vision was proposed as a new direction for TVM. Its goal was to break boundaries between computational graphs, tensor programs, libraries, runtimes, and hardware primitives, enabling deeper cross-layer optimisation.
2022: Relax Appears as Relay Next
Relax emerged as the high-level IR designed for Unity. It aimed to evolve Relay around symbolic shape, dynamic workload representation, TensorIR integration, modular transformation, and improved portability.
2022: Unity Connection
The Unity connection work aimed to bring Relax into the broader TVM ecosystem as a key component of the new end-to-end modular flow, together with TensorIR, MetaSchedule, and TVMScript.
2023: Relax Remains Actively Developed
Relax continued to develop actively in the Unity-oriented workflow. The community direction became increasingly clear: Relax was no longer merely an experiment, but the intended successor direction for high-level graph compilation.
2024 and After: Transition from Legacy Components to the New Modular Flow
The TVM community increasingly encouraged new development around Relax, TensorIR, MetaSchedule, TVMScript, and modular IRModule-level transformation. Legacy components such as Relay, TE schedule, AutoTVM, and AutoScheduler remained relevant for existing workflows, but the long-term direction moved towards the Unity stack.
13. How a Relay Engineer Should Understand Relax
For engineers familiar with Relay, the following conceptual mapping is useful:
| Relay Concept | Relax / Unity Interpretation |
|---|---|
| Relay Function | Relax Function |
| Relay Expr | Relax Expr / Binding |
| Relay Type | StructInfo |
| Relay graph pass | Relax / IRModule transformation |
| Relay operator fusion | Relax pattern rewriting plus call_tir or external call |
| Relay build | Relax / Unity build flow |
| TE lowering | TensorIR PrimFunc and call_tir |
| Relay VM | Relax VM |
| BYOC | Relax external call / codegen integration / modular backend |
| Static-shape graph | Symbolic-shape-aware program |
| Graph-only optimisation | Cross-level graph + TIR optimisation |
However, this table only captures surface-level correspondence. The deeper shift is that Relax encourages engineers to treat model compilation as a sequence of transformations over an IRModule, rather than as a fixed frontend-import-to-build pipeline.
14. A Practical Mental Model of the Relax/Unity Flow
A typical Relax/Unity compilation flow can be understood as:
1. Import or construct the model
PyTorch / ONNX / TensorFlow / NNModule / manual TVMScript
|
V
2. Produce a Relax IRModule
Relax functions + symbolic shape + high-level operations
|
V
3. Apply graph-level transformations
legalisation, fusion, pattern rewriting, quantisation, layout rewriting
|
V
4. Lower selected parts to TensorIR
call_tir, PrimFunc generation, external calls
|
V
5. Schedule TensorIR
manual schedule, DLight, MetaSchedule, tensorisation
|
V
6. Integrate with runtime
Relax VM, external libraries, packed functions, device placement
|
V
7. Build deployable module
target-specific code generation and runtime module
Compared with the Relay-era pipeline, the key difference is that steps 3, 4, and 5 are no longer separated by rigid one-way boundaries. They can interact through IRModule-level transformations.
15. Why This Matters for AI Infrastructure Engineers
From the perspective of AI infrastructure, the transition from Relay to Relax and Unity means that TVM is evolving from a neural-network compiler into a programmable ML systems compilation infrastructure.
For embedded deployment, autonomous driving, LLM inference, NPU/GPU backends, model compression, runtime optimisation, and custom kernel integration, Relax/Unity offers several practical advantages.
First, model structure becomes more controllable. Engineers can perform structured rewrites at the Relax level rather than relying on runtime hacks or low-level kernel replacement alone.
Second, dynamic shape becomes a first-class concern. This is essential for Transformers, BEV models, VLA models, multimodal models, and LLM workloads.
Third, integration with TIR becomes more direct. Hand-written TIR, generated TIR, scheduled TIR, and Relax graph programs can coexist in the same IRModule.
Fourth, external high-performance kernels become easier to integrate. Examples include FlashInfer, CUTLASS, cuBLASLt, TensorRT plugins, vendor libraries, and custom CUDA kernels.
Fifth, end-to-end performance experiments become more systematic. One can compare different implementation paths:
Relax operator path
call_tir path
external library path
manual TIR schedule path
MetaSchedule-tuned path
Sixth, compiler engineering becomes more maintainable. TVMScript, structural equality, and IRModule-level transformation provide a more inspectable and testable workflow than opaque build pipelines.
16. Conclusion
Relay was the decisive step that transformed TVM from an early tensor compiler with graph support into a mature deep learning compiler. It introduced a typed functional high-level IR and enabled robust graph-level optimisation.
Relax is the successor direction of Relay under the Unity vision. It does not reject Relay’s achievements; it reconstructs the high-level abstraction to address the requirements of modern workloads: symbolic shape, dynamic execution, LLM inference, runtime objects, external kernels, and cross-level optimisation with TensorIR.
Unity is the broader architectural framework. It attempts to integrate Relax, TensorIR, TVMScript, MetaSchedule, runtime systems, libraries, external code generation, and hardware-specific primitives into a modular and composable compilation system.
The historical evolution can therefore be summarised as:
NNVM:
computation graph as model representation
Relay:
typed functional IR as graph-level compiler IR
Relax:
symbolic-shape-aware dataflow/programme IR integrated with TensorIR
Unity:
modular cross-layer ML compilation system
For current and future work in TVM, MLC-LLM, AI Engine engineering, and high-performance model deployment, the most important direction is not merely to understand Relay passes or traditional TE schedules. Those remain historically and technically valuable. However, the main trajectory is now Relax + TensorIR + TVMScript + MetaSchedule + runtime integration.
Relay explains how TVM became a mature graph compiler. Relax and Unity explain where TVM is going.