Publication Date: 4 May 2026
Last Update Date: 7 May 2026
Author: Ailven LIU 刘翔
All Rights Reserved
Overview
This guide is intended for practitioners who wish to build the complete machine learning compilation stack entirely from source on an NVIDIA H800 system (24-core CPU, 220 GB memory) within a Linux container environment, as referenced in the mlc-ai/mlc-llm source repository. The stack in question comprises MLC-LLM, XGrammar, Apache TVM, FlashInfer, and CUDATile.
The guide covers the full process: pulling a base Docker image, configuring the container, resolving system-level read-only filesystem and dependency issues, accelerating package management (APT and PyPI), and configuring and compiling TVM. Each step adheres strictly to a ‘one step, one verification’ principle to ensure robustness and reproducibility.
Critical Note: In the target environment described in this guide, the container is managed by a Kubernetes orchestration platform, with the root filesystem configured as read-only by default. This causes the system package manager (dpkg/APT) to fail when attempting to write to
/etc/group, blocking all subsequent operations.
This guide not only addresses how to resolve such read-only container constraints gracefully, but treats the issue as an opportunity to establish an independent, writable, and stable high-performance compilation environment from first principles.
1. Preparing the container
1. Container Environment Preparation
1.1 Pulling the Base Image
MLC-LLM requires CUDA version 12.0 or later to correctly support its compilation and runtime operators. We select nvidia/cuda:13.2.0-cudnn-devel-ubuntu22.04 as the base container. The devel tag is built atop the runtime image and includes the CUDA compilation toolchain, header files, and development libraries – precisely what is required for building CUDA applications from source.
# Base pull command (exact invocation depends on your container runtime)
docker pull docker.io/nvidia/cuda:13.2.0-cudnn-devel-ubuntu22.04
# If operating within a Chinese network where docker.io access is slow or unavailable,
# a domestic mirror may be used instead:
# swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/nvidia/cuda:13.2.0-cudnn-devel-ubuntu22.04
1.2 Launching an Interactive Container
The following command launches a persistent interactive container from the Docker image. To ensure that compiled artefacts are not lost when the container exits, the host-side source directory must be mounted into the container. In a Kubernetes environment in particular, the root filesystem must be explicitly declared as writable; failure to do so will trigger the dpkg errors described later in this guide.
# Launch an interactive container with host directory mount and writable root filesystem
docker run -it \
--gpus all \
--shm-size=32g \
--mount type=bind,source=/path/to/your/local/repos,target=/workspace/repos \
--read-only=false \ # Critical in Kubernetes environments
docker.io/nvidia/cuda:13.2.0-cudnn-devel-ubuntu22.04 /bin/bash
If using a company-managed Kubernetes-based experimental platform (such as KTP or an equivalent), the image address may be entered directly, provided that the platform is capable of pulling from the specified registry.
1.3 Base Directory Structure
Upon entering the container, it is advisable to create the source code directory in advance before proceeding with subsequent steps.
mkdir -p /workspace/repos
cd /workspace/repos
2. System Configuration and Package Mirror Acceleration
When provisioning a Pod on a corporate Kubernetes-based experimental platform, the resulting environment will very likely be a read-write–restricted Pod whose root filesystem is read-only. This can cause apt configuration failures or system package corruption. Under any circumstances, a writable system environment is a prerequisite for compiling software. Attempting to run advanced compilation tasks on an unstable read-only filesystem is inadvisable.
2.1 Configuring APT and PIP Mirror Sources
To ensure the fastest possible download speeds during compilation, it is recommended to switch to high-speed domestic mirrors.
APT acceleration (Tsinghua University mirror): https://mirrors.tuna.tsinghua.edu.cn/help/ubuntu/
PyPI acceleration (Tsinghua University mirror): https://mirrors.tuna.tsinghua.edu.cn/help/pypi/
# Configure a trusted host to suppress SSL-related warnings pip config set install.trusted-host mirrors.tuna.tsinghua.edu.cn
2.2 Installing Essential Build Tools
The following installs the foundational components required for building TVM:
# Install core build dependencies
sudo apt install -y \
vim \
build-essential \
cmake \
curl \
wget \
git \
ninja-build \
python3 \
python3-pip \
python3-dev
# Install version control and additional dependencies
sudo apt install -y git-lfs
# Verify installations
cmake --version # Should be >= 3.22
ninja --version # Should be >= 1.10
python3 --version # Should be >= 3.10
If the system has entered a dpkg deadlock – for instance, if groupadd: cannot open /etc/group or E: Sub-process /usr/bin/dpkg returned an error code (1) is reported – the obstructing post-installation scripts may be safely removed as follows:
# Remove the corrupted package configuration script (openssh-client shown as an example)
sudo rm -f /var/lib/dpkg/info/openssh-client.postinst
# Force completion of the dpkg database repair
sudo dpkg --configure -a
sudo apt install -f
2. Installing the toolchains
3. Installing the LLVM and CUDA Toolchains
3.1 Installing LLVM 18
LLVM 18 is installed via the ISCAS (Institute of Software, Chinese Academy of Sciences) mirror, which provides one of the most stable and accessible open-source LLVM repositories.
set -e
# Basic Dependencies
sudo apt update
sudo apt install -y ca-certificates wget gnupg
# Remove personal pre-existing LLVM source list files
sudo rm -f /etc/apt/sources.list.d/llvm-iscas.list
# Add the ISCAS LLVM mirror
wget -qO- https://apt.llvm.org/llvm-snapshot.gpg.key \
| sudo gpg --dearmor -o /usr/share/keyrings/llvm-archive-keyring.gpg
# Verify fingerprint
gpg --show-keys --fingerprint /usr/share/keyrings/llvm-archive-keyring.gpg
# Add ISCAS LLVM mirror
echo "deb [signed-by=/usr/share/keyrings/llvm-archive-keyring.gpg] https://mirror.iscas.ac.cn/llvm-apt/jammy/ llvm-toolchain-jammy-18 main" \
| sudo tee /etc/apt/sources.list.d/llvm-iscas.list
sudo apt update
sudo apt install -y \
clang-18 \
lld-18 \
lldb-18 \
clangd-18 \
clang-format-18 \
clang-tidy-18 \
llvm-18-dev \
llvm-18-tools
# Verify the installation
llvm-config-18 --version # Expected output: 18.1.8
clang-18 --version
which clang-18
Outcome at this stage: the ISCAS mirror provides llvm-config-18 at version 18.1.8, supplying TVM with a complete LLVM backend for JIT compilation.
3.2 Configuring CUDA and cuDNN
This guide uses NVIDIA CUDA 13.2.0. As the container image already contains the full CUDA toolchain, no additional installation is necessary. Should the toolchain be absent, installation instructions are available at the NVIDIA CUDA download page.
When compiling TVM, CMake must be able to locate the cuDNN library files precisely. If the default path does not exist – for example, if /usr/lib/x86_64-linux-gnu/libcudnn.so is a broken symbolic link or is absent – it must be specified manually.
Confirm the actual location of the cuDNN libraries first:
find /usr -name "libcudnn*" 2>/dev/null
find /usr -name "cudnn*.h" 2>/dev/null
Adjust CUDNN_LIBRARY and the cuDNN INCLUDE_DIR within config.cmake according to the paths returned by these commands.
4. Configuring the proxy for GitHub
In some cases, especially you are operating on a domestic machine, you may encounter problem that github unaccessible. Please make sure you have a proxy server in your server for github clone:
sudo apt update
sudo apt install shadowsocks-libev
Create a configuration file, link ~/shadowsockets.json:
{
"server": "your.ss.node.ip",
"server_port": {port},
"local_address": "127.0.0.1",
"local_port": 1082,
"password": "your ss node password",
"timeout": 300,
"method": "entropy method",
"fast_open": false,
"workers": 1
}
Check where ss-local is: which ss-local, it always is /usr/bin/ss-local
self-start
Create systemd: sudo vim /etc/systemd/system/ss-local.service, paste content below:
[Unit]
Description=Shadowsocks Local Proxy
After=network.target
[Service]
Type=simple
User=ailven.liu
ExecStart=/usr/bin/ss-local -c /home/ailven.liu/shadowsockets.json
Restart=on-failure
RestartSec=5
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.target
Note: replace ExecStart and User to what they really are.
Reload configuration and set to self-start:
sudo systemctl daemon-reload
sudo systemctl enable --now ss-local.service
Check status and log:
systemctl status ss-local.service
journalctl -u ss-local.service -f
Run background
nohup ss-local -c ~/shadowsockets.json > ~/ss-local.log 2>&1 &
ps aux | grep ss-local # check status
tail -f ~/ss-local.log # check logs
Setting the proxy for GitHub only
Then run command building proxy only for github:
git config --global http.https://github.com/.proxy socks5://127.0.0.1:1082
git config --global https.https://github.com/.proxy socks5://127.0.0.1:1082
or you can also edit file ~/.gitconfig directly:
[http "https://github.com"]
proxy = socks5://127.0.0.1:1082
[https "https://github.com"]
proxy = socks5://127.0.0.1:1082
Check it:
git config --global --list --show-origin | grep github
You can also delete the proxy config from github:
git config --global --unset http.https://github.com/.proxy
git config --global --unset https.https://github.com/.proxy
3. Cloning and pinning the sources
5. Cloning MLC-LLM and Its Pinned Source Dependencies
All source code is placed under ~/repos.
mkdir -p ~/repos
cd ~/repos
Before cloning the full repository, verify that GitHub access is working through the configured Git proxy:
git config --global --list --show-origin | grep -Ei 'proxy|github' || true
GIT_CURL_VERBOSE=1 git ls-remote https://github.com/mlc-ai/mlc-llm.git HEAD
GIT_CURL_VERBOSE=1 git ls-remote https://github.com/apache/tvm.git HEAD
The GitHub-specific SOCKS5 proxy used in this environment is:
socks5://127.0.0.1:1082
Clone mlc-llm recursively:
cd ~/repos
git clone --recursive https://github.com/mlc-ai/mlc-llm.git mlc-llm
cd ~/repos/mlc-llm
git rev-parse HEAD
git submodule status --recursive
In this run, the checked-out mlc-llm commit was:
d1ea69a87280e821611643958bfec385b62dafd3
A transient network failure occurred while cloning one nested CUTLASS submodule:
error: RPC failed; curl 56 GnuTLS recv error (-9): Error decoding the received TLS packet.
fatal: early EOF
fatal: fetch-pack: invalid index-pack output
However, Git automatically retried the submodule clone and succeeded. The final git submodule status --recursive output showed all submodules checked out cleanly, with no leading -, +, or U.
Important observation:
~/repos/mlc-llm/3rdparty/tvm
is not a direct checkout of apache/tvm.git. It is pinned by mlc-llm to:
https://github.com/mlc-ai/relax.git
The checked-out TVM commit was:
c2028da5ac03055f30a52b492260f37b9650ac03
For MLC-LLM development, this pinned TVM source tree should be built and used. Building an unrelated apache/tvm HEAD separately may cause API or ABI mismatches.
Verify repository cleanliness:
cd ~/repos/mlc-llm
git status --short
git rev-parse HEAD
git submodule foreach --recursive '
echo "----- $name :: $path -----"
git status --short
git rev-parse HEAD
'
A clean checkout should produce no git status --short output in the main repository or in submodules.
6. Configuring the Pinned TVM Source Tree
Enter the pinned TVM source tree:
cd ~/repos/mlc-llm/3rdparty/tvm
The TVM source tree contains a default CMake configuration file at:
cmake/config.cmake
However, TVM does not automatically read this file from cmake/config.cmake. Its top-level CMakeLists.txt loads configuration in the following order:
if(EXISTS ${CMAKE_BINARY_DIR}/config.cmake)
include(${CMAKE_BINARY_DIR}/config.cmake)
else()
if(EXISTS ${CMAKE_SOURCE_DIR}/config.cmake)
include(${CMAKE_SOURCE_DIR}/config.cmake)
endif()
endif()
Therefore, after modifying cmake/config.cmake, it must be copied into the build directory as:
build/config.cmake
6.1 Confirm CUDA, cuDNN, and NCCL State
CUDA root:
readlink -f /usr/local/cuda
Expected result in this environment:
/usr/local/cuda-13.2
Check key CUDA libraries:
ls -lh \
/usr/local/cuda/include/cuda.h \
/usr/local/cuda/lib64/libcudart.so \
/usr/local/cuda/lib64/libcublas.so
cuDNN 9 development files were installed through the CUDA repository:
dpkg -l | grep -Ei 'cudnn|libcudnn' || true
Installed packages:
libcudnn9-cuda-13 9.20.0.48-1
libcudnn9-dev-cuda-13 9.20.0.48-1
libcudnn9-headers-cuda-13 9.20.0.48-1
cuDNN headers and libraries are located at:
/usr/include/x86_64-linux-gnu/cudnn.h
/usr/include/x86_64-linux-gnu/cudnn_version.h
/usr/lib/x86_64-linux-gnu/libcudnn.so
Check via:
dpkg -L libcudnn9-headers-cuda-13 libcudnn9-dev-cuda-13 \
| grep -Ei 'include|header|cudnn|\.h$' \
| sort
NCCL was not installed in this environment:
dpkg -l | grep -Ei 'nccl|libnccl' || true
find /usr /opt -type f \( -name 'nccl.h' -o -name 'libnccl.so*' \) 2>/dev/null | sort
Because the build policy is source-only and refuses precompiled binary packages, USE_NCCL was kept OFF for the TVM build. NCCL can be built later from NVIDIA’s NCCL source tree if required.
6.2 Patch TVM CMake Configuration
Back up the original TVM CMake configuration:
cd ~/repos/mlc-llm/3rdparty/tvm
cp -av cmake/config.cmake cmake/config.cmake.bak.$(date +%Y%m%d_%H%M%S)
Apply the local configuration:
cd ~/repos/mlc-llm/3rdparty/tvm
python3 - <<'PY'
from pathlib import Path
p = Path("cmake/config.cmake")
s = p.read_text()
repls = {
"set(USE_CUDA OFF)": "set(USE_CUDA /usr/local/cuda)",
"set(USE_NCCL OFF)": "set(USE_NCCL OFF)",
"set(USE_ROCM OFF)": "set(USE_ROCM OFF)",
"set(USE_OPENCL OFF)": "set(USE_OPENCL OFF)",
"set(USE_METAL OFF)": "set(USE_METAL OFF)",
"set(USE_VULKAN OFF)": "set(USE_VULKAN OFF)",
"set(USE_LLVM OFF)": "set(USE_LLVM /usr/bin/llvm-config-18)",
"set(USE_CUDNN OFF)": "set(USE_CUDNN ON)",
"set(USE_CUDNN_FRONTEND OFF)": "set(USE_CUDNN_FRONTEND OFF)",
"set(USE_CUBLAS OFF)": "set(USE_CUBLAS ON)",
"set(TVM_FFI_USE_LIBBACKTRACE ON)": "set(TVM_FFI_USE_LIBBACKTRACE ON)",
"set(USE_CUTLASS OFF)": "set(USE_CUTLASS ON)",
"set(USE_FLASHINFER OFF)": "set(USE_FLASHINFER ON)",
}
missing = []
for old, new in repls.items():
if old not in s:
missing.append(old)
else:
s = s.replace(old, new)
if missing:
print("MISSING_PATTERNS:")
for x in missing:
print(x)
raise SystemExit(1)
extra = """
# ---- Local explicit paths for CUDA/cuDNN on this H800 CUDA 13.2 environment ----
set(CUDNN_INCLUDE_DIR /usr/include/x86_64-linux-gnu)
set(CUDNN_LIBRARY /usr/lib/x86_64-linux-gnu/libcudnn.so)
"""
if "Local explicit paths for CUDA/cuDNN" not in s:
s += extra
p.write_text(s)
PY
Verify the relevant options:
grep -nE 'USE_CUDA|USE_NCCL|USE_LLVM|USE_CUDNN|USE_CUBLAS|USE_CUTLASS|USE_FLASHINFER|TVM_FFI_USE_LIBBACKTRACE|CUDNN_INCLUDE_DIR|CUDNN_LIBRARY|USE_OPENCL|USE_METAL|USE_ROCM|USE_VULKAN' \
cmake/config.cmake
Expected important entries:
set(USE_CUDA /usr/local/cuda)
set(USE_NCCL OFF)
set(USE_ROCM OFF)
set(USE_OPENCL OFF)
set(USE_METAL OFF)
set(USE_VULKAN OFF)
set(USE_LLVM /usr/bin/llvm-config-18)
set(USE_CUDNN ON)
set(USE_CUDNN_FRONTEND OFF)
set(USE_CUBLAS ON)
set(TVM_FFI_USE_LIBBACKTRACE ON)
set(USE_CUTLASS ON)
set(USE_FLASHINFER ON)
set(CUDNN_INCLUDE_DIR /usr/include/x86_64-linux-gnu)
set(CUDNN_LIBRARY /usr/lib/x86_64-linux-gnu/libcudnn.so)
4. Configuring TVM
7. Configuring TVM for NVIDIA H800 / Hopper
The first CMake configure attempt showed that, if cmake/config.cmake is not copied into the build directory, TVM silently keeps important options disabled:
USE_CUDA=OFF
USE_LLVM=OFF
USE_CUDNN=OFF
USE_CUBLAS=OFF
USE_CUTLASS=OFF
The correct procedure is:
cd ~/repos/mlc-llm/3rdparty/tvm
rm -rf build
mkdir -p build
cp -av cmake/config.cmake build/config.cmake
cd build
cmake .. \
-G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_COMPILER=/usr/bin/gcc \
-DCMAKE_CXX_COMPILER=/usr/bin/g++ \
-DCMAKE_CUDA_ARCHITECTURES=90
The explicit CUDA architecture is important. H800 is a Hopper GPU and should target:
sm_90 / compute_90
Without this override, CMake detected or defaulted to:
CMAKE_CUDA_ARCHITECTURES=75
which is inappropriate for H800.
A successful configure should include messages similar to:
-- Custom CUDA_PATH=/usr/local/cuda
-- Found CUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda
-- Found CUDA_CUDART_LIBRARY=/usr/local/cuda/lib64/libcudart.so
-- Found CUDA_CUDNN_LIBRARY=/usr/lib/x86_64-linux-gnu/libcudnn.so
-- Found CUDA_CUBLAS_LIBRARY=/usr/local/cuda/lib64/libcublas.so
-- Found CUDA_CUBLASLT_LIBRARY=/usr/local/cuda/lib64/libcublasLt.so
-- Build with CUDA support
-- Build with cuDNN support
-- Build with cuBLAS support
-- Found CMAKE_CUDA_ARCHITECTURES=90
-- Use llvm-config=/usr/bin/llvm-config-18
-- Found TVM_LLVM_VERSION=181
-- Build with LLVM
-- Build with CUTLASS
Verify CUDA architecture in build.ninja:
grep -nE 'compute_90|sm_90' build.ninja | sed -n '1,80p'
Expected form:
--generate-code=arch=compute_90,code=[compute_90,sm_90]
8. Patching libflash_attn CUDA Architecture from sm_80 to sm_90
After configuring TVM with CMAKE_CUDA_ARCHITECTURES=90, build.ninja still contained some compute_80 / sm_80 flags.
The source was identified as:
3rdparty/libflash_attn/src/CMakeLists.txt
Relevant original content:
set(CMAKE_CUDA_FLAGS "${CMAKE_CUDA_FLAGS} --expt-relaxed-constexpr --expt-relaxed-constexpr --use_fast_math -t 8 \
-gencode=arch=compute_80,code=\\\"sm_80,compute_80\\\" \
")
set_target_properties(flash_attn PROPERTIES CUDA_ARCHITECTURES "80")
Patch it for H800 / Hopper:
cd ~/repos/mlc-llm/3rdparty/tvm
cp -av 3rdparty/libflash_attn/src/CMakeLists.txt \
3rdparty/libflash_attn/src/CMakeLists.txt.bak.h800_sm90.$(date +%Y%m%d_%H%M%S)
python3 - <<'PY'
from pathlib import Path
p = Path("3rdparty/libflash_attn/src/CMakeLists.txt")
s = p.read_text()
repls = {
"compute_80": "compute_90",
"sm_80": "sm_90",
'CUDA_ARCHITECTURES "80"': 'CUDA_ARCHITECTURES "90"',
}
for old, new in repls.items():
if old not in s:
print(f"WARNING: pattern not found: {old}")
s = s.replace(old, new)
p.write_text(s)
PY
Reconfigure TVM after this patch:
cd ~/repos/mlc-llm/3rdparty/tvm
rm -rf build
mkdir -p build
cp -av cmake/config.cmake build/config.cmake
cd build
cmake .. \
-G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_COMPILER=/usr/bin/gcc \
-DCMAKE_CXX_COMPILER=/usr/bin/g++ \
-DCMAKE_CUDA_ARCHITECTURES=90
Verify that compute_80 / sm_80 no longer appear in generated build rules:
grep -nE 'compute_80|sm_80' build.ninja | sed -n '1,80p' || true
grep -nE 'compute_90|sm_90' build.ninja | sed -n '1,80p' || true
Expected result:
No compute_80 / sm_80 compile flags remain.
compute_90 / sm_90 compile flags are present.
Note: several source filenames still contain sm80, for example:
flash_fwd_hdim128_fp16_sm80.cu
This is only a filename. The actual NVCC target architecture is controlled by -gencode and CUDA_ARCHITECTURES.
5. Building TVM and its bindings
9. Building TVM Native Libraries
Build TVM with Ninja. A conservative parallelism level is used first to keep failures easier to debug:
cd ~/repos/mlc-llm/3rdparty/tvm/build
ninja -j64
Successful final link messages include:
[634/634] Linking CXX shared library libtvm.so
Generated libraries include:
./lib/libtvm_ffi_testing.so
./lib/libtvm_ffi.so
./libtvm_runtime.so
./libtvm.so
In this run:
libtvm_runtime.so 6.5M
libtvm.so 86M
libtvm_ffi.so 2.0M
Check linked dependencies:
cd ~/repos/mlc-llm/3rdparty/tvm/build
ldd ./libtvm.so | grep -Ei 'cuda|cudnn|cublas|llvm|stdc\+\+|not found' || true
ldd ./libtvm_runtime.so | grep -Ei 'cuda|cudnn|cublas|llvm|stdc\+\+|not found' || true
Expected important dependencies for libtvm.so:
libLLVM-18.so.18.1
libcudart.so.13
libcuda.so.1
libcudnn.so.9
libcublas.so.13
libcublasLt.so.13
libstdc++.so.6
Expected important dependencies for libtvm_runtime.so:
libcudart.so.13
libcuda.so.1
libcudnn.so.9
libcublas.so.13
libcublasLt.so.13
libstdc++.so.6
No not found entries should appear.
10. Building and Installing apache-tvm-ffi from Source
After TVM native libraries were built, directly importing TVM from source initially failed:
ModuleNotFoundError: No module named 'tvm_ffi'
The correct tvm_ffi source path is:
~/repos/mlc-llm/3rdparty/tvm/3rdparty/tvm-ffi/python
After adding that path, import proceeded further but failed because package metadata was missing:
importlib.metadata.PackageNotFoundError: No package metadata was found for apache-tvm-ffi
10.1 Local Development Patch for tvm_ffi/libinfo.py
tvm_ffi/libinfo.py attempted to call:
dist = im.distribution(package)
before entering its fallback search logic. In a source-tree development build, apache-tvm-ffi package metadata may not exist yet, even though libtvm_ffi.so has already been built.
Patch 3rdparty/tvm-ffi/python/tvm_ffi/libinfo.py so that PackageNotFoundError falls through to the existing fallback search:
cd ~/repos/mlc-llm/3rdparty/tvm
cp -av 3rdparty/tvm-ffi/python/tvm_ffi/libinfo.py \
3rdparty/tvm-ffi/python/tvm_ffi/libinfo.py.bak.dev_fallback.$(date +%Y%m%d_%H%M%S)
python3 - <<'PY'
from pathlib import Path
p = Path("3rdparty/tvm-ffi/python/tvm_ffi/libinfo.py")
s = p.read_text()
old = ''' # Use `importlib.metadata` is the most reliable way to find package data files
dist: im.PathDistribution = im.distribution(package) # ty: ignore[invalid-assignment]
record = dist.read_text("RECORD") or ""
for line in record.splitlines():
partial_path, *_ = line.split(",")
if partial_path.endswith(lib_dll_names):
try:
path = (dist._path.parent / partial_path).resolve()
except OSError:
continue
if path.name in lib_dll_names:
return path
'''
new = ''' # Use `importlib.metadata` is the most reliable way to find package data files.
# In a source-tree development build, package metadata may not exist yet.
# In that case, continue to the fallback search below.
try:
dist: im.PathDistribution = im.distribution(package) # ty: ignore[invalid-assignment]
except im.PackageNotFoundError:
dist = None
if dist is not None:
record = dist.read_text("RECORD") or ""
for line in record.splitlines():
partial_path, *_ = line.split(",")
if partial_path.endswith(lib_dll_names):
try:
path = (dist._path.parent / partial_path).resolve()
except OSError:
continue
if path.name in lib_dll_names:
return path
'''
if old not in s:
raise SystemExit("ERROR: target block not found; do not patch blindly")
p.write_text(s.replace(old, new))
PY
This patch allows tvm_ffi to continue searching:
$PROJECT_ROOT/build/lib
$PROJECT_ROOT/lib
$DEV_TOP/build/lib
$DEV_TOP/lib
LD_LIBRARY_PATH
PATH
instead of failing immediately when Python package metadata is absent.
10.2 Install Python Build Tools from Source
tvm_ffi.core is a Cython native extension. It is not produced by the TVM C++ Ninja build alone.
The source files include:
3rdparty/tvm-ffi/python/tvm_ffi/cython/core.pyx
3rdparty/tvm-ffi/python/tvm_ffi/core.pyi
The required Python build tools are:
scikit-build-core
Cython
setuptools-scm
setuptools
ninja
In this environment, scikit-build-core, tomli, packaging, and ninja were already installed in user-site, but Cython and setuptools-scm were missing.
Correct import name for scikit-build-core:
import scikit_build_core
not:
import skbuild_core
Verify Python build tools:
python3 - <<'PY'
mods = [
"scikit_build_core",
"Cython",
"setuptools_scm",
"tomli",
"packaging",
"setuptools",
"wheel",
"ninja",
]
for m in mods:
try:
mod = __import__(m)
print(f"{m}: OK -> {getattr(mod, '__version__', 'unknown')} -> {getattr(mod, '__file__', 'built-in')}")
except Exception as e:
print(f"{m}: MISSING/ERROR -> {type(e).__name__}: {e}")
PY
Upgrade setuptools from source:
python3 -m pip install \
--user \
--no-binary=:all: \
--upgrade \
"setuptools>=77.0.3"
In this run, setuptools was built from source and installed as:
setuptools 82.0.1
/home/ailven.liu/.local/lib/python3.10/site-packages/setuptools
Warning observed:
torch 2.11.0 requires setuptools<82, but you have setuptools 82.0.1 which is incompatible.
This warning was not blocking for TVM/tvm-ffi. If later Torch-related tooling fails, isolate this stack in a virtual environment or adjust setuptools.
Install Cython from source:
python3 -m pip install \
--user \
--no-binary=:all: \
--upgrade \
"cython>=3.0"
Verify:
python3 - <<'PY'
import Cython
print("Cython:", Cython.__version__, Cython.__file__)
PY
Expected result in this run:
Cython 3.2.4
/home/ailven.liu/.local/lib/python3.10/site-packages/Cython
Installing setuptools-scm directly from source initially failed because its build isolation environment lacked tomli while building vcs-versioning.
The successful procedure was:
python3 -m pip install \
--user \
--no-binary=:all: \
--no-build-isolation \
--upgrade \
"vcs-versioning"
Then:
python3 -m pip install \
--user \
--no-binary=:all: \
--no-build-isolation \
--upgrade \
"setuptools-scm"
Verify:
python3 - <<'PY'
import vcs_versioning
import setuptools_scm
print("vcs_versioning:", getattr(vcs_versioning, "__version__", "unknown"), vcs_versioning.__file__)
print("setuptools_scm:", getattr(setuptools_scm, "__version__", "unknown"), setuptools_scm.__file__)
PY
10.3 Editable Source Build of apache-tvm-ffi
Build and install apache-tvm-ffi from the local source tree in editable mode:
cd ~/repos/mlc-llm/3rdparty/tvm/3rdparty/tvm-ffi
export LD_LIBRARY_PATH=$PWD/../../build/lib:$PWD/../../build:/usr/local/cuda/lib64:${LD_LIBRARY_PATH:-}
python3 -m pip install \
--user \
--no-build-isolation \
--no-binary=:all: \
-e .
Although pip reports a locally created wheel during editable installation:
Created wheel for apache-tvm-ffi: ...
this is a PEP 517 editable-install build artefact produced from the local source tree. It is not a downloaded precompiled binary package.
Verify:
python3 - <<'PY'
import tvm_ffi
import tvm_ffi.core
import importlib.metadata as im
print("tvm_ffi:", tvm_ffi.__file__)
print("tvm_ffi.core:", tvm_ffi.core.__file__)
print("dist version:", im.version("apache-tvm-ffi"))
PY
Successful result:
tvm_ffi: /home/ailven.liu/repos/mlc-llm/3rdparty/tvm/3rdparty/tvm-ffi/python/tvm_ffi/__init__.py
tvm_ffi.core: /home/ailven.liu/.local/lib/python3.10/site-packages/tvm_ffi/core.cpython-310-x86_64-linux-gnu.so
dist version: 0.1.11.dev0+g1fed0ae04.d20260505
11. Verifying TVM Python Binding
Set the TVM environment for the current shell:
cd ~/repos/mlc-llm/3rdparty/tvm
export TVM_HOME=$PWD
export PYTHONPATH=$TVM_HOME/python
export LD_LIBRARY_PATH=$TVM_HOME/build:$TVM_HOME/build/lib:/usr/local/cuda/lib64:${LD_LIBRARY_PATH:-}
Verify tvm, tvm_ffi, and CUDA visibility:
python3 - <<'PY'
import os
import tvm
import tvm_ffi
import tvm_ffi.core
print("tvm module:", tvm.__file__)
print("tvm_ffi module:", tvm_ffi.__file__)
print("tvm_ffi.core:", tvm_ffi.core.__file__)
from tvm.support import libinfo
info = libinfo()
for k in sorted(info):
if k.startswith("USE_") or k in ("GIT_COMMIT_HASH", "LLVM_VERSION"):
print(f"{k}={info[k]}")
print("cuda exist:", tvm.cuda().exist)
print("cuda device:", tvm.cuda(0))
PY
Successful result:
tvm module: /home/ailven.liu/repos/mlc-llm/3rdparty/tvm/python/tvm/__init__.py
tvm_ffi module: /home/ailven.liu/repos/mlc-llm/3rdparty/tvm/3rdparty/tvm-ffi/python/tvm_ffi/__init__.py
tvm_ffi.core: /home/ailven.liu/.local/lib/python3.10/site-packages/tvm_ffi/core.cpython-310-x86_64-linux-gnu.so
GIT_COMMIT_HASH=c2028da5ac03055f30a52b492260f37b9650ac03
LLVM_VERSION=18.1.8
USE_CUBLAS=ON
USE_CUDA=/usr/local/cuda
USE_CUDNN=ON
USE_CUTLASS=ON
USE_LLVM=/usr/bin/llvm-config-18
USE_NCCL=OFF
USE_RPC=ON
USE_THREADS=ON
cuda exist: True
cuda device: cuda:0
At this point, the source-built TVM stack is functional from Python and can see the CUDA device.
12. Persisting the TVM Source Environment
To avoid manually exporting environment variables every time, create a sourceable environment script:
cat > ~/repos/mlc-llm/env-tvm-source.sh <<'EOF'
# Source-built TVM environment for this mlc-llm checkout.
# Usage:
# source ~/repos/mlc-llm/env-tvm-source.sh
export MLC_LLM_HOME="$HOME/repos/mlc-llm"
export TVM_HOME="$MLC_LLM_HOME/3rdparty/tvm"
export PYTHONPATH="$TVM_HOME/python:$MLC_LLM_HOME/python:${PYTHONPATH:-}"
export LD_LIBRARY_PATH="$TVM_HOME/build:$TVM_HOME/build/lib:/usr/local/cuda/lib64:${LD_LIBRARY_PATH:-}"
export PATH="$HOME/.local/bin:/usr/local/cuda/bin:${PATH:-}"
EOF
chmod +x ~/repos/mlc-llm/env-tvm-source.sh
Use it with:
source ~/repos/mlc-llm/env-tvm-source.sh
Verify from a fresh subshell:
bash -lc '
source ~/repos/mlc-llm/env-tvm-source.sh
python3 - <<PY
import tvm
print("tvm:", tvm.__file__)
print("cuda:", tvm.cuda().exist)
PY
'
Expected result:
tvm: /home/ailven.liu/repos/mlc-llm/3rdparty/tvm/python/tvm/__init__.py
cuda: True
6. Building the remaining dependencies
13. Continuing from TVM to XGrammar
After the source-built TVM stack had been verified, the next stage was to build and install XGrammar from the mlc-llm pinned submodule.
The XGrammar source tree is located at:
~/repos/mlc-llm/3rdparty/xgrammar
The checked-out XGrammar commit in this environment is:
d4f57c440f3da8e7330a1e5d50bba9c31f9433ea
Its dlpack submodule was already initialised:
3rdparty/dlpack: bbd2f4d32427e548797929af08cfe2a9cbb3cf12
The relevant build entry points are:
~/repos/mlc-llm/3rdparty/xgrammar/CMakeLists.txt
~/repos/mlc-llm/3rdparty/xgrammar/cpp/pybind/CMakeLists.txt
~/repos/mlc-llm/3rdparty/xgrammar/python/setup.py
~/repos/mlc-llm/3rdparty/xgrammar/python/pyproject.toml
A key observation is that python/setup.py does not invoke CMake by itself. It expects the native binding library to have already been built and placed under:
~/repos/mlc-llm/3rdparty/xgrammar/python/xgrammar/xgrammar_bindings.*.so
Therefore, the correct build order is:
- configure and build the native C++/pybind11 extension with CMake/Ninja;
- install the Python package from the local source tree.
The available toolchain used for XGrammar was:
cmake: 3.22.1
ninja: 1.13.0.git.kitware.jobserver-pipe-1
gcc/g++: 11.4.0
Python: 3.10.12
pybind11: 3.0.4
torch: 2.11.0+cu130
The Torch CMake prefix path was available at:
~/.local/lib/python3.10/site-packages/torch/share/cmake
The pybind11 CMake directory was available at:
~/.local/lib/python3.10/site-packages/pybind11/share/cmake/pybind11
13.1 Configuring XGrammar
Configure XGrammar with CMake:
source ~/repos/mlc-llm/env-tvm-source.sh
cd ~/repos/mlc-llm/3rdparty/xgrammar
rm -rf build
mkdir -p build
cmake -S . -B build \
-G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_COMPILER=/usr/bin/gcc \
-DCMAKE_CXX_COMPILER=/usr/bin/g++
The configure stage completed successfully and generated Ninja build files under:
~/repos/mlc-llm/3rdparty/xgrammar/build
During configuration, CMake emitted the following non-blocking Torch warning:
library kineto not found
This warning did not prevent configuration. Torch was still found successfully:
Found Torch: ~/.local/lib/python3.10/site-packages/torch/lib/libtorch.so
13.2 Building XGrammar Native Binding
Build the native binding:
source ~/repos/mlc-llm/env-tvm-source.sh
cd ~/repos/mlc-llm/3rdparty/xgrammar
cmake --build build -j64
The build completed successfully and produced:
~/repos/mlc-llm/3rdparty/xgrammar/python/xgrammar/xgrammar_bindings.cpython-310-x86_64-linux-gnu.so
13.3 Installing XGrammar from Local Source
Install the Python package from the local source tree in editable mode:
source ~/repos/mlc-llm/env-tvm-source.sh
cd ~/repos/mlc-llm/3rdparty/xgrammar/python
python3 -m pip install \
--user \
--no-build-isolation \
--no-binary=:all: \
-e .
The installation completed successfully:
Successfully built xgrammar
Successfully installed xgrammar-0.0.3
Although pip created an editable wheel locally, this was a local PEP 660 editable-install artefact, not a downloaded precompiled binary package:
xgrammar-0.0.3-0.editable-cp310-cp310-linux_x86_64.whl
The final import locations were:
xgrammar:
~/repos/mlc-llm/3rdparty/xgrammar/python/xgrammar/__init__.py
xgrammar_bindings:
~/repos/mlc-llm/3rdparty/xgrammar/python/xgrammar/xgrammar_bindings.cpython-310-x86_64-linux-gnu.so
The visible public API included:
BNFGrammar
BuiltinGrammar
CachedGrammarCompiler
CompiledGrammar
GrammarMatcher
TokenizerInfo
VocabType
At this point, XGrammar was successfully built and installed from local source.
14. Updating the Source Environment Script for XGrammar
The environment script was extended so that XGrammar is explicitly part of the source development environment.
The script is located at:
~/repos/mlc-llm/env-tvm-source.sh
The following variable was added:
export XGRAMMAR_HOME="$MLC_LLM_HOME/3rdparty/xgrammar"
The PYTHONPATH was updated to include:
$XGRAMMAR_HOME/python
The relevant environment entries became:
export MLC_LLM_HOME="$HOME/repos/mlc-llm"
export TVM_HOME="$MLC_LLM_HOME/3rdparty/tvm"
export XGRAMMAR_HOME="$MLC_LLM_HOME/3rdparty/xgrammar"
export PYTHONPATH="$TVM_HOME/python:$MLC_LLM_HOME/python:$XGRAMMAR_HOME/python:${PYTHONPATH:-}"
export LD_LIBRARY_PATH="$TVM_HOME/build:$TVM_HOME/build/lib:/usr/local/cuda/lib64:${LD_LIBRARY_PATH:-}"
export PATH="$HOME/.local/bin:/usr/local/cuda/bin:${PATH:-}"
A fresh shell confirmed that the source-built TVM, XGrammar, and CUDA visibility remained valid.
15. FlashInfer Source State in This Checkout
The mlc-llm/3rdparty directory in this checkout does not contain an independent flashinfer source tree.
The available immediate third-party directories were:
argparse
googletest
stb
tokenizers-cpp
tvm
xgrammar
The only FlashInfer or FlashAttention-related source discovered in the current checkout was under the pinned TVM source tree:
~/repos/mlc-llm/3rdparty/tvm/3rdparty/libflash_attn
The TVM build had already incorporated this internal flash attention component, and the MLC-LLM native libraries later linked against:
libflash_attn.so
libfpA_intB_gemm.so
However, the MLC-LLM Python package metadata declares a Linux dependency on:
flashinfer-python
This appears in:
~/repos/mlc-llm/pyproject.toml
~/repos/mlc-llm/python/requirements.txt
The relevant dependency entry is:
flashinfer-python; sys_platform == 'linux'
The current environment did not have flashinfer-python installed:
flashinfer-python: NOT_FOUND
The MLC-LLM Python source tree did not contain a direct Python import of FlashInfer, such as:
import flashinfer
from flashinfer import ...
Instead, FlashInfer references were concentrated in compiler/runtime paths and optimisation flags, including:
python/mlc_llm/compiler_pass
python/mlc_llm/interface/compiler_flags.py
python/mlc_llm/op/extern.py
python/mlc_llm/op/attention.py
Therefore, flashinfer-python was deliberately not installed at this stage. To avoid downloading a precompiled wheel, MLC-LLM itself was later installed with dependency resolution disabled.
For model compilation before building flashinfer-python from source, FlashInfer can be explicitly disabled via the optimisation string, for example:
--opt "O2;flashinfer=0"
16. Building tokenizers-cpp from Source
The tokenizers-cpp source tree is located at:
~/repos/mlc-llm/3rdparty/tokenizers-cpp
The checked-out commit was:
34885cfd7b9ef27b859c28a41e71413dd31926f5
The following submodules were already initialised:
msgpack: 092bc69b6e815980bce7808595c914dd3a29f905
sentencepiece: 11051e3b73b3a6222a52acd720e39805dc7545ab
The project is a CMake-based C++ library with a Rust/Cargo component. It produces static libraries rather than a Python package.
Important build characteristics:
CMake project: tokenizers_cpp
Rust crate: tokenizers-c
Rust crate type: staticlib
SentencePiece tokenizer: enabled
msgpack: local submodule
sentencepiece: local submodule
The Rust dependency declaration is located at:
~/repos/mlc-llm/3rdparty/tokenizers-cpp/rust/Cargo.toml
The key dependencies were:
tokenizers = { version = "0.21.2", default-features = false, features = ["onig"] }
serde = { version = "1.0", features = [ "derive" ] }
serde_json = "1.0"
The Rust toolchain available in this environment was:
rustc 1.95.0
cargo 1.95.0
Cargo was configured to use the USTC crates.io mirror:
[source.crates-io]
replace-with = "ustc"
[source.ustc]
registry = "https://mirrors.ustc.edu.cn/crates.io-index"
[net]
git-fetch-with-cli = true
16.1 Generating Cargo.lock
The repository did not include rust/Cargo.lock. Generate it first:
source ~/repos/mlc-llm/env-tvm-source.sh
cd ~/repos/mlc-llm/3rdparty/tokenizers-cpp/rust
cargo generate-lockfile
This successfully generated:
~/repos/mlc-llm/3rdparty/tokenizers-cpp/rust/Cargo.lock
Cargo locked 78 packages. The resolved tokenizers crate version was:
tokenizers v0.21.4
The generated Cargo.lock is ignored by the repository’s Git configuration and appears as an ignored file:
!! rust/Cargo.lock
16.2 Configuring tokenizers-cpp
Configure the build:
source ~/repos/mlc-llm/env-tvm-source.sh
cd ~/repos/mlc-llm/3rdparty/tokenizers-cpp
rm -rf build
mkdir -p build
cmake -S . -B build \
-G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_COMPILER=/usr/bin/gcc \
-DCMAKE_CXX_COMPILER=/usr/bin/g++ \
-DMLC_ENABLE_SENTENCEPIECE_TOKENIZER=ON
The configure stage completed successfully and wrote build files to:
~/repos/mlc-llm/3rdparty/tokenizers-cpp/build
16.3 Building tokenizers-cpp
Build the static libraries:
source ~/repos/mlc-llm/env-tvm-source.sh
cd ~/repos/mlc-llm/3rdparty/tokenizers-cpp
cmake --build build -j64
The build completed successfully and generated:
~/repos/mlc-llm/3rdparty/tokenizers-cpp/build/libtokenizers_c.a
~/repos/mlc-llm/3rdparty/tokenizers-cpp/build/libtokenizers_cpp.a
~/repos/mlc-llm/3rdparty/tokenizers-cpp/build/release/libtokenizers_c.a
~/repos/mlc-llm/3rdparty/tokenizers-cpp/build/sentencepiece/src/libsentencepiece.a
At this point, tokenizers-cpp had been successfully built from local source.
7. Installing and validating MLC-LLM
17. Installing MLC-LLM from Local Source
After TVM, apache-tvm-ffi, XGrammar, and tokenizers-cpp had been built, MLC-LLM itself was installed from the local checkout.
The source tree is located at:
~/repos/mlc-llm
The checked-out MLC-LLM commit was:
d1ea69a87280e821611643958bfec385b62dafd3
The project metadata declares the following relevant dependencies:
apache-tvm-ffi
flashinfer-python; sys_platform == 'linux'
requests
torch
tqdm
transformers
Since flashinfer-python was not available as a local source tree and should not be pulled as a prebuilt wheel, MLC-LLM was installed with dependency resolution disabled:
source ~/repos/mlc-llm/env-tvm-source.sh
cd ~/repos/mlc-llm
python3 -m pip install \
--user \
--no-build-isolation \
--no-binary=:all: \
--no-deps \
-e .
The installation completed successfully:
Successfully built mlc_llm
Successfully installed mlc_llm-0.20.0.dev0
The editable wheel created during installation was a local build artefact:
mlc_llm-0.20.0.dev0-py3-none-linux_x86_64.whl
No flashinfer-python wheel was downloaded.
The installed package was confirmed to point back to the local source tree:
mlc_llm:
~/repos/mlc-llm/python/mlc_llm/__init__.py
distribution:
mlc_llm 0.20.0.dev0
editable: true
The current editable install path did not generate a mlc_llm console script under:
~/.local/bin
However, the module CLI is available and should be used as:
python3 -m mlc_llm ...
For example:
python3 -m mlc_llm --help
python3 -m mlc_llm compile --help
python3 -m mlc_llm chat --help
python3 -m mlc_llm serve --help
18. MLC-LLM Native Runtime Libraries
The MLC-LLM native runtime libraries were present under the local build directory:
~/repos/mlc-llm/build/libmlc_llm.so
~/repos/mlc-llm/build/libmlc_llm_module.so
They were discoverable through mlc_llm.libinfo:
mlc_llm:
~/repos/mlc-llm/build/libmlc_llm.so
mlc_llm_module:
~/repos/mlc-llm/build/libmlc_llm_module.so
The dynamic linker resolved their dependencies successfully, including:
libtvm.so
libtvm_runtime.so
libtvm_ffi.so
libflash_attn.so
libfpA_intB_gemm.so
libcudart.so.13
libcuda.so.1
libcudnn.so.9
libcublas.so.13
libcublasLt.so.13
No missing shared-library dependency was observed.
Representative resolved paths included:
libtvm.so:
~/repos/mlc-llm/3rdparty/tvm/build/libtvm.so
libtvm_runtime.so:
~/repos/mlc-llm/3rdparty/tvm/build/libtvm_runtime.so
libtvm_ffi.so:
~/repos/mlc-llm/3rdparty/tvm/build/lib/libtvm_ffi.so
libflash_attn.so:
~/repos/mlc-llm/3rdparty/tvm/build/3rdparty/libflash_attn/src/libflash_attn.so
libfpA_intB_gemm.so:
~/repos/mlc-llm/3rdparty/tvm/build/3rdparty/cutlass_fpA_intB_gemm/cutlass_kernels/libfpA_intB_gemm.so
19. MLC-LLM Python and Runtime Smoke Validation
The following Python modules loaded successfully from the local source-built stack:
tvm
tvm_ffi
tvm_ffi.core
xgrammar
xgrammar.xgrammar_bindings
mlc_llm
mlc_llm.base
mlc_llm.support
mlc_llm.interface.compiler_flags
mlc_llm.interface.compile
mlc_llm.op.extern
mlc_llm.op.attention
The resolved source paths were:
tvm:
~/repos/mlc-llm/3rdparty/tvm/python/tvm/__init__.py
tvm_ffi:
~/repos/mlc-llm/3rdparty/tvm/3rdparty/tvm-ffi/python/tvm_ffi/__init__.py
tvm_ffi.core:
~/.local/lib/python3.10/site-packages/tvm_ffi/core.cpython-310-x86_64-linux-gnu.so
xgrammar:
~/repos/mlc-llm/3rdparty/xgrammar/python/xgrammar/__init__.py
xgrammar_bindings:
~/repos/mlc-llm/3rdparty/xgrammar/python/xgrammar/xgrammar_bindings.cpython-310-x86_64-linux-gnu.so
mlc_llm:
~/repos/mlc-llm/python/mlc_llm/__init__.py
CUDA was visible through TVM:
cuda: True
The MLC-LLM module CLI loaded successfully:
python3 -m mlc_llm --help
The following subcommands also loaded their help pages successfully without traceback:
compile
convert_weight
gen_config
chat
serve
package
The compile command exposed the expected model compilation interface, including:
--quantization
--model-type
--device
--host
--opt
--output
--overrides
--debug-dump
The compiler option string can be used to control optimisation flags, for example:
--opt "O2;flashinfer=0"
This is useful while flashinfer-python remains intentionally uninstalled.
20. Preventing Environment Variable Duplication
The first version of env-tvm-source.sh prepended paths directly. Repeatedly sourcing it caused PATH, PYTHONPATH, and LD_LIBRARY_PATH to grow with duplicate entries.
The script was rewritten to be idempotent.
The final environment script is:
# Source-built MLC-LLM environment for this checkout.
# Usage:
# source ~/repos/mlc-llm/env-tvm-source.sh
#
# This script is intentionally sourceable and idempotent.
# It does not modify shell startup files.
export MLC_LLM_HOME="$HOME/repos/mlc-llm"
export TVM_HOME="$MLC_LLM_HOME/3rdparty/tvm"
export XGRAMMAR_HOME="$MLC_LLM_HOME/3rdparty/xgrammar"
export TOKENIZERS_CPP_HOME="$MLC_LLM_HOME/3rdparty/tokenizers-cpp"
_path_prepend_once() {
case ":${PATH:-}:" in
*":$1:"*) ;;
*) export PATH="$1:${PATH:-}" ;;
esac
}
_pythonpath_prepend_once() {
case ":${PYTHONPATH:-}:" in
*":$1:"*) ;;
*) export PYTHONPATH="$1:${PYTHONPATH:-}" ;;
esac
}
_ldpath_prepend_once() {
case ":${LD_LIBRARY_PATH:-}:" in
*":$1:"*) ;;
*) export LD_LIBRARY_PATH="$1:${LD_LIBRARY_PATH:-}" ;;
esac
}
_pythonpath_prepend_once "$XGRAMMAR_HOME/python"
_pythonpath_prepend_once "$MLC_LLM_HOME/python"
_pythonpath_prepend_once "$TVM_HOME/python"
_ldpath_prepend_once "$TVM_HOME/build/lib"
_ldpath_prepend_once "$TVM_HOME/build"
_ldpath_prepend_once "/usr/local/cuda/lib64"
_path_prepend_once "/usr/local/cuda/bin"
_path_prepend_once "$HOME/.local/bin"
unset -f _path_prepend_once
unset -f _pythonpath_prepend_once
unset -f _ldpath_prepend_once
The script passed shell syntax validation:
bash -n ~/repos/mlc-llm/env-tvm-source.sh
Repeatedly sourcing the script no longer duplicates the relevant paths.
The final effective variables are:
MLC_LLM_HOME=~/repos/mlc-llm
TVM_HOME=~/repos/mlc-llm/3rdparty/tvm
XGRAMMAR_HOME=~/repos/mlc-llm/3rdparty/xgrammar
TOKENIZERS_CPP_HOME=~/repos/mlc-llm/3rdparty/tokenizers-cpp
The script remains local and does not modify .bashrc or any system-wide shell configuration.
21. Verifying That No Critical Prebuilt Package Was Accidentally Installed
The key installed Python distributions were:
apache-tvm-ffi:
0.1.11.dev0+g1fed0ae04.d20260505
editable local source:
~/repos/mlc-llm/3rdparty/tvm/3rdparty/tvm-ffi
xgrammar:
0.0.3
editable local source:
~/repos/mlc-llm/3rdparty/xgrammar/python
mlc_llm:
0.20.0.dev0
editable local source:
~/repos/mlc-llm
The following packages were not installed:
flashinfer-python
apache-tvm
tlcpack
mlc-ai-nightly
mlc-llm-nightly
This confirms that the stack did not accidentally switch to an unrelated TVM wheel or MLC nightly package.
8. Local modifications and build state
22. Local Source Modifications Introduced During the Build
The build required several local source changes. These changes are intentional and should be preserved or documented before any reset or clean checkout.
22.1 TVM CMake Configuration
Modified file:
~/repos/mlc-llm/3rdparty/tvm/cmake/config.cmake
The important local changes are:
USE_CUDA=/usr/local/cuda
USE_LLVM=/usr/bin/llvm-config-18
USE_CUDNN=ON
USE_CUBLAS=ON
USE_CUTLASS=ON
USE_FLASHINFER=ON
CUDNN_INCLUDE_DIR=/usr/include/x86_64-linux-gnu
CUDNN_LIBRARY=/usr/lib/x86_64-linux-gnu/libcudnn.so
22.2 libflash_attn H800 Patch
Modified file:
~/repos/mlc-llm/3rdparty/tvm/3rdparty/libflash_attn/src/CMakeLists.txt
The local patch changed the hard-coded Ampere target to Hopper:
compute_80 / sm_80 -> compute_90 / sm_90
CUDA_ARCHITECTURES "80" -> "90"
This was required for the H800 environment.
22.3 tvm-ffi Source-Development Fallback Patch
Modified file:
~/repos/mlc-llm/3rdparty/tvm/3rdparty/tvm-ffi/python/tvm_ffi/libinfo.py
The patch allows apache-tvm-ffi to continue to its existing fallback search if importlib.metadata.distribution("apache-tvm-ffi") raises PackageNotFoundError.
The purpose is to support a source-tree development build where the native library already exists but Python package metadata may not yet exist.
22.4 Local Backup Files
The following local backup files were created during patching:
~/repos/mlc-llm/3rdparty/tvm/3rdparty/libflash_attn/src/CMakeLists.txt.bak.h800_sm90.20260505_072157
~/repos/mlc-llm/3rdparty/tvm/cmake/config.cmake.bak.20260505_071332
~/repos/mlc-llm/env-tvm-source.sh.bak.20260505_090200
~/repos/mlc-llm/env-tvm-source.sh.bak.idempotent.20260505_085858
These are local recovery artefacts and are not required for normal runtime use.
23. Local Build-State Snapshot
A local build-state summary file was created at:
~/repos/mlc-llm/BUILD_STATE_LOCAL.md
It records:
source roots
verified commits
environment script
built TVM state
apache-tvm-ffi state
XGrammar state
tokenizers-cpp state
MLC-LLM state
FlashInfer state
local patches
known non-blocking issues
fresh-shell smoke test result
The file was also normalised to use standard markdown code fences.
24. Current Completed State
The following parts of the stack have now been successfully built or configured from source:
MLC-LLM source tree:
~/repos/mlc-llm
Pinned TVM source:
~/repos/mlc-llm/3rdparty/tvm
TVM native libraries:
built with Ninja
TVM Python source import:
working
apache-tvm-ffi:
editable local source build installed
tvm_ffi.core native extension:
built from Cython source
XGrammar:
native binding built and Python package installed from editable local source
tokenizers-cpp:
C++ / Rust / SentencePiece static libraries built from local source
MLC-LLM:
editable local source install completed with --no-deps
MLC-LLM native runtime:
libmlc_llm.so and libmlc_llm_module.so available and linkable
CUDA visibility from TVM:
working
LLVM backend:
LLVM 18.1.8
CUDA architecture:
sm_90 / compute_90 for H800
cuDNN:
ON
cuBLAS:
ON
CUTLASS:
ON
NCCL:
OFF
flashinfer-python:
not installed
Important built artefacts:
~/repos/mlc-llm/3rdparty/tvm/build/libtvm.so
~/repos/mlc-llm/3rdparty/tvm/build/libtvm_runtime.so
~/repos/mlc-llm/3rdparty/tvm/build/lib/libtvm_ffi.so
~/.local/lib/python3.10/site-packages/tvm_ffi/core.cpython-310-x86_64-linux-gnu.so
~/repos/mlc-llm/3rdparty/xgrammar/python/xgrammar/xgrammar_bindings.cpython-310-x86_64-linux-gnu.so
~/repos/mlc-llm/3rdparty/tokenizers-cpp/build/libtokenizers_c.a
~/repos/mlc-llm/3rdparty/tokenizers-cpp/build/libtokenizers_cpp.a
~/repos/mlc-llm/3rdparty/tokenizers-cpp/build/sentencepiece/src/libsentencepiece.a
~/repos/mlc-llm/build/libmlc_llm.so
~/repos/mlc-llm/build/libmlc_llm_module.so
~/repos/mlc-llm/env-tvm-source.sh
~/repos/mlc-llm/BUILD_STATE_LOCAL.md
The correct environment entry point is:
source ~/repos/mlc-llm/env-tvm-source.sh
The correct CLI invocation form is:
python3 -m mlc_llm ...
rather than:
mlc_llm ...
because the editable install path did not generate a ~/.local/bin/mlc_llm console script.
A final fresh-shell smoke test passed:
bash -lc '
source ~/repos/mlc-llm/env-tvm-source.sh
python3 - <<PY
import tvm
import tvm_ffi
import tvm_ffi.core
import xgrammar
import xgrammar.xgrammar_bindings
import mlc_llm
print("cuda:", tvm.cuda().exist)
PY
python3 -m mlc_llm --help
'
Successful result:
cuda: True
MLC-LLM CLI help loads successfully
At this point, the local source-built MLC-LLM development stack is functional for Python import, native runtime loading, CLI entry, and subsequent model compilation work. The remaining unresolved item is whether to build flashinfer-python from source or to continue compiling models with FlashInfer explicitly disabled via:
--opt "O2;flashinfer=0"
9. Appendix: session log — FlashInfer source integration
Date: 2026-05-05
Scope
This session continued the source-build setup of the MLC-LLM development stack on the Ubuntu/H800 machine.
The explicit objective was:
Integrate flashinfer-python from source into the existing MLC-LLM source-built environment.
The session did not download model weights, did not run a large model compilation, and did not modify global shell configuration such as .bashrc.
All source repositories remain under:
~/repos
The main MLC-LLM repository remains:
~/repos/mlc-llm
1. Initial Fresh-Shell Validation
A fresh-shell smoke test was run first by sourcing:
source ~/repos/mlc-llm/env-tvm-source.sh
The following components were verified to remain valid in a fresh shell:
TVM
TVM-FFI
XGrammar
MLC-LLM
MLC-LLM native libraries
MLC-LLM CLI
CUDA visibility
Observed paths:
tvm:
/home/ailven.liu/repos/mlc-llm/3rdparty/tvm/python/tvm/__init__.py
tvm_ffi:
/home/ailven.liu/repos/mlc-llm/3rdparty/tvm/3rdparty/tvm-ffi/python/tvm_ffi/__init__.py
tvm_ffi.core:
/home/ailven.liu/.local/lib/python3.10/site-packages/tvm_ffi/core.cpython-310-x86_64-linux-gnu.so
xgrammar:
/home/ailven.liu/repos/mlc-llm/3rdparty/xgrammar/python/xgrammar/__init__.py
xgrammar_bindings:
/home/ailven.liu/repos/mlc-llm/3rdparty/xgrammar/python/xgrammar/xgrammar_bindings.cpython-310-x86_64-linux-gnu.so
mlc_llm:
/home/ailven.liu/repos/mlc-llm/python/mlc_llm/__init__.py
MLC-LLM native libraries were located successfully:
/home/ailven.liu/repos/mlc-llm/build/libmlc_llm.so
/home/ailven.liu/repos/mlc-llm/build/libmlc_llm_module.so
CUDA was visible to TVM:
tvm.cuda().exist: True
At the beginning of the session, FlashInfer was confirmed as absent:
flashinfer-python: NOT_FOUND
2. MLC-LLM Source Inspection for FlashInfer Usage
The current MLC-LLM commit was confirmed:
d1ea69a87280e821611643958bfec385b62dafd3
The local repository state contained expected local changes/untracked state:
m 3rdparty/tvm
?? BUILD_STATE_LOCAL.md
?? env-tvm-source.sh
?? env-tvm-source.sh.bak.*
A local grep over the MLC-LLM and pinned TVM source tree found FlashInfer references in:
pyproject.toml
python/requirements.txt
python/mlc_llm/interface/compiler_flags.py
python/mlc_llm/compiler_pass/dispatch_kv_cache_creation.py
python/mlc_llm/compiler_pass/pipeline.py
python/mlc_llm/op/attention.py
python/mlc_llm/op/extern.py
cpp/serve/function_table.cc
cpp/serve/sampler/gpu_sampler.cc
3rdparty/tvm/python/tvm/relax/backend/cuda/flashinfer.py
3rdparty/tvm/python/tvm/relax/frontend/nn/llm/kv_cache.py
3rdparty/tvm/src/runtime/vm/attn_backend.*
3rdparty/tvm/src/runtime/vm/paged_kv_cache.cc
The MLC-LLM dependency declaration was confirmed as unpinned:
"flashinfer-python; sys_platform == 'linux'"
and:
python/requirements.txt:
flashinfer-python; sys_platform == "linux"
No exact FlashInfer version was pinned by MLC-LLM.
3. FlashInfer API Requirements Identified
The pinned TVM FlashInfer backend requires these Python APIs:
flashinfer.jit.gen_customize_batch_prefill_module
flashinfer.jit.gen_customize_batch_decode_module
flashinfer.jit.gen_batch_mla_module
flashinfer.gemm.gen_gemm_sm100_module
The main H800 attention and paged KV cache paths require the first three APIs.
The flashinfer.gemm.gen_gemm_sm100_module path is only used by the grouped GEMM generation path for compute capability 100 in the current TVM wrapper logic. The current machine is H800 / SM90, so this is not an immediate blocker for H800 FlashInfer attention, prefill, decode, paged KV cache, or MLA.
4. Python, PyTorch, CUDA, and Build Tool State
The Python environment was checked before building FlashInfer.
Observed state:
python3:
/usr/bin/python3
Python:
3.10.12
pip:
22.0.2
setuptools:
82.0.1
wheel:
0.37.1
ninja:
1.13.0
cmake Python package:
NOT_FOUND
python3 -m build:
NOT_FOUND
system cmake:
/usr/bin/cmake
cmake version 3.22.1
system git:
/usr/bin/git
git version 2.34.1
PyTorch/CUDA state:
torch:
2.11.0+cu130
torch.version.cuda:
13.0
torch.cuda.is_available:
True
torch.cuda.device_count:
8
device0:
NVIDIA H800
capability0:
(9, 0)
System CUDA compiler:
/usr/local/cuda/bin/nvcc
nvcc:
CUDA 13.2
V13.2.51
Important compatibility note:
PyTorch is built for CUDA 13.0, while the system nvcc is CUDA 13.2.
This combination was later validated successfully by FlashInfer decode, prefill, and MLA JIT build-and-load probes.
5. FlashInfer Version Selection
The official FlashInfer repository was queried:
https://github.com/flashinfer-ai/flashinfer.git
Remote HEAD was accessible:
HEAD -> refs/heads/main
The following tag was selected:
v0.6.10
Tag commit:
943d0ff0dfdbb5019b9733bd78ac730c5bff6f92
Reason for selecting v0.6.10:
It is a current stable FlashInfer tag.
It exposes the required flashinfer.jit APIs used by the pinned TVM backend.
Its pyproject.toml declares package name flashinfer-python.
It supports Python >=3.10,<4.0.
6. Remote Source API Check Before Clone
Before cloning, the tag contents were inspected through remote source reads and a temporary one-tag fetch.
Confirmed in v0.6.10:
flashinfer.jit.gen_customize_batch_prefill_module: present
flashinfer.jit.gen_customize_batch_decode_module: present
flashinfer.jit.gen_batch_mla_module: present
Confirmed layout:
flashinfer.gemm is a package, not a single flashinfer/gemm.py file.
Relevant files:
flashinfer/gemm/__init__.py
flashinfer/gemm/gemm_base.py
flashinfer/jit/gemm/__init__.py
flashinfer/jit/gemm/core.py
The SM100 GEMM generator exists here:
flashinfer.jit.gemm.gen_gemm_sm100_module
but is not exported from:
flashinfer.gemm
Therefore:
from flashinfer.gemm import gen_gemm_sm100_module
is incompatible with FlashInfer v0.6.10.
However:
from flashinfer.jit.gemm import gen_gemm_sm100_module
works.
This was recorded as a known non-blocking compatibility issue for H800.
7. FlashInfer v0.6.10 Requirements Check
The v0.6.10 requirements were inspected:
apache-tvm-ffi>=0.1.6,!=0.1.8,!=0.1.8.post0,<0.2
click
cuda-tile
einops
ninja
numpy
nvidia-cudnn-frontend>=1.13.0
nvidia-cutlass-dsl>=4.4.2
nvidia-ml-py
packaging>=24.2
requests
tabulate
torch
tqdm
The current Python environment already contained these distributions:
apache-tvm-ffi: 0.1.11.dev0+g1fed0ae04.d20260505
click: 8.3.3
cuda-tile: 1.3.0
einops: 0.8.2
ninja: 1.13.0
numpy: 2.2.6
nvidia-cudnn-frontend: 1.23.0
nvidia-cutlass-dsl: 4.4.2
nvidia-ml-py: 13.595.45
packaging: 26.2
requests: 2.33.1
tabulate: 0.10.0
torch: 2.11.0
tqdm: 4.67.3
Non-obvious import checks:
cutlass: IMPORT_OK
cuda: IMPORT_OK
cuda.bindings: IMPORT_OK
cuda_tile: IMPORT_FAIL
cudnn: IMPORT_OK
nvidia.cudnn_frontend: IMPORT_FAIL
pynvml: IMPORT_OK
The failed import checks were not treated as immediate blockers because the distribution metadata was present, and import names do not always match distribution names.
8. FlashInfer Source Clone
FlashInfer was cloned under the unified source root:
~/repos/flashinfer
Clone command shape:
cd ~/repos
git clone --recursive --branch v0.6.10 https://github.com/flashinfer-ai/flashinfer.git flashinfer
Verified state:
repository:
/home/ailven.liu/repos/flashinfer
tag:
v0.6.10
commit:
943d0ff0dfdbb5019b9733bd78ac730c5bff6f92
Submodules:
3rdparty/cccl:
876867684f7fac130e0f5911236e0a92a970d4fd
v3.3.2
3rdparty/cutlass:
da5e086dab31d63815acafdac9a9c5893b1c69e2
v4.4.2
3rdparty/spdlog:
c3aed4b68373955e1cc94307683d44dca1515d2b
The detached HEAD state is expected because v0.6.10 is a tag.
9. FlashInfer Build Metadata Probe
Before installation, PEP 517 editable metadata generation was tested manually.
The source contains:
pyproject.toml
build_backend.py
build_utils.py
requirements.txt
Initially, this generated file did not exist:
flashinfer/_build_meta.py
The metadata probe successfully created it:
"""Build metadata for flashinfer package."""
__version__ = "0.6.10"
__git_version__ = "943d0ff0dfdbb5019b9733bd78ac730c5bff6f92"
Editable metadata output:
flashinfer_python-0.6.10.dist-info
Metadata content confirmed:
Name: flashinfer-python
Version: 0.6.10
Requires-Python: <4.0,>=3.10
No CUDA compilation, wheel download, dependency resolution, or installation was performed during the metadata-only probe.
Warnings observed during metadata generation:
pyproject.toml does not contain setuptools_scm / vcs-versioning tool sections
Pattern 'LICENSE*.txt' did not match any files
These warnings did not block metadata generation.
10. FlashInfer Editable Source Installation
FlashInfer was installed as an editable local source package.
Command used:
cd ~/repos/flashinfer
python3 -m pip install \
--user \
--no-build-isolation \
--no-binary=:all: \
--no-deps \
-e . \
-v
Rationale:
--no-build-isolation:
Preserve the current source-built Python environment, especially local apache-tvm-ffi.
Avoid creating an isolated build environment that might resolve packages from PyPI.
--no-deps:
Prevent pip from downloading, replacing, or resolving dependencies such as torch, cuda-tile,
nvidia-cudnn-frontend, nvidia-cutlass-dsl, or apache-tvm-ffi.
--no-binary=:all:
Preserve the source-build policy and reject prebuilt binary packages.
-e .:
Install the local FlashInfer source tree in editable mode.
Installation result:
Successfully installed flashinfer-python-0.6.10
The install log only contained a locally generated PEP 660 editable wheel:
flashinfer_python-0.6.10-0.editable-py3-none-any.whl
This was not a downloaded binary wheel. It was a local editable-install metadata artifact.
Installed metadata:
flashinfer-python: 0.6.10
location: /home/ailven.liu/.local/lib/python3.10/site-packages
Editable import path:
/home/ailven.liu/repos/flashinfer/flashinfer/__init__.py
11. FlashInfer Python Import and API Validation
After editable installation:
flashinfer.__file__:
/home/ailven.liu/repos/flashinfer/flashinfer/__init__.py
flashinfer.__version__:
0.6.10
flashinfer.jit.__file__:
/home/ailven.liu/repos/flashinfer/flashinfer/jit/__init__.py
Required JIT APIs were available:
flashinfer.jit.gen_customize_batch_prefill_module: True
flashinfer.jit.gen_customize_batch_decode_module: True
flashinfer.jit.gen_batch_mla_module: True
Known non-blocking incompatibility:
from flashinfer.gemm import gen_gemm_sm100_module
failed with:
ImportError: cannot import name 'gen_gemm_sm100_module' from 'flashinfer.gemm'
but:
from flashinfer.jit.gemm import gen_gemm_sm100_module
succeeded.
This remains non-blocking for the current H800 / SM90 path.
12. FlashInfer JIT Spec Generation Probe
Before compiling JIT kernels, FlashInfer JIT spec generation was tested.
Direct FlashInfer JIT generation succeeded for:
prefill
decode
MLA
Generated source locations:
/home/ailven.liu/.cache/flashinfer/0.6.10/90a/generated/...
Observed source counts:
prefill: 10 CUDA source files
decode: 3 CUDA source files
MLA: 3 CUDA source files
The 90a cache path indicates that FlashInfer generated Hopper-oriented sm_90a JIT artifacts for the H800 environment.
No CUDA compilation was performed in this spec-generation probe.
13. FlashInfer Direct JIT Build-and-Load Probes
The following environment variables were set for JIT compilation probes:
export CUDA_HOME=/usr/local/cuda
export CUDA_PATH=/usr/local/cuda
export CUDACXX=/usr/local/cuda/bin/nvcc
export TORCH_CUDA_ARCH_LIST="9.0a"
Decode JIT Probe
Direct FlashInfer decode JIT:
gen_customize_batch_decode_module(...).build_and_load()
Result:
build_and_load: OK
module_type: tvm_ffi.module.Module
Prefill JIT Probe
Direct FlashInfer prefill JIT:
gen_customize_batch_prefill_module(...).build_and_load()
Result:
build_and_load: OK
module_type: tvm_ffi.module.Module
MLA JIT Probe
Direct FlashInfer MLA JIT:
gen_batch_mla_module(...).build_and_load()
Result:
build_and_load: OK
module_type: tvm_ffi.module.Module
Conclusion:
FlashInfer direct JIT generation, CUDA compilation, linking, and runtime loading work for decode, prefill, and MLA on this H800 environment.
Validated toolchain combination:
FlashInfer: 0.6.10
PyTorch: 2.11.0+cu130
torch.version.cuda: 13.0
System nvcc: CUDA 13.2
GPU: NVIDIA H800
GPU capability: (9, 0)
TORCH_CUDA_ARCH_LIST: 9.0a
14. Environment Script Patch for FlashInfer JIT
The local source environment script was patched:
~/repos/mlc-llm/env-tvm-source.sh
A backup was created:
/home/ailven.liu/repos/mlc-llm/env-tvm-source.sh.bak.flashinfer.20260505_100638
The following block was appended:
# FlashInfer JIT CUDA toolchain for Hopper/H800.
# Keep these in the source-able local environment script instead of .bashrc.
# PyTorch is cu130, while system nvcc is CUDA 13.2; this combination has been
# validated by FlashInfer decode/prefill/MLA JIT build_and_load probes.
export CUDA_HOME="${CUDA_HOME:-/usr/local/cuda}"
export CUDA_PATH="${CUDA_PATH:-/usr/local/cuda}"
export CUDACXX="${CUDACXX:-/usr/local/cuda/bin/nvcc}"
export TORCH_CUDA_ARCH_LIST="${TORCH_CUDA_ARCH_LIST:-9.0a}"
Fresh-shell validation after patch:
CUDA_HOME=/usr/local/cuda
CUDA_PATH=/usr/local/cuda
CUDACXX=/usr/local/cuda/bin/nvcc
TORCH_CUDA_ARCH_LIST=9.0a
CUDACXX was executable:
/usr/local/cuda/bin/nvcc
No .bashrc or global system shell configuration was modified.
15. Fresh-Shell Full Stack Validation After FlashInfer Integration
After patching the environment script, a fresh-shell full stack check was run.
Command pattern:
bash -lc '
source ~/repos/mlc-llm/env-tvm-source.sh
...
'
Confirmed environment:
CUDA_HOME=/usr/local/cuda
CUDA_PATH=/usr/local/cuda
CUDACXX=/usr/local/cuda/bin/nvcc
TORCH_CUDA_ARCH_LIST=9.0a
Confirmed stack imports:
torch: 2.11.0+cu130
torch.version.cuda: 13.0
torch.cuda.is_available: True
device0: NVIDIA H800
capability0: (9, 0)
tvm:
/home/ailven.liu/repos/mlc-llm/3rdparty/tvm/python/tvm/__init__.py
mlc_llm:
/home/ailven.liu/repos/mlc-llm/python/mlc_llm/__init__.py
flashinfer:
/home/ailven.liu/repos/flashinfer/flashinfer/__init__.py
flashinfer.__version__:
0.6.10
flashinfer.jit:
/home/ailven.liu/repos/flashinfer/flashinfer/jit/__init__.py
tvm.cuda().exist:
True
MLC-LLM native libraries remained discoverable:
/home/ailven.liu/repos/mlc-llm/build/libmlc_llm.so
/home/ailven.liu/repos/mlc-llm/build/libmlc_llm_module.so
Distribution metadata:
apache-tvm-ffi: 0.1.11.dev0+g1fed0ae04.d20260505
xgrammar: 0.0.3
mlc_llm: 0.20.0.dev0
flashinfer-python: 0.6.10
MLC-LLM CLI still worked:
python3 -m mlc_llm --help
16. TVM FlashInfer Backend Wrapper Validation
After validating direct FlashInfer JIT paths, the actual TVM wrapper path was tested:
from tvm.relax.backend.cuda import flashinfer as tvm_flashinfer
The following TVM wrapper APIs were compiled and loaded successfully:
tvm.relax.backend.cuda.flashinfer.gen_flashinfer_decode_module
tvm.relax.backend.cuda.flashinfer.gen_flashinfer_prefill_module
tvm.relax.backend.cuda.flashinfer.gen_flashinfer_mla_module
Results:
decode_mods_len: 1
prefill_mods_len: 1
mla_mods_len: 1
Returned module types:
<class 'tvm.runtime.module.Module'>
Final wrapper result:
TVM_BACKEND_FLASHINFER_WRAPPER_BUILD: OK
Conclusion:
The TVM backend wrapper path used by MLC-LLM can call FlashInfer, generate CUDA code, compile it, load it, and return TVM runtime modules.
17. MLC-LLM Optimisation Flag Behaviour
The compile optimisation flag behaviour was inspected through Python API and CLI help.
Detected target:
target: cuda
TVM target arch: sm_90
detected_cuda_arch_list: [90]
Quantisation used for check:
q0f16
Observed preset behaviour:
O0 -> flashinfer=False
O1 -> flashinfer=False
O2 -> flashinfer=True
O3 -> flashinfer=True
Therefore, for this H800 machine:
The default --opt O2 enables FlashInfer.
Explicit detailed knobs were tested:
flashinfer=0 -> flashinfer=False
flashinfer=1 -> flashinfer=True
Important parser constraint:
O2 is a preset and cannot be mixed with detailed knobs in the same semicolon-separated string.
Invalid:
--opt "O2;flashinfer=0"
Observed failure:
unrecognized arguments: --O2
Reason:
When the string is not exactly one of O0/O1/O2/O3, the parser treats each semicolon-separated segment as a detailed knob and prepends "--".
Therefore "O2;flashinfer=0" becomes "--O2 --flashinfer=0", but "--O2" is not a recognised detailed knob.
Valid examples:
--opt O2
--opt O3
--opt "flashinfer=0"
--opt "flashinfer=1"
--opt "cublas_gemm=1;cudagraph=0"
Note:
--opt "flashinfer=1" is not equivalent to "O2 plus only changing FlashInfer".
It explicitly sets only the provided detailed knobs and leaves other optimisation flags at their detailed-knob defaults.
18. MLC-LLM Compile CLI Help Check
The compile subcommand help was checked:
python3 -m mlc_llm compile --help
Confirmed important options:
--quantization
--model-type
--device
--host
--enable-subgroups
--opt
--system-lib-prefix
--output / -o
--overrides
--debug-dump
The CLI help states:
--opt OPT
Optimization flags. MLC LLM maintains a predefined set of optimization flags,
denoted as O0, O1, O2, O3, where O0 means no optimization, O2 means majority
of them, and O3 represents extreme optimization that could potentially break
the system. Meanwhile, optimization flags could be explicitly specified via
details knobs, e.g. --opt="cublas_gemm=1;cudagraph=0".
(default: "O2")
19. BUILD_STATE_LOCAL.md Update
The FlashInfer integration state and compile option semantics were appended to:
~/repos/mlc-llm/BUILD_STATE_LOCAL.md
The file was then validated.
Observed file state:
-rw-r--r-- 1 ailven.liu ailven.liu 11K 5 May 10:29 BUILD_STATE_LOCAL.md
Key sections found:
MLC-LLM Compile CLI and FlashInfer Integration Update
Current FlashInfer Integration Status
Next Recommended Step
Tail validation confirmed that the appended section was not truncated.
20. Local Model Config Search
A local search was performed for existing MLC model configs:
find ~/repos \
-path "*/.git" -prune -o \
-name "mlc-chat-config.json" \
-type f \
-print
Result:
No mlc-chat-config.json was found under ~/repos.
Python recursive summary over ~/repos also found no local mlc-chat-config.json.
Conclusion:
There is currently no local MLC model config available for the next minimal compile-path probe.
No model was downloaded, no compile was run, and no model artifact was generated.
21. Current End-of-Session State
Current completed status:
TVM source build: OK
TVM Python import: OK
TVM CUDA visibility: OK
apache-tvm-ffi source/editable integration: OK
XGrammar source/editable integration: OK
tokenizers-cpp source build: OK
MLC-LLM source/editable integration: OK
MLC-LLM native libraries: OK
MLC-LLM CLI: OK
FlashInfer source clone: OK
FlashInfer submodules: OK
FlashInfer metadata generation: OK
FlashInfer editable source install: OK
FlashInfer Python import: OK
FlashInfer required JIT APIs: OK
FlashInfer direct decode JIT build/load: OK
FlashInfer direct prefill JIT build/load: OK
FlashInfer direct MLA JIT build/load: OK
TVM FlashInfer wrapper decode build/load: OK
TVM FlashInfer wrapper prefill build/load: OK
TVM FlashInfer wrapper MLA build/load: OK
env-tvm-source.sh FlashInfer JIT variables: OK
MLC-LLM --opt O2 FlashInfer enablement on H800: OK
Local mlc-chat-config.json availability: NOT_FOUND
Current unresolved item:
No local mlc-chat-config.json exists under ~/repos.
Known non-blocking compatibility issue:
FlashInfer v0.6.10 does not export gen_gemm_sm100_module from flashinfer.gemm.
The symbol exists under flashinfer.jit.gemm.
This affects the SM100 grouped GEMM wrapper path only and is not currently blocking H800 / SM90 attention, paged KV cache, decode, prefill, or MLA paths.
22. Recommended Next Step
Since no local mlc-chat-config.json exists under ~/repos, the next step should not be a normal model compilation yet.
Recommended next action:
Create or locate a minimal local MLC model config suitable for a compiler-path probe,
without downloading model weights and without running a large compilation.
Possible next directions:
1. Search the MLC-LLM test tree for minimal model config examples.
2. Inspect MLC-LLM compiler tests for synthetic config construction.
3. Build a minimal local test config only if it can exercise the FlashInfer paged KV cache creation path.
4. Run a minimal compile-path probe with --opt O2 only after a suitable local config is available.
Do not proceed directly to downloading a HuggingFace model or compiling a large model until the minimal compile-path probe strategy is selected.