← Writing
DevOps

Building the MLC-LLM Compilation Stack from Source on NVIDIA H800

Publication Date: 4 May 2026
Last Update Date: 7 May 2026
Author: Ailven LIU 刘翔
All Rights Reserved

Overview

This guide is intended for practitioners who wish to build the complete machine learning compilation stack entirely from source on an NVIDIA H800 system (24-core CPU, 220 GB memory) within a Linux container environment, as referenced in the mlc-ai/mlc-llm source repository. The stack in question comprises MLC-LLM, XGrammar, Apache TVM, FlashInfer, and CUDATile.

The guide covers the full process: pulling a base Docker image, configuring the container, resolving system-level read-only filesystem and dependency issues, accelerating package management (APT and PyPI), and configuring and compiling TVM. Each step adheres strictly to a ‘one step, one verification’ principle to ensure robustness and reproducibility.

Critical Note: In the target environment described in this guide, the container is managed by a Kubernetes orchestration platform, with the root filesystem configured as read-only by default. This causes the system package manager (dpkg/APT) to fail when attempting to write to /etc/group, blocking all subsequent operations.

This guide not only addresses how to resolve such read-only container constraints gracefully, but treats the issue as an opportunity to establish an independent, writable, and stable high-performance compilation environment from first principles.

1. Preparing the container

1. Container Environment Preparation

1.1 Pulling the Base Image

MLC-LLM requires CUDA version 12.0 or later to correctly support its compilation and runtime operators. We select nvidia/cuda:13.2.0-cudnn-devel-ubuntu22.04 as the base container. The devel tag is built atop the runtime image and includes the CUDA compilation toolchain, header files, and development libraries – precisely what is required for building CUDA applications from source.

# Base pull command (exact invocation depends on your container runtime)
docker pull docker.io/nvidia/cuda:13.2.0-cudnn-devel-ubuntu22.04

# If operating within a Chinese network where docker.io access is slow or unavailable,
# a domestic mirror may be used instead:
# swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/nvidia/cuda:13.2.0-cudnn-devel-ubuntu22.04

1.2 Launching an Interactive Container

The following command launches a persistent interactive container from the Docker image. To ensure that compiled artefacts are not lost when the container exits, the host-side source directory must be mounted into the container. In a Kubernetes environment in particular, the root filesystem must be explicitly declared as writable; failure to do so will trigger the dpkg errors described later in this guide.

# Launch an interactive container with host directory mount and writable root filesystem
docker run -it \
  --gpus all \
  --shm-size=32g \
  --mount type=bind,source=/path/to/your/local/repos,target=/workspace/repos \
  --read-only=false \     # Critical in Kubernetes environments
  docker.io/nvidia/cuda:13.2.0-cudnn-devel-ubuntu22.04 /bin/bash

If using a company-managed Kubernetes-based experimental platform (such as KTP or an equivalent), the image address may be entered directly, provided that the platform is capable of pulling from the specified registry.

1.3 Base Directory Structure

Upon entering the container, it is advisable to create the source code directory in advance before proceeding with subsequent steps.

mkdir -p /workspace/repos
cd /workspace/repos

2. System Configuration and Package Mirror Acceleration

When provisioning a Pod on a corporate Kubernetes-based experimental platform, the resulting environment will very likely be a read-write–restricted Pod whose root filesystem is read-only. This can cause apt configuration failures or system package corruption. Under any circumstances, a writable system environment is a prerequisite for compiling software. Attempting to run advanced compilation tasks on an unstable read-only filesystem is inadvisable.

2.1 Configuring APT and PIP Mirror Sources

To ensure the fastest possible download speeds during compilation, it is recommended to switch to high-speed domestic mirrors.

2.2 Installing Essential Build Tools

The following installs the foundational components required for building TVM:

# Install core build dependencies
sudo apt install -y \
    vim \
    build-essential \
    cmake \
    curl \
    wget \ 
    git \
    ninja-build \
    python3 \
    python3-pip \
    python3-dev

# Install version control and additional dependencies
sudo apt install -y git-lfs

# Verify installations
cmake --version          # Should be >= 3.22
ninja --version          # Should be >= 1.10
python3 --version        # Should be >= 3.10

If the system has entered a dpkg deadlock – for instance, if groupadd: cannot open /etc/group or E: Sub-process /usr/bin/dpkg returned an error code (1) is reported – the obstructing post-installation scripts may be safely removed as follows:

# Remove the corrupted package configuration script (openssh-client shown as an example)
sudo rm -f /var/lib/dpkg/info/openssh-client.postinst

# Force completion of the dpkg database repair
sudo dpkg --configure -a
sudo apt install -f

2. Installing the toolchains

3. Installing the LLVM and CUDA Toolchains

3.1 Installing LLVM 18

LLVM 18 is installed via the ISCAS (Institute of Software, Chinese Academy of Sciences) mirror, which provides one of the most stable and accessible open-source LLVM repositories.

set -e

# Basic Dependencies
sudo apt update
sudo apt install -y ca-certificates wget gnupg

# Remove personal pre-existing LLVM source list files
sudo rm -f /etc/apt/sources.list.d/llvm-iscas.list

# Add the ISCAS LLVM mirror
wget -qO- https://apt.llvm.org/llvm-snapshot.gpg.key \
  | sudo gpg --dearmor -o /usr/share/keyrings/llvm-archive-keyring.gpg

# Verify fingerprint 
gpg --show-keys --fingerprint /usr/share/keyrings/llvm-archive-keyring.gpg

# Add ISCAS LLVM mirror
echo "deb [signed-by=/usr/share/keyrings/llvm-archive-keyring.gpg] https://mirror.iscas.ac.cn/llvm-apt/jammy/ llvm-toolchain-jammy-18 main" \
  | sudo tee /etc/apt/sources.list.d/llvm-iscas.list

sudo apt update

sudo apt install -y \
  clang-18 \
  lld-18 \
  lldb-18 \
  clangd-18 \
  clang-format-18 \
  clang-tidy-18 \
  llvm-18-dev \
  llvm-18-tools

# Verify the installation
llvm-config-18 --version   # Expected output: 18.1.8
clang-18 --version
which clang-18

Outcome at this stage: the ISCAS mirror provides llvm-config-18 at version 18.1.8, supplying TVM with a complete LLVM backend for JIT compilation.

3.2 Configuring CUDA and cuDNN

This guide uses NVIDIA CUDA 13.2.0. As the container image already contains the full CUDA toolchain, no additional installation is necessary. Should the toolchain be absent, installation instructions are available at the NVIDIA CUDA download page.

When compiling TVM, CMake must be able to locate the cuDNN library files precisely. If the default path does not exist – for example, if /usr/lib/x86_64-linux-gnu/libcudnn.so is a broken symbolic link or is absent – it must be specified manually.

Confirm the actual location of the cuDNN libraries first:

find /usr -name "libcudnn*" 2>/dev/null
find /usr -name "cudnn*.h" 2>/dev/null

Adjust CUDNN_LIBRARY and the cuDNN INCLUDE_DIR within config.cmake according to the paths returned by these commands.

4. Configuring the proxy for GitHub

In some cases, especially you are operating on a domestic machine, you may encounter problem that github unaccessible. Please make sure you have a proxy server in your server for github clone:

sudo apt update
sudo apt install shadowsocks-libev

Create a configuration file, link ~/shadowsockets.json:

{
    "server": "your.ss.node.ip",
    "server_port": {port},
    "local_address": "127.0.0.1",
    "local_port": 1082,
    "password": "your ss node password",
    "timeout": 300,
    "method": "entropy method",
    "fast_open": false,
    "workers": 1
}

Check where ss-local is: which ss-local, it always is /usr/bin/ss-local

self-start

Create systemd: sudo vim /etc/systemd/system/ss-local.service, paste content below:

[Unit]
Description=Shadowsocks Local Proxy
After=network.target

[Service]
Type=simple
User=ailven.liu
ExecStart=/usr/bin/ss-local -c /home/ailven.liu/shadowsockets.json
Restart=on-failure
RestartSec=5
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

Note: replace ExecStart and User to what they really are.

Reload configuration and set to self-start:

sudo systemctl daemon-reload
sudo systemctl enable --now ss-local.service

Check status and log:

systemctl status ss-local.service
journalctl -u ss-local.service -f

Run background

nohup ss-local -c ~/shadowsockets.json > ~/ss-local.log 2>&1 &
ps aux | grep ss-local # check status
tail -f ~/ss-local.log # check logs

Setting the proxy for GitHub only

Then run command building proxy only for github:

git config --global http.https://github.com/.proxy socks5://127.0.0.1:1082
git config --global https.https://github.com/.proxy socks5://127.0.0.1:1082

or you can also edit file ~/.gitconfig directly:

[http "https://github.com"]
    proxy = socks5://127.0.0.1:1082
[https "https://github.com"]
    proxy = socks5://127.0.0.1:1082

Check it:

git config --global --list --show-origin | grep github

You can also delete the proxy config from github:

git config --global --unset http.https://github.com/.proxy
git config --global --unset https.https://github.com/.proxy

3. Cloning and pinning the sources

5. Cloning MLC-LLM and Its Pinned Source Dependencies

All source code is placed under ~/repos.

mkdir -p ~/repos
cd ~/repos

Before cloning the full repository, verify that GitHub access is working through the configured Git proxy:

git config --global --list --show-origin | grep -Ei 'proxy|github' || true

GIT_CURL_VERBOSE=1 git ls-remote https://github.com/mlc-ai/mlc-llm.git HEAD
GIT_CURL_VERBOSE=1 git ls-remote https://github.com/apache/tvm.git HEAD

The GitHub-specific SOCKS5 proxy used in this environment is:

socks5://127.0.0.1:1082

Clone mlc-llm recursively:

cd ~/repos

git clone --recursive https://github.com/mlc-ai/mlc-llm.git mlc-llm

cd ~/repos/mlc-llm
git rev-parse HEAD
git submodule status --recursive

In this run, the checked-out mlc-llm commit was:

d1ea69a87280e821611643958bfec385b62dafd3

A transient network failure occurred while cloning one nested CUTLASS submodule:

error: RPC failed; curl 56 GnuTLS recv error (-9): Error decoding the received TLS packet.
fatal: early EOF
fatal: fetch-pack: invalid index-pack output

However, Git automatically retried the submodule clone and succeeded. The final git submodule status --recursive output showed all submodules checked out cleanly, with no leading -, +, or U.

Important observation:

~/repos/mlc-llm/3rdparty/tvm

is not a direct checkout of apache/tvm.git. It is pinned by mlc-llm to:

https://github.com/mlc-ai/relax.git

The checked-out TVM commit was:

c2028da5ac03055f30a52b492260f37b9650ac03

For MLC-LLM development, this pinned TVM source tree should be built and used. Building an unrelated apache/tvm HEAD separately may cause API or ABI mismatches.

Verify repository cleanliness:

cd ~/repos/mlc-llm

git status --short
git rev-parse HEAD

git submodule foreach --recursive '
  echo "----- $name :: $path -----"
  git status --short
  git rev-parse HEAD
'

A clean checkout should produce no git status --short output in the main repository or in submodules.

6. Configuring the Pinned TVM Source Tree

Enter the pinned TVM source tree:

cd ~/repos/mlc-llm/3rdparty/tvm

The TVM source tree contains a default CMake configuration file at:

cmake/config.cmake

However, TVM does not automatically read this file from cmake/config.cmake. Its top-level CMakeLists.txt loads configuration in the following order:

if(EXISTS ${CMAKE_BINARY_DIR}/config.cmake)
  include(${CMAKE_BINARY_DIR}/config.cmake)
else()
  if(EXISTS ${CMAKE_SOURCE_DIR}/config.cmake)
    include(${CMAKE_SOURCE_DIR}/config.cmake)
  endif()
endif()

Therefore, after modifying cmake/config.cmake, it must be copied into the build directory as:

build/config.cmake

6.1 Confirm CUDA, cuDNN, and NCCL State

CUDA root:

readlink -f /usr/local/cuda

Expected result in this environment:

/usr/local/cuda-13.2

Check key CUDA libraries:

ls -lh \
  /usr/local/cuda/include/cuda.h \
  /usr/local/cuda/lib64/libcudart.so \
  /usr/local/cuda/lib64/libcublas.so

cuDNN 9 development files were installed through the CUDA repository:

dpkg -l | grep -Ei 'cudnn|libcudnn' || true

Installed packages:

libcudnn9-cuda-13         9.20.0.48-1
libcudnn9-dev-cuda-13     9.20.0.48-1
libcudnn9-headers-cuda-13 9.20.0.48-1

cuDNN headers and libraries are located at:

/usr/include/x86_64-linux-gnu/cudnn.h
/usr/include/x86_64-linux-gnu/cudnn_version.h
/usr/lib/x86_64-linux-gnu/libcudnn.so

Check via:

dpkg -L libcudnn9-headers-cuda-13 libcudnn9-dev-cuda-13 \
  | grep -Ei 'include|header|cudnn|\.h$' \
  | sort

NCCL was not installed in this environment:

dpkg -l | grep -Ei 'nccl|libnccl' || true
find /usr /opt -type f \( -name 'nccl.h' -o -name 'libnccl.so*' \) 2>/dev/null | sort

Because the build policy is source-only and refuses precompiled binary packages, USE_NCCL was kept OFF for the TVM build. NCCL can be built later from NVIDIA’s NCCL source tree if required.

6.2 Patch TVM CMake Configuration

Back up the original TVM CMake configuration:

cd ~/repos/mlc-llm/3rdparty/tvm

cp -av cmake/config.cmake cmake/config.cmake.bak.$(date +%Y%m%d_%H%M%S)

Apply the local configuration:

cd ~/repos/mlc-llm/3rdparty/tvm

python3 - <<'PY'
from pathlib import Path

p = Path("cmake/config.cmake")
s = p.read_text()

repls = {
    "set(USE_CUDA OFF)": "set(USE_CUDA /usr/local/cuda)",
    "set(USE_NCCL OFF)": "set(USE_NCCL OFF)",
    "set(USE_ROCM OFF)": "set(USE_ROCM OFF)",
    "set(USE_OPENCL OFF)": "set(USE_OPENCL OFF)",
    "set(USE_METAL OFF)": "set(USE_METAL OFF)",
    "set(USE_VULKAN OFF)": "set(USE_VULKAN OFF)",
    "set(USE_LLVM OFF)": "set(USE_LLVM /usr/bin/llvm-config-18)",
    "set(USE_CUDNN OFF)": "set(USE_CUDNN ON)",
    "set(USE_CUDNN_FRONTEND OFF)": "set(USE_CUDNN_FRONTEND OFF)",
    "set(USE_CUBLAS OFF)": "set(USE_CUBLAS ON)",
    "set(TVM_FFI_USE_LIBBACKTRACE ON)": "set(TVM_FFI_USE_LIBBACKTRACE ON)",
    "set(USE_CUTLASS OFF)": "set(USE_CUTLASS ON)",
    "set(USE_FLASHINFER OFF)": "set(USE_FLASHINFER ON)",
}

missing = []
for old, new in repls.items():
    if old not in s:
        missing.append(old)
    else:
        s = s.replace(old, new)

if missing:
    print("MISSING_PATTERNS:")
    for x in missing:
        print(x)
    raise SystemExit(1)

extra = """
# ---- Local explicit paths for CUDA/cuDNN on this H800 CUDA 13.2 environment ----
set(CUDNN_INCLUDE_DIR /usr/include/x86_64-linux-gnu)
set(CUDNN_LIBRARY /usr/lib/x86_64-linux-gnu/libcudnn.so)
"""

if "Local explicit paths for CUDA/cuDNN" not in s:
    s += extra

p.write_text(s)
PY

Verify the relevant options:

grep -nE 'USE_CUDA|USE_NCCL|USE_LLVM|USE_CUDNN|USE_CUBLAS|USE_CUTLASS|USE_FLASHINFER|TVM_FFI_USE_LIBBACKTRACE|CUDNN_INCLUDE_DIR|CUDNN_LIBRARY|USE_OPENCL|USE_METAL|USE_ROCM|USE_VULKAN' \
  cmake/config.cmake

Expected important entries:

set(USE_CUDA /usr/local/cuda)
set(USE_NCCL OFF)
set(USE_ROCM OFF)
set(USE_OPENCL OFF)
set(USE_METAL OFF)
set(USE_VULKAN OFF)
set(USE_LLVM /usr/bin/llvm-config-18)
set(USE_CUDNN ON)
set(USE_CUDNN_FRONTEND OFF)
set(USE_CUBLAS ON)
set(TVM_FFI_USE_LIBBACKTRACE ON)
set(USE_CUTLASS ON)
set(USE_FLASHINFER ON)
set(CUDNN_INCLUDE_DIR /usr/include/x86_64-linux-gnu)
set(CUDNN_LIBRARY /usr/lib/x86_64-linux-gnu/libcudnn.so)

4. Configuring TVM

7. Configuring TVM for NVIDIA H800 / Hopper

The first CMake configure attempt showed that, if cmake/config.cmake is not copied into the build directory, TVM silently keeps important options disabled:

USE_CUDA=OFF
USE_LLVM=OFF
USE_CUDNN=OFF
USE_CUBLAS=OFF
USE_CUTLASS=OFF

The correct procedure is:

cd ~/repos/mlc-llm/3rdparty/tvm

rm -rf build
mkdir -p build
cp -av cmake/config.cmake build/config.cmake

cd build

cmake .. \
  -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_C_COMPILER=/usr/bin/gcc \
  -DCMAKE_CXX_COMPILER=/usr/bin/g++ \
  -DCMAKE_CUDA_ARCHITECTURES=90

The explicit CUDA architecture is important. H800 is a Hopper GPU and should target:

sm_90 / compute_90

Without this override, CMake detected or defaulted to:

CMAKE_CUDA_ARCHITECTURES=75

which is inappropriate for H800.

A successful configure should include messages similar to:

-- Custom CUDA_PATH=/usr/local/cuda
-- Found CUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda
-- Found CUDA_CUDART_LIBRARY=/usr/local/cuda/lib64/libcudart.so
-- Found CUDA_CUDNN_LIBRARY=/usr/lib/x86_64-linux-gnu/libcudnn.so
-- Found CUDA_CUBLAS_LIBRARY=/usr/local/cuda/lib64/libcublas.so
-- Found CUDA_CUBLASLT_LIBRARY=/usr/local/cuda/lib64/libcublasLt.so
-- Build with CUDA  support
-- Build with cuDNN support
-- Build with cuBLAS support
-- Found CMAKE_CUDA_ARCHITECTURES=90
-- Use llvm-config=/usr/bin/llvm-config-18
-- Found TVM_LLVM_VERSION=181
-- Build with LLVM
-- Build with CUTLASS

Verify CUDA architecture in build.ninja:

grep -nE 'compute_90|sm_90' build.ninja | sed -n '1,80p'

Expected form:

--generate-code=arch=compute_90,code=[compute_90,sm_90]

8. Patching libflash_attn CUDA Architecture from sm_80 to sm_90

After configuring TVM with CMAKE_CUDA_ARCHITECTURES=90, build.ninja still contained some compute_80 / sm_80 flags.

The source was identified as:

3rdparty/libflash_attn/src/CMakeLists.txt

Relevant original content:

set(CMAKE_CUDA_FLAGS "${CMAKE_CUDA_FLAGS} --expt-relaxed-constexpr --expt-relaxed-constexpr --use_fast_math -t 8 \
                      -gencode=arch=compute_80,code=\\\"sm_80,compute_80\\\" \
                      ")

set_target_properties(flash_attn PROPERTIES CUDA_ARCHITECTURES "80")

Patch it for H800 / Hopper:

cd ~/repos/mlc-llm/3rdparty/tvm

cp -av 3rdparty/libflash_attn/src/CMakeLists.txt \
  3rdparty/libflash_attn/src/CMakeLists.txt.bak.h800_sm90.$(date +%Y%m%d_%H%M%S)

python3 - <<'PY'
from pathlib import Path

p = Path("3rdparty/libflash_attn/src/CMakeLists.txt")
s = p.read_text()

repls = {
    "compute_80": "compute_90",
    "sm_80": "sm_90",
    'CUDA_ARCHITECTURES "80"': 'CUDA_ARCHITECTURES "90"',
}

for old, new in repls.items():
    if old not in s:
        print(f"WARNING: pattern not found: {old}")
    s = s.replace(old, new)

p.write_text(s)
PY

Reconfigure TVM after this patch:

cd ~/repos/mlc-llm/3rdparty/tvm

rm -rf build
mkdir -p build
cp -av cmake/config.cmake build/config.cmake

cd build

cmake .. \
  -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_C_COMPILER=/usr/bin/gcc \
  -DCMAKE_CXX_COMPILER=/usr/bin/g++ \
  -DCMAKE_CUDA_ARCHITECTURES=90

Verify that compute_80 / sm_80 no longer appear in generated build rules:

grep -nE 'compute_80|sm_80' build.ninja | sed -n '1,80p' || true
grep -nE 'compute_90|sm_90' build.ninja | sed -n '1,80p' || true

Expected result:

No compute_80 / sm_80 compile flags remain.
compute_90 / sm_90 compile flags are present.

Note: several source filenames still contain sm80, for example:

flash_fwd_hdim128_fp16_sm80.cu

This is only a filename. The actual NVCC target architecture is controlled by -gencode and CUDA_ARCHITECTURES.

5. Building TVM and its bindings

9. Building TVM Native Libraries

Build TVM with Ninja. A conservative parallelism level is used first to keep failures easier to debug:

cd ~/repos/mlc-llm/3rdparty/tvm/build

ninja -j64

Successful final link messages include:

[634/634] Linking CXX shared library libtvm.so

Generated libraries include:

./lib/libtvm_ffi_testing.so
./lib/libtvm_ffi.so
./libtvm_runtime.so
./libtvm.so

In this run:

libtvm_runtime.so  6.5M
libtvm.so          86M
libtvm_ffi.so      2.0M

Check linked dependencies:

cd ~/repos/mlc-llm/3rdparty/tvm/build

ldd ./libtvm.so | grep -Ei 'cuda|cudnn|cublas|llvm|stdc\+\+|not found' || true
ldd ./libtvm_runtime.so | grep -Ei 'cuda|cudnn|cublas|llvm|stdc\+\+|not found' || true

Expected important dependencies for libtvm.so:

libLLVM-18.so.18.1
libcudart.so.13
libcuda.so.1
libcudnn.so.9
libcublas.so.13
libcublasLt.so.13
libstdc++.so.6

Expected important dependencies for libtvm_runtime.so:

libcudart.so.13
libcuda.so.1
libcudnn.so.9
libcublas.so.13
libcublasLt.so.13
libstdc++.so.6

No not found entries should appear.

10. Building and Installing apache-tvm-ffi from Source

After TVM native libraries were built, directly importing TVM from source initially failed:

ModuleNotFoundError: No module named 'tvm_ffi'

The correct tvm_ffi source path is:

~/repos/mlc-llm/3rdparty/tvm/3rdparty/tvm-ffi/python

After adding that path, import proceeded further but failed because package metadata was missing:

importlib.metadata.PackageNotFoundError: No package metadata was found for apache-tvm-ffi

10.1 Local Development Patch for tvm_ffi/libinfo.py

tvm_ffi/libinfo.py attempted to call:

dist = im.distribution(package)

before entering its fallback search logic. In a source-tree development build, apache-tvm-ffi package metadata may not exist yet, even though libtvm_ffi.so has already been built.

Patch 3rdparty/tvm-ffi/python/tvm_ffi/libinfo.py so that PackageNotFoundError falls through to the existing fallback search:

cd ~/repos/mlc-llm/3rdparty/tvm

cp -av 3rdparty/tvm-ffi/python/tvm_ffi/libinfo.py \
  3rdparty/tvm-ffi/python/tvm_ffi/libinfo.py.bak.dev_fallback.$(date +%Y%m%d_%H%M%S)

python3 - <<'PY'
from pathlib import Path

p = Path("3rdparty/tvm-ffi/python/tvm_ffi/libinfo.py")
s = p.read_text()

old = '''    # Use `importlib.metadata` is the most reliable way to find package data files
    dist: im.PathDistribution = im.distribution(package)  # ty: ignore[invalid-assignment]
    record = dist.read_text("RECORD") or ""
    for line in record.splitlines():
        partial_path, *_ = line.split(",")
        if partial_path.endswith(lib_dll_names):
            try:
                path = (dist._path.parent / partial_path).resolve()
            except OSError:
                continue
            if path.name in lib_dll_names:
                return path
'''

new = '''    # Use `importlib.metadata` is the most reliable way to find package data files.
    # In a source-tree development build, package metadata may not exist yet.
    # In that case, continue to the fallback search below.
    try:
        dist: im.PathDistribution = im.distribution(package)  # ty: ignore[invalid-assignment]
    except im.PackageNotFoundError:
        dist = None

    if dist is not None:
        record = dist.read_text("RECORD") or ""
        for line in record.splitlines():
            partial_path, *_ = line.split(",")
            if partial_path.endswith(lib_dll_names):
                try:
                    path = (dist._path.parent / partial_path).resolve()
                except OSError:
                    continue
                if path.name in lib_dll_names:
                    return path
'''

if old not in s:
    raise SystemExit("ERROR: target block not found; do not patch blindly")

p.write_text(s.replace(old, new))
PY

This patch allows tvm_ffi to continue searching:

$PROJECT_ROOT/build/lib
$PROJECT_ROOT/lib
$DEV_TOP/build/lib
$DEV_TOP/lib
LD_LIBRARY_PATH
PATH

instead of failing immediately when Python package metadata is absent.

10.2 Install Python Build Tools from Source

tvm_ffi.core is a Cython native extension. It is not produced by the TVM C++ Ninja build alone.

The source files include:

3rdparty/tvm-ffi/python/tvm_ffi/cython/core.pyx
3rdparty/tvm-ffi/python/tvm_ffi/core.pyi

The required Python build tools are:

scikit-build-core
Cython
setuptools-scm
setuptools
ninja

In this environment, scikit-build-core, tomli, packaging, and ninja were already installed in user-site, but Cython and setuptools-scm were missing.

Correct import name for scikit-build-core:

import scikit_build_core

not:

import skbuild_core

Verify Python build tools:

python3 - <<'PY'
mods = [
    "scikit_build_core",
    "Cython",
    "setuptools_scm",
    "tomli",
    "packaging",
    "setuptools",
    "wheel",
    "ninja",
]
for m in mods:
    try:
        mod = __import__(m)
        print(f"{m}: OK -> {getattr(mod, '__version__', 'unknown')} -> {getattr(mod, '__file__', 'built-in')}")
    except Exception as e:
        print(f"{m}: MISSING/ERROR -> {type(e).__name__}: {e}")
PY

Upgrade setuptools from source:

python3 -m pip install \
  --user \
  --no-binary=:all: \
  --upgrade \
  "setuptools>=77.0.3"

In this run, setuptools was built from source and installed as:

setuptools 82.0.1
/home/ailven.liu/.local/lib/python3.10/site-packages/setuptools

Warning observed:

torch 2.11.0 requires setuptools<82, but you have setuptools 82.0.1 which is incompatible.

This warning was not blocking for TVM/tvm-ffi. If later Torch-related tooling fails, isolate this stack in a virtual environment or adjust setuptools.

Install Cython from source:

python3 -m pip install \
  --user \
  --no-binary=:all: \
  --upgrade \
  "cython>=3.0"

Verify:

python3 - <<'PY'
import Cython
print("Cython:", Cython.__version__, Cython.__file__)
PY

Expected result in this run:

Cython 3.2.4
/home/ailven.liu/.local/lib/python3.10/site-packages/Cython

Installing setuptools-scm directly from source initially failed because its build isolation environment lacked tomli while building vcs-versioning.

The successful procedure was:

python3 -m pip install \
  --user \
  --no-binary=:all: \
  --no-build-isolation \
  --upgrade \
  "vcs-versioning"

Then:

python3 -m pip install \
  --user \
  --no-binary=:all: \
  --no-build-isolation \
  --upgrade \
  "setuptools-scm"

Verify:

python3 - <<'PY'
import vcs_versioning
import setuptools_scm

print("vcs_versioning:", getattr(vcs_versioning, "__version__", "unknown"), vcs_versioning.__file__)
print("setuptools_scm:", getattr(setuptools_scm, "__version__", "unknown"), setuptools_scm.__file__)
PY

10.3 Editable Source Build of apache-tvm-ffi

Build and install apache-tvm-ffi from the local source tree in editable mode:

cd ~/repos/mlc-llm/3rdparty/tvm/3rdparty/tvm-ffi

export LD_LIBRARY_PATH=$PWD/../../build/lib:$PWD/../../build:/usr/local/cuda/lib64:${LD_LIBRARY_PATH:-}

python3 -m pip install \
  --user \
  --no-build-isolation \
  --no-binary=:all: \
  -e .

Although pip reports a locally created wheel during editable installation:

Created wheel for apache-tvm-ffi: ...

this is a PEP 517 editable-install build artefact produced from the local source tree. It is not a downloaded precompiled binary package.

Verify:

python3 - <<'PY'
import tvm_ffi
import tvm_ffi.core
import importlib.metadata as im

print("tvm_ffi:", tvm_ffi.__file__)
print("tvm_ffi.core:", tvm_ffi.core.__file__)
print("dist version:", im.version("apache-tvm-ffi"))
PY

Successful result:

tvm_ffi: /home/ailven.liu/repos/mlc-llm/3rdparty/tvm/3rdparty/tvm-ffi/python/tvm_ffi/__init__.py
tvm_ffi.core: /home/ailven.liu/.local/lib/python3.10/site-packages/tvm_ffi/core.cpython-310-x86_64-linux-gnu.so
dist version: 0.1.11.dev0+g1fed0ae04.d20260505

11. Verifying TVM Python Binding

Set the TVM environment for the current shell:

cd ~/repos/mlc-llm/3rdparty/tvm

export TVM_HOME=$PWD
export PYTHONPATH=$TVM_HOME/python
export LD_LIBRARY_PATH=$TVM_HOME/build:$TVM_HOME/build/lib:/usr/local/cuda/lib64:${LD_LIBRARY_PATH:-}

Verify tvm, tvm_ffi, and CUDA visibility:

python3 - <<'PY'
import os
import tvm
import tvm_ffi
import tvm_ffi.core

print("tvm module:", tvm.__file__)
print("tvm_ffi module:", tvm_ffi.__file__)
print("tvm_ffi.core:", tvm_ffi.core.__file__)

from tvm.support import libinfo
info = libinfo()
for k in sorted(info):
    if k.startswith("USE_") or k in ("GIT_COMMIT_HASH", "LLVM_VERSION"):
        print(f"{k}={info[k]}")

print("cuda exist:", tvm.cuda().exist)
print("cuda device:", tvm.cuda(0))
PY

Successful result:

tvm module: /home/ailven.liu/repos/mlc-llm/3rdparty/tvm/python/tvm/__init__.py
tvm_ffi module: /home/ailven.liu/repos/mlc-llm/3rdparty/tvm/3rdparty/tvm-ffi/python/tvm_ffi/__init__.py
tvm_ffi.core: /home/ailven.liu/.local/lib/python3.10/site-packages/tvm_ffi/core.cpython-310-x86_64-linux-gnu.so

GIT_COMMIT_HASH=c2028da5ac03055f30a52b492260f37b9650ac03
LLVM_VERSION=18.1.8
USE_CUBLAS=ON
USE_CUDA=/usr/local/cuda
USE_CUDNN=ON
USE_CUTLASS=ON
USE_LLVM=/usr/bin/llvm-config-18
USE_NCCL=OFF
USE_RPC=ON
USE_THREADS=ON

cuda exist: True
cuda device: cuda:0

At this point, the source-built TVM stack is functional from Python and can see the CUDA device.

12. Persisting the TVM Source Environment

To avoid manually exporting environment variables every time, create a sourceable environment script:

cat > ~/repos/mlc-llm/env-tvm-source.sh <<'EOF'
# Source-built TVM environment for this mlc-llm checkout.
# Usage:
#   source ~/repos/mlc-llm/env-tvm-source.sh

export MLC_LLM_HOME="$HOME/repos/mlc-llm"
export TVM_HOME="$MLC_LLM_HOME/3rdparty/tvm"

export PYTHONPATH="$TVM_HOME/python:$MLC_LLM_HOME/python:${PYTHONPATH:-}"
export LD_LIBRARY_PATH="$TVM_HOME/build:$TVM_HOME/build/lib:/usr/local/cuda/lib64:${LD_LIBRARY_PATH:-}"
export PATH="$HOME/.local/bin:/usr/local/cuda/bin:${PATH:-}"
EOF

chmod +x ~/repos/mlc-llm/env-tvm-source.sh

Use it with:

source ~/repos/mlc-llm/env-tvm-source.sh

Verify from a fresh subshell:

bash -lc '
  source ~/repos/mlc-llm/env-tvm-source.sh
  python3 - <<PY
import tvm
print("tvm:", tvm.__file__)
print("cuda:", tvm.cuda().exist)
PY
'

Expected result:

tvm: /home/ailven.liu/repos/mlc-llm/3rdparty/tvm/python/tvm/__init__.py
cuda: True

6. Building the remaining dependencies

13. Continuing from TVM to XGrammar

After the source-built TVM stack had been verified, the next stage was to build and install XGrammar from the mlc-llm pinned submodule.

The XGrammar source tree is located at:

~/repos/mlc-llm/3rdparty/xgrammar

The checked-out XGrammar commit in this environment is:

d4f57c440f3da8e7330a1e5d50bba9c31f9433ea

Its dlpack submodule was already initialised:

3rdparty/dlpack: bbd2f4d32427e548797929af08cfe2a9cbb3cf12

The relevant build entry points are:

~/repos/mlc-llm/3rdparty/xgrammar/CMakeLists.txt
~/repos/mlc-llm/3rdparty/xgrammar/cpp/pybind/CMakeLists.txt
~/repos/mlc-llm/3rdparty/xgrammar/python/setup.py
~/repos/mlc-llm/3rdparty/xgrammar/python/pyproject.toml

A key observation is that python/setup.py does not invoke CMake by itself. It expects the native binding library to have already been built and placed under:

~/repos/mlc-llm/3rdparty/xgrammar/python/xgrammar/xgrammar_bindings.*.so

Therefore, the correct build order is:

  1. configure and build the native C++/pybind11 extension with CMake/Ninja;
  2. install the Python package from the local source tree.

The available toolchain used for XGrammar was:

cmake: 3.22.1
ninja: 1.13.0.git.kitware.jobserver-pipe-1
gcc/g++: 11.4.0
Python: 3.10.12
pybind11: 3.0.4
torch: 2.11.0+cu130

The Torch CMake prefix path was available at:

~/.local/lib/python3.10/site-packages/torch/share/cmake

The pybind11 CMake directory was available at:

~/.local/lib/python3.10/site-packages/pybind11/share/cmake/pybind11

13.1 Configuring XGrammar

Configure XGrammar with CMake:

source ~/repos/mlc-llm/env-tvm-source.sh

cd ~/repos/mlc-llm/3rdparty/xgrammar

rm -rf build
mkdir -p build

cmake -S . -B build \
  -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_C_COMPILER=/usr/bin/gcc \
  -DCMAKE_CXX_COMPILER=/usr/bin/g++

The configure stage completed successfully and generated Ninja build files under:

~/repos/mlc-llm/3rdparty/xgrammar/build

During configuration, CMake emitted the following non-blocking Torch warning:

library kineto not found

This warning did not prevent configuration. Torch was still found successfully:

Found Torch: ~/.local/lib/python3.10/site-packages/torch/lib/libtorch.so

13.2 Building XGrammar Native Binding

Build the native binding:

source ~/repos/mlc-llm/env-tvm-source.sh

cd ~/repos/mlc-llm/3rdparty/xgrammar

cmake --build build -j64

The build completed successfully and produced:

~/repos/mlc-llm/3rdparty/xgrammar/python/xgrammar/xgrammar_bindings.cpython-310-x86_64-linux-gnu.so

13.3 Installing XGrammar from Local Source

Install the Python package from the local source tree in editable mode:

source ~/repos/mlc-llm/env-tvm-source.sh

cd ~/repos/mlc-llm/3rdparty/xgrammar/python

python3 -m pip install \
  --user \
  --no-build-isolation \
  --no-binary=:all: \
  -e .

The installation completed successfully:

Successfully built xgrammar
Successfully installed xgrammar-0.0.3

Although pip created an editable wheel locally, this was a local PEP 660 editable-install artefact, not a downloaded precompiled binary package:

xgrammar-0.0.3-0.editable-cp310-cp310-linux_x86_64.whl

The final import locations were:

xgrammar:
~/repos/mlc-llm/3rdparty/xgrammar/python/xgrammar/__init__.py

xgrammar_bindings:
~/repos/mlc-llm/3rdparty/xgrammar/python/xgrammar/xgrammar_bindings.cpython-310-x86_64-linux-gnu.so

The visible public API included:

BNFGrammar
BuiltinGrammar
CachedGrammarCompiler
CompiledGrammar
GrammarMatcher
TokenizerInfo
VocabType

At this point, XGrammar was successfully built and installed from local source.

14. Updating the Source Environment Script for XGrammar

The environment script was extended so that XGrammar is explicitly part of the source development environment.

The script is located at:

~/repos/mlc-llm/env-tvm-source.sh

The following variable was added:

export XGRAMMAR_HOME="$MLC_LLM_HOME/3rdparty/xgrammar"

The PYTHONPATH was updated to include:

$XGRAMMAR_HOME/python

The relevant environment entries became:

export MLC_LLM_HOME="$HOME/repos/mlc-llm"
export TVM_HOME="$MLC_LLM_HOME/3rdparty/tvm"
export XGRAMMAR_HOME="$MLC_LLM_HOME/3rdparty/xgrammar"

export PYTHONPATH="$TVM_HOME/python:$MLC_LLM_HOME/python:$XGRAMMAR_HOME/python:${PYTHONPATH:-}"
export LD_LIBRARY_PATH="$TVM_HOME/build:$TVM_HOME/build/lib:/usr/local/cuda/lib64:${LD_LIBRARY_PATH:-}"
export PATH="$HOME/.local/bin:/usr/local/cuda/bin:${PATH:-}"

A fresh shell confirmed that the source-built TVM, XGrammar, and CUDA visibility remained valid.

15. FlashInfer Source State in This Checkout

The mlc-llm/3rdparty directory in this checkout does not contain an independent flashinfer source tree.

The available immediate third-party directories were:

argparse
googletest
stb
tokenizers-cpp
tvm
xgrammar

The only FlashInfer or FlashAttention-related source discovered in the current checkout was under the pinned TVM source tree:

~/repos/mlc-llm/3rdparty/tvm/3rdparty/libflash_attn

The TVM build had already incorporated this internal flash attention component, and the MLC-LLM native libraries later linked against:

libflash_attn.so
libfpA_intB_gemm.so

However, the MLC-LLM Python package metadata declares a Linux dependency on:

flashinfer-python

This appears in:

~/repos/mlc-llm/pyproject.toml
~/repos/mlc-llm/python/requirements.txt

The relevant dependency entry is:

flashinfer-python; sys_platform == 'linux'

The current environment did not have flashinfer-python installed:

flashinfer-python: NOT_FOUND

The MLC-LLM Python source tree did not contain a direct Python import of FlashInfer, such as:

import flashinfer
from flashinfer import ...

Instead, FlashInfer references were concentrated in compiler/runtime paths and optimisation flags, including:

python/mlc_llm/compiler_pass
python/mlc_llm/interface/compiler_flags.py
python/mlc_llm/op/extern.py
python/mlc_llm/op/attention.py

Therefore, flashinfer-python was deliberately not installed at this stage. To avoid downloading a precompiled wheel, MLC-LLM itself was later installed with dependency resolution disabled.

For model compilation before building flashinfer-python from source, FlashInfer can be explicitly disabled via the optimisation string, for example:

--opt "O2;flashinfer=0"

16. Building tokenizers-cpp from Source

The tokenizers-cpp source tree is located at:

~/repos/mlc-llm/3rdparty/tokenizers-cpp

The checked-out commit was:

34885cfd7b9ef27b859c28a41e71413dd31926f5

The following submodules were already initialised:

msgpack:       092bc69b6e815980bce7808595c914dd3a29f905
sentencepiece: 11051e3b73b3a6222a52acd720e39805dc7545ab

The project is a CMake-based C++ library with a Rust/Cargo component. It produces static libraries rather than a Python package.

Important build characteristics:

CMake project: tokenizers_cpp
Rust crate: tokenizers-c
Rust crate type: staticlib
SentencePiece tokenizer: enabled
msgpack: local submodule
sentencepiece: local submodule

The Rust dependency declaration is located at:

~/repos/mlc-llm/3rdparty/tokenizers-cpp/rust/Cargo.toml

The key dependencies were:

tokenizers = { version = "0.21.2", default-features = false, features = ["onig"] }
serde = { version = "1.0", features = [ "derive" ] }
serde_json = "1.0"

The Rust toolchain available in this environment was:

rustc 1.95.0
cargo 1.95.0

Cargo was configured to use the USTC crates.io mirror:

[source.crates-io]
replace-with = "ustc"

[source.ustc]
registry = "https://mirrors.ustc.edu.cn/crates.io-index"

[net]
git-fetch-with-cli = true

16.1 Generating Cargo.lock

The repository did not include rust/Cargo.lock. Generate it first:

source ~/repos/mlc-llm/env-tvm-source.sh

cd ~/repos/mlc-llm/3rdparty/tokenizers-cpp/rust

cargo generate-lockfile

This successfully generated:

~/repos/mlc-llm/3rdparty/tokenizers-cpp/rust/Cargo.lock

Cargo locked 78 packages. The resolved tokenizers crate version was:

tokenizers v0.21.4

The generated Cargo.lock is ignored by the repository’s Git configuration and appears as an ignored file:

!! rust/Cargo.lock

16.2 Configuring tokenizers-cpp

Configure the build:

source ~/repos/mlc-llm/env-tvm-source.sh

cd ~/repos/mlc-llm/3rdparty/tokenizers-cpp

rm -rf build
mkdir -p build

cmake -S . -B build \
  -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_C_COMPILER=/usr/bin/gcc \
  -DCMAKE_CXX_COMPILER=/usr/bin/g++ \
  -DMLC_ENABLE_SENTENCEPIECE_TOKENIZER=ON

The configure stage completed successfully and wrote build files to:

~/repos/mlc-llm/3rdparty/tokenizers-cpp/build

16.3 Building tokenizers-cpp

Build the static libraries:

source ~/repos/mlc-llm/env-tvm-source.sh

cd ~/repos/mlc-llm/3rdparty/tokenizers-cpp

cmake --build build -j64

The build completed successfully and generated:

~/repos/mlc-llm/3rdparty/tokenizers-cpp/build/libtokenizers_c.a
~/repos/mlc-llm/3rdparty/tokenizers-cpp/build/libtokenizers_cpp.a
~/repos/mlc-llm/3rdparty/tokenizers-cpp/build/release/libtokenizers_c.a
~/repos/mlc-llm/3rdparty/tokenizers-cpp/build/sentencepiece/src/libsentencepiece.a

At this point, tokenizers-cpp had been successfully built from local source.

7. Installing and validating MLC-LLM

17. Installing MLC-LLM from Local Source

After TVM, apache-tvm-ffi, XGrammar, and tokenizers-cpp had been built, MLC-LLM itself was installed from the local checkout.

The source tree is located at:

~/repos/mlc-llm

The checked-out MLC-LLM commit was:

d1ea69a87280e821611643958bfec385b62dafd3

The project metadata declares the following relevant dependencies:

apache-tvm-ffi
flashinfer-python; sys_platform == 'linux'
requests
torch
tqdm
transformers

Since flashinfer-python was not available as a local source tree and should not be pulled as a prebuilt wheel, MLC-LLM was installed with dependency resolution disabled:

source ~/repos/mlc-llm/env-tvm-source.sh

cd ~/repos/mlc-llm

python3 -m pip install \
  --user \
  --no-build-isolation \
  --no-binary=:all: \
  --no-deps \
  -e .

The installation completed successfully:

Successfully built mlc_llm
Successfully installed mlc_llm-0.20.0.dev0

The editable wheel created during installation was a local build artefact:

mlc_llm-0.20.0.dev0-py3-none-linux_x86_64.whl

No flashinfer-python wheel was downloaded.

The installed package was confirmed to point back to the local source tree:

mlc_llm:
~/repos/mlc-llm/python/mlc_llm/__init__.py

distribution:
mlc_llm 0.20.0.dev0
editable: true

The current editable install path did not generate a mlc_llm console script under:

~/.local/bin

However, the module CLI is available and should be used as:

python3 -m mlc_llm ...

For example:

python3 -m mlc_llm --help
python3 -m mlc_llm compile --help
python3 -m mlc_llm chat --help
python3 -m mlc_llm serve --help

18. MLC-LLM Native Runtime Libraries

The MLC-LLM native runtime libraries were present under the local build directory:

~/repos/mlc-llm/build/libmlc_llm.so
~/repos/mlc-llm/build/libmlc_llm_module.so

They were discoverable through mlc_llm.libinfo:

mlc_llm:
~/repos/mlc-llm/build/libmlc_llm.so

mlc_llm_module:
~/repos/mlc-llm/build/libmlc_llm_module.so

The dynamic linker resolved their dependencies successfully, including:

libtvm.so
libtvm_runtime.so
libtvm_ffi.so
libflash_attn.so
libfpA_intB_gemm.so
libcudart.so.13
libcuda.so.1
libcudnn.so.9
libcublas.so.13
libcublasLt.so.13

No missing shared-library dependency was observed.

Representative resolved paths included:

libtvm.so:
~/repos/mlc-llm/3rdparty/tvm/build/libtvm.so

libtvm_runtime.so:
~/repos/mlc-llm/3rdparty/tvm/build/libtvm_runtime.so

libtvm_ffi.so:
~/repos/mlc-llm/3rdparty/tvm/build/lib/libtvm_ffi.so

libflash_attn.so:
~/repos/mlc-llm/3rdparty/tvm/build/3rdparty/libflash_attn/src/libflash_attn.so

libfpA_intB_gemm.so:
~/repos/mlc-llm/3rdparty/tvm/build/3rdparty/cutlass_fpA_intB_gemm/cutlass_kernels/libfpA_intB_gemm.so

19. MLC-LLM Python and Runtime Smoke Validation

The following Python modules loaded successfully from the local source-built stack:

tvm
tvm_ffi
tvm_ffi.core
xgrammar
xgrammar.xgrammar_bindings
mlc_llm
mlc_llm.base
mlc_llm.support
mlc_llm.interface.compiler_flags
mlc_llm.interface.compile
mlc_llm.op.extern
mlc_llm.op.attention

The resolved source paths were:

tvm:
~/repos/mlc-llm/3rdparty/tvm/python/tvm/__init__.py

tvm_ffi:
~/repos/mlc-llm/3rdparty/tvm/3rdparty/tvm-ffi/python/tvm_ffi/__init__.py

tvm_ffi.core:
~/.local/lib/python3.10/site-packages/tvm_ffi/core.cpython-310-x86_64-linux-gnu.so

xgrammar:
~/repos/mlc-llm/3rdparty/xgrammar/python/xgrammar/__init__.py

xgrammar_bindings:
~/repos/mlc-llm/3rdparty/xgrammar/python/xgrammar/xgrammar_bindings.cpython-310-x86_64-linux-gnu.so

mlc_llm:
~/repos/mlc-llm/python/mlc_llm/__init__.py

CUDA was visible through TVM:

cuda: True

The MLC-LLM module CLI loaded successfully:

python3 -m mlc_llm --help

The following subcommands also loaded their help pages successfully without traceback:

compile
convert_weight
gen_config
chat
serve
package

The compile command exposed the expected model compilation interface, including:

--quantization
--model-type
--device
--host
--opt
--output
--overrides
--debug-dump

The compiler option string can be used to control optimisation flags, for example:

--opt "O2;flashinfer=0"

This is useful while flashinfer-python remains intentionally uninstalled.

20. Preventing Environment Variable Duplication

The first version of env-tvm-source.sh prepended paths directly. Repeatedly sourcing it caused PATH, PYTHONPATH, and LD_LIBRARY_PATH to grow with duplicate entries.

The script was rewritten to be idempotent.

The final environment script is:

# Source-built MLC-LLM environment for this checkout.
# Usage:
#   source ~/repos/mlc-llm/env-tvm-source.sh
#
# This script is intentionally sourceable and idempotent.
# It does not modify shell startup files.

export MLC_LLM_HOME="$HOME/repos/mlc-llm"
export TVM_HOME="$MLC_LLM_HOME/3rdparty/tvm"
export XGRAMMAR_HOME="$MLC_LLM_HOME/3rdparty/xgrammar"
export TOKENIZERS_CPP_HOME="$MLC_LLM_HOME/3rdparty/tokenizers-cpp"

_path_prepend_once() {
  case ":${PATH:-}:" in
    *":$1:"*) ;;
    *) export PATH="$1:${PATH:-}" ;;
  esac
}

_pythonpath_prepend_once() {
  case ":${PYTHONPATH:-}:" in
    *":$1:"*) ;;
    *) export PYTHONPATH="$1:${PYTHONPATH:-}" ;;
  esac
}

_ldpath_prepend_once() {
  case ":${LD_LIBRARY_PATH:-}:" in
    *":$1:"*) ;;
    *) export LD_LIBRARY_PATH="$1:${LD_LIBRARY_PATH:-}" ;;
  esac
}

_pythonpath_prepend_once "$XGRAMMAR_HOME/python"
_pythonpath_prepend_once "$MLC_LLM_HOME/python"
_pythonpath_prepend_once "$TVM_HOME/python"

_ldpath_prepend_once "$TVM_HOME/build/lib"
_ldpath_prepend_once "$TVM_HOME/build"
_ldpath_prepend_once "/usr/local/cuda/lib64"

_path_prepend_once "/usr/local/cuda/bin"
_path_prepend_once "$HOME/.local/bin"

unset -f _path_prepend_once
unset -f _pythonpath_prepend_once
unset -f _ldpath_prepend_once

The script passed shell syntax validation:

bash -n ~/repos/mlc-llm/env-tvm-source.sh

Repeatedly sourcing the script no longer duplicates the relevant paths.

The final effective variables are:

MLC_LLM_HOME=~/repos/mlc-llm
TVM_HOME=~/repos/mlc-llm/3rdparty/tvm
XGRAMMAR_HOME=~/repos/mlc-llm/3rdparty/xgrammar
TOKENIZERS_CPP_HOME=~/repos/mlc-llm/3rdparty/tokenizers-cpp

The script remains local and does not modify .bashrc or any system-wide shell configuration.

21. Verifying That No Critical Prebuilt Package Was Accidentally Installed

The key installed Python distributions were:

apache-tvm-ffi:
0.1.11.dev0+g1fed0ae04.d20260505
editable local source:
~/repos/mlc-llm/3rdparty/tvm/3rdparty/tvm-ffi

xgrammar:
0.0.3
editable local source:
~/repos/mlc-llm/3rdparty/xgrammar/python

mlc_llm:
0.20.0.dev0
editable local source:
~/repos/mlc-llm

The following packages were not installed:

flashinfer-python
apache-tvm
tlcpack
mlc-ai-nightly
mlc-llm-nightly

This confirms that the stack did not accidentally switch to an unrelated TVM wheel or MLC nightly package.

8. Local modifications and build state

22. Local Source Modifications Introduced During the Build

The build required several local source changes. These changes are intentional and should be preserved or documented before any reset or clean checkout.

22.1 TVM CMake Configuration

Modified file:

~/repos/mlc-llm/3rdparty/tvm/cmake/config.cmake

The important local changes are:

USE_CUDA=/usr/local/cuda
USE_LLVM=/usr/bin/llvm-config-18
USE_CUDNN=ON
USE_CUBLAS=ON
USE_CUTLASS=ON
USE_FLASHINFER=ON
CUDNN_INCLUDE_DIR=/usr/include/x86_64-linux-gnu
CUDNN_LIBRARY=/usr/lib/x86_64-linux-gnu/libcudnn.so

22.2 libflash_attn H800 Patch

Modified file:

~/repos/mlc-llm/3rdparty/tvm/3rdparty/libflash_attn/src/CMakeLists.txt

The local patch changed the hard-coded Ampere target to Hopper:

compute_80 / sm_80 -> compute_90 / sm_90
CUDA_ARCHITECTURES "80" -> "90"

This was required for the H800 environment.

22.3 tvm-ffi Source-Development Fallback Patch

Modified file:

~/repos/mlc-llm/3rdparty/tvm/3rdparty/tvm-ffi/python/tvm_ffi/libinfo.py

The patch allows apache-tvm-ffi to continue to its existing fallback search if importlib.metadata.distribution("apache-tvm-ffi") raises PackageNotFoundError.

The purpose is to support a source-tree development build where the native library already exists but Python package metadata may not yet exist.

22.4 Local Backup Files

The following local backup files were created during patching:

~/repos/mlc-llm/3rdparty/tvm/3rdparty/libflash_attn/src/CMakeLists.txt.bak.h800_sm90.20260505_072157
~/repos/mlc-llm/3rdparty/tvm/cmake/config.cmake.bak.20260505_071332
~/repos/mlc-llm/env-tvm-source.sh.bak.20260505_090200
~/repos/mlc-llm/env-tvm-source.sh.bak.idempotent.20260505_085858

These are local recovery artefacts and are not required for normal runtime use.

23. Local Build-State Snapshot

A local build-state summary file was created at:

~/repos/mlc-llm/BUILD_STATE_LOCAL.md

It records:

source roots
verified commits
environment script
built TVM state
apache-tvm-ffi state
XGrammar state
tokenizers-cpp state
MLC-LLM state
FlashInfer state
local patches
known non-blocking issues
fresh-shell smoke test result

The file was also normalised to use standard markdown code fences.

24. Current Completed State

The following parts of the stack have now been successfully built or configured from source:

MLC-LLM source tree:
~/repos/mlc-llm

Pinned TVM source:
~/repos/mlc-llm/3rdparty/tvm

TVM native libraries:
built with Ninja

TVM Python source import:
working

apache-tvm-ffi:
editable local source build installed

tvm_ffi.core native extension:
built from Cython source

XGrammar:
native binding built and Python package installed from editable local source

tokenizers-cpp:
C++ / Rust / SentencePiece static libraries built from local source

MLC-LLM:
editable local source install completed with --no-deps

MLC-LLM native runtime:
libmlc_llm.so and libmlc_llm_module.so available and linkable

CUDA visibility from TVM:
working

LLVM backend:
LLVM 18.1.8

CUDA architecture:
sm_90 / compute_90 for H800

cuDNN:
ON

cuBLAS:
ON

CUTLASS:
ON

NCCL:
OFF

flashinfer-python:
not installed

Important built artefacts:

~/repos/mlc-llm/3rdparty/tvm/build/libtvm.so
~/repos/mlc-llm/3rdparty/tvm/build/libtvm_runtime.so
~/repos/mlc-llm/3rdparty/tvm/build/lib/libtvm_ffi.so
~/.local/lib/python3.10/site-packages/tvm_ffi/core.cpython-310-x86_64-linux-gnu.so
~/repos/mlc-llm/3rdparty/xgrammar/python/xgrammar/xgrammar_bindings.cpython-310-x86_64-linux-gnu.so
~/repos/mlc-llm/3rdparty/tokenizers-cpp/build/libtokenizers_c.a
~/repos/mlc-llm/3rdparty/tokenizers-cpp/build/libtokenizers_cpp.a
~/repos/mlc-llm/3rdparty/tokenizers-cpp/build/sentencepiece/src/libsentencepiece.a
~/repos/mlc-llm/build/libmlc_llm.so
~/repos/mlc-llm/build/libmlc_llm_module.so
~/repos/mlc-llm/env-tvm-source.sh
~/repos/mlc-llm/BUILD_STATE_LOCAL.md

The correct environment entry point is:

source ~/repos/mlc-llm/env-tvm-source.sh

The correct CLI invocation form is:

python3 -m mlc_llm ...

rather than:

mlc_llm ...

because the editable install path did not generate a ~/.local/bin/mlc_llm console script.

A final fresh-shell smoke test passed:

bash -lc '
  source ~/repos/mlc-llm/env-tvm-source.sh
  python3 - <<PY
import tvm
import tvm_ffi
import tvm_ffi.core
import xgrammar
import xgrammar.xgrammar_bindings
import mlc_llm
print("cuda:", tvm.cuda().exist)
PY
  python3 -m mlc_llm --help
'

Successful result:

cuda: True
MLC-LLM CLI help loads successfully

At this point, the local source-built MLC-LLM development stack is functional for Python import, native runtime loading, CLI entry, and subsequent model compilation work. The remaining unresolved item is whether to build flashinfer-python from source or to continue compiling models with FlashInfer explicitly disabled via:

--opt "O2;flashinfer=0"

9. Appendix: session log — FlashInfer source integration

Date: 2026-05-05

Scope

This session continued the source-build setup of the MLC-LLM development stack on the Ubuntu/H800 machine.

The explicit objective was:

Integrate flashinfer-python from source into the existing MLC-LLM source-built environment.

The session did not download model weights, did not run a large model compilation, and did not modify global shell configuration such as .bashrc.

All source repositories remain under:

~/repos

The main MLC-LLM repository remains:

~/repos/mlc-llm

1. Initial Fresh-Shell Validation

A fresh-shell smoke test was run first by sourcing:

source ~/repos/mlc-llm/env-tvm-source.sh

The following components were verified to remain valid in a fresh shell:

TVM
TVM-FFI
XGrammar
MLC-LLM
MLC-LLM native libraries
MLC-LLM CLI
CUDA visibility

Observed paths:

tvm:
  /home/ailven.liu/repos/mlc-llm/3rdparty/tvm/python/tvm/__init__.py

tvm_ffi:
  /home/ailven.liu/repos/mlc-llm/3rdparty/tvm/3rdparty/tvm-ffi/python/tvm_ffi/__init__.py

tvm_ffi.core:
  /home/ailven.liu/.local/lib/python3.10/site-packages/tvm_ffi/core.cpython-310-x86_64-linux-gnu.so

xgrammar:
  /home/ailven.liu/repos/mlc-llm/3rdparty/xgrammar/python/xgrammar/__init__.py

xgrammar_bindings:
  /home/ailven.liu/repos/mlc-llm/3rdparty/xgrammar/python/xgrammar/xgrammar_bindings.cpython-310-x86_64-linux-gnu.so

mlc_llm:
  /home/ailven.liu/repos/mlc-llm/python/mlc_llm/__init__.py

MLC-LLM native libraries were located successfully:

/home/ailven.liu/repos/mlc-llm/build/libmlc_llm.so
/home/ailven.liu/repos/mlc-llm/build/libmlc_llm_module.so

CUDA was visible to TVM:

tvm.cuda().exist: True

At the beginning of the session, FlashInfer was confirmed as absent:

flashinfer-python: NOT_FOUND

2. MLC-LLM Source Inspection for FlashInfer Usage

The current MLC-LLM commit was confirmed:

d1ea69a87280e821611643958bfec385b62dafd3

The local repository state contained expected local changes/untracked state:

m 3rdparty/tvm
?? BUILD_STATE_LOCAL.md
?? env-tvm-source.sh
?? env-tvm-source.sh.bak.*

A local grep over the MLC-LLM and pinned TVM source tree found FlashInfer references in:

pyproject.toml
python/requirements.txt
python/mlc_llm/interface/compiler_flags.py
python/mlc_llm/compiler_pass/dispatch_kv_cache_creation.py
python/mlc_llm/compiler_pass/pipeline.py
python/mlc_llm/op/attention.py
python/mlc_llm/op/extern.py
cpp/serve/function_table.cc
cpp/serve/sampler/gpu_sampler.cc
3rdparty/tvm/python/tvm/relax/backend/cuda/flashinfer.py
3rdparty/tvm/python/tvm/relax/frontend/nn/llm/kv_cache.py
3rdparty/tvm/src/runtime/vm/attn_backend.*
3rdparty/tvm/src/runtime/vm/paged_kv_cache.cc

The MLC-LLM dependency declaration was confirmed as unpinned:

"flashinfer-python; sys_platform == 'linux'"

and:

python/requirements.txt:
flashinfer-python; sys_platform == "linux"

No exact FlashInfer version was pinned by MLC-LLM.

3. FlashInfer API Requirements Identified

The pinned TVM FlashInfer backend requires these Python APIs:

flashinfer.jit.gen_customize_batch_prefill_module
flashinfer.jit.gen_customize_batch_decode_module
flashinfer.jit.gen_batch_mla_module
flashinfer.gemm.gen_gemm_sm100_module

The main H800 attention and paged KV cache paths require the first three APIs.

The flashinfer.gemm.gen_gemm_sm100_module path is only used by the grouped GEMM generation path for compute capability 100 in the current TVM wrapper logic. The current machine is H800 / SM90, so this is not an immediate blocker for H800 FlashInfer attention, prefill, decode, paged KV cache, or MLA.

4. Python, PyTorch, CUDA, and Build Tool State

The Python environment was checked before building FlashInfer.

Observed state:

python3:
  /usr/bin/python3

Python:
  3.10.12

pip:
  22.0.2

setuptools:
  82.0.1

wheel:
  0.37.1

ninja:
  1.13.0

cmake Python package:
  NOT_FOUND

python3 -m build:
  NOT_FOUND

system cmake:
  /usr/bin/cmake
  cmake version 3.22.1

system git:
  /usr/bin/git
  git version 2.34.1

PyTorch/CUDA state:

torch:
  2.11.0+cu130

torch.version.cuda:
  13.0

torch.cuda.is_available:
  True

torch.cuda.device_count:
  8

device0:
  NVIDIA H800

capability0:
  (9, 0)

System CUDA compiler:

/usr/local/cuda/bin/nvcc

nvcc:
  CUDA 13.2
  V13.2.51

Important compatibility note:

PyTorch is built for CUDA 13.0, while the system nvcc is CUDA 13.2.

This combination was later validated successfully by FlashInfer decode, prefill, and MLA JIT build-and-load probes.

5. FlashInfer Version Selection

The official FlashInfer repository was queried:

https://github.com/flashinfer-ai/flashinfer.git

Remote HEAD was accessible:

HEAD -> refs/heads/main

The following tag was selected:

v0.6.10

Tag commit:

943d0ff0dfdbb5019b9733bd78ac730c5bff6f92

Reason for selecting v0.6.10:

It is a current stable FlashInfer tag.
It exposes the required flashinfer.jit APIs used by the pinned TVM backend.
Its pyproject.toml declares package name flashinfer-python.
It supports Python >=3.10,<4.0.

6. Remote Source API Check Before Clone

Before cloning, the tag contents were inspected through remote source reads and a temporary one-tag fetch.

Confirmed in v0.6.10:

flashinfer.jit.gen_customize_batch_prefill_module: present
flashinfer.jit.gen_customize_batch_decode_module: present
flashinfer.jit.gen_batch_mla_module: present

Confirmed layout:

flashinfer.gemm is a package, not a single flashinfer/gemm.py file.

Relevant files:

flashinfer/gemm/__init__.py
flashinfer/gemm/gemm_base.py
flashinfer/jit/gemm/__init__.py
flashinfer/jit/gemm/core.py

The SM100 GEMM generator exists here:

flashinfer.jit.gemm.gen_gemm_sm100_module

but is not exported from:

flashinfer.gemm

Therefore:

from flashinfer.gemm import gen_gemm_sm100_module

is incompatible with FlashInfer v0.6.10.

However:

from flashinfer.jit.gemm import gen_gemm_sm100_module

works.

This was recorded as a known non-blocking compatibility issue for H800.

7. FlashInfer v0.6.10 Requirements Check

The v0.6.10 requirements were inspected:

apache-tvm-ffi>=0.1.6,!=0.1.8,!=0.1.8.post0,<0.2
click
cuda-tile
einops
ninja
numpy
nvidia-cudnn-frontend>=1.13.0
nvidia-cutlass-dsl>=4.4.2
nvidia-ml-py
packaging>=24.2
requests
tabulate
torch
tqdm

The current Python environment already contained these distributions:

apache-tvm-ffi: 0.1.11.dev0+g1fed0ae04.d20260505
click: 8.3.3
cuda-tile: 1.3.0
einops: 0.8.2
ninja: 1.13.0
numpy: 2.2.6
nvidia-cudnn-frontend: 1.23.0
nvidia-cutlass-dsl: 4.4.2
nvidia-ml-py: 13.595.45
packaging: 26.2
requests: 2.33.1
tabulate: 0.10.0
torch: 2.11.0
tqdm: 4.67.3

Non-obvious import checks:

cutlass: IMPORT_OK
cuda: IMPORT_OK
cuda.bindings: IMPORT_OK
cuda_tile: IMPORT_FAIL
cudnn: IMPORT_OK
nvidia.cudnn_frontend: IMPORT_FAIL
pynvml: IMPORT_OK

The failed import checks were not treated as immediate blockers because the distribution metadata was present, and import names do not always match distribution names.

8. FlashInfer Source Clone

FlashInfer was cloned under the unified source root:

~/repos/flashinfer

Clone command shape:

cd ~/repos
git clone --recursive --branch v0.6.10 https://github.com/flashinfer-ai/flashinfer.git flashinfer

Verified state:

repository:
  /home/ailven.liu/repos/flashinfer

tag:
  v0.6.10

commit:
  943d0ff0dfdbb5019b9733bd78ac730c5bff6f92

Submodules:

3rdparty/cccl:
  876867684f7fac130e0f5911236e0a92a970d4fd
  v3.3.2

3rdparty/cutlass:
  da5e086dab31d63815acafdac9a9c5893b1c69e2
  v4.4.2

3rdparty/spdlog:
  c3aed4b68373955e1cc94307683d44dca1515d2b

The detached HEAD state is expected because v0.6.10 is a tag.

9. FlashInfer Build Metadata Probe

Before installation, PEP 517 editable metadata generation was tested manually.

The source contains:

pyproject.toml
build_backend.py
build_utils.py
requirements.txt

Initially, this generated file did not exist:

flashinfer/_build_meta.py

The metadata probe successfully created it:

"""Build metadata for flashinfer package."""
__version__ = "0.6.10"
__git_version__ = "943d0ff0dfdbb5019b9733bd78ac730c5bff6f92"

Editable metadata output:

flashinfer_python-0.6.10.dist-info

Metadata content confirmed:

Name: flashinfer-python
Version: 0.6.10
Requires-Python: <4.0,>=3.10

No CUDA compilation, wheel download, dependency resolution, or installation was performed during the metadata-only probe.

Warnings observed during metadata generation:

pyproject.toml does not contain setuptools_scm / vcs-versioning tool sections
Pattern 'LICENSE*.txt' did not match any files

These warnings did not block metadata generation.

10. FlashInfer Editable Source Installation

FlashInfer was installed as an editable local source package.

Command used:

cd ~/repos/flashinfer

python3 -m pip install \
  --user \
  --no-build-isolation \
  --no-binary=:all: \
  --no-deps \
  -e . \
  -v

Rationale:

--no-build-isolation:
  Preserve the current source-built Python environment, especially local apache-tvm-ffi.
  Avoid creating an isolated build environment that might resolve packages from PyPI.

--no-deps:
  Prevent pip from downloading, replacing, or resolving dependencies such as torch, cuda-tile,
  nvidia-cudnn-frontend, nvidia-cutlass-dsl, or apache-tvm-ffi.

--no-binary=:all:
  Preserve the source-build policy and reject prebuilt binary packages.

-e .:
  Install the local FlashInfer source tree in editable mode.

Installation result:

Successfully installed flashinfer-python-0.6.10

The install log only contained a locally generated PEP 660 editable wheel:

flashinfer_python-0.6.10-0.editable-py3-none-any.whl

This was not a downloaded binary wheel. It was a local editable-install metadata artifact.

Installed metadata:

flashinfer-python: 0.6.10
location: /home/ailven.liu/.local/lib/python3.10/site-packages

Editable import path:

/home/ailven.liu/repos/flashinfer/flashinfer/__init__.py

11. FlashInfer Python Import and API Validation

After editable installation:

flashinfer.__file__:
  /home/ailven.liu/repos/flashinfer/flashinfer/__init__.py

flashinfer.__version__:
  0.6.10

flashinfer.jit.__file__:
  /home/ailven.liu/repos/flashinfer/flashinfer/jit/__init__.py

Required JIT APIs were available:

flashinfer.jit.gen_customize_batch_prefill_module: True
flashinfer.jit.gen_customize_batch_decode_module: True
flashinfer.jit.gen_batch_mla_module: True

Known non-blocking incompatibility:

from flashinfer.gemm import gen_gemm_sm100_module

failed with:

ImportError: cannot import name 'gen_gemm_sm100_module' from 'flashinfer.gemm'

but:

from flashinfer.jit.gemm import gen_gemm_sm100_module

succeeded.

This remains non-blocking for the current H800 / SM90 path.

12. FlashInfer JIT Spec Generation Probe

Before compiling JIT kernels, FlashInfer JIT spec generation was tested.

Direct FlashInfer JIT generation succeeded for:

prefill
decode
MLA

Generated source locations:

/home/ailven.liu/.cache/flashinfer/0.6.10/90a/generated/...

Observed source counts:

prefill: 10 CUDA source files
decode:  3 CUDA source files
MLA:     3 CUDA source files

The 90a cache path indicates that FlashInfer generated Hopper-oriented sm_90a JIT artifacts for the H800 environment.

No CUDA compilation was performed in this spec-generation probe.

13. FlashInfer Direct JIT Build-and-Load Probes

The following environment variables were set for JIT compilation probes:

export CUDA_HOME=/usr/local/cuda
export CUDA_PATH=/usr/local/cuda
export CUDACXX=/usr/local/cuda/bin/nvcc
export TORCH_CUDA_ARCH_LIST="9.0a"

Decode JIT Probe

Direct FlashInfer decode JIT:

gen_customize_batch_decode_module(...).build_and_load()

Result:

build_and_load: OK
module_type: tvm_ffi.module.Module

Prefill JIT Probe

Direct FlashInfer prefill JIT:

gen_customize_batch_prefill_module(...).build_and_load()

Result:

build_and_load: OK
module_type: tvm_ffi.module.Module

MLA JIT Probe

Direct FlashInfer MLA JIT:

gen_batch_mla_module(...).build_and_load()

Result:

build_and_load: OK
module_type: tvm_ffi.module.Module

Conclusion:

FlashInfer direct JIT generation, CUDA compilation, linking, and runtime loading work for decode, prefill, and MLA on this H800 environment.

Validated toolchain combination:

FlashInfer: 0.6.10
PyTorch: 2.11.0+cu130
torch.version.cuda: 13.0
System nvcc: CUDA 13.2
GPU: NVIDIA H800
GPU capability: (9, 0)
TORCH_CUDA_ARCH_LIST: 9.0a

14. Environment Script Patch for FlashInfer JIT

The local source environment script was patched:

~/repos/mlc-llm/env-tvm-source.sh

A backup was created:

/home/ailven.liu/repos/mlc-llm/env-tvm-source.sh.bak.flashinfer.20260505_100638

The following block was appended:

# FlashInfer JIT CUDA toolchain for Hopper/H800.
# Keep these in the source-able local environment script instead of .bashrc.
# PyTorch is cu130, while system nvcc is CUDA 13.2; this combination has been
# validated by FlashInfer decode/prefill/MLA JIT build_and_load probes.
export CUDA_HOME="${CUDA_HOME:-/usr/local/cuda}"
export CUDA_PATH="${CUDA_PATH:-/usr/local/cuda}"
export CUDACXX="${CUDACXX:-/usr/local/cuda/bin/nvcc}"
export TORCH_CUDA_ARCH_LIST="${TORCH_CUDA_ARCH_LIST:-9.0a}"

Fresh-shell validation after patch:

CUDA_HOME=/usr/local/cuda
CUDA_PATH=/usr/local/cuda
CUDACXX=/usr/local/cuda/bin/nvcc
TORCH_CUDA_ARCH_LIST=9.0a

CUDACXX was executable:

/usr/local/cuda/bin/nvcc

No .bashrc or global system shell configuration was modified.

15. Fresh-Shell Full Stack Validation After FlashInfer Integration

After patching the environment script, a fresh-shell full stack check was run.

Command pattern:

bash -lc '
  source ~/repos/mlc-llm/env-tvm-source.sh
  ...
'

Confirmed environment:

CUDA_HOME=/usr/local/cuda
CUDA_PATH=/usr/local/cuda
CUDACXX=/usr/local/cuda/bin/nvcc
TORCH_CUDA_ARCH_LIST=9.0a

Confirmed stack imports:

torch: 2.11.0+cu130
torch.version.cuda: 13.0
torch.cuda.is_available: True
device0: NVIDIA H800
capability0: (9, 0)

tvm:
  /home/ailven.liu/repos/mlc-llm/3rdparty/tvm/python/tvm/__init__.py

mlc_llm:
  /home/ailven.liu/repos/mlc-llm/python/mlc_llm/__init__.py

flashinfer:
  /home/ailven.liu/repos/flashinfer/flashinfer/__init__.py

flashinfer.__version__:
  0.6.10

flashinfer.jit:
  /home/ailven.liu/repos/flashinfer/flashinfer/jit/__init__.py

tvm.cuda().exist:
  True

MLC-LLM native libraries remained discoverable:

/home/ailven.liu/repos/mlc-llm/build/libmlc_llm.so
/home/ailven.liu/repos/mlc-llm/build/libmlc_llm_module.so

Distribution metadata:

apache-tvm-ffi: 0.1.11.dev0+g1fed0ae04.d20260505
xgrammar: 0.0.3
mlc_llm: 0.20.0.dev0
flashinfer-python: 0.6.10

MLC-LLM CLI still worked:

python3 -m mlc_llm --help

16. TVM FlashInfer Backend Wrapper Validation

After validating direct FlashInfer JIT paths, the actual TVM wrapper path was tested:

from tvm.relax.backend.cuda import flashinfer as tvm_flashinfer

The following TVM wrapper APIs were compiled and loaded successfully:

tvm.relax.backend.cuda.flashinfer.gen_flashinfer_decode_module
tvm.relax.backend.cuda.flashinfer.gen_flashinfer_prefill_module
tvm.relax.backend.cuda.flashinfer.gen_flashinfer_mla_module

Results:

decode_mods_len: 1
prefill_mods_len: 1
mla_mods_len: 1

Returned module types:

<class 'tvm.runtime.module.Module'>

Final wrapper result:

TVM_BACKEND_FLASHINFER_WRAPPER_BUILD: OK

Conclusion:

The TVM backend wrapper path used by MLC-LLM can call FlashInfer, generate CUDA code, compile it, load it, and return TVM runtime modules.

17. MLC-LLM Optimisation Flag Behaviour

The compile optimisation flag behaviour was inspected through Python API and CLI help.

Detected target:

target: cuda
TVM target arch: sm_90
detected_cuda_arch_list: [90]

Quantisation used for check:

q0f16

Observed preset behaviour:

O0 -> flashinfer=False
O1 -> flashinfer=False
O2 -> flashinfer=True
O3 -> flashinfer=True

Therefore, for this H800 machine:

The default --opt O2 enables FlashInfer.

Explicit detailed knobs were tested:

flashinfer=0 -> flashinfer=False
flashinfer=1 -> flashinfer=True

Important parser constraint:

O2 is a preset and cannot be mixed with detailed knobs in the same semicolon-separated string.

Invalid:

--opt "O2;flashinfer=0"

Observed failure:

unrecognized arguments: --O2

Reason:

When the string is not exactly one of O0/O1/O2/O3, the parser treats each semicolon-separated segment as a detailed knob and prepends "--".
Therefore "O2;flashinfer=0" becomes "--O2 --flashinfer=0", but "--O2" is not a recognised detailed knob.

Valid examples:

--opt O2
--opt O3
--opt "flashinfer=0"
--opt "flashinfer=1"
--opt "cublas_gemm=1;cudagraph=0"

Note:

--opt "flashinfer=1" is not equivalent to "O2 plus only changing FlashInfer".
It explicitly sets only the provided detailed knobs and leaves other optimisation flags at their detailed-knob defaults.

18. MLC-LLM Compile CLI Help Check

The compile subcommand help was checked:

python3 -m mlc_llm compile --help

Confirmed important options:

--quantization
--model-type
--device
--host
--enable-subgroups
--opt
--system-lib-prefix
--output / -o
--overrides
--debug-dump

The CLI help states:

--opt OPT

Optimization flags. MLC LLM maintains a predefined set of optimization flags,
denoted as O0, O1, O2, O3, where O0 means no optimization, O2 means majority
of them, and O3 represents extreme optimization that could potentially break
the system. Meanwhile, optimization flags could be explicitly specified via
details knobs, e.g. --opt="cublas_gemm=1;cudagraph=0".
(default: "O2")

19. BUILD_STATE_LOCAL.md Update

The FlashInfer integration state and compile option semantics were appended to:

~/repos/mlc-llm/BUILD_STATE_LOCAL.md

The file was then validated.

Observed file state:

-rw-r--r-- 1 ailven.liu ailven.liu 11K 5 May 10:29 BUILD_STATE_LOCAL.md

Key sections found:

MLC-LLM Compile CLI and FlashInfer Integration Update
Current FlashInfer Integration Status
Next Recommended Step

Tail validation confirmed that the appended section was not truncated.

A local search was performed for existing MLC model configs:

find ~/repos \
  -path "*/.git" -prune -o \
  -name "mlc-chat-config.json" \
  -type f \
  -print

Result:

No mlc-chat-config.json was found under ~/repos.

Python recursive summary over ~/repos also found no local mlc-chat-config.json.

Conclusion:

There is currently no local MLC model config available for the next minimal compile-path probe.

No model was downloaded, no compile was run, and no model artifact was generated.

21. Current End-of-Session State

Current completed status:

TVM source build: OK
TVM Python import: OK
TVM CUDA visibility: OK
apache-tvm-ffi source/editable integration: OK
XGrammar source/editable integration: OK
tokenizers-cpp source build: OK
MLC-LLM source/editable integration: OK
MLC-LLM native libraries: OK
MLC-LLM CLI: OK
FlashInfer source clone: OK
FlashInfer submodules: OK
FlashInfer metadata generation: OK
FlashInfer editable source install: OK
FlashInfer Python import: OK
FlashInfer required JIT APIs: OK
FlashInfer direct decode JIT build/load: OK
FlashInfer direct prefill JIT build/load: OK
FlashInfer direct MLA JIT build/load: OK
TVM FlashInfer wrapper decode build/load: OK
TVM FlashInfer wrapper prefill build/load: OK
TVM FlashInfer wrapper MLA build/load: OK
env-tvm-source.sh FlashInfer JIT variables: OK
MLC-LLM --opt O2 FlashInfer enablement on H800: OK
Local mlc-chat-config.json availability: NOT_FOUND

Current unresolved item:

No local mlc-chat-config.json exists under ~/repos.

Known non-blocking compatibility issue:

FlashInfer v0.6.10 does not export gen_gemm_sm100_module from flashinfer.gemm.
The symbol exists under flashinfer.jit.gemm.
This affects the SM100 grouped GEMM wrapper path only and is not currently blocking H800 / SM90 attention, paged KV cache, decode, prefill, or MLA paths.

Since no local mlc-chat-config.json exists under ~/repos, the next step should not be a normal model compilation yet.

Recommended next action:

Create or locate a minimal local MLC model config suitable for a compiler-path probe,
without downloading model weights and without running a large compilation.

Possible next directions:

1. Search the MLC-LLM test tree for minimal model config examples.
2. Inspect MLC-LLM compiler tests for synthetic config construction.
3. Build a minimal local test config only if it can exercise the FlashInfer paged KV cache creation path.
4. Run a minimal compile-path probe with --opt O2 only after a suitable local config is available.

Do not proceed directly to downloading a HuggingFace model or compiling a large model until the minimal compile-path probe strategy is selected.