> ## Documentation Index > Fetch the complete documentation index at: https://lmsysorg-dsv4-1.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # XPU The document addresses how to set up the [SGLang](https://github.com/sgl-project/sglang) environment and run LLM inference on Intel GPU, [see more context about Intel GPU support within PyTorch ecosystem](https://docs.pytorch.org/docs/stable/notes/get_start_xpu.html). Specifically, SGLang is optimized for [Intel® Arc™ Pro B-Series Graphics](https://www.intel.com/content/www/us/en/ark/products/series/242616/intel-arc-pro-b-series-graphics.html) and [ Intel® Arc™ B-Series Graphics](https://www.intel.com/content/www/us/en/ark/products/series/240391/intel-arc-b-series-graphics.html). ## Optimized Model List A list of LLMs have been optimized on Intel GPU, and more are on the way:
Model Name BF16
Llama-3.2-3B [meta-llama/Llama-3.2-3B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct)
Llama-3.1-8B [meta-llama/Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct)
Qwen2.5-1.5B [Qwen/Qwen2.5-1.5B](https://huggingface.co/Qwen/Qwen2.5-1.5B)
**Note:** The model identifiers listed in the table above have been verified on [Intel® Arc™ B580 Graphics](https://www.intel.com/content/www/us/en/products/sku/241598/intel-arc-b580-graphics/specifications.html). Quantized MoE models are covered separately in [MXFP4 MoE Quantization](#mxfp4-moe-quantization) below. ## Installation ### Install From Source Currently SGLang XPU only supports installation from source. Please refer to ["Getting Started on Intel GPU"](https://docs.pytorch.org/docs/stable/notes/get_start_xpu.html) to install XPU dependency. ```bash Command theme={null} # Below commands shows Ubuntu/Debian OS as example, other OS requires similar command change # add sudo if required to install below packages # Install curl and protobuf compiler apt-get update && apt-get install -y curl protobuf-compiler # Install Rust (non-interactive) curl --proto '=https' --tlsv1.2 --retry 3 --retry-delay 2 -sSf https://sh.rustup.rs | sh -s -- -y # Make cargo available in the current shell source "$HOME/.cargo/env" # Create and activate a conda environment conda create -n sgl-xpu python=3.12 -y conda activate sgl-xpu # Set PyTorch XPU as primary pip install channel to avoid installing the larger CUDA-enabled version and prevent potential runtime issues. pip3 install torch==2.13.0+xpu torchvision==0.28.0+xpu torchaudio==2.11.0+xpu --index-url https://download.pytorch.org/whl/xpu pip3 install xgrammar==0.1.33 --no-deps # xgrammar will introduce CUDA-enabled triton which might conflict with XPU # Clone the SGLang code git clone https://github.com/sgl-project/sglang.git cd sglang git checkout # Use dedicated toml file cd python cp pyproject_xpu.toml pyproject.toml # Install SGLang dependent libs, and build SGLang main package pip install --upgrade pip setuptools pip install -v . --extra-index-url https://download.pytorch.org/whl/xpu ``` ### Install Using Docker [The SGLang XPU Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/xpu.Dockerfile) is provided to facilitate the installation. Replace `` below with your [HuggingFace access token](https://huggingface.co/docs/hub/en/security-tokens). ```bash Command theme={null} # Clone the SGLang repository git clone https://github.com/sgl-project/sglang.git cd sglang/docker # Build the docker image docker build -t sglang-xpu:latest -f xpu.Dockerfile . # Initiate a docker container docker run \ -it \ --privileged \ --ipc=host \ --network=host \ --user root \ --group-add $(getent group video | cut -d: -f3) \ --device /dev/dri \ -v /dev/dri/by-path:/dev/dri/by-path \ -v /dev/shm:/dev/shm \ -v ~/.cache/huggingface:/root/.cache/huggingface \ -p 30000:30000 \ -e "HF_TOKEN=" \ sglang-xpu:latest /bin/bash ``` ## Launch of the Serving Engine Example command to launch SGLang serving: ```bash theme={null} sglang serve \ --model-path \ --trust-remote-code \ --disable-overlap-schedule \ --device xpu \ --host 0.0.0.0 \ --tp 2 \ # using multi GPUs --attention-backend intel_xpu \ # using intel optimized XPU attention backend --page-size \ # intel_xpu attention backend supports [32, 64, 128] ``` ## MXFP4 MoE Quantization Native MXFP4 MoE checkpoints (OCP microscaling FP4: packed `e2m1` weights plus per-32-element `ue8m0` block scales) run on Intel GPUs through the `sgl-kernel-xpu` W4A16 grouped GEMM. The expert weights stay in the checkpoint's packed layout end to end — there is no dequantization to BF16 — so GPT-OSS-20B loads in roughly 13 GB rather than the \~42 GB a BF16 upcast would need, which is what lets it fit on a single 24 GB card. The `mxfp4` method is registered automatically on `--device xpu`, which already requires `sgl-kernel-xpu` — no extra flag is needed: ```bash theme={null} sglang serve \ --model-path openai/gpt-oss-20b \ --device xpu \ --attention-backend intel_xpu ``` **Tested models:** | Model | Notes | | :-------------------------------------------------------------- | :--------------------------------------------------------------------------------------------------------------------------------- | | [openai/gpt-oss-20b](https://huggingface.co/openai/gpt-oss-20b) | 32 experts, top-k 4, hidden 2880, intermediate 2880; clamped-swiglu activation (`gemm1_alpha` 1.702, limit 7.0) with expert biases | **Requirements and limitations:** | Item | Status | | ------------------ | --------------------------------------------------------------------------------------------------------------------------- | | GPU architecture | Xe2 / BMG only (Intel® Arc™ B-Series and Arc™ Pro B-Series) | | Quantized layers | Fused MoE experts only; attention, router, embeddings and LM head stay BF16 (per the checkpoint's `modules_to_not_convert`) | | Activations | BF16 (W4A16). Clamped swiglu (GPT-OSS) and plain SiLU are supported | | Expert parallelism | Not yet validated on this path | ## Benchmarking with Requests You can benchmark the performance via the `bench_serving` script. Run the command in another terminal. ```bash theme={null} python -m sglang.bench_serving \ --dataset-name random \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 1 \ --request-rate inf \ --random-range-ratio 1.0 ``` The detail explanations of the parameters can be looked up by the command: ```bash theme={null} python -m sglang.bench_serving -h ``` Additionally, the requests can be formed with [OpenAI Completions API](../basic_usage/openai_api_completions) and sent via the command line (e.g. using `curl`) or via your own script. ## XPU Graph \[Experimental] SGLang enables XPU graph capture to reduce per-step kernel-launch overhead. | Phase | Backend | Mechanism | Default | | ------- | -------------- | -------------------------------------------------------------------------------------------------------------------- | ---------------- | | Decode | `full` | One `torch.xpu.XPUGraph` per batch size, captured on startup | **Off** (opt-in) | | Prefill | `tc_piecewise` | `torch.compile` + XPU graph, one graph segment per token-length bucket | **Off** (opt-in) | | Prefill | `breakable` | Segmented `torch.xpu.XPUGraph` capture/replay (no `torch.compile`); eager break points at attention / MoE boundaries | **Off** (opt-in) | ### Enable Decode Graph Decode graph capture is **opt-in** on XPU. Enable it explicitly: ```bash theme={null} python -m sglang.launch_server --model-path --device xpu \ --cuda-graph-backend-decode full ``` ### Enable Prefill Graph Prefill graph capture is **opt-in** on XPU and must be enabled explicitly. Two backends are available: `tc_piecewise` and `breakable`. #### tc\_piecewise Uses `torch.compile` plus an XPU graph, one graph segment per token-length bucket: ```bash theme={null} python -m sglang.launch_server --model-path --device xpu \ --cuda-graph-backend-prefill tc_piecewise ``` By default the prefill subgraphs are compiled with `eager` mode. Switch to `inductor` for higher-quality generated code at the cost of longer startup: ```bash theme={null} python -m sglang.launch_server --model-path --device xpu \ --cuda-graph-backend-prefill tc_piecewise \ --cuda-graph-tc-compiler inductor ``` #### breakable Captures the transformer stack as segmented `XPUGraph`s with eager break points at attention / MoE boundaries, without `torch.compile`: ```bash theme={null} python -m sglang.launch_server --model-path --device xpu \ --cuda-graph-backend-prefill breakable ``` You can also configure both phases together with a single `--cuda-graph-config` JSON argument: ```bash theme={null} python -m sglang.launch_server --model-path --device xpu \ --cuda-graph-config '{"decode":{"backend":"full"},"prefill":{"backend":"tc_piecewise","tc_compiler":"eager"}}' ``` ### Enable torch.compile for Decode `--enable-torch-compile` adds a `torch.compile` pass on top of the decode XPU graph: the model forward is compiled first, and the compiled forward is then captured as an `XPUGraph`. This can reduce per-kernel overhead further but increases startup time. ```bash theme={null} python -m sglang.launch_server --model-path --device xpu \ --enable-torch-compile ``` > **Note:** `--enable-torch-compile` is mutually exclusive with the prefill > `tc_piecewise` graph (the compatibility rules auto-disable it). Use them > separately or lock the prefill backend explicitly via `--cuda-graph-config` > if you need both. ### Disable XPU Graph Both phases are disabled by default. To explicitly disable them anyway: ```bash theme={null} # Disable decode graph (already off by default; explicit form) python -m sglang.launch_server --model-path --device xpu \ --cuda-graph-backend-decode=disabled # Disable prefill graph (already off by default; explicit form) python -m sglang.launch_server --model-path --device xpu \ --cuda-graph-backend-prefill=disabled # Disable both phases python -m sglang.launch_server --model-path --device xpu \ --cuda-graph-backend-decode=disabled \ --cuda-graph-backend-prefill=disabled ``` ### Customize Capture Buckets By default, prefill capture sizes are derived from `--chunked-prefill-size`. To specify explicit token-length buckets: ```bash theme={null} python -m sglang.launch_server \ --model-path --device xpu \ --cuda-graph-backend-prefill tc_piecewise \ --cuda-graph-bs-prefill 64 128 256 512 ``` To specify explicit decode graph batch sizes: ```bash theme={null} python -m sglang.launch_server \ --model-path --device xpu \ --cuda-graph-bs-decode 1 2 4 8 ``` ### Server Args | Argument | XPU allowed values | Default | Description | | ------------------------------ | --------------------------------------- | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--cuda-graph-backend-decode` | `full`, `disabled` | `disabled` | Backend for the decode phase. Only `full` is supported on XPU. Set to `full` to enable. | | `--cuda-graph-backend-prefill` | `tc_piecewise`, `breakable`, `disabled` | `disabled`\* | Backend for the prefill phase. Set to `tc_piecewise` or `breakable` explicitly to enable. | | `--cuda-graph-tc-compiler` | `eager`, `inductor` | `eager` | Compiler for `tc_piecewise` prefill subgraphs. `inductor` produces more optimized code but has longer startup. | | `--cuda-graph-bs-prefill` | list of ints | auto | Explicit token-length buckets to capture for prefill. | | `--cuda-graph-bs-decode` | list of ints | auto | Explicit batch sizes to capture for decode. | | `--cuda-graph-config` | JSON string | — | One-shot JSON config for both phases, e.g. `'{"decode":{"backend":"full"},"prefill":{"backend":"tc_piecewise","tc_compiler":"eager"}}'`. Overrides all per-phase flags. | | `--disable-decode-cuda-graph` | — | `False` | Shorthand for `--cuda-graph-backend-decode=disabled`. | | `--disable-prefill-cuda-graph` | — | `False` | Shorthand for `--cuda-graph-backend-prefill=disabled`. | | `--enable-torch-compile` | — | `False` | Apply `torch.compile` on top of the decode XPU graph for further kernel optimization. | | `--torch-compile-max-bs` | int | `32` | Maximum batch size compiled by `torch.compile` when `--enable-torch-compile` is set. | \* Prefill graph is auto-disabled on XPU unless you lock the backend explicitly via `--cuda-graph-backend-prefill` or `--cuda-graph-config`. ### Limitations | Feature | Status | | ------------------------------------------------ | ------------------- | | Two-batch overlap (`--enable-two-batch-overlap`) | Not yet supported | | Speculative decoding | Not yet implemented | ## Memory Saver (release/resume memory occupation) on Intel XPU \[Experimental] SGLang can temporarily release most of the GPU memory it holds — model weights and/or KV cache — and reclaim it later without restarting the process. This is the same `release_memory_occupation` / `resume_memory_occupation` feature available on CUDA, used for RL rollout/training hand-off and for freeing the device between inference bursts. This is backed by the [`torch_memory_saver`](https://github.com/fzyzcjy/torch_memory_saver) package — the same package used on CUDA — which gained an Intel XPU backend built natively on Level Zero (keeping virtual addresses fixed while releasing/re-committing physical pages via `zeVirtualMemUnmap` / `zeVirtualMemMap`). **Install `torch_memory_saver`.** Unlike CUDA (prebuilt wheel), the XPU backend is built from source against your local oneAPI + `torch+xpu` runtime (the `.so` links `libsycl.so.`, which must match the installed `intel-sycl-rt`). `TMS_PLATFORM=xpu` forces the XPU backend, and `--no-build-isolation` lets the build import your installed `torch` so it can match the `libsycl` major to it: The published wheels are CUDA-only, so install from git and let it build. The ref below is the `v0.0.10b2` release, pinned so the build is reproducible. ```bash theme={null} source /opt/intel/oneapi/setvars.sh TMS_PLATFORM=xpu pip install --no-build-isolation \ git+https://github.com/fzyzcjy/torch_memory_saver.git@a5c99f11b18ebb8e9fda71a68812e476ae49e417 ``` **Use it** by launching with `--enable-memory-saver` (the XPU backend is selected automatically); optionally add `--enable-weights-cpu-backup` to keep weights in host RAM across a release: ```bash theme={null} python -m sglang.launch_server --model-path Qwen/Qwen3-0.6B \ --trust-remote-code --device xpu --enable-memory-saver ``` ```bash theme={null} # Release GPU memory while idle, then reclaim it (server must be idle). curl -X POST http://127.0.0.1:30000/release_memory_occupation curl -X POST http://127.0.0.1:30000/resume_memory_occupation ``` The Python engine API (`engine.release_memory_occupation(tags=...)` / `engine.resume_memory_occupation(tags=...)`) and the `weights` / `kv_cache` tags behave the same as on CUDA. Pauseable CUDA-graph capture is not used on XPU, so the `cuda_graph` tag is a no-op there. > **Verifying memory was freed:** neither `torch.xpu.memory_allocated()` nor > `torch.xpu.mem_get_info()` drops when physical pages are released — the first is > allocator accounting, and the second stays flat because torch keeps the freed block > cached. Query actual device memory via sysman (`ZES_ENABLE_SYSMAN=1`) instead. ## Prefill-Decode (P/D) Disaggregation on Intel XPU \[Experimental] SGLang supports prefill-decode disaggregation on Intel XPU using the [NIXL](https://github.com/ai-dynamo/nixl) KV-transfer backend. **Tested models:** | Model | Notes | | :-------------------------------------------------------------------------: | :----------------------------------------------------------------------------------------------: | | [Qwen/Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B) | Used in integration tests; verified on Intel XPU with homogeneous P/D (XPU prefill + XPU decode) | | [Qwen/Qwen2.5-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct) | Verified on Intel XPU with homogeneous P/D (XPU prefill + XPU decode) | **Prerequisites:** `pip install nixl sglang-router` **Start the prefill server (GPU 0):** ```bash theme={null} ZE_AFFINITY_MASK=0 UCX_POSIX_USE_PROC_LINK=n python -m sglang.launch_server \ --model-path Qwen/Qwen3-0.6B --trust-remote-code --device xpu \ --disaggregation-mode prefill --disaggregation-transfer-backend nixl \ --disaggregation-bootstrap-port 12335 --host 0.0.0.0 --port 30000 ``` **Start the decode server (GPU 1):** ```bash theme={null} ZE_AFFINITY_MASK=1 UCX_POSIX_USE_PROC_LINK=n python -m sglang.launch_server \ --model-path Qwen/Qwen3-0.6B --trust-remote-code --device xpu \ --disaggregation-mode decode --disaggregation-transfer-backend nixl \ --disaggregation-bootstrap-port 12335 --host 0.0.0.0 --port 30001 ``` **Start the router:** ```bash theme={null} python -m sglang_router.launch_router \ --pd-disaggregation \ --prefill http://127.0.0.1:30000 \ --decode http://127.0.0.1:30001 \ --host 0.0.0.0 --port 8000 ``` **Send a request:** ```bash theme={null} curl http://127.0.0.1:8000/v1/completions \ -H "Content-Type: application/json" \ -d '{"model": "Qwen/Qwen3-0.6B", "prompt": "The capital of France is", "max_tokens": 32}' ``` > **Note:** `UCX_POSIX_USE_PROC_LINK=n` is required on Intel XPU to avoid UCX shared-memory transport issues.