> ## Documentation Index > Fetch the complete documentation index at: https://lmsysorg-dsv4-1.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Attention Backends > Select and configure attention backends for SGLang diffusion pipelines. This document describes the attention backends available in sglang diffusion (`sglang.multimodal_gen`) and how to select them. ## Overview Attention backends are defined by `AttentionBackendEnum` (`sglang.multimodal_gen.runtime.platforms.interface.AttentionBackendEnum`) and selected via the CLI flag `--attention-backend`. Backend selection is performed by the shared attention layers (e.g. `LocalAttention` / `USPAttention` / `UlyssesAttention` in `sglang.multimodal_gen.runtime.layers.attention.layer`). `--attention-backend` is strict for the diffusion transformer / DiT. Auxiliary components such as encoders and VAEs use it when compatible, then fall back to a component default or a platform-compatible backend. Use `--component-attention-backends` when an auxiliary component must use a specific backend; incompatible overrides, including an override for a component that constructs no SGLang attention layer, fail unless a sparse backend is being replaced for cross-attention. When using the diffusers backend, `--attention-backend` is passed through to diffusers' `set_attention_backend` (e.g., `flash`, `_flash_3_hub`, `sage`, `xformers`, `native`). * **CUDA**: prefers FlashAttention (FA3/FA4) when supported; otherwise falls back to PyTorch SDPA. On SM100/B200, dense non-causal fp16/bf16 native attention prefers cuDNN SDPA and falls back to FA4 if cuDNN has no compatible kernel. * **ROCm**: uses FlashAttention when available; otherwise falls back to PyTorch SDPA. * **Intel XPU**: uses XPU Flash Attention backend (fp16/bf16, head sizes 64/96/128/192/256); otherwise falls back to PyTorch SDPA. * **MUSA**: uses FlashAttention when available; also supports Sage Attention when installed; otherwise falls back to PyTorch SDPA. * **MPS**: always uses PyTorch SDPA. * **NPU**: for ring attention uses FA otherwise uses PyTorch SDPA. ## Backend options For SGLang-native pipelines, the CLI accepts the lowercase names of `AttentionBackendEnum`. The table below lists the backends implemented by the built-in platforms. `fa3`/`fa4` are accepted as aliases for `fa`.
CLI value Enum value Notes
`fa` / `fa3` / `fa4` `FA` FlashAttention. fa3/fa4 are normalized to fa during argument parsing (ServerArgs.**post\_init**).
`torch_sdpa` `TORCH_SDPA` PyTorch scaled\_dot\_product\_attention.
`sliding_tile_attn` `SLIDING_TILE_ATTN` Sliding Tile Attention (STA). Requires st\_attn. Configure via --attention-backend-config.
`sage_attn` `SAGE_ATTN` Requires sageattention. On Hopper (SM90), PyPI sageattention==2.2.0 is unsupported because it lacks the upstream SM90 binding fix. Install pip install --force-reinstall git+[https://github.com/thu-ml/SageAttention.git@d9704247a5139ab4c03bf7fc6b35cc0e2cbb5ea4](https://github.com/thu-ml/SageAttention.git@d9704247a5139ab4c03bf7fc6b35cc0e2cbb5ea4) --no-build-isolation. Upstream SageAttention CUDA extensions target SM80/SM86/SM89/SM90/SM120; see upstream setup.py.
`sage_attn_3` `SAGE_ATTN_3` Requires SageAttention3 installed per upstream instructions.
`sparge_attn` `SPARGE_ATTN` Training-free sparse SageAttention2. CUDA SM80/86/87/89/90, FP16/BF16, head dim 64/128, and square self-attention with sequence length at least 128. Other attention shapes use dense SDPA. Install pip install git+[https://github.com/thu-ml/SpargeAttn.git](https://github.com/thu-ml/SpargeAttn.git) --no-build-isolation; configure retained block ratio with topk.
`sol_attn` `SOL_ATTN` Requires the upstream sol-attn package. Install with pip install git+[https://github.com/NVlabs/Sana.git@sol-engine#subdirectory=techniques/sparse\_backends](https://github.com/NVlabs/Sana.git@sol-engine#subdirectory=techniques/sparse_backends). BF16, head dim 128. Configure via --attention-backend-config.
`video_sparse_attn` `VIDEO_SPARSE_ATTN` Requires vsa. Configure sparsity via --attention-backend-config.
`subblock_sparse_attn` `SUBBLOCK_SPARSE_ATTN` Training-free SubBlock sparsity for MiniMax-H3 on CUDA SM90, SM100, and SM120. The default compute mode is BF16; compute\_mode=sage\_fp8 is an approximate SM90-only path that requires SpargeAttn. See the MiniMax-H3 recipe.
`video_sparse_attn_h3` `VIDEO_SPARSE_ATTN_H3` Video Sparse Attention for MiniMax-H3 / FastH3 (VSA-H3). In-tree Triton block-sparse kernel (SM90 / SM100 / SM103); no external package. Configure via --attention-backend-config.
`hybrid_window_attn_h3` `HYBRID_WINDOW_ATTN_H3` VDN-H3 hybrid attention: chunk-aligned window softmax (exact, gated) plus the Video Delta linear branch over the window complement. The window runs as a union of FlashAttention varlen calls (FA4 on SM100 / SM103 / SM120, FA3 on SM90; on SM80 / SM86 / SM89 the same FA3 build runs its Sm80 mainloop at FA2-class speed). Configure via --attention-backend-config.
`vmoba_attn` `VMOBA_ATTN` Requires kernel.attn.vmoba\_attn.vmoba. Configure via --attention-backend-config.
`aiter` `AITER` Requires aiter.
aiter\_sage AITER\_SAGE Requires aiter.
sla\_attn SLA\_ATTN Sparse Linear Attention. Requires SpargeAttn. Install with pip install git+[https://github.com/thu-ml/SpargeAttn.git](https://github.com/thu-ml/SpargeAttn.git) --no-build-isolation.
sage\_sla\_attn SAGE\_SLA\_ATTN SageAttention + Sparse Linear Attention. Requires SpargeAttn (same install as SLA).
`sparse_video_gen_2_attn` `SPARSE_VIDEO_GEN_2_ATTN` Requires svg. See installation instructions at [https://github.com/svg-project/Sparse-VideoGen](https://github.com/svg-project/Sparse-VideoGen).
laser\_attn LASER\_ATTN Requires attentions which can be installed with sgl\_kernel\_npu; available only for NPU.
block\_sparse\_attn BLOCK\_SPARSE\_ATTN Requires attentions which can be installed with sgl\_kernel\_npu; available only for NPU.
rain\_fusion\_attn RAIN\_FUSION\_ATTN Requires attentions which can be installed with sgl\_kernel\_npu; available only for NPU.
## Selection priority The selection order in `runtime/layers/attention/selector.py` is: 1. `global_force_attn_backend(...)` / `global_force_attn_backend_context_manager(...)` 2. Component override from `--component-attention-backends` while that component is being constructed 3. CLI `--attention-backend` (`ServerArgs.attention_backend`) 4. Layer or component default, when declared 5. Auto selection (platform capability, dtype, and installed packages) An explicit global backend mismatch fails for DiT self-attention. Auxiliary components may fall back to their declared default or another compatible backend. Sparse backends selected for self-attention similarly fall back to a compatible dense backend for cross-attention. Explicit component overrides are otherwise strict. Model and layer backend sets guide automatic selection; they are not positive allowlists for explicit requests. A backend selected with `--attention-backend` or `--component-attention-backends` can therefore be used without first adding it to every model's set. It still fails closed when the platform cannot resolve it or when it lacks a required operation such as dense cross-attention, packed variable-length attention, or ring-attention LSE. Model paths that require one backend for correctness declare it as required; those layers keep that backend even when the surrounding component is overridden. ## Configuration Some backends require additional configuration. You can pass these parameters via `--attention-backend-config`. This argument accepts: * A path to a JSON or YAML configuration file. * A JSON string (e.g., `'{"sparsity": 0.5}'`). * Key-value pairs (e.g., `"sparsity=0.5,enable_x=true"`). ### Supported Configuration Parameters **Sliding Tile Attention (`sliding_tile_attn`)**
Parameter Type Description Default
`mask_strategy_file_path` `str` **Required.** Path to the mask strategy JSON file. -
`sta_mode` `str` Mode of STA. `STA_inference`
`skip_time_steps` `int` Number of steps to use full attention before switching to sparse attention. `15`
**Video Sparse Attention (`video_sparse_attn`)**
Parameter Type Description Default
`sparsity` `float` Validation sparsity (0.0 - 1.0). `0.0`
**SpargeAttention (`sparge_attn`)** | Parameter | Type | Description | Default | | --------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | `topk` | `float` | Fraction of predicted attention blocks retained per head. Higher values preserve more attention work and generally improve quality. Must be in `(0, 1]`. | `0.5` | SpargeAttention is approximate even when `topk=1`: the recommended upstream kernel quantizes attention through SageAttention2. Validate output quality and end-to-end latency on the target model and resolution before deployment. **Video Sparse Attention for H3 (`video_sparse_attn_h3`)**
Parameter Type Description Default
`VSA_sparsity` `float` Fraction of video tiles excluded from the top-k selection (0.0 - 1.0). `0.9` is the FastH3 trained policy. `0.9`
`vsa_mode` `str` `exempt`: non-video keys (text/audio prefix tiles) are always selected. `compete`: they compete with video tiles in the top-k. `exempt`
`vsa_dense_first_n_steps` `int` Use dense attention for the first N denoising steps. `0`
`vsa_dense_layers` `list[int]` Layer indices kept dense, e.g. `[0, 1]`. `[]`
`vsa_tile_size` `int` Kernel tile size. Only `64` (the trained (4, 4, 4) geometry) is accepted. `64`
VSA-H3 constraints: * Only the DiT runs sparse; the token refiner, text encoder, and VAEs keep their dense defaults. An explicit `--component-attention-backends text_encoder=fa` is rejected because the H3 text encoder has SDPA-only layers. * Uses the checkpoint's trained `to_gate_compress` compression branch. Base MiniMax-H3 weights load zero gates and run pure sparse. * Ulysses sequence parallelism is supported; `--ring-degree` greater than 1, `torch.compile`, and breakable CUDA graph execution are rejected. **Hybrid window attention for VDN-H3 (`hybrid_window_attn_h3`)** [VDN-H3](/cookbook/diffusion/MiniMax/MiniMax-H3#7-vdn-h3-hybrid-attention-8-step-distill) replaces every DiT block's dense self-attention with two branches. The softmax branch is an exact softmax over a chunk-aligned frame window (frame `t` belongs to chunk `t // 5` and attends to chunks `c - 1 .. c + 1`; frames 0 and F-1 are dense as rows and columns; every text / audio pair stays dense) scaled by a per-head sigmoid gate. The linear branch (a frame-wise Video Delta rule recurrence, forward and reverse over frames) covers exactly the window's complement and is driven by the attention module. The mask is request-static, so the metadata is built once per request. The checkpoint's `transformer/config.json` carries the window geometry; the backend reads it.
Parameter Type Description Default
`vdn_h3_dense_smoke` `bool` Allows a dense transformer backend (`fa`) on the VDN-H3 weights for the base-H3 + LoRA equivalence smoke. The gates and the linear branch are skipped: a study, not a sample. `false`
`vdn_max_gather_rows` `int` Upper bound on the gathered K/V rows per window FlashAttention call; consecutive chunk groups fill one call up to it. Splitting changes no query's kept key set, only the pass count and the gather's peak memory. `200000`
Hybrid window attention constraints: * VDN-H3 requires this backend for the transformer; a dense backend on these weights would silently skip the linear branch and the gates, so it is rejected unless `vdn_h3_dense_smoke` is set. The token refiner, text encoder, and VAEs keep dense attention. * Base MiniMax-H3 and FastH3 checkpoints have no linear branch and are rejected by this backend. * Ulysses sequence parallelism is supported (QK-norm + RoPE run after the all-to-all on the head shard); `--ring-degree` greater than 1, `torch.compile`, and breakable CUDA graph execution are rejected. **V-MoBA (`vmoba_attn`)**
Parameter Type Description Default
`temporal_chunk_size` `int` Chunk size for temporal dimension. -
`temporal_topk` `int` Top-K tokens to select in temporal dimension. -
`spatial_chunk_size` `list[int]` Chunk size for spatial dimension (H, W). -
`spatial_topk` `int` Top-K tokens to select in spatial dimension. -
`st_chunk_size` `list[int]` Chunk size for spatiotemporal dimension (T, H, W). -
`st_topk` `int` Top-K tokens to select in spatiotemporal dimension. -
`moba_select_mode` `str` Selection mode (e.g., `threshold`). `threshold`
`moba_threshold` `float` Threshold value for selection. `0.25`
`moba_threshold_type` `str` Type of thresholding (e.g., `query_head`). `query_head`
`first_full_step` `int` Number of initial steps to use full attention. `12`
`first_full_layer` `int` Number of initial layers to use full attention. `0`
`temporal_layer` `int` Number of temporal layers. `1`
`spatial_layer` `int` Number of spatial layers. `1`
`st_layer` `int` Number of spatiotemporal layers. `1`
**Block Sparse Attention (`block_sparse_attn`)**
Parameter Type Description Default
`skip_first_steps` `int` Number of steps to use laser attention before switching to sparse attention. `10`
`sparsity` `float` The sparsity coefficient must be in the range (0, 1). `0.2`
**Sol-Attn (`sol_attn`)**
Parameter Type Description Default
`tau` `float` Routing threshold scale. Higher values select fewer exact KV blocks. `1.0`
`thresh_type` `str` Threshold mode: `diag` or `exact`. `diag`
`sink_tokens` `int` Exact KV sink length for prefix tokens such as text/audio rows. `0`
`sink_start` `int` Start index of the exact KV sink range. `0`
`dense_steps` `int` Use dense attention for the first N denoising steps. `10`
`dense_layers` `str` Layer indices kept dense, e.g. `0,1` or `0-2`. `0,1`
`dense_backend` `str` Backend used for the dense prefix: `fa` (default) or `sage_attn`. `sage_attn` is approximate. `fa`
`kv_splits` `int | str` KV split factor passed to the Sol-Attn kernel. Use `auto` on long sequences. `auto`
## Platform support matrix
Backend CUDA ROCm XPU MUSA MPS NPU Notes
`fa` Yes Yes ✅ ✅ ❌ ✅ CUDA requires SM80+ and fp16/bf16. XPU uses its own flash attention backend. FlashAttention is only used when the required runtime is installed; otherwise it falls back to torch\_sdpa. No extra installations are required for NPU
`torch_sdpa` Yes Yes Yes Yes ✅ ✅ Most compatible option across platforms.
`sliding_tile_attn` Yes No No No ❌ ❌ CUDA-only. Requires st\_attn. Configure via --attention-backend-config.
`sage_attn` Yes No No Yes ❌ ❌ Optional dependency on CUDA and MUSA. On Hopper, also falls back to FlashAttention when the installed package lacks the SM90 binding fix.
`sage_attn_3` Yes No No No ❌ ❌ CUDA-only (optional dependency).
`sparge_attn` Yes No No No ❌ ❌ CUDA SM80/86/87/89/90 only. Requires SpargeAttn; head dim 64/128 and square self-attention with sequence length at least 128.
`sol_attn` Yes No No No ❌ ❌ CUDA-only. Requires sol-attn. Install with pip install git+[https://github.com/NVlabs/Sana.git@sol-engine#subdirectory=techniques/sparse\_backends](https://github.com/NVlabs/Sana.git@sol-engine#subdirectory=techniques/sparse_backends). Configure via --attention-backend-config.
`video_sparse_attn` Yes No No No ❌ ❌ CUDA-only. Requires vsa. Configure sparsity via --attention-backend-config.
`video_sparse_attn_h3` Yes No No No ❌ ❌ CUDA-only (SM90 / SM100 / SM103). In-tree Triton kernel, no external dependency. Configure via --attention-backend-config.
sla\_attn Yes No No No ❌ ❌ CUDA-only. Requires SpargeAttn.
sage\_sla\_attn Yes No No No ❌ ❌ CUDA-only. Requires SpargeAttn.
vmoba\_attn Yes No No No ❌ ❌ CUDA-only. Requires kernel.attn.vmoba\_attn.vmoba. Configure via --attention-backend-config.
aiter ❌ ✅ No No ❌ ❌ Requires aiter.
aiter\_sage ❌ ✅ No No ❌ ❌ Requires aiter.
`sparse_video_gen_2_attn` Yes No No No ❌ ❌ CUDA-only. Requires svg.
laser\_attn ❌ ❌ ❌ ❌ ❌ ✅ NPU-only. Requires attentions from sgl\_kernel\_npu. Uses SDPA if seqlen less than 2048.
block\_sparse\_attn ❌ ❌ ❌ ❌ ❌ ✅ NPU-only. Requires attentions from sgl\_kernel\_npu. Configuration via --attention-backend-config.
rain\_fusion\_attn ❌ ❌ ❌ ❌ ❌ ✅ NPU-only. Requires attentions from sgl\_kernel\_npu Configuration via --attention-backend-config.
## Usage ### Select a backend via CLI ```bash theme={null} sglang generate \ --model-path \ --prompt "..." \ --attention-backend fa ``` ```bash theme={null} sglang generate \ --model-path \ --prompt "..." \ --attention-backend torch_sdpa ``` ### Override one component Use component overrides when a specific module needs different attention semantics from the main transformer: ```bash theme={null} sglang generate \ --model-path \ --prompt "..." \ --attention-backend fa \ --component-attention-backends text_encoder=torch_sdpa ``` Component keys match pipeline module names from `model_index.json`, such as `text_encoder`, `text_encoder_2`, `transformer`, `transformer_2`, or `connectors`. Use this override when the fallback must be pinned: unlike the global backend, an incompatible component override raises an error instead of selecting another backend. The one role-based exception is a sparse self-attention backend, which uses a compatible dense backend for cross-attention layers in the same component. The component must construct SGLang-selectable attention or explicitly defer selection until first use; components with fixed attention reject the override. Per-component overrides apply only to native pipelines. The Diffusers backend accepts the global `--attention-backend` passthrough instead. ### Per-request override (denoise loop) A single server can serve exact and approximate attention side by side: requests may switch the DiT denoise attention backend via the `attention_backend_override` sampling param. Valid values are the exact/drop-in dense kernels — `fa`, `torch_sdpa`, `sage_attn`, `sage_attn_3`. The field participates in the dynamic-batch signature, so requests with different backends never share a batch. ```bash theme={null} sglang generate \ --model-path \ --prompt "..." \ --attention-backend-override sage_attn ``` ```python theme={null} client.images.generate( model="", prompt="...", extra_body={"attention_backend_override": "sage_attn"}, ) ``` Incompatible server settings **reject the request** (with a server log) instead of silently falling back: breakable CUDA graphs and `torch.compile` bake the attention kernel into a captured/traced graph; sparse server backends (`sliding_tile_attn`, `video_sparse_attn`, ...) cannot be mixed with per-request dense switching; under ring parallelism the target must be ring-capable. Note `sage_attn` / `sage_attn_3` are lossy (quantized attention) — validate quality on your workload. ### Request-scoped Skip Softmax Skip Softmax (BLASST) keeps the QK matmul, but skips the exponential, softmax-state update, V load, and PV matmul for attention tiles whose estimated softmax mass is below a threshold. It is an explicit lossy optimization backed by the FlashInfer kernels shipped with SGLang's pinned dependencies. See the [BLASST paper](https://arxiv.org/abs/2512.12087) and NVIDIA's [video-generation study](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/blogs/tech_blog/blog28_Accelerating_Video_Generation_with_GEMM_Quantization_Attention_Quantization_and_Skip_Softmax_Attention_in_TensorRT-LLM.md) for the algorithm and measured quality/performance trade-offs. Pass `skip_softmax_params` in one image or video request. The runtime routes compatible self-attention layers through the FA/FlashInfer path for that batch; cross-attention keeps its normal backend, and the next request restores the server default. ```python theme={null} client.videos.create( model="", prompt="...", extra_body={ "skip_softmax_params": { "threshold_scale_factor": 500.0, "start_step": 14, } }, ) ``` * `threshold_scale_factor` is required and must be positive. The kernel uses `threshold_scale_factor / context_length`; larger values skip more work and usually increase quality loss. * `start_step` is the zero-based denoising step at which sparse execution starts. It defaults to `0`; keeping early high-noise steps dense is generally safer. There is intentionally no default threshold. Calibrate the threshold and start step against output-quality metrics for each model, resolution, step count, and deployment backend; values tuned for one workload are not portable. Current support is FP16/BF16, head dimension 128 or 256, and unmasked self-attention on Hopper SM90 and Blackwell SM100/SM103/SM107. Ulysses sequence parallelism is supported because the kernel runs after its all-to-all. Ring Attention, attention masks, `torch.compile`, and breakable CUDA graphs reject the request instead of silently running dense attention. Models that merely offer an FA backend do not automatically qualify: the runtime still checks the GPU, dtype, head dimension, attention role, and execution mode. ### Using SpargeAttention Install the optional CUDA extension, then select the backend explicitly: ```bash theme={null} pip install ninja pip install git+https://github.com/thu-ml/SpargeAttn.git --no-build-isolation sglang generate \ --model-path \ --prompt "..." \ --attention-backend sparge_attn \ --attention-backend-config topk=0.5 ``` `sparge_attn` applies to compatible self-attention layers. Cross-attention uses the existing dense fallback; short self-attention and asymmetric Q/KV also use dense SDPA. LTX-2, LTX-2.3, and LTX-2.5 have compatible video (128) and audio (64) head dimensions, so their sufficiently long, unmasked self-attention uses SpargeAttention while prompt and audio-video cross-attention remain dense. Ulysses sequence parallelism is supported, but ring attention is not because the upstream kernel does not expose the softmax LSE needed for ring merging. K/V-gather SP therefore follows the normal sparse-backend rule and uses Ulysses instead. ### Sage then Sol hybrid `sol_attn` keeps the first `dense_steps` steps dense. Set `dense_backend=sage_attn` to run that prefix on SageAttention and the tail on Sol sparse attention. Keep the text encoder on `torch_sdpa`: ```bash theme={null} sglang generate \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant fl2va \ --attention-backend sol_attn \ --attention-backend-config dense_backend=sage_attn,dense_steps=10 \ --component-attention-backends text_encoder=torch_sdpa,transformer=sol_attn \ --prompt "A cat walking on a sunny beach, gentle waves." \ --save-output ``` Both `sage_attn` and `sol_attn` are approximate. The default DiT backend remains `fa`. ### Using Sliding Tile Attention (STA) ```bash theme={null} # Pass the mask strategy file path via config sglang generate \ --model-path \ --prompt "..." \ --attention-backend sliding_tile_attn \ --attention-backend-config "mask_strategy_file_path=/abs/path/to/mask_strategy.json" ``` ### Notes for ROCm / MPS * ROCm: use `--attention-backend torch_sdpa` or `fa` depending on what is available in your environment. * MPS: the platform implementation always uses `torch_sdpa`.