> ## Documentation Index
> Fetch the complete documentation index at: https://lmsysorg-dsv4-1.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.
# Fused Kernels
> The fused CUDA/Triton kernels SGLang Diffusion ships, what each one replaces, and which are on by default.
Diffusion transformers and VAEs spend a large share of their non-GEMM time on short elementwise and selection chains — adaLN modulate, residual gating, QK-norm, RoPE, norm epilogues, and MoE routing — each of which can require several kernel launches and HBM round trips in eager PyTorch. SGLang Diffusion replaces these chains with fused kernels under [`sglang/kernels/ops/diffusion`](https://github.com/sgl-project/sglang/tree/main/python/sglang/kernels/ops/diffusion).
This page is an inventory: what each kernel fuses, what its numerical contract is, and which models use it. It is not a lever you tune — most of these kernels are on by default and require no flag. The one switch is `--quality`, described below.
## Numerical contracts and quality tiers
Multi-step denoising amplifies a per-step rounding difference into visible quality loss, so "close enough" and "bit-exact" are different products here. The `quality` switch distinguishes unconditional bit-exact replacements from non-bit-exact eager-chain fusions:
**Bit-exact — mounted unconditionally.** The kernel reproduces every rounding boundary of the eager chain, so `torch.equal` holds against the reference. Some go quite far to get there: the fused LayerNorm+modulate kernel replicates PyTorch's `vectorized_layer_norm_kernel` down to its Welford update order, guarded reciprocal, and warp-fold tree; the fused RMSNorm+scale/shift kernel replicates FlashInfer's CuTe-DSL `RMSNormKernel` fragment order and `shfl.bfly` fold. Because the dispatch they replicate can change underneath them, each one still verifies itself against the live eager chain on first sight and falls back permanently on any mismatch.
**Not bit-exact — request-gated.** These differ from eager only at half-precision rounding-order level, but that is enough to matter, so they are mounted only for `quality="extra-high"` and `quality="high"` requests, at batch boundaries, all-or-nothing per transformer. The default `quality="lossless"` runs the unmodified reference chain.
**Model/checkpoint-native.** Generic close-contract kernels, sparse operators, and FP8/NVFP4 producers can be part of a model implementation or a separately selected deployment path. They are documented in the inventory, but `quality` does not select or undo those choices.
**Selection-equivalent routing — enabled unconditionally.** LingBot Video's fused group-limited top-k returns the same selected expert-id set as its guarded CUDA `torch.topk(..., sorted=False)` reference chain. The order of those ids is not part of either path's contract. Because the selected experts are unchanged, this path does not depend on the request `quality` tier.
A plain fp32 single-pass norm fusion looks harmless and is not. On ERNIE-Image it moved the 50-step trajectory to 18.83 dB PSNR, which is what motivated the bit-exact rewrite of that path.
The quality levels are cumulative:
| `quality` | Included optimization set |
| ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `lossless` | The selected deployment's reference path plus every unconditional bit-exact replacement |
| `extra-high` | Everything in `lossless`, plus request-gated DiT and VAE kernel fusions; this level does not itself enable sparse, caching, or another approximate path |
| `high` | Everything in `extra-high`, plus any model-owned high-only optimization, such as an audited Cache-DiT policy or lower-precision VAE decode |
If a model has no eligible request-gated fusion, `extra-high` can execute the same path as `lossless`. Likewise, `high` adds only the model-specific high-only paths that the active pipeline implements.
`quality` is not a master precision switch. A quantized checkpoint, an explicitly selected approximate attention backend, or an independently enabled cache remains active at every quality tier.
## Enabling the request-gated set
```bash theme={null}
sglang generate --model-path MODEL_PATH --prompt "..." --quality extra-high
```
The server default stays `lossless`; the OpenAI-compatible endpoints carry it per request. Images:
```bash theme={null}
curl -X POST http://${HOST}:${PORT}/v1/images/generations \
-H 'Content-Type: application/json' \
-d '{"model": "MODEL_PATH", "prompt": "...", "quality": "extra-high"}'
```
Video, same field:
```bash theme={null}
curl -X POST http://${HOST}:${PORT}/v1/videos \
-H 'Content-Type: application/json' \
-d '{"model": "MODEL_PATH", "prompt": "...", "quality": "extra-high"}'
```
The `quality` field in a **video response** body is unrelated. It is Sora-compatible response metadata and is always reported as `"standard"`; it does not reflect the sampling quality the request ran with.
`quality` participates in the dynamic-batch signature, so mixed-quality traffic is batched separately and the transition happens safely at a batch boundary. Mounting is all-or-nothing: if any marked site on a transformer fails its static guards, no site on that transformer is fused.
Do not combine request-gated DiT fusions with `--enable-breakable-cuda-graph`.
BCG warmup captures the lossless module branches before an `extra-high` or
`high` request mounts its DiT fusions, so replay would bypass the requested
kernels. SGLang rejects this combination for models with eligible DiT quality
sites. Models whose request-gated path changes only VAE decode remain allowed
because BCG captures the DiT only.
These fusion families mount under both `quality="extra-high"` and
`quality="high"`:
| Fusion | What it folds |
| --------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Linear + tanh-GELU | Bias-add and GELU into the GEMM epilogue (cublasLt), removing the `[tokens, 4*dim]` intermediate round trip |
| Wan NVFP4 linear + GELU | Bias-add and GELU fused into Wan's NVFP4 FFN projection output |
| Qwen-Image added-QKV | Added Q/K/V projections fused into joint-buffer production |
| LayerNorm + modulate | `layer_norm(x, weight=(1 + scale), bias=shift)` in place of affine-free LN plus a separate modulate |
| LTX-2 RMSNorm + modulate | `rms_norm(x) * (1 + scale) + shift` in one launch |
| Gate RMSNorm (BF16-native) | `RMSNorm + tanh + mul + add` in one pass |
| HunyuanVideo strided QK RMSNorm | Per-head QK RMSNorm over the packed QKV layout |
| LingBot Video fused RMSNorm | Replaces the handwritten cast, square, mean, rsqrt, and multiply chain with existing Triton RMSNorm kernels |
| LingBot Video per-token gated residual + RMSNorm modulate | Folds `residual + gate * update` (per-token `[B, S, 1]` gate) and the `rmsnorm(x) * (1 + scale) + shift` adaLN chain (strided `[B, S, 6D]` chunk views) into single kernels |
| SANA-Video BF16-input linear attention | Keeps the first linear-attention GEMM's inputs in BF16 with FP32 accumulation/output; the second GEMM remains FP32 |
| FLUX-family VAE fast paths | Channels-last decode, GroupNorm(+SiLU), upsample, and attention replacements for FLUX.2 and AutoencoderKL-based FLUX.1, Z-Image, and SD3 pipelines |
| Wan VAE RMSNorm + SiLU | Replaces the channel-first RMSNorm/SiLU chain while keeping the decode in `channels_last_3d` |
## Kernel inventory
45 operators are registered in the kernel registry across 51 implementations (some operators carry several backends). Backends are named by provenance, not device: `KDA` identifies Kernel Design Agents implementations, `JIT` compiles under nvcc *and* hipcc, `TRITON` identifies Triton sources, `CUTE_DSL` needs CUTLASS, `FLYDSL` is ROCm gfx950 only, and `AOT` comes from the `sgl_kernel` wheel. Per-operator capability metadata determines which devices can load each implementation.
### Normalization
| Operator | Backend | Contract | Replaces |
| -------------------------------------------- | -------------------------------- | ----------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- |
| `rmsnorm_scale_shift` | Triton | bit-exact | RMSNorm + `* (1 + scale) + shift` (4 kernels) |
| `scale_residual_norm_scale_shift` | KDA / Triton / CuTe-DSL / FlyDSL | bit-exact (Triton) or backend-specific close contract | the above plus the preceding `residual + gate * update` |
| `scale_residual_norm_scale_shift_nvfp4` | JIT CUDA | matches the selected NVFP4 producer contract | Qwen residual LayerNorm/modulation + FC1 NVFP4 quantization |
| `layernorm_modulate` | Triton | bit-exact | affine-free LayerNorm + adaLN modulate |
| `qk_head_layernorm` | Triton | bit-exact | per-head LayerNorm on q/k |
| `qk_rmsnorm_native` | Triton | bit-exact | Z-Image per-head QK RMSNorm |
| `norm_scale_shift` | KDA / CuTe-DSL / FlyDSL | backend-specific close contract | LN-or-RMS + scale/shift, many broadcast modes |
| `rmsnorm_scale`, `rmsnorm_tanh_residual` | Triton | bf16-native statistics | `RMSNorm(x) * scale`, `x + tanh(gate) * RMSNorm(y)` |
| `apply_group_norm_silu` | Triton | close | `GroupNorm + SiLU`, NCHW-contiguous |
| `group_norm_silu_4d`, `group_norm_silu_rows` | Triton | close | channels-last GroupNorm(+SiLU); what lets a VAE decoder run channels\_last end to end with no `nchwToNhwc` transposes |
| `wan_rmsnorm_silu` | Triton | close | Wan VAE `channels_last_3d` RMSNorm + SiLU |
### adaLN modulation and gating
| Operator | Backend | Contract | Replaces |
| ---------------------- | -------------- | --------- | ------------------------------------------------------------ |
| `modulate_scale_shift` | JIT CUDA | bit-exact | `x * (1 + scale) + shift` |
| `residual_gate_add` | KDA (JIT CUDA) | bit-exact | `residual + gate * update` |
| `timestep_embedding` | JIT CUDA | close | sinusoidal timestep embedding |
| `temb_table_slices` | Triton | bit-exact | see note below |
| `ltx2_ada_values` | Triton | bit-exact | LTX-2 nine-way adaLN value split, slices come out contiguous |
`temb_table_slices` is worth knowing about. The eager `(scale_shift_table + temb.float()).chunk(6, dim=2)` materializes roughly 8 GB of fp32 at 704p/121f *and* hands six **strided** slices downstream, whose `.contiguous()` calls then copy each one again. The fused kernel produces the six slices in one pass, each naturally contiguous, so the downstream copies become no-ops.
### RoPE and QK-norm
| Operator | Backend | Contract | Replaces |
| --------------------------- | -------------- | -------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- |
| `fused_inplace_qknorm_rope` | JIT CUDA | one bf16 rounding step vs the split baseline; exact with `round_norm_before_rope=True` | separate QK-norm kernel + RoPE |
| `flux2_qkv_epilogue` | KDA (JIT CUDA) | bit-exact against its selected BF16 reference chain | FLUX.2 QK RMSNorm + RoPE + joint text/image QKV packing |
| `qwen_qkv_epilogue` | JIT CUDA | bit-exact against its selected BF16 reference chain | Qwen-Image QK RMSNorm + RoPE + joint QKV writes on SM100+ |
| `rope_rotate_half` | Triton | bit-exact | `chunk` → `cat(-x2, x1)` → two muls + add → `cat(tail)`, about 7 kernels per projection |
| `interleaved_rope_fp64` | JIT CUDA | bit-exact | paired SANA-Video Q/K RoPE with fp64 tables, about 14 eager kernels |
| `helios_qk_rope` | JIT CUDA | bit-exact | paired in-place Helios Q/K RoPE with transposed frequency layout |
| `ltx2_qknorm_split_rope` | KDA (JIT CUDA) | close (validated on B200) | LTX-2 QK-norm + split RoPE |
| `ltx25_decoder_rope` | JIT CUDA | bit-exact | paired LTX-2.5 decoder 3D RoPE from cached compact axis tables |
| `hunyuan_qkv_rope_pack` | Triton | bit-exact | QKV pack and RoPE in one pass |
### Activation
| Operator | Backend | Contract | Replaces |
| ----------------------- | -------------- | ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| `silu_mul` | Triton | bit-exact | `F.silu(a) * b` for split-projection SwiGLU, where the concatenated `silu_and_mul` kernels do not apply without an extra full-width `cat` |
| `bias_silu`, `bias_glu` | Triton | bit-exact | Sana GLUMB conv bias + SiLU / GLU post-processing |
| `linear_gelu_tanh` | AOT (cublasLt) | not bit-exact, request-gated | bias-add and tanh-GELU folded into the GEMM epilogue |
### Attention
| Operator | Backend | Notes |
| ------------------------ | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `sparse_linear_attn_fwd` | Triton | block-map, compression, and forward for sparse linear attention |
| `bigdn` | Triton | Sana-WM bidirectional gated delta-net; the chunkwise form splits phase A along the KV and Z streams so two blocks stay resident per SM, and stores `(I - P)` so phase B's MMA folds the identity-add in |
### MoE routing
| Operator | Backend | Contract | Replaces |
| -------------------- | ------- | -------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| `group_limited_topk` | Triton | selected expert-id set matches the guarded CUDA reference; output order is unspecified | LingBot Video's per-group top-2 reduction, group top-k, mask construction, masked expert scores, and final expert top-k |
### Data movement
Every kernel here only moves values (plus zero fill, plus at most one same-order add), so each is bitwise identical to the aten chain it replaces.
| Operator | Backend | Replaces |
| --------------------------------------------- | ----------------------- | ------------------------------------------------------------------------------------------- |
| `usp_merge_heads` | JIT CUDA | USP all-to-all output head merge (`permute` + `contiguous`) |
| `pack_qkv_destination_major` | Triton | Ulysses destination-major QKV pack |
| `varlen_pack_qkv`, `varlen_scatter_to_padded` | Triton | varlen gather/scatter around the masked attention path |
| `varlen_pack_segmented_qkv` | Triton | varlen gather from a virtual prefix/main Q/K/V sequence |
| `causal_conv3d_cat_pad` | KDA (JIT CUDA) / Triton | causal Conv3d `cat` + `pad` |
| `cat_pad_channels_last_3d` | Triton | Wan causal VAE `cat + F.pad + contiguous` (three passes plus cache bookkeeping) in one pass |
| `dup_up3d_add` | Triton | `repeat_interleave + permute().contiguous() + add` |
### Quantized layout producers
These kernels preserve the quantized checkpoint path's selected reference operation. They are not a claim that FP8 or NVFP4 is equivalent to an unquantized BF16 checkpoint.
| Operator | Backend | Replaces |
| ------------------------------------ | ------------ | ---------------------------------------------------------------------------- |
| `flux2_layernorm_modulate_fp8_quant` | KDA (Triton) | FLUX.2 LayerNorm plus adaLN modulation directly into static FP8 output |
| `flux2_token_cat_fp8` | KDA (Triton) | FLUX.2 single-block attention/MLP concatenation plus static FP8 quantization |
| `flux2_token_cat_nvfp4` | JIT CUDA | FLUX.2 single-block attention/MLP concatenation plus NVFP4 quantization |
## Coverage by model
Kernels are written against a specific eager chain in a specific model, so coverage is per-model rather than universal.
| Model | Fused paths |
| ------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| FLUX.1 | LN+modulate, modulate, residual-gate add, linear+GELU |
| FLUX.2 | LN+modulate, fused LN+modulate-to-FP8, packed SwiGLU, gated residual/norm, residual-gate add, QK RMSNorm+RoPE+joint QKV packing, FP8/NVFP4 token-cat producers |
| Qwen-Image | linear+GELU, select-0/1 LN modulation, added-QKV fusion, QK RMSNorm+RoPE+joint QKV writes, residual norm/modulate+NVFP4 producer |
| GLM-Image | LN+modulate, per-head qk LN, residual-gate add, linear+GELU |
| ERNIE-Image | RMSNorm+scale/shift, residual-gated variant, rotate-half RoPE, residual-gate add |
| Z-Image | BF16-native RMSNorm scale / tanh-residual, per-head QK RMSNorm |
| Ideogram 4 | gate RMSNorm, SwiGLU, rotate-half RoPE, modulate, residual-gate add |
| LTX-2 | QK-norm + split RoPE, ada-values split, RMSNorm+modulate, modulate, residual-gate add, linear+GELU |
| LTX-2.5 decoder | paired 3D RoPE with shared axis-table cache |
| HunyuanVideo / Helios | QKV+RoPE pack, strided QK RMSNorm, linear+GELU; Helios also has paired in-place Q/K RoPE |
| LingBot Video MoE | Default-on group-limited top-k expert selection; fused RMSNorm, per-token gated residual, and fused RMSNorm+modulate at `quality=extra-high` or `quality=high` |
| Sana | LN+modulate, GLUMB bias+SiLU / bias+GLU, residual-gate add |
| SANA-Video | Packed QKV/KV; paired fp64 interleaved RoPE; LN+modulate, GLUMB bias+SiLU / bias+GLU, and residual-gate add during BCG; BF16-input linear attention at `quality=extra-high` or `quality=high` |
| Sana-WM | bidirectional gated delta-net, fused QK inverse-RMS |
| Wan | temb table slices; VAE cat+pad and DupUp3D add, `channels_last_3d` RMSNorm+SiLU |
| Cosmos3 / Krea2 / MiniMax-H3 | QK-norm + RoPE (Krea2 also CuTe-DSL norm+scale/shift; MiniMax-H3 also indexed modulation) |
| FLUX.2 VAE / HunyuanVAE / latent upsampler | GroupNorm + SiLU (channels-last two-pass for FLUX.2) |
## Inspecting what is registered
Every kernel is described by a `KernelSpec` in the process-wide registry, so the inventory is queryable without importing any backend:
```python theme={null}
from sglang.kernels.registry import registry
diffusion_ops = [op for op in registry.ops() if op.startswith("diffusion.")]
for spec in registry.get("diffusion.scale_residual_norm_scale_shift"):
print(spec.backend, spec.target, spec.capabilities)
```
Registration is metadata only — it imports neither torch nor a backend and triggers no JIT build. To pick a specific implementation of an operator that has several:
```python theme={null}
from sglang.kernels import select_kernel, KernelBackend
fn = select_kernel(
"diffusion.scale_residual_norm_scale_shift", backend=KernelBackend.CUTE_DSL
).load()
```
## Importing the kernels
Runtime code imports from the package, never from a submodule:
```python theme={null}
from sglang.kernels.ops.diffusion import fused_rmsnorm_scale_shift_bitexact
```
Resolution is lazy: the backends have disjoint, heavy dependencies (Triton, CUTLASS/CuTe-DSL, and FlyDSL on ROCm), so an eager re-export would make every one of them an import-time requirement on every platform. Each public kernel is a predicate-plus-kernel pair — call `can_use_(...)` first and fall back to the reference chain when it returns `False`; the kernel raises on an unsupported input rather than silently returning `None`.
The package `README.md` carries a selection matrix for the cases where several kernels look interchangeable and are not. The normalization domain alone holds more than a dozen implementations that differ by numerical contract, activation layout, and backend rather than by speed.
## References
* [Performance Optimization](/docs/sglang-diffusion/performance-optimization)
* [Attention Backends](/docs/sglang-diffusion/attention_backends)
* [Quantization](/docs/sglang-diffusion/quantization)
* [Profiling](/docs/sglang-diffusion/profiling)
* [`sglang/kernels/ops/diffusion`](https://github.com/sgl-project/sglang/tree/main/python/sglang/kernels/ops/diffusion) — source and selection matrix
* [RFC #29630](https://github.com/sgl-project/sglang/issues/29630) — the unified `sglang.kernels` namespace