> ## Documentation Index > Fetch the complete documentation index at: https://lmsysorg-dsv4-1.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Deployment and Performance Modes > Choose component residency, FSDP, CFG parallelism, SP, TP, and performance-mode presets in SGLang Diffusion. This page gives practical defaults for choosing `--performance-mode`, component residency, FSDP, CFG parallelism, SP, and TP. ## Quick Rule Use the simplest setting that fits your memory target:
Goal Recommended setting
Fastest single-GPU run when the model fits Use resident components and do not use FSDP.
Lower single-GPU memory usage Use component offload, then layerwise offload when a complete component still does not fit comfortably.
Faster multi-GPU Qwen/Wan CFG generation Use FSDP with CFG parallelism and keep the sharded component resident.
Sequence length or video-shape scaling Use SP/Ulysses/Ring when the model benefits from sequence parallelism.
TP compatibility or encoder-heavy paths Set TP explicitly; do not treat TP as the default latency optimization.
Base the decision on available memory on the selected GPU(s). * For multi-GPU deployment: the least-free selected GPU is the bottleneck. A busy 80GiB GPU can behave like a much smaller GPU. * For single-GPU deployment: FSDP shards weights across multiple GPUs. It is not useful for keeping a single-GPU deployment on one GPU; use component or layerwise offload instead. ## Health Probes Use `/liveness` to check that the HTTP process is alive and `/health` to check that the server is ready for inference. During server-based warmup, `/liveness` returns `200` while `/health` returns `503`. Configure the startup probe with a failure budget large enough for model loading and compilation: ```yaml theme={null} startupProbe: httpGet: path: /health port: 30010 periodSeconds: 10 failureThreshold: 180 readinessProbe: httpGet: path: /health port: 30010 livenessProbe: httpGet: path: /liveness port: 30010 ``` See [Health endpoints](/docs/sglang-diffusion/api/cli#health-endpoints) for the status-code contract and warmup-mode behavior. ## Stable Model Identity Use `--served-model-name` when the public model name must remain stable across replicas, hosts, or checkpoint mount paths: ```bash theme={null} sglang serve \ --model-path /mnt/checkpoints/Qwen-Image \ --model-id Qwen-Image \ --served-model-name image-production \ --port 30010 ``` The resolved public name follows `--served-model-name`, then `--model-id`, then `--model-path`. `--model-id` remains an internal model registry and configuration-resolution hint; it is not a replacement for a deployment alias. The resolved name is exposed through `/server_info` and `/v1/models` and is used by video and action responses when a request does not supply its own model. See [OpenAI API: Served model name](/docs/sglang-diffusion/api/openai_api#served-model-name) for discovery and retrieval examples. ## Performance Modes `--performance-mode` applies safe presets without overriding explicit offload, FSDP, or parallelism flags. `auto` is the default. Use `manual` when you need to keep performance-related server args under explicit user control. `--mode` is a short alias.
Mode Meaning
`manual` Keeps performance-related server args under explicit user control.
`auto` Default. Keeps legacy safe offload defaults and uses FSDP/CFG only on validated multi-GPU deployments where FSDP can replace DiT offload.
`speed` Favors GPU-resident execution for lower latency and higher throughput. Disables CPU offload when unset. `torch.compile` stays off unless the model has a validated default or it is enabled explicitly; may OOM.
`memory` Favors lower GPU memory. Uses component offload, or Wan/MOVA layerwise DiT offload when supported.
`auto` checks selected GPU memory before applying FSDP. In multi-GPU runs it uses the least available memory across selected GPUs, and only turns on FSDP automatically when doing so can replace DiT offload. For image workloads with at least 45 GiB available per selected GPU, it keeps the repeatedly reused DiT resident and uses layerwise offload for large auxiliary encoders; below that threshold it keeps the DiT offloaded. Model-specific components such as VAEs become resident only when their configured memory threshold is met. `memory` instead keeps the VAE in its default layerwise set to maximize memory headroom; use `--component-residency vae=resident` for a measured recipe with sufficient capacity. Video DiT residency remains model- and workload-specific because frame count and resolution change its peak memory substantially. When the model default uses CFG and the user did not set a parallelism policy, `auto` may also enable CFG parallelism. `speed` intentionally does not check memory; it is the mode for users who prefer latency/throughput and accept OOM risk. It keeps `torch.compile` disabled by default because its effect varies by model and workload. A model-specific deployment config may enable a validated compile path, and `--enable-torch-compile true` always opts in explicitly. The modes tune native pipeline components declared to the component residency manager. DiTs, text/image encoders, VAEs, vocoders, adapters, and upsamplers can use layerwise offload when their native module declares its executable layer structure. Explicitly selecting an unsupported component fails at startup instead of falling back to another residency mode. For direct control, assign one of `resident`, `component-offload`, `snapshot-offload`, or `layerwise-offload` with `--component-residency COMPONENT=MODE`: ```bash theme={null} sglang generate \ --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \ --component-residency dit=layerwise-offload text_encoder=component-offload vae=resident ``` Existing per-component CPU-offload and layerwise flags remain supported. Canonical selectors override matching legacy settings only; unmatched legacy settings and automatic defaults remain effective. See [Component Residency](/docs/sglang-diffusion/api/cli#component-residency) for the complete precedence rules. When `torch.compile` is enabled, `--offload-during-compile` stays on by default. During compile warmup it temporarily offloads the DiT and evicts resident non-DiT components so `max-autotune` fits on tighter-memory GPUs, then restores the configured serving residency before real traffic. Breakable CUDA graph is a separate manual opt-in for supported image pipelines. If you enable `--enable-breakable-cuda-graph`, declare every served resolution in `--warmup-resolutions` so warmup captures matching graph signatures. The preset is intentionally coarse. A future continuous value such as `0.0` to `1.0` could express the speed-memory tradeoff more precisely, but it would need model-specific memory models and clearer user expectations. Until then, use the preset plus explicit flags for overrides. Examples: ```bash theme={null} sglang generate \ --model-path Qwen/Qwen-Image \ --num-gpus 2 \ --performance-mode auto ``` ```bash theme={null} sglang generate \ --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \ --performance-mode memory ``` Explicit flags win over the mode: ```bash theme={null} sglang generate \ --model-path Qwen/Qwen-Image \ --num-gpus 2 \ --performance-mode auto \ --use-fsdp-inference false ``` In this example, `auto` will not re-enable FSDP. The same applies to parallelism; for example, `--enable-cfg-parallel false` keeps CFG parallelism disabled. ## Interpreting The Levers **Resident** keeps the complete component on the accelerator. It is usually fastest when memory is sufficient. **Component offload** keeps a complete component on CPU between declared uses. It is simple and robust, but each use pays a whole-component transfer. **Layerwise offload** streams the declared layers of any supported native weighted component. It lowers peak accelerator memory further, but may increase latency and lower throughput. ### Which components to stream `--layerwise-offload-components` takes a list, and more is not better. Streaming pays one host-to-device transfer of a component's weights **every time that component runs**, overlapped with that run's compute. What matters is how often it runs per request, not how large it is. * The **DiT** runs once per denoising step. Its transfer amortizes over every step and hides behind attention, so it is the component layerwise offload is for. * A **video VAE** runs once per temporal chunk, not once: H3's decoder re-runs whole for each chunk, so streaming its blocks pays their transfer per chunk. * A **text encoder** runs once. Keep it resident if it fits; otherwise stream it with its blocks resident. There are three placements, not two. A component can be in the list and still hold its blocks — `--layerwise-offload-components dit,text_encoder,vae` together with `--layerwise-resident-layers video_vae=36` transfers them once instead of per pass, and gives the VRAM back when the component finishes rather than holding it through denoise. On MiniMax-H3 at 864x480 / 124 frames that is 13 s of decode against 150 s streamed. So do not read "drop it from the list" as the fix for a one-shot component: that keeps it resident for the whole process. With no resident blocks on either side, moving H3's 10.4 GB video VAE out of the list took decode from 39.9 s to 5.3 s on one RTX 4090 — but it also added 5.8 GB of peak, which is the budget a 12 GB card does not have. ### How deep to prefetch `--dit-offload-prefetch-size` is not monotonic. Deeper prefetch hides more of the copy, but its staging buffers crowd out activations, so latency turns back up and peak memory keeps climbing. On the H3 configuration above, 1/2/3/4 layers measured 17.3 / 15.3 / 15.8 / 16.7 s of denoise, with the deepest setting reaching 96% of a 24 GB card. Sweep two or three values rather than trusting the default. ### When not to bother Before tuning residency or prefetch, measure whether the transfer is exposed — and measure it rather than inferring it from the checkpoint size. Bytes over bandwidth bounds what *could* be exposed, not what is, because prefetch exists to hide exactly that. A 1.3B video DiT moves about 2.6 GB a step, on the order of a tenth of a one-second step if none of it overlapped, and its residency settings measure flat: it overlaps, so there is nothing to recover. A 50-layer model at 1.4 GB a layer moves 66 GB a step, and there the same flags decide whether the model runs at all. Sweep two or three values on the target configuration and keep the measured winner. **FSDP** shards DiT weights across multiple GPUs and all-gathers weights during forward. It can reduce DiT CPU offload cost on multi-GPU deployments, especially for validated Wan I2V workloads. FSDP sharding granularity matters. SGLang Diffusion prefers sharding direct repeated transformer block entries such as `transformer_blocks.0` or `blocks.0`. Coarser sharding lowers wrapper count but can increase all-gather peak memory; finer sharding can reduce transient memory but adds communication and scheduling overhead. If a model does not define an explicit sharding rule, the loader falls back to repeated block class names and common direct numbered block paths. **CFG parallelism** splits positive and negative CFG branches across GPUs. For Qwen/Wan workloads with normal step counts, this is the most reliable multi-GPU speedup observed so far. **SP/Ulysses/Ring** splits sequence work. It can help video workloads, but validated Qwen/Wan runs showed CFG parallelism outperforming SP for latency. **TP** is supported for compatibility and some model structures, but current measurements do not make it the default latency path for Qwen/Wan. ## Current Benchmark Takeaways Observed regular-scale trends: * Z-Image: single-GPU no-offload was faster than FSDP/SP in the tested setting; keep FSDP off unless memory or parallelism requires it. * Qwen-Image: keep the default non-FSDP path unless a specific FSDP/SP/Ring setting has been benchmarked on the target hardware. * Wan: FSDP can replace DiT offload on validated multi-GPU workloads, while text/image encoders may still need component offload. Keep model-specific precision checks before making FSDP automatic for a path. * Component offload mainly reduced memory; it did not improve latency in the tested no-offload-vs-offload runs. Always benchmark with your actual resolution, frame count, step count, and GPU type before locking production defaults.