DeepSeek-V4

​

Deployment

pip install --upgrade pip
pip install uv
uv pip install --prerelease=allow sglang

docker pull lmsysorg/sglang:latest

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<your-hf-token>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    sglang serve <use args below>

docker pull lmsysorg/sglang-rocm:v0.5.19-rocm720-{mi35x,mi30x}-20260911

docker run \
    --device=/dev/kfd --device=/dev/dri \
    --group-add video \
    --cap-add=SYS_PTRACE --security-opt seccomp=unconfined \
    --shm-size 32g --ipc=host \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<your-hf-token>" \
    lmsysorg/sglang-rocm:v0.5.19-rocm720-{mi35x,mi30x}-20260911 \
    sglang serve <use args below>

• Low-Latency — fastest reply for a single user. Pick for chat.
• Balanced — good speed with several users at once. Use for typical multi-user serving.
• High-Throughput — most tokens per second across many users. Best for batch jobs.

​

Playground

• Built-in SGLang features — parallelism overrides (TP / CP / DP-Attention — DP-Attention’s value is the DP degree, with off to disable), MoE backend + EP, reasoning / tool-call parsers, speculative-decoding presets, prefill/decode disaggregation, HiCache tiers, and HiSparse hierarchical sparse attention (decode-role only — the card appears once PD-Disagg mode is set to decode).
• DeepSeek-V4 specific features — MegaMoE W4A8 / W4A4 fused kernel (Blackwell only; Hopper SM90 uses a separate all-FP8 MegaMoE path — see Configuration Tips below).

​

1. Model Introduction

​

2. Configuration Tips

• low-latency: steps=3, draft-tokens=4 → largest win at bs=1.
• balanced: steps=1, draft-tokens=2 → gentler MTP, reduces throughput hit at higher batch.
• high-throughput: MTP disabled — at saturation the verify step costs more than it saves.
• MTP runs on the v2 speculative path.

• Docker image — all three cells use lmsysorg/sglang:dev-v4f-2dgx-v2, a preview build made only for DGX Spark (branch b12x-vision @ 452239a74f): it bakes in the SM12x b12x MoE (W4A8) and compressed-MLA attention kernels (#34878, #35899, #34018), the Flash Vision model support (#37253), the b12x image-prefill fix that lets Flash Vision serve images on SM12x, the NVFP4 MTP-layer dispatch fix, and the CuTeDSL and NCCL pins the GB10 pair needs. Do not use it on other hardware, and use the panel’s Docker mode — the bare Python command needs the b12x kernel package this image ships.
• Run the same command on both Sparks with --node-rank 0 / --node-rank 1 and --dist-init-addr pointing at node 0 over the ConnectX-7 link. The docker run flags the panel emits (--network host --ulimit memlock=-1:-1 --cap-add IPC_LOCK --device /dev/infiniband) are what let NCCL use RDMA; without them NCCL silently falls back to TCP and decode slows by roughly 40%.
• Env knobs in the cells are part of the recipe: SGLANG_SM120_FLASHMLA_BACKEND=b12x selects the b12x attention path, SGLANG_B12X_MAX_TOKENS must equal --chunked-prefill-size, and PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True avoids unified-memory fragmentation OOMs on GB10.
• NVFP4 (nvidia/DeepSeek-V4-Flash-0731-NVFP4) — only the routed experts are NVFP4; attention, shared experts and the DSpark MTP layer stay in the checkpoint’s native formats. On SM12x that means three extra flags: --moe-runner-backend flashinfer_cutlass (b12x’s MoE is MXFP4-only and trtllm-gen kernels are sm100-only), --speculative-moe-runner-backend b12x (the DSpark draft’s MTP experts are MXFP4 and run on b12x), and --disable-shared-experts-fusion (HashTopK rejects fused shared experts under the cutlass runner). Throughput and DSpark acceptance match the FP4 cell within noise.
• Flash Vision — images are served natively on the b12x recipe with the same flags as Flash Official (send image_url content on /v1/chat/completions, see Vision); text-only requests work unchanged. Expect roughly 15–20% lower text throughput than Flash Official on this checkpoint — its bundled DSpark head accepts fewer drafts (~3.2 vs ~3.9) — with text accuracy intact.

• Preview build required — support lands via sgl-project/sglang#37253 and has not shipped in a release. Docker mode on the Flash Vision cells already emits the preview image lmsysorg/sglang:dev-dsv4-flash-vision; for a Python environment, install SGLang from that PR’s branch. The DGX Spark Flash Vision cell is the exception: it uses the DGX Spark image lmsysorg/sglang:dev-v4f-2dgx-v2 (see the DGX Spark notes).
• Verified matrix — MMMU-Pro via sgl-eval at temperature 1.0, top-p 0.95, --reasoning-effort max.
• Engine auto-configuration — the engine picks the flashinfer_mxfp4 MoE runner and auto-disables shared-experts fusion for this checkpoint (its HashTopK routing rejects fused shared experts); don’t pass --enforce-shared-experts-fusion.
• Chunked prefill & radix cache stay enabled — the scheduler keeps image spans consistent automatically: chunked-prefill truncation points are span-aligned (an image span always prefills within a single extend, overshooting the chunk budget by at most one span), and a radix-cache prefix match ending deep inside an image span is re-issued from the span start.
• Speculative decoding — the checkpoint bundles a DSpark head, and the low-latency recipes enable it with --speculative-algorithm DSPARK (verified with image batches on B200 via the MMMU-Pro round; the other hardware rows are pending verification). The balanced and high-throughput recipes run target-only: they use DP Attention, which DSpark is incompatible with on current releases. As on the 0731/0813 checkpoints, do not pass the EAGLE flags.

--enforce-shared-experts-fusion

SGLANG_DSV4_COMPRESS_STATE_DTYPE=bf16 \
sglang serve \
  --model-path deepseek-ai/DeepSeek-V4-Flash \
  <other args>

--moe-a2a-backend deepep \
--deepep-mode auto \
--init-expert-location /path/to/expert_distribution_recorder_*.pt \
--enable-waterfill

# prefill
--moe-a2a-backend deepep \
--deepep-mode normal \
--init-expert-location /path/to/expert_distribution_recorder_*.pt \
--enable-waterfill

# decode
--moe-a2a-backend deepep \
--deepep-mode low_latency \
--init-expert-location /path/to/expert_distribution_recorder_*.pt \
--enable-waterfill

# Please use the latest main branch for this feature.
sglang serve \
  --model-path deepseek-ai/DeepSeek-V4-Flash \
  --tp 4 \
  --moe-runner-backend flashinfer_mxfp4 \
  --enable-deepseek-v4-fp4-indexer

# Please use the latest ROCm image for this feature.
sglang serve \
  --model-path deepseek-ai/DeepSeek-V4-Pro \
  --tp 8 \
  --attention-backend dsv4 \
  --enable-deepseek-v4-fp4-indexer \
  <other args from the Deploy panel>

sglang serve \
  --model-path nvidia/DeepSeek-V4-Pro-NVFP4 \
  --tp 8

sglang serve \
  --model-path nvidia/DeepSeek-V4-Flash-NVFP4 \
  --tp 8

• Original FP4 checkpoints — run the MoE experts with W4A16 kernels (Marlin or the FlashInfer SM90 CUTLASS runner) as the command generator picks for Hopper cells. With FlashInfer >= 0.6.18 you can instead select the W4A8 path — MXFP4 weights with FP8 activations via FlashInfer’s Humming kernels — by adding --flashinfer-mxfp4-moe-precision fp8 to --moe-runner-backend flashinfer_mxfp4; the low-latency Hopper cells now generate this form. Both work on H100 and H200; FP4 is the only option for H100 (no FP8 path). It is TP-only; on H200 the Pro variant fits on a single 8-GPU node, while H100 Pro needs 2 nodes (TP=16).
• Converted FP8 checkpoints (H100 and H200 only) — pre-repackaged FP8 weights at sgl-project/DeepSeek-V4-Flash-FP8 and sgl-project/DeepSeek-V4-Pro-FP8 unlock DP-attention + DeepEP and richer parallelism (e.g. Pro TP=16 across 2 nodes).

• Model checkpoints — for correct accuracy, the FP4 model uses the stock deepseek-ai/DeepSeek-V4-{Flash,Pro}, and the FP8 model uses the repackaged sgl-project/DeepSeek-V4-{Flash,Pro}-FP8.
• Supported models — MI300X supports DeepSeek-V4-Flash in FP8; MI355X supports DeepSeek-V4-Flash / Pro in both FP4 and FP8. All recipes run single-node.
• TP / DP setting (MI355X) — both TP=4 and TP=8 are supported. At low concurrency we recommend TP-only; at high concurrency use TP + DP (balanced / high-throughput recipes). The verified MI355X DP recipes additionally set --dp 8 --enable-dp-attention --enable-dp-attention-local-control-broadcast --tokenizer-worker-num 8 --stream-interval 20 --prefill-decode-interval 10.
• MTP — speculative decoding on the original Flash / Pro checkpoints; add --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4. The 0813 Pro Official checkpoint keeps its MTP head too, so the same flags are the speculative option under PD disaggregation — see the DSpark note below.
• DSpark (MI355X Pro Official 0813) — the 0813 checkpoint bundles the DSpark draft head. Prefer it over EAGLE on 0813 for ordinary serving: enable --speculative-algorithm DSPARK (low-latency) and see the MI355X agentic recipe for the long-context path. The one exception is PD disaggregation, where DSpark cannot run at all and the Playground strips it — there, fall back to the MTP 3-1-4 shape above, as in §3.8.
• Kernels — uses the Unified KV attention and the flydsl MoE.
• FP4 indexer (MI355X) — FP4 C4 indexer is supported via --enable-deepseek-v4-fp4-indexer on top of the standard ROCm recipe.
• Agentic long-context (MI355X Pro Official FP4) — TP-only serving adds --prefill-decode-interval 10 for scheduler stability. The DP path additionally needs --enable-dp-lm-head (required for DSpark under DP attention), --enable-prefill-delayer --prefill-delayer-token-usage-low-watermark 0.7 so a single long prefill does not monopolise the engine, plus the flags in the MI355X agentic recipe.

export SGLANG_MORI_DISPATCH_DTYPE=mxfp8
export SGLANG_MORI_RECV_BOUND=1

sglang serve \
  --model-path {MODEL_PATH} \
  --tp 8 --dp 8 --enable-dp-attention \
  --ep-size 8 --moe-a2a-backend mori --deepep-mode normal \
  <other args from the Deploy panel>

• FP4: enable both env vars.
• FP8: use SGLANG_MORI_RECV_BOUND=1 only; omit SGLANG_MORI_DISPATCH_DTYPE=mxfp8.

• W4A8 — default MegaMoE kernel (FP4 weights, FP8 activations).
• W4A4 — adds --enable-w4a4-mxfp4-megamoe to run the custom W4A4 kernel (FP4 activations). The flag configures the required DeepGEMM settings. Higher throughput with negligible accuracy drop (~89.5 GPQA on Pro).

• The W4A8 / W4A4 variants above are Blackwell-only (B200 / B300 / GB200 / GB300). On Hopper (SM90, H100 / H200) use the all-FP8 MegaMoE path described below instead.
• MegaMoE is only wired into the high-throughput recipe on Blackwell (per sgl-project/sglang#26451). The chip is hidden on low-latency and balanced — switch to high-throughput to expose it.
• When running MegaMoE, don’t set --moe-runner-backend manually.
• Adjust SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK based on your workload and memory usage. Setting higher number of tokens for MegaMoE requires more HBM space (recommended: 8320 for high-throughput).

SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=4096 \
SGLANG_DSV4_FP4_EXPERTS=0 \
sglang serve \
  --model-path sgl-project/DeepSeek-V4-Flash-FP8 \
  --tp 8 \
  --moe-a2a-backend megamoe \
  --chunked-prefill-size 4096

​

3. Advanced Usage

​

3.1 Reasoning

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V4-Flash",
    messages=[
        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
    ],
    max_tokens=2048,
    extra_body={"chat_template_kwargs": {"thinking": True}},
    stream=True,
)

thinking_started = False
has_thinking = False
has_answer = False

for chunk in response:
    if not chunk.choices:
        continue
    delta = chunk.choices[0].delta

    if getattr(delta, "reasoning_content", None):
        if not thinking_started:
            print("=============== Thinking =================", flush=True)
            thinking_started = True
        has_thinking = True
        print(delta.reasoning_content, end="", flush=True)

    if delta.content:
        if has_thinking and not has_answer:
            print("\n=============== Content =================", flush=True)
            has_answer = True
        print(delta.content, end="", flush=True)

print()

We are asked: "What is 15% of 240?" This is a simple percentage problem. I need to provide a step-by-step solution. The user wants the solution explained step by step. I'll calculate 15% of 240: 0.15 * 240 = 36. I'll break it down into steps: understand what percent means, convert percentage to decimal or fraction, then multiply. I'll present the answer clearly.</think>To find 15% of 240, follow these steps:

**Step 1: Understand the meaning of percent**
"Percent" means "per hundred," so 15% means 15 out of every100, or \( \frac{15}{100} \).

**Step2: Convert the percentage to a decimal or fraction**
\( 15\% = \frac{15}{100} = 0.15 \)

**Step3: Multiply by the given number**
Multiply the decimal form by 240:
\( 0.15 \times 240 \)

**Step4: Perform the multiplication**
\( 0.15 \times 240 = 36 \)

**Answer:** 15% of 240 is **36**.

​

3.2 Tool Calling

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {"type": "string", "description": "The city name"},
                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
                },
                "required": ["location"],
            },
        },
    }
]

response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V4-Flash",
    messages=[{"role": "user", "content": "What's the weather in Beijing?"}],
    tools=tools,
    extra_body={"chat_template_kwargs": {"thinking": True}},
    stream=True,
)

thinking_started = False
has_thinking = False
tool_calls_accumulator = {}

for chunk in response:
    if not chunk.choices:
        continue
    delta = chunk.choices[0].delta

    if getattr(delta, "reasoning_content", None):
        if not thinking_started:
            print("=============== Thinking =================", flush=True)
            thinking_started = True
        has_thinking = True
        print(delta.reasoning_content, end="", flush=True)

    if getattr(delta, "tool_calls", None):
        if has_thinking and thinking_started:
            print("\n=============== Content =================\n", flush=True)
            thinking_started = False
        for tool_call in delta.tool_calls:
            index = tool_call.index
            if index not in tool_calls_accumulator:
                tool_calls_accumulator[index] = {"name": None, "arguments": ""}
            if tool_call.function:
                if tool_call.function.name:
                    tool_calls_accumulator[index]["name"] = tool_call.function.name
                if tool_call.function.arguments:
                    tool_calls_accumulator[index]["arguments"] += tool_call.function.arguments

    if delta.content:
        print(delta.content, end="", flush=True)

for index, tool_call in sorted(tool_calls_accumulator.items()):
    print(f"Tool Call: {tool_call['name']}")
    print(f"   Arguments: {tool_call['arguments']}")

print()

The user wants to know the weather in Beijing. I'll use the get_weather function with Beijing as the location. I don't need to specify a unit, so I'll just use the default.</think>

<｜DSML｜tool_calls>
<｜DSML｜invoke name="get_weather">
<｜DSML｜parameter name="location" string="true">Beijing</｜DSML｜parameter>
</｜DSML｜invoke>
</｜DSML｜tool_calls>

​

3.3 HiCache (Hierarchical KV Caching)

• L2 (GPU + CPU) — leave Storage on auto (default). Cold KV pages spill to CPU pinned memory only.
• L3 (GPU + CPU + Storage) — pick a Storage backend (file / mooncake / hf3fs / nixl); the Playground emits the canonical page_first_direct mem-layout + direct IO backend + wait_complete prefetch policy, matching the HiCache best-practices recipe.

• L2 (GPU + CPU) — leave Storage on auto (default). Cold KV pages spill to CPU pinned memory only. Use direct IO backend + page_first_direct or layer-first mem-layout.
• L3 (GPU + CPU + Storage) — pick a Storage backend (file); the Playground emits the canonical page_first_direct mem-layout + direct IO backend + wait_complete prefetch policy, matching the HiCache best-practices recipe.

​

3.4 DSpark (Speculative Decoding)

sglang serve \
  --trust-remote-code \
  --model-path deepseek-ai/DeepSeek-V4-Flash-0731 \
  --tp 4 \
  --moe-runner-backend flashinfer_mxfp4 \
  --speculative-algorithm DSPARK \
  --mem-fraction-static 0.90 \
  --chunked-prefill-size 4096 \
  --swa-full-tokens-ratio 0.1 \
  --host 0.0.0.0 \
  --port 30000

​

3.5 Vision (Image Inputs)

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V4-Flash-Vision-Exp",
    messages=[{
        "role": "user",
        "content": [
            {"type": "image_url",
             "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"}},
            {"type": "text", "text": "Describe this image in a few sentences."},
        ],
    }],
)

message = response.choices[0].message
if getattr(message, "reasoning_content", None):
    print("=============== Thinking =================")
    print(message.reasoning_content)
print("=============== Content =================")
print(message.content)

Pending update...

​

3.6 Agentic Long-Context with HiCache DRAM Offload (B200 FP4, DSpark)

SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 \
python3 -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V4-Pro-0813 \
  --trust-remote-code \
  --tp 8 \
  --moe-runner-backend flashinfer_mxfp4 \
  --enable-deepseek-v4-fp4-indexer \
  --disable-flashinfer-autotune \
  --mem-fraction-static 0.90 \
  --swa-full-tokens-ratio 0.1 \
  --chunked-prefill-size 8192 \
  --tool-call-parser deepseekv4 \
  --reasoning-parser deepseek-v4 \
  --speculative-algorithm DSPARK \
  --speculative-dspark-block-size 6 \
  --enable-hierarchical-cache \
  --hicache-ratio 2.75 \
  --hicache-write-policy write_through \
  --hicache-io-backend direct \
  --hicache-mem-layout page_first_direct

SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 \
SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320 \
python3 -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V4-Pro-0813 \
  --trust-remote-code \
  --tp 8 \
  --dp 8 \
  --enable-dp-attention \
  --enable-dp-lm-head \
  --ep-size 8 \
  --moe-a2a-backend megamoe \
  --enable-w4a4-mxfp4-megamoe \
  --enable-deepseek-v4-fp4-indexer \
  --disable-shared-experts-fusion \
  --disable-flashinfer-autotune \
  --mem-fraction-static 0.88 \
  --swa-full-tokens-ratio 0.02 \
  --chunked-prefill-size 49152 \
  --tool-call-parser deepseekv4 \
  --reasoning-parser deepseek-v4 \
  --speculative-algorithm DSPARK \
  --speculative-dspark-block-size 6 \
  --enable-hierarchical-cache \
  --hicache-ratio 8 \
  --hicache-write-policy write_through \
  --hicache-io-backend direct \
  --hicache-mem-layout page_first_direct

​

3.7 Agentic Long-Context with HiCache DRAM Offload (MI355X FP4, DSpark)

SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 \
SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS=1 \
SGLANG_USE_ROCM700A=0 \
TORCH_BLAS_PREFER_HIPBLASLT=1 \
SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton \
AITER_BF16_FP8_MOE_BOUND=0 \
SGLANG_OPT_USE_AITER_BATCHED_GEMM=1 \
python3 -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V4-Pro-0813 \
  --trust-remote-code \
  --tp 8 \
  --prefill-decode-interval 10 \
  --attention-backend dsv4 \
  --enable-deepseek-v4-fp4-indexer \
  --page-size 256 \
  --swa-full-tokens-ratio 0.1 \
  --kv-cache-dtype fp8_e4m3 \
  --enforce-shared-experts-fusion \
  --mem-fraction-static 0.86 \
  --chunked-prefill-size 16384 \
  --tool-call-parser deepseekv4 \
  --reasoning-parser deepseek-v4 \
  --speculative-algorithm DSPARK \
  --speculative-dspark-block-size 6 \
  --watchdog-timeout 3600

  --enable-hierarchical-cache \
  --hicache-ratio 1.5 \
  --hicache-write-policy write_through \
  --hicache-io-backend direct \
  --hicache-mem-layout page_first_direct

SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 \
SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS=1 \
SGLANG_USE_ROCM700A=0 \
TORCH_BLAS_PREFER_HIPBLASLT=1 \
SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton \
AITER_BF16_FP8_MOE_BOUND=0 \
SGLANG_OPT_USE_AITER_BATCHED_GEMM=1 \
SGLANG_SHARED_EXPERT_TP1=1 \
SGLANG_DP_SHARED_EXPERT_LOCAL=1 \
SGLANG_DP_USE_GATHERV=1 \
SGLANG_DP_USE_REDUCE_SCATTER=1 \
python3 -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V4-Pro-0813 \
  --trust-remote-code \
  --tp 8 \
  --dp 8 \
  --enable-dp-attention \
  --enable-dp-lm-head \
  --enable-prefill-delayer \
  --enable-dp-attention-local-control-broadcast \
  --tokenizer-worker-num 8 \
  --stream-interval 20 \
  --prefill-decode-interval 10 \
  --prefill-delayer-token-usage-low-watermark 0.7 \
  --attention-backend dsv4 \
  --enable-deepseek-v4-fp4-indexer \
  --page-size 256 \
  --swa-full-tokens-ratio 0.1 \
  --kv-cache-dtype fp8_e4m3 \
  --enforce-shared-experts-fusion \
  --mem-fraction-static 0.92 \
  --chunked-prefill-size 65536 \
  --tool-call-parser deepseekv4 \
  --reasoning-parser deepseek-v4 \
  --speculative-algorithm DSPARK \
  --speculative-dspark-block-size 6 \
  --enable-hierarchical-cache \
  --hicache-ratio 1.5 \
  --hicache-write-policy write_through \
  --hicache-io-backend direct \
  --hicache-mem-layout page_first_direct \
  --watchdog-timeout 3600

​

3.8 PD Disaggregation on MI355X (MORI-IO)

docker run \
    --device=/dev/kfd --device=/dev/dri --device /dev/infiniband \
    --group-add video --cap-add IPC_LOCK \
    --cap-add=SYS_PTRACE --security-opt seccomp=unconfined \
    --ulimit memlock=-1 --ulimit stack=67108864 \
    --ulimit nofile=1048576:1048576 \
    --network host --ipc=host --shm-size 32g \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    lmsysorg/sglang-rocm:v0.5.19-rocm720-mi35x-20260911 \
    python3 -m sglang.launch_server <role args below>

• Low-Latency — TP-only, --mem-fraction-static 0.86, and a running-request ceiling of 8. Decode captures graphs for batches 1–8. No offload tier: the KV pool is all there is.
• Balanced — still TP-only, with the ceiling at --max-running-requests 96 and a 16384 chunked-prefill budget. Decode ladders all the way to 96, since without DP the server-wide ceiling is the per-rank batch. Prefill runs a HiCache host tier (--hicache-ratio 2.5, page_first, write-through, best-effort prefetch); decode does not, because HiCache is not recommended on the decode role with MORI.
• High-Throughput — inherits TP8 / DP8 and the 65536 chunked-prefill budget from the base cell, raises the ceiling to --max-running-requests 256 and --mem-fraction-static 0.92, and ladders decode to 32 — the per-rank batch 256 leaves across DP 8. It adds --enable-cache-report, which surfaces the prefix hit rate you need to judge whether an offload tier is paying for itself at that concurrency.

• Cuda graphs. Prefill runs eager (--disable-cuda-graph) because the FP4 indexer’s prefill path is not captured. Decode captures the small batch ladder that matches its running-request ceiling (--cuda-graph-bs-decode 1 2 3 4 5 6 7 8).
• MORI dispatch budget. SGLANG_MORI_NUM_MAX_DISPATCH_TOKENS_PER_RANK is 16384 on prefill, sized for a whole chunked-prefill batch, and 128 on decode, where a step never dispatches more than a token per running request.

--speculative-algorithm EAGLE --speculative-eagle-topk 1 \
--speculative-num-steps 3 --speculative-num-draft-tokens 4

​

3.9 UMBP (Direct External KV Store)

--enable-unified-cache-external-linker \
--unified-cache-external-linker-backend mori