Qwen3.8-27B

​

Deployment

pip install --upgrade pip
pip install uv

uv pip install --prerelease=allow sglang

docker pull lmsysorg/sglang:latest

​

Mamba ratio calculator

ratio = (S + D) x state_bytes / (L x kv_bytes_per_token)

​

Playground

​

1. Model Introduction

​

2. Configuration Tips

• SM120/SM121 (RTX PRO 6000 Blackwell, RTX 5090, DGX Spark): use --attention-backend flashinfer; trtllm_mha is SM100-only. MTP with the FlashInfer backend requires a FlashInfer build whose prefill plan accepts uniform_q_len (newer than 0.6.15.post1); otherwise run spec with --attention-backend triton. On DGX Spark the 128GB is unified memory shared with the host CPU, so all three checkpoints fit, and its cells reuse the RTX PRO 6000 recipe at --mem-fraction-static 0.80 rather than a separate operating point. The one lower pin is the unified pool pricing the host’s memory too: 0.85 of 128GB leaves ~8GB for the OS — exactly DGX OS earlyoom’s SIGTERM threshold — and the first long prefill or boot-time graph capture dips under it and gets the scheduler killed with exit code -15 and no traceback (journalctl -u earlyoom shows the kill). At 0.85, 15 of the 48 cells were killed that way, and which cells is margin noise; at 0.80 every cell served on every attempt. Validated on SM121 / aarch64: all 80 configurations (5 checkpoints x Speculative Decoding x Serving Strategy x Mamba SSM Dtype, DFLASH2 included) served on GB10 on v0.5.19 at ISL 8192 / OSL 1024, concurrency 1, and each scored the full 1319-question GSM8K (93.18-95.15%); the float32 and bfloat16 halves ran on two separate GB10 boxes. No throughput or acceptance-length numbers were re-taken. The sweep exercises the FlashInfer plan / uniform_q_len path above, which raised no arity error on that build. Three host quirks when reproducing on GB10: docker GPU access is CDI-only (--device nvidia.com/gpu=all, as no nvidia runtime is registered); nvidia-smi reports Not Supported for memory because it is unified with the CPU — gate a relaunch on MemAvailable in /proc/meminfo instead; and the BF16 checkpoint takes ~6.5 minutes just to load its 18 shards from NVMe, so budget ~10 minutes to READY before calling a boot hung.
• H200 (SM90): BF16 and FP8 only — the card has no FP4 tensor cores, so an NVFP4 checkpoint’s MLP would fall back to the Marlin W4A16 weight-only path, and all three NVFP4 cells are greyed out. The H200 recipes use 32768-token prefill chunks (SM90 prefill is fast enough that a big chunk barely stalls decode, unlike the SM120 guidance below), and the FlashInfer GDN prefill backend engages by default under them. --attention-backend fa3 is a valid alternative, measured slightly faster at bs=1.
• MTP: --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 uses the in-checkpoint MTP head. (This recipe was originally documented with NEXTN, an alias of EAGLE — same algorithm.)
• DSpark: the trained draft model is a separate checkpoint — add --speculative-algorithm DSPARK --speculative-draft-model-path RadixArk/Qwen3.8-27B-DSpark (the Playground’s Speculative Decoding card emits this pair). DSpark does not take --speculative-num-draft-tokens: its verify window is --speculative-dspark-block-size (gamma) + 1, and gamma is auto-inferred from the draft checkpoint when the flag is omitted (7 for this checkpoint, so D = 8). That D is a term in the balanced ratio — r = (S + D) x token_equiv / L, where token_equiv is the state slot expressed in KV tokens, state_bytes / kv_bytes_per_token (4698 at fp32 state / 2394 at bf16, over fp8 KV) — so DSpark needs a materially higher --mamba-full-memory-ratio than no-spec at the same S, and pinning a different gamma changes the ratio with it. MTP is the opposite case: with --enable-linear-replayssm-spec its draft intermediates move onto a fixed ring, so D = 0 and the ratio returns to the no-spec value. The calculator applies both rules.
• DFlash2: a trained block-diffusion draft in a separate checkpoint — add --speculative-algorithm DFLASH --speculative-draft-model-path incoai/Qwen3.8-27B-DFlash2 --speculative-num-draft-tokens 8 (8 is the draft’s block size, and it is the D term in the ratio, same value as DSpark’s). Runs on Ascend NPUs as well (#35629): the selector verify falls back to argmax there, matching what EAGLE and first-generation DFlash drafts (such as z-lab/Qwen3-8B-DFlash-b16) already do on NPU. Therefore, NPU currently guarantees lossless verification only for greedy requests; use temperature=0 and top_k=1. Non-greedy requests log a warning; both draft proposal and target verification fall back to greedy, so their requested sampling distribution is not preserved. The selector projects candidates through the target lm_head, including quantized heads, so it runs on the NVFP4 checkpoint (whose head is NVFP4-packed; the BF16 and FP8 checkpoints keep a dense head). The Ascend comparison in #35629 used an A3 Series device with BF16 target weights, --tp-size 2 --attention-backend ascend --mamba-ssm-dtype bfloat16 --mamba-radix-cache-strategy extra_buffer, and disabled RadixCache for both baseline and DFlash2 to exclude cache warm-up and prefix reuse. The DFlash2 run added the three flags shown above. That comparison’s accuracy used zero-shot GSM8K with greedy sampling, max_new_tokens=2048, 128 examples, and concurrency levels 1, 2, 4, 8 and 16 — a different protocol from this page’s own sweep below. Validation: every SM12x cell on this page is measured end to end on v0.5.19 — 202 cells over the five checkpoints, four speculative options, two serving tiers and two GDN state dtypes, full 1319-question GSM8K on each, 93.18-95.15%. The RTX PRO 6000 and DGX Spark recipes need no changes. On the 32GB RTX 5090 the panel applies the measured pins automatically: DFlash2 at --mem-fraction-static 0.91 with --chunked-prefill-size 1024 — at 0.91 the pools fit but a 2048-token chunk’s activations do not — DSpark at 0.88 (bfloat16), 0.91 (float32) and 0.92 on the dense-lm_head export, all three with their pools pinned and the last two also cutting the prefill chunk to 1024 and 512, EAGLE at 0.93 (bfloat16) and 0.94 (float32), and no-speculation at 0.90. Whether float32 is available with a draft model depends on the lm_head: on the BF16-head export it is greyed out for both DSpark and DFlash2, since the dense head’s ~3.2 GB leave no fp32 state pool that also clears prefill graph capture. The FP4-head export frees that headroom back — DSpark serves at 0.91 and DFlash2 High-Throughput at 0.895 with --mamba-full-memory-ratio 10 overriding the balanced value — and only DFlash2 Low-Latency stays out of reach, where five fp32 slots and a full request’s KV never coexist. bfloat16 remains the faster choice regardless: DFlash2 posts 4.92 ms median TPOT at an accept length of 4.29, the best result on this card.
• Hardware fit: FP8 weights ~28.5GB (not serviceable beyond bs≤2 on 32GB cards); NVFP4 weights ~16.5GB (recommended for RTX 5090-class GPUs).
• --mamba-radix-cache-strategy extra_buffer_lazy lowers the state cost per request from 5 slots to 4 at no accuracy cost. On small-VRAM cards (RTX 5090 32GB) the state pool bounds concurrency long before KV does — prefer lowering S (lazy strategy, or --disable-radix-cache for S=1); the calculator re-derives the ratio for the new S. The balanced ratio itself is VRAM-independent.
• --mamba-ssm-dtype: the GDN state slot is 153.9 MB at float32 (the checkpoint’s declared precision) and 78.4 MB at bfloat16, so bf16 roughly halves the state pool and hands the difference to KV — measured on an RTX 5090 with no speculation, 97,280 KV tokens at bf16 against 68,588 at fp32. On 32GB cards it also decides whether a config fits at all: EAGLE needs --mem-fraction-static 0.94 at fp32 but 0.92 at bf16. Speed is not a one-way trade — with speculative decoding fp32 sometimes wins (NVFP4 + EAGLE: 152.9 vs 144.5 tok/s/user) and sometimes loses (FP8 + EAGLE: 106.3 vs 116.1); measure both for your quantization. Treat bfloat16 as an accuracy gate and validate it for your workload. On SM120 both precisions run the Triton linear-attn prefill path — the FlashInfer GDN prefill fast path gates on SM100, where its validated domain is in fact a bf16 state pool — so no dtype forces an extra flag here. One interaction to know: --enable-linear-replayssm-spec auto-selects fp32 state when --mamba-ssm-dtype is unset, and an explicit non-fp32 value logs a state-drift warning at boot. The SSM dtype row always emits the flag explicitly, so the bf16 + EAGLE cells run with that warning — accounted for in their validation.
• --chunked-prefill-size 2048: decode steps stall behind each prefill chunk on hybrid GDN models, and 8192-token chunks stall them ~600ms at a time. 2048 keeps decode inter-token latency smooth under mixed load and also improves single-wave TTFT.

​

3. Agent Harnesses

​

3.1 OpenCode

opencode
/connect

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "sglang": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "SGLang (Qwen3.8-27B)",
      "options": {
        "baseURL": "http://localhost:30000/v1"
      },
      "models": {
        "RadixArk/Qwen3.8-27B-NVFP4": {
          "name": "Qwen3.8-27B NVFP4"
        }
      }
    }
  }
}

​

3.2 Pi

pi.registerProvider("sglang", {
  baseUrl: "http://localhost:30000/v1",
  api: "openai-completions",
  apiKey: "$SGLANG_API_KEY",
  models: [
    {
      id: "RadixArk/Qwen3.8-27B-NVFP4",
      name: "Qwen3.8-27B",
      reasoning: true,
      input: ["text", "image"],
      cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
      contextWindow: 262144,
      maxTokens: 32768,
    },
  ],
});

​

3.3 Claude Code

export ANTHROPIC_BASE_URL=http://localhost:30000
export ANTHROPIC_AUTH_TOKEN=placeholder

{
  "env": {
    "ANTHROPIC_BASE_URL": "http://localhost:30000",
    "ANTHROPIC_AUTH_TOKEN": "placeholder"
  }
}

​

3.4 Hermes Agent

hermes model
# choose "Custom endpoint (self-hosted / VLLM / etc.)", then enter the
# base URL, an API key (blank for a local server) and the model name

model:
  default: RadixArk/Qwen3.8-27B-NVFP4
  provider: custom
  base_url: http://localhost:30000/v1
  api_key: ""
  context_length: 262144

providers:
  workstation:
    api: http://localhost:30000/v1
  server:
    api: https://gpu-host.internal:30000/v1
    key_env: SGLANG_API_KEY