Pi0.5 - SGLang Documentation

dVLAOpenPI / LeRobotflow matching actionrobot edge

1. Model Introduction

Pi0.5 is an OpenPI / LeRobot diffusion Vision-Language-Action (dVLA) policy. It consumes camera images, a language instruction, and robot state, then returns a continuous action chunk for robot control. SGLang serves Pi0.5 through the native multimodal_gen runtime. The implementation uses a SigLIP/PaliGemma prefix encoder and a Gemma action expert: the prefix is encoded once, then the action expert runs the flow-matching denoising loop. This is not a token decode workload, so the Pi0.5 path does not use the LLM sampler, logits processor, token streaming, paged decode KV cache, or a separate SRT serving engine. The SigLIP vision tower, PaliGemma language stack, and action expert are all SGLang-native modules. Transformers is used for checkpoint configuration and tokenization, not for the runtime neural network. The prefix encoder covers both stages of observation encoding: SigLIP turns resized camera pixels into continuous patch embeddings, then the PaliGemma transformer jointly encodes those patches with tokenized task/state inputs and produces per-layer prefix K/V. At flow timestep t, the action expert projects the noisy continuous action chunk x_t into action embeddings. Its queries attend to both the fixed prefix K/V and the current action K/V, while the timestep follows a separate sinusoidal-MLP path and conditions every action-expert layer through AdaRMSNorm gates. Supported public checkpoints:

Checkpoint	Cameras	State Dim	Output Action Dim	Action Horizon	Denoise Steps
`lerobot/pi05_base`	`base_0_rgb`, `left_wrist_0_rgb`, `right_wrist_0_rgb`	32	32	50	10
`lerobot/pi05_libero_base`	`image`, `image2`, one empty camera	8	7	50	10

References:

2. Installation

Install SGLang with the diffusion extra. Pi0.5 lives in multimodal_gen, and the extra includes the runtime dependencies used by the policy server.

Command

git clone https://github.com/sgl-project/sglang.git
cd sglang
pip install -e "python[diffusion]"

For general environment setup, see the SGLang Diffusion installation guide.

3. Model Deployment

Serve the base Pi0.5 policy:

Command

sglang serve lerobot/pi05_base \
  --model-type diffusion \
  --host 127.0.0.1 \
  --port 30000

Serve the LIBERO checkpoint:

Command

sglang serve lerobot/pi05_libero_base \
  --model-type diffusion \
  --host 127.0.0.1 \
  --port 30000

These registered LeRobot checkpoints dispatch to the native multimodal_gen Pi0.5 pipeline automatically. The --pipeline / --pipeline-class-name flag is only an advanced override for local or private checkpoints that cannot be resolved from the model registry. Pi0.5 serving does not start a second SRT LLM serving engine.

3.1 Action Request Schema

Field	Type	Description
`model`	string, optional	Served model name. When omitted, the server uses the currently loaded policy.
`input.task` or `input.prompt`	string	Language instruction for the policy.
`input.observation.images`	object	Map from camera name to RGB image. Images can be nested here or sent as OpenPI observation keys such as `observation.images.base_0_rgb` through the websocket adapter.
`input.observation.state`	array or tensor object	Robot state vector. Use the same normalization convention as the OpenPI / LeRobot checkpoint.
`input.observation.noise`	array or tensor object, optional	Initial action noise with shape `[action_horizon, action_dim]`. Use this for deterministic debugging.
`parameters.num_inference_steps`	integer, optional	Flow-matching denoise steps. Defaults to `10`.
`parameters.action_horizon`	integer, optional	Output action horizon. Defaults to the checkpoint config.
`parameters.action_dim`	integer, optional	Internal padded action dimension. Defaults to the checkpoint config.
`runtime.return_timing`	boolean, optional	Return stage timing fields. Defaults to `true`.
`runtime.prefix_cache`	boolean or `"auto"`, optional	Enable exact full-prefix lookup for this request when the server has `enable_global_prefix_cache=true`. Defaults to `"auto"`.
`runtime.cuda_graph`	boolean or `"auto"`, optional	Enable the available prefix and action-denoise CUDA graph paths for this request. Defaults to `"auto"`.
`runtime.output_format`	`"list"` or `"numpy"`, optional	Use `"list"` for JSON compatibility. Use `"numpy"` with msgpack or Python clients to avoid Python-list materialization. Defaults to `"list"`.
`runtime.response_format`	`"envelope"` or `"raw"`, optional	HTTP-only response shape. `"envelope"` returns the generic action envelope. `"raw"` returns the policy payload directly. Defaults to `"envelope"`.

4. API Usage

4.1 Generic Action HTTP API

Use /v1/actions/generations for direct policy calls and debugging. Use /v1/actions/metadata to discover the camera keys, state size, action shape, defaults, and websocket capabilities of the currently served policy.

Example

import numpy as np
import requests

image = np.zeros((224, 224, 3), dtype=np.uint8)
payload = {
    "model": "lerobot/pi05_base",
    "input": {
        "task": "pick up the block",
        "observation": {
            "images": {
                "base_0_rgb": image.tolist(),
                "left_wrist_0_rgb": image.tolist(),
                "right_wrist_0_rgb": image.tolist(),
            },
            "state": np.zeros(32, dtype=np.float32).tolist(),
        },
    },
    "runtime": {
        "return_timing": True,
        "prefix_cache": "auto",
        "cuda_graph": "auto",
    },
}

response = requests.post(
    "http://127.0.0.1:30000/v1/actions/generations",
    json=payload,
    timeout=60,
)
response.raise_for_status()
data = response.json()

actions = data["data"][0]["action"]["values"]
print(len(actions), len(actions[0]))
print(data.get("timings"))

The same /v1/actions/generations endpoint also accepts Content-Type: application/msgpack and can return msgpack when Accept: application/msgpack is set. For msgpack clients, send numpy arrays directly using the pack_numpy_payload helper from the websocket example below. Msgpack requests default to numpy action output on the server side; set runtime.output_format to "list" only when a client explicitly needs nested Python lists. For the lowest-overhead generic HTTP path, use msgpack with the raw policy response:

Example

payload["runtime"] = {
    "return_timing": True,
    "prefix_cache": False,
    "cuda_graph": "auto",
    "response_format": "raw",
}

response = requests.post(
    "http://127.0.0.1:30000/v1/actions/generations",
    data=packb(payload),
    headers={
        "Content-Type": "application/msgpack",
        "Accept": "application/msgpack",
    },
    timeout=60,
)
result = unpackb(response.content)
actions = result["actions"]

For lerobot/pi05_libero_base, use the LIBERO camera names and an 8-dimensional state vector:

Example

payload = {
    "input": {
        "task": "pick up the object",
        "observation": {
            "images": {
                "image": image.tolist(),
                "image2": image.tolist(),
            },
            "state": np.zeros(8, dtype=np.float32).tolist(),
        },
    },
}

4.2 Generic Realtime WebSocket

/v1/actions/realtime is the generic msgpack websocket path. It sends action.metadata on connect and returns the same action.generation envelope as the HTTP API for each request.

4.3 OpenPI-Compatible WebSocket

Robot clients can use the OpenPI-compatible msgpack websocket endpoint at /openpi/policy. The server sends metadata immediately after connection, then each client message should contain one observation.

Example

import asyncio

import msgpack
import numpy as np
import websockets


def pack_array(obj):
    if isinstance(obj, np.ndarray):
        return {
            b"__ndarray__": True,
            b"data": obj.tobytes(),
            b"dtype": obj.dtype.str,
            b"shape": obj.shape,
        }
    if isinstance(obj, np.generic):
        return {
            b"__npgeneric__": True,
            b"data": obj.item(),
            b"dtype": obj.dtype.str,
        }
    return obj


def unpack_array(obj):
    ndarray_marker = obj.get("__ndarray__") or obj.get(b"__ndarray__")
    npgeneric_marker = obj.get("__npgeneric__") or obj.get(b"__npgeneric__")
    data = obj.get("data", obj.get(b"data"))
    dtype = obj.get("dtype", obj.get(b"dtype"))
    shape = obj.get("shape", obj.get(b"shape"))
    if ndarray_marker:
        return np.ndarray(
            buffer=data,
            dtype=np.dtype(dtype),
            shape=shape,
        )
    if npgeneric_marker:
        return np.dtype(dtype).type(data)
    return obj


def packb(payload):
    return msgpack.packb(payload, default=pack_array, use_bin_type=True)


def unpackb(payload):
    return msgpack.unpackb(payload, object_hook=unpack_array, raw=False)


async def main():
    image = np.zeros((224, 224, 3), dtype=np.uint8)
    observation = {
        "task": "pick up the block",
        "observation.images.base_0_rgb": image,
        "observation.images.left_wrist_0_rgb": image,
        "observation.images.right_wrist_0_rgb": image,
        "observation.state": np.zeros(32, dtype=np.float32),
    }

    async with websockets.connect(
        "ws://127.0.0.1:30000/openpi/policy",
        max_size=None,
    ) as websocket:
        metadata = unpackb(await websocket.recv())
        print(metadata)

        await websocket.send(packb(observation))
        result = unpackb(await websocket.recv())
        actions = result["actions"]
        print(len(actions), len(actions[0]))
        print(result.get("server_timing"))


asyncio.run(main())

5. Configuration Tips

Request-local PrefixContext is always reused across all denoise steps in one request. The prefix K/V is not cloned per step.
The optional global prefix cache is a bounded exact-match LRU. It is disabled by default because changing robot frames rarely hit it and enabling it prevents unrelated misses from entering grouped prefix execution. Set enable_global_prefix_cache=true for repeated observations, retries, or multiple policy calls over the same camera/state sample; runtime.prefix_cache can then disable lookup per request.
Partial-prefix reuse is not supported because Pi0.5 combines image and tokenized task/state inputs under full attention. Changing any input can change every deeper-layer prefix K/V tensor. The exact key hashes resized and normalized pixels before SigLIP, plus effective token IDs, token masks, camera masks, model revision, dtype, and parallel layout. Tensor content hashing reuses SRT’s CPU/CUDA implementation; hashing the pre-SigLIP input lets an exact hit skip both the vision encoder and prefix transformer.
CUDA graph capture targets single-request prefix encoding and one action-denoise step. The lossless default keeps exact prompt lengths and one resident prefix graph. Prefix and action graph residency are independently bounded by prefix_cuda_graph_max_entries (default 1) and action_cuda_graph_max_entries (default 4). Without prompt buckets, unseen signatures run eagerly after the cache reaches capacity, avoiding capture churn. The denoise signature includes whether prefix attention is full or masked, plus batch size, prefix length, action horizon, action dim, dtype, and parallel layout. Mask-aware action graphs rebuild position IDs from the current request mask inside the captured graph. With action SP enabled, the denoise bucket uses the local action shard length and rank-specific position offset.
prompt_token_buckets optionally right-pads prompt tokens to fixed sizes such as [32, 64, 128, 200], allowing nearby prompt lengths to reuse both graphs. In this mode the bounded graph caches use LRU replacement and reset evicted CUDA graphs; prompts beyond the largest configured bucket stay exact-length and eager. This is opt-in because padding changes GPU reduction shapes: an H200 sweep over lengths around all four boundaries was structurally correct and bounded to four prefix/action graphs, but non-boundary prompts differed from the exact-length path by up to 0.09589 in normalized action space over five denoise steps. Validate closed-loop policy quality before enabling it; empty buckets preserve the numerically lossless path.

An opt-in four-bucket serving config is:

File

{
  "prompt_token_buckets": [32, 64, 128, 200],
  "prefix_cuda_graph_max_entries": 4,
  "action_cuda_graph_max_entries": 4
}

Cache-DiT is not used in the default Pi0.5 path. The current robot policy target is numerically lossless inference, while Cache-DiT-style reuse is an image/video DiT approximation that needs separate policy-quality validation before it can be recommended for action control.
Do not use CFG parallelism to split the 10 Euler steps. Use it only for independent branches such as multiple candidate actions or future conditional/unconditional branches.
Prefix TP uses native SGLang parallel linear layers for the PaliGemma language prefix model when model parallel TP is initialized and the VLA split broadcast group is not active. The action expert does not share that TP layout. The v1 split prefix/action path instead uses the SP group: prefix root computes/broadcasts PrefixContext, while action ranks run the SP action path.
The split path uses the SP group as the action group: prefix root computes/broadcasts PrefixContext, all action ranks broadcast the initial action noise once, shard the action horizon, and run the action expert through Ulysses attention when the prefix is full-attention, ring degree is one, heads are divisible by SP size, and the horizon is evenly shardable. Otherwise it falls back to action-root execution.
lerobot/pi05_libero_base returns 7 action dimensions even though the internal padded action tensor uses 32 dimensions.

6. VRAM Tuning

For current public Pi0.5 checkpoints, a stable 16GB discrete GPU target is a reasonable v1 deployment bar for robot workstations. In practice, leave headroom for the driver, camera middleware, robot process, and allocator fragmentation. Jetson/Orin unified-memory devices need extra caution because system RAM and GPU memory share the same pool. OpenPI inference is mixed precision, not full fp32: most weights and compute run in bf16, selected stability-sensitive weights stay in fp32, and returned actions are float32. SGLang mirrors that policy by default. The validated pi05_aloha OpenPI PyTorch checkpoint keeps 119,720,608 parameters in fp32; SGLang reports the same fp32 stability set, plus 3,233,713,264 bf16 runtime parameters after skipping unused LM heads for continuous action inference. The fp32 set includes SigLIP patch/position embeddings, Gemma layer norms/final norms, and the action/time projection heads. Keep materialize_dtype at bf16 unless you are debugging numerical parity. Current H100 pressure validation shows that the bf16 model path fits a 16GB-free budget without layerwise offload. The Pi0.5 path batches all camera frames for a grouped request into one SigLIP forward before splitting the embeddings back by camera, which is important for multi-camera robot workloads. The latency numbers below use the Python grouped API with global prefix cache disabled and CUDA graph enabled on unconstrained H100; rerun HTTP/OpenPI websocket on your target server before using the policy in a closed-loop robot.

Mode	Command Shape	Steady VRAM Snapshot	Notes
Single GPU bf16	`--num-gpus 1`, no offload	fit with `16383 MiB` free before load in H100 pressure test	Recommended first path for 16GB-class discrete GPUs.
ALOHA bf16 grouped	Python API, batch size 1 / 2 / 4 / 8	unconstrained H100	`52.4 ms` single; `65.1 ms / 2`; `91.9 ms / 4`; `147.4 ms / 8`.
ALOHA 16GB-free pressure	Python API, no offload	same 16GB-free pressure	Fit was validated after the bf16 correction; rerun latency on the target because older pressure latency was collected before the final OpenPI precision fix.
Global prefix cache disabled	`enable_global_prefix_cache=false` or request `runtime.prefix_cache=false`	avoids cache growth across changing frames	Recommended for tight edge budgets unless repeated exact frames are common.
Offload fallback	config in 6.2	target-dependent	Use only if the default bf16 path OOMs on the target device.

Use these knobs first because they do not change action numerics for a fixed input/noise:

6.1 16GB Edge Config

Use this single-GPU config first for 16GB-class robot workstations. It keeps parameters resident on GPU, keeps CUDA graph enabled, and disables global prefix cache growth.

File

{
  "materialize_dtype": "bf16",
  "enable_global_prefix_cache": false,
  "prefix_cache_max_entries": 0,
  "enable_prefix_cuda_graph": true,
  "prefix_cuda_graph_max_entries": 1,
  "enable_action_cuda_graph": true,
  "action_cuda_graph_max_entries": 1
}

Start a single-GPU server with the override:

Command

sglang serve lerobot/pi05_base \
  --model-type diffusion \
  --pipeline-config-path pi05_edge_16gb.json \
  --num-gpus 1 \
  --warmup-mode off \
  --host 127.0.0.1 \
  --port 30000

Treat the H100 pressure result as a memory-budget validation, not a Jetson latency guarantee. Jetson/Orin devices use shared system memory and have much lower effective memory bandwidth than H100, so measure closed-loop control latency on the target device before deciding the action chunk cadence.

6.2 Offload Fallback

Use offload only when the default bf16 config still does not fit on the target. These modes are numerically lossless for fixed inputs/noise, but they move weights between CPU and GPU and can hurt latency substantially. For a moderate fallback, offload cache growth and selected stage-resident modules first:

File

{
  "materialize_dtype": "bf16",
  "enable_global_prefix_cache": false,
  "prefix_cache_max_entries": 0,
  "enable_prefix_cuda_graph": false,
  "enable_action_cuda_graph": false,
  "offload_prefix_image_encoder_after_embed": true,
  "offload_prefix_token_embedding": true,
  "offload_prefix_language_layer_count_after_prefix": 2,
  "offload_action_expert_after_denoise": true,
  "empty_cache_after_prefix": true
}

If that still does not fit, full prefix layerwise CPU offload keeps every PaliGemma language layer on CPU and moves one layer at a time to GPU during prefix compute:

File

{
  "materialize_dtype": "bf16",
  "enable_global_prefix_cache": false,
  "prefix_cache_max_entries": 0,
  "enable_prefix_cuda_graph": false,
  "enable_action_cuda_graph": false,
  "offload_prefix_image_encoder": true,
  "offload_prefix_token_embedding": true,
  "offload_prefix_language_layers": true,
  "offload_prefix_language_layers_empty_cache": true,
  "empty_cache_after_prefix": true
}

Offload validation should be repeated on the target hardware after any dtype or loader change. Earlier fp32-runtime offload numbers are not comparable to the current bf16 path.

6.3 Per-Request Controls

For HTTP calls, disable cache or both CUDA graph paths without restarting the server:

Example

payload = {
    "input": {
        "task": "pick up the block",
        "observation": {
            "images": images,
            "state": state,
        },
    },
    "runtime": {
        "prefix_cache": False,
        "cuda_graph": False,
    },
}

For OpenPI websocket clients, include equivalent enable_prefix_cache and enable_cuda_graph fields in each raw msgpack observation if you need per-request compatibility controls.

6.4 Deployment Choices

Keep batch size to one control stream unless grouped robot streams are explicitly validated. More concurrent observations increase activation and PrefixContext residency.
Keep materialize_dtype at the default bf16. fp32 is useful only for debugging numerical issues and will increase memory and latency.
Use split prefix/action only when you have multiple GPUs and have validated the robot control latency on that topology. On two GPUs with --sp-degree 2 --ulysses-degree 2, the action horizon can be sequence-sharded while the prefix root still computes and broadcasts PrefixContext. A single 16GB-class GPU should try the default bf16 config first.
Reducing num_inference_steps lowers latency but is not a pure memory fix and can change policy behavior. Validate closed-loop task success before using fewer than the checkpoint default.
CPU offload is a compatibility fallback, not the preferred 16GB path. Quantization and deeper prefix/vision sharding are the next steps for sub-16GB devices.

6.5 Loader And Run:ai Model Streamer

Run:ai Model Streamer can improve cold-start and checkpoint loading by reading safetensors concurrently and streaming tensors toward GPU memory. It is useful for local SSD, object storage, and cloud deployments where startup time is dominated by model file IO. The python[diffusion] extra includes runai_model_streamer. If the package is installed, SGLang enables it by default through SGLANG_USE_RUNAI_MODEL_STREAMER=true. Set it to false to force the plain safetensors loader:

Command

SGLANG_USE_RUNAI_MODEL_STREAMER=false \
sglang serve lerobot/pi05_base \
  --model-type diffusion \
  --host 127.0.0.1 \
  --port 30000

Pi0.5 uses the direct SSD-to-GPU Run:ai path when all load targets for the current process are GPU-resident. This check is rank-local: distributed action ranks can stream their action-expert subset directly to GPU, while any rank with CPU/offloaded target tensors stays on the CPU safetensors fallback. The single-GPU validation streamed 13.5 GiB of safetensors directly to cuda:0 in about 1.5 s, then returned [50, 32] actions over both HTTP and OpenPI websocket. For mixed CPU offload and low-VRAM 16GB-class modes, Pi0.5 still uses the header-filtered safe loader for ranks whose target tensors are not fully CUDA-resident. Direct GPU streaming can increase the exact VRAM pressure the low-memory path is trying to avoid. When debugging distributed startup, compare with SGLANG_USE_RUNAI_MODEL_STREAMER=false to separate streamer behavior from model execution behavior. Run:ai Model Streamer does not reduce steady inference VRAM after parameters, caches, activations, CUDA contexts, and graph buffers are resident. For Pi0.5 low-VRAM work, prioritize component placement, prefix cache size, CUDA graph residency, and CPU/offload first. Model Streamer is a cold-start optimization after the steady-memory budget is correct.

7. OpenPI Comparison Benchmark

Use bench_pi05_openpi.py when you need a side-by-side latency and action-difference report against the OpenPI policy implementation. Start the SGLang Pi0.5 server first for HTTP or websocket modes, then run the benchmark from the SGLang repository root:

Command

python python/sglang/multimodal_gen/benchmarks/bench_pi05_openpi.py \
  --profile aloha \
  --sglang-url http://127.0.0.1:30000 \
  --openpi-checkpoint gs://openpi-assets/checkpoints/pi05_base \
  --num-inference-steps 10 \
  --batch-size 4 \
  --repeats 20 \
  --warmup 3 \
  --deterministic-noise \
  --output pi05_openpi_compare.json

The benchmark reports:

SGLang HTTP latency for /v1/actions/generations, including stage timings when the server returns them.
SGLang msgpack HTTP latency when --sglang-api http_msgpack is set. This keeps the generic HTTP endpoint but avoids JSON image-array overhead. Use --sglang-http-response-format raw to benchmark the compact raw policy response.
SGLang OpenPI-compatible websocket latency when --sglang-api openpi_ws is set. This uses persistent msgpack websocket connections and is closer to the robot client path than JSON-over-HTTP.
SGLang Python in-process latency when --sglang-api python is set. This loads the native Pi0.5 pipeline in the benchmark process and avoids HTTP, websocket, scheduler, and serialization overhead. Use --sglang-python-batch-mode grouped to exercise the conservative native grouped-batch path. The Python path also reports actual SGLang module parameter dtype counts and example parameter names.
OpenPI single-request latency through Policy.infer.
Batch latency for grouped robot streams. SGLang uses concurrent HTTP requests in HTTP mode and persistent multi-connection msgpack calls in websocket mode. The Python backend can use true grouped model execution for fresh-prefix requests. OpenPI defaults to its internal direct model batch path because the public Policy.infer API is single-observation.
Action difference in normalized model space from identical OpenPI-transformed model inputs and noise. The check requires --deterministic-noise and fails when either --action-max-abs-diff or --action-mean-abs-diff is exceeded. This mode isolates model parity from robot-specific normalization and action postprocessing. The LIBERO policy returns only 10 actions after policy postprocessing, but its flow-matching model still generates a 50-step chunk; the benchmark compares that model output with SGLang before OpenPI unnormalization and horizon slicing.

For one-sided 16GB-class checks, run each backend separately under the same VRAM pressure. The SGLang Python path accepts the same pipeline config override as serving:

Command

python python/sglang/multimodal_gen/benchmarks/bench_pi05_openpi.py \
  --profile aloha \
  --sglang-api python \
  --sglang-python-batch-mode grouped \
  --skip-openpi \
  --sglang-pipeline-config-path pi05_edge_16gb.json \
  --num-inference-steps 10 \
  --batch-size 4 \
  --repeats 5 \
  --warmup 2 \
  --deterministic-noise \
  --disable-prefix-cache \
  --disable-cuda-graph

Use --skip-sglang --openpi-pytorch-compile-mode none to measure an OpenPI eager baseline in the same environment. For a PyTorch-native OpenPI baseline, point --openpi-checkpoint at a checkpoint directory containing model.safetensors and the OpenPI assets/ norm-stat tree. The keep mode preserves OpenPI’s checkpoint default compile setting. The grouped Python path currently requires compatible fresh-prefix requests without split prefix/action workers or effective prefix-cache hits; other cases fall back to per-request execution.

8. Validation Notes

The following checks were run on H100 GPUs with the native SGLang Pi0.5 path:

Check	Result
`lerobot/pi05_base` direct end-to-end	Prefix length `968`, output shape `[1, 50, 32]`, peak allocated memory `12.817 GiB`.
Official OpenPI parity	Against OpenPI PyTorch revision `15a9616`, with the same LeRobot checkpoint revision, observation, and noise: first-step velocity max/mean absolute difference `0.02677` / `0.00344`; production 10-step normalized action `0.00813` / `0.00092`.
Action denoise CUDA graph	Eager 10-step denoise `125.4 ms`; steady graph replay `50.8 ms`; max output difference `0`.
Prefix CUDA graph	On H200 with action graph already enabled, ALOHA batch=1 p50 improved from `48.04 ms` to `42.98 ms` (`1.118x`). Two observations at 5 and 10 steps were bit-exact with graph disabled. One prefix shape bucket added about `48.5 MiB`; batch=4 showed no benefit and stays eager.
Exact full-prefix cache	First prefix pass about `203 ms`; exact cache hit prefix stage about `0.2 ms`.
`lerobot/pi05_libero_base` direct end-to-end	Image keys `image`, `image2`, `empty_camera_0`; state dim `8`; output action dim `7`; output tensor shape `[1, 50, 32]`.
Python grouped execution	ALOHA batch=4 grouped path measured `91.9 ms / 4` on current mixed precision: prefix `18.4 ms`, action denoise `61.2 ms`, preprocess about `2.4 ms` per request. Sequential Python loop batch=4 measured `211.9 ms / 4`.
`sglang serve` HTTP	`/v1/actions/generations` returned action shape `[50, 32]`. JSON-over-HTTP remains compatible but image-array serialization dominates. Msgpack HTTP with prefix cache disabled measured `57.2 ms` single and `221.9 ms / 4` for the envelope response, and `56.4 ms` single and `219.7 ms / 4` for `runtime.response_format="raw"`.
OpenPI websocket	`/openpi/policy` returned action shape `[50, 32]`; with persistent msgpack connections and prefix cache disabled, ALOHA measured about `77.0 ms` single and `162.0 ms / 4`.
2-GPU prefix/action split	Earlier split validation returned action shape `[50, 32]` and matched single-GPU HTTP with max absolute difference `0`. After the true action-SP change, rerun this check with `--num-gpus 2 --sp-degree 2 --ulysses-degree 2` and verify both action ranks enter denoise kernels.
OpenPI/SGLang precision	Official OpenPI JAX inference restores the public GCS checkpoint as bf16 with selected fp32 stability compute and returns float32 actions. The converted OpenPI PyTorch `pi05_aloha` checkpoint keeps `119,720,608` fp32 stability params; SGLang reports the same fp32 set and `3,233,713,264` bf16 runtime params after skipping unused LM heads.
Native attention dtype	Checkpoint source tensors may be fp32, but SGLang finalizes PiGemma and SigLIP compute dtype before native attention backend selection; backend logs showed `Using fa attention backend` for the PiGemma path in the prior run.
16GB-free Python pressure	With an H100 artificially constrained to `16381 MiB` free before model load, single-GPU bf16 no-offload Python grouped path completed without OOM. Re-run latency after precision or loader changes before using pressure numbers for deployment sizing.
Low-VRAM switches	Disabling prefix cache prevents cache growth across changing robot frames. Prefix graph residency defaults to `1`, action graph residency defaults to `4`, and setting either limit to `0` disables that runner. One validated ALOHA prefix graph used about `48.5 MiB`; action graph residency is now bounded separately. Keep `prompt_token_buckets` empty for the lossless path.
Offload fallback	CPU/offload modes are retained as numerically lossless compatibility fallbacks, but earlier fp32-runtime offload latency numbers are stale after the bf16 dtype correction and should be revalidated before deployment decisions.
Run:ai direct loader	Single-GPU serve streamed `13.5 GiB` safetensors to `cuda:0` in about `1.5 s` and returned `[50, 32]` actions. Distributed direct streaming is now rank-local and should be revalidated on the target split topology; offload ranks with CPU targets still use the safe loader.
OpenPI comparison status	Official OpenPI GCS `pi05_base` is a JAX checkpoint; converted PyTorch eager was validated without `torch.compile`. On 80GB H100, ALOHA OpenPI PyTorch eager was about `125-130 ms` single and about `164 ms / 4` in the direct-model batch path. Current SGLang Python grouped measured `52.4 ms` single and `91.9 ms / 4`; JAX OpenPI was `53.0 ms` single and `59.5 ms / 2` in a short check.

Run a full HTTP or websocket smoke test in your robot deployment environment before using the policy in a closed-loop controller.

​1. Model Introduction

​2. Installation

​3. Model Deployment

​3.1 Action Request Schema

​4. API Usage

​4.1 Generic Action HTTP API

​4.2 Generic Realtime WebSocket

​4.3 OpenPI-Compatible WebSocket

​5. Configuration Tips

​6. VRAM Tuning

​6.1 16GB Edge Config

​6.2 Offload Fallback

​6.3 Per-Request Controls

​6.4 Deployment Choices

​6.5 Loader And Run:ai Model Streamer

​7. OpenPI Comparison Benchmark

​8. Validation Notes

1. Model Introduction

2. Installation

3. Model Deployment

3.1 Action Request Schema

4. API Usage

4.1 Generic Action HTTP API

4.2 Generic Realtime WebSocket

4.3 OpenPI-Compatible WebSocket

5. Configuration Tips

6. VRAM Tuning

6.1 16GB Edge Config

6.2 Offload Fallback

6.3 Per-Request Controls

6.4 Deployment Choices

6.5 Loader And Run:ai Model Streamer

7. OpenPI Comparison Benchmark

8. Validation Notes