GLM-5.3 served at 607.8 tok/s on DGX with vLLM
Archived — this story has rotated out of today’s deck. It is kept here in full.
The gist
NVIDIA published a guide to serve Z.ai's GLM-5.3 on two DGX Stations, peaking at 607.8 tok/s.
The model needs about 464 GB of GPU memory, more than one GB300 GPU's 252 GB.
Background
NVIDIA is the chipmaker whose technical blog published the serving guide, and Z.ai makes GLM-5.3, a mixture-of-experts model in which only 40 billion of its 753 billion parameters run for any one token. A router picks different experts per token, so every expert must stay in GPU memory: NVIDIA says the NVFP4 version needs about 464 GB, exceeding the 252 GB of HBM3e (high-bandwidth memory) on a single GB300 GPU. The guide, written by Fatima Fall, shows how two machines split the model instead.
How it unfolded
- Nov 13, 2026NVIDIA publishes a step-by-step guide to serving GLM-5.3 on two DGX Stations with vLLM, using tensor parallelism across both machines.
- Nov 13, 2026NVIDIA reports overall throughput peaking at 607.8 tok/s at concurrency 16, falling to 385.0 tok/s at concurrency 32 with mean request latency of 453.5 s; single-user speed is 107.4 tok/s.
- Nov 14, 2026The NVIDIA post is shared on Hacker News (12 points, posted by wondersky) and on Threads.
Key terms
- mixture-of-experts
- A machine learning technique using multiple expert networks to divide a problem space into regions.
- HBM3e
- A memory interface using 3D-stacked DRAM chips to deliver very high bandwidth, often paired with graphics accelerators.
Source: Wikipedia
Who’s saying what
- Official
- NVIDIA says two DGX Stations connected over ConnectX-8 RDMA serve the model through one OpenAI-compatible endpoint with tensor parallelism (TP=2), each GPU holding and computing roughly half the weights.
- Caution
- explainx.ai's Yash Thakker notes the API is unauthenticated unless you add a firewall, auth and TLS.