GLM-5.3 served at 607.8 tok/s on DGX with vLLM
explainx.ai

GLM-5.3 served at 607.8 tok/s on DGX with vLLM

Archived — this story has rotated out of today’s deck. It is kept here in full.

The gist

NVIDIA published a guide to serve Z.ai's GLM-5.3 on two DGX Stations, peaking at 607.8 tok/s.
The model needs about 464 GB of GPU memory, more than one GB300 GPU's 252 GB.

Background

NVIDIA is the chipmaker whose technical blog published the serving guide, and Z.ai makes GLM-5.3, a mixture-of-experts model in which only 40 billion of its 753 billion parameters run for any one token. A router picks different experts per token, so every expert must stay in GPU memory: NVIDIA says the NVFP4 version needs about 464 GB, exceeding the 252 GB of HBM3e (high-bandwidth memory) on a single GB300 GPU. The guide, written by Fatima Fall, shows how two machines split the model instead.

How it unfolded

  1. Nov 13, 2026NVIDIA publishes a step-by-step guide to serving GLM-5.3 on two DGX Stations with vLLM, using tensor parallelism across both machines.
  2. Nov 13, 2026NVIDIA reports overall throughput peaking at 607.8 tok/s at concurrency 16, falling to 385.0 tok/s at concurrency 32 with mean request latency of 453.5 s; single-user speed is 107.4 tok/s.
  3. Nov 14, 2026The NVIDIA post is shared on Hacker News (12 points, posted by wondersky) and on Threads.

Key terms

mixture-of-experts
A machine learning technique using multiple expert networks to divide a problem space into regions.
HBM3e
A memory interface using 3D-stacked DRAM chips to deliver very high bandwidth, often paired with graphics accelerators.

Source: Wikipedia

Who’s saying what

Official
NVIDIA says two DGX Stations connected over ConnectX-8 RDMA serve the model through one OpenAI-compatible endpoint with tensor parallelism (TP=2), each GPU holding and computing roughly half the weights.
Caution
explainx.ai's Yash Thakker notes the API is unauthenticated unless you add a firewall, auth and TLS.

Sources

See today’s stories in the app Gistline — free on the App Store