NVIDIA demos GLM-5.3 served at 607.8 tok/s across two DGX Stations with vLLM
NVIDIA published a tutorial showing how to serve Z.ai's GLM-5.3 NVFP4 model, a 753-billion-parameter mixture-of-experts model with 40 billion parameters activated per token, across two GB300 DGX Stations linked via ConnectX-8 RDMA. Because the roughly 464 GB quantized checkpoint exceeds the 252 GB of HBM3e on a single GB300 GPU, the setup uses tensor parallelism (TP=2) and vLLM to split weights and computation across both machines while exposing one OpenAI-compatible API endpoint, reportedly measuring throughput of 607.8 tokens per second.
GoKawiil's interpretation of the reporting above, not reported fact.
The demonstration suggests that frontier-scale, sparsely-activated models can be run on multi-node workstation setups rather than requiring full datacenter clusters, which could broaden who can experiment with such large open models locally. It also illustrates a practical constraint of MoE architectures: despite sparse per-token activation, all expert weights must stay resident in memory, driving hardware and networking requirements like RDMA interconnects between machines.
- GLM-5.3 NVFP4 has 753B total parameters with only 40B activated per token, yet requires all weights in memory due to expert routing.
- The quantized checkpoint (~464 GB) exceeds a single GB300 GPU's 252 GB HBM3e, necessitating a two-node setup.
- NVIDIA reports serving the model across two DGX Stations via vLLM and tensor parallelism at 607.8 tokens per second through one unified API endpoint.
NVIDIA DGX Station — This article is literally about serving a massive model across two of these workstations, making the DGX Station the obvious centerpiece hardware for anyone exploring local frontier-scale inference. Its GPU memory capacity is exactly what enables keeping huge MoE model weights resident without offloading to CPU.
See NVIDIA DGX Station on Amazon → Affiliate link — we may earn a commission on purchases, at no extra cost to you. Product picked by AI based on this article; it is not a tested recommendation.Source: developer.nvidia.com — Fatima Fall, 2026-11-14
Published there as: “Serving GLM-5.3 at 607.8 tok/s across two Nvidia DGX stations with vLLM”
Read the original report → The summary and analysis above are GoKawiil's own, written from reporting by the source above. Facts and quotes belong to the original publisher.