Skip to content
Tech News
← Back to articles

NVIDIA demos GLM-5.3 served at 607.8 tok/s across two DGX Stations with vLLM

read original get NVIDIA DGX Station → more articles
GoKawiil Brief

NVIDIA published a tutorial showing how to serve Z.ai's GLM-5.3 NVFP4 model, a 753-billion-parameter mixture-of-experts model with 40 billion parameters activated per token, across two GB300 DGX Stations linked via ConnectX-8 RDMA. Because the roughly 464 GB quantized checkpoint exceeds the 252 GB of HBM3e on a single GB300 GPU, the setup uses tensor parallelism (TP=2) and vLLM to split weights and computation across both machines while exposing one OpenAI-compatible API endpoint, reportedly measuring throughput of 607.8 tokens per second.

Why It Matters

GoKawiil's interpretation of the reporting above, not reported fact.

The demonstration suggests that frontier-scale, sparsely-activated models can be run on multi-node workstation setups rather than requiring full datacenter clusters, which could broaden who can experiment with such large open models locally. It also illustrates a practical constraint of MoE architectures: despite sparse per-token activation, all expert weights must stay resident in memory, driving hardware and networking requirements like RDMA interconnects between machines.

Key Takeaways
Worth a Look

NVIDIA DGX Station — This article is literally about serving a massive model across two of these workstations, making the DGX Station the obvious centerpiece hardware for anyone exploring local frontier-scale inference. Its GPU memory capacity is exactly what enables keeping huge MoE model weights resident without offloading to CPU.

See NVIDIA DGX Station on Amazon → Affiliate link — we may earn a commission on purchases, at no extra cost to you. Product picked by AI based on this article; it is not a tested recommendation.

Source: developer.nvidia.com — Fatima Fall, 2026-11-14

Published there as: “Serving GLM-5.3 at 607.8 tok/s across two Nvidia DGX stations with vLLM”

Read the original report → The summary and analysis above are GoKawiil's own, written from reporting by the source above. Facts and quotes belong to the original publisher.