Serving GLM-5.3 Across Two NVIDIA DGX Stations Using Tensor Parallelism

The NVIDIA GLM-5.3 NVFP4 model, a quantized mixture-of-experts (MoE) model with 753 billion parameters, requires approximately 464 GB of GPU memory—a footprint that exceeds the 252 GB HBM3e capacity of a single GB300 GPU. To overcome this hardware limitation, NVIDIA utilized two DGX Stations linked via ConnectX-8 RDMA to serve the model as a single, unified endpoint.

Distributed Weight Management

Because each token in a mixture-of-experts model can be routed to different experts, all model weights must remain accessible in GPU memory. The tutorial demonstrates a distributed service where two DGX Stations run the same software and checkpoint in different roles: Station A (rank 0) hosts the API, while Station B (rank 1) acts as a worker.

Chart drawn by HN Digest from figures reported by huggingface.co; values are shown as written in the source.

Tensor parallelism (TP=2) is the mechanism used to split the model's tensors across the two GB300 GPUs. Both stations compute every request, but only Station A returns the final answer through an OpenAI-compatible API.

This image illustrates the physical and logical setup of the two DGX Stations working together to host the GLM-5.3 model.

“Two NVIDIA DGX Stations connected over ConnectX-8 RDMA serve the model through one OpenAI-compatible endpoint using tensor parallelism (TP=2), with each GPU holding and computing roughly half the weights.”

— When One GPU Isn’t Enough: Serving GLM-5.3 Across Two NVIDIA DGX Stations with vLLM | NVIDIA Technical Blog

Networking and Inference Optimizations

The two stations are connected over ConnectX-8 RDMA, providing the high-speed interconnect necessary for tensor parallelism. This setup enables vLLM to launch a distributed service that supports multi-token prediction (MTP), where the model drafts four tokens per step to increase speed.

To further optimize performance under concurrent load, the configuration employs FP8 KV-cache storage to halve memory usage compared to 16-bit storage. It also utilizes prefix caching to avoid reprocessing prompt prefixes that the server has already encountered.

Performance Benchmarks and Trade-offs

AIPerf benchmarks on the SPEED-Bench Coding dataset showed that the system peaked at 607.8 tokens per second at a concurrency of 16. However, as concurrency increased to 32, the throughput fell to 385.0 tokens per second, with a mean request latency of 453.5 seconds.

The chart provides the specific data points for the performance benchmarks mentioned in the text.

While this architecture allows for the local execution of frontier-scale models, it involves significant hardware requirements and high power consumption. The setup is designed for high-throughput environments where privacy or specific local requirements outweigh the costs of maintaining a multi-station DGX cluster.