NVIDIA DGX Spark Cluster: Benchmarking Distributed Inference

What is the practical utility of a clustered DGX Spark?

The NVIDIA DGX Spark has emerged as a distinctive piece of hardware in the current artificial intelligence landscape. It combines 128 GB of unified memory within a desktop form factor priced at approximately $4,000, a specification that challenges traditional boundaries between workstation and datacenter hardware. Central to its architecture is the inclusion of a 200 Gb network interface, which allows for direct clustering capabilities previously reserved for rack-mounted server infrastructure. This review examines the performance of two-node clusters formed by three major OEMs: Dell, GIGABYTE, and HP.

The primary motivation for clustering these units is the physical limitation of memory capacity. A single DGX Spark cannot house models exceeding its 128 GB limit, such as the 120 billion parameter GPT-OSS-120B. By connecting two units, engineers can stretch these large models across both boxes, enabling workloads that were previously inaccessible on desktop hardware. However, the platform is also heavily positioned as an educational tool, allowing individuals to learn distributed computing concepts without the capital expenditure of enterprise datacenters.

The networking implementation relies on two QSFP56 cages driven by an integrated NVIDIA ConnectX-7 SmartNIC. Although the physical ports suggest higher aggregate bandwidth, the PCIe Gen5 x4 link limits usable bandwidth to 200 Gb. This constraint is critical because it dictates how efficiently data moves between nodes. The cluster supports direct linking, ring-like topologies, or split-role configurations with high-speed storage, providing flexibility for various deployment scenarios.

While NVIDIA has demonstrated four-unit configurations, the two-node setup remains the most practical for most users. The platform is not designed for high-throughput production serving, but rather for exploration and learning. Understanding the limitations of the 200 Gb fabric is essential for interpreting performance benchmarks, as the network becomes a significant bottleneck when strategies like tensor parallelism are employed inefficiently.

Why does the parallelism strategy matter more than the OEM?

A common assumption in hardware benchmarking is that the manufacturer's implementation dictates performance. However, this analysis demonstrates that the choice of parallelism strategy is the dominant factor in cluster efficiency. NVIDIA’s default guidance favors tensor parallelism (TP), which splits matrix multiplications across both GPUs. This method requires an all-reduce operation after every attention and MLP block. On a 200 GbE link, this creates excessive cross-box traffic that leaves compute units idle, particularly as batch sizes increase.

Pipeline parallelism (PP) offers a different approach by cutting the model in half by layer and streaming activations between the two boxes. While this introduces a pipeline bubble cost, it drastically reduces the volume of data transmitted over the network. For batched inference, where many requests are processed simultaneously, the PP=2 configuration amortizes this bubble cost effectively, leading to higher throughput.

The performance gap is most pronounced with large models like GPT-OSS-120B. In Equal ISL/OSL workloads with a batch size of 128, pipeline parallelism achieved 554.69 tokens per second, compared to 252.01 tokens per second for tensor parallelism. This 2.20x advantage highlights the inefficiency of TP on this specific network fabric. The advantage persists in Prefill Heavy workloads, where PP=2 reached 310.63 tok/s against TP=2’s 164.99 tok/s.

Tensor parallelism does retain a narrow advantage in single-stream interactive serving. At batch size 1, TP=2 delivers lower latency for the initial token, making it suitable for chat-style applications where time-to-first-token is critical. However, for infrastructure-scale serving with concurrent requests, pipeline parallelism is the superior choice. This finding aligns with the physics of the hardware: the network cannot sustain the per-token traffic of TP without significant performance penalties.

Smaller models, such as Llama-3.1-8B-Instruct, tell a different story. Because the computation per layer is faster, the all-reduce traffic in TP is less dominant. Consequently, TP=2 leads across nearly the entire batch sweep for this model. This counterexample reinforces that parallelism strategy must be tuned to the model size and workload type, rather than applied universally.

How do Dell, GIGABYTE, and HP hardware implementations compare?

The benchmarking process involved testing three specific OEM implementations of the DGX Spark: the Dell Pro Max with GB 10, the GIGABYTE AI TOP ATOM, and the HP ZGX Nano G1n. The goal was to determine if minor hardware variations resulted in significant performance deltas when clustered. The results indicate that the three systems perform within a very narrow band across all tested models and workload shapes.

In the GPT-OSS-120B Equal ISL/OSL workload, HP led the group at the highest batch sizes with 1,009.75 tok/s, followed closely by GIGABYTE at 994.53 tok/s and Dell at 927.93 tok/s. The spread remains tight, suggesting that the networking fabric and GPU architecture are the limiting factors, not the chassis design. In Prefill Heavy scenarios, HP again posted the strongest result at 2,208.16 tok/s, while Dell and GIGABYTE remained tightly grouped.

For the GPT-OSS-20B model, Dell demonstrated stronger scaling in the Equal ISL/OSL workload, reaching 1,953.55 tok/s at batch size 64, compared to GIGABYTE’s 1,904.62 tok/s and HP’s 1,831.45 tok/s. Dell also led in Prefill Heavy throughput, scaling to 4,261.96 tok/s. However, these differences are marginal and often fall within the run-to-run variance expected of desktop-class systems under sustained load.

The Llama-3.1-8B-Instruct tests further illustrate the consistency across OEMs. In Equal ISL/OSL, Dell scaled to 1,376.38 tok/s, GIGABYTE to 1,372.27 tok/s, and HP to 1,235.32 tok/s. In Prefill Heavy, GIGABYTE took a slight lead with 2,694.25 tok/s, while Dell followed closely at 2,575.25 tok/s. The performance hierarchy shifts slightly depending on the specific batch size and model, with no single OEM maintaining a consistent advantage across all metrics.

The Mistral-Small-3.1-24B and Qwen3-coder-30B-A3B models showed similar trends. GIGABYTE and Dell traded leads in various scenarios, with HP generally trailing slightly in high-concurrency decode workloads. The data suggests that buyers should base their decision on chassis design, thermal behavior, warranty terms, and support relationships rather than expecting significant performance gains from one OEM over another.

What are the implications for future AI infrastructure?

The findings from this cluster review have broader implications for how engineers approach distributed inference on desktop hardware. The DGX Spark serves as a critical bridge between individual exploration and large-scale datacenter operations. It allows users to develop an intuition for distributed computing concepts, such as tensor parallelism and pipeline parallelism, in a controlled environment.

The performance limitations observed, particularly the degradation of tensor parallelism on the 200 GbE fabric, highlight the importance of software optimization alongside hardware selection. As models grow larger, the bottleneck shifts from compute to communication. This cluster configuration exposes these bottlenecks clearly, providing valuable insights for developers who will eventually work with larger-scale systems.

While the two-node cluster is a powerful learning tool, it is not a replacement for production inference servers. The inter-node fabric becomes the dominant cost, and collective performance degrades sharply as more nodes are added. This reinforces NVIDIA’s positioning of the Spark as an entry point for learning rather than a high-throughput serving platform.

Future work will explore training sub-1B parameter models from scratch on dual-Spark clusters. This will provide further insight into the limits of this hardware for pre-training tasks. The ongoing development of the 800 Gb lab core switch will also play a role in these future experiments, potentially altering the performance dynamics observed in this review.

Ultimately, the DGX Spark cluster represents a significant step toward democratizing access to distributed AI infrastructure. By providing a realistic simulation of datacenter networking constraints in a desktop form factor, it enables a deeper understanding of the challenges and opportunities in modern AI deployment.