The Spark pair: serving one local model on two NVIDIA DGX Sparks

Published Oct 13, 2026 · Updated Oct 15, 2026

1. REQUEST Client selects the local route Local is one route, not every client request.
2. HEAD API + serving container Receives requests and coordinates model execution.
3. WORKER Tensor-parallel partner Works with the Head on the same model, not a second model.
4. MEASURE Separate the workloads Solo, concurrent, prefill and long-context decode are different measurements.

This is a dated build log, corrected October 15, 2026. Operational measurements are scoped to their cited audit dates; see the evidence note below for the private-source limit.

Evidence note — October 15, 2026: the operational counts and incident history below come from retained private receipts, not public downloads or a new live audit. Private receipt paths have been removed from the reader copy; the editorial record retains their mapping. Diagrams explain the design and do not prove runtime state.

Why local at all

The automation fleet here is real: seven Mac Hermes profiles, a mini-host running a nightly blog-factory, and assorted agents that all want a capable model at unpredictable hours. Cloud APIs priced per token don’t love that pattern, and rate limits don’t either. The alternative — a small pair of machines serving an open model 24/7 — looked feasible on paper. This post is the as-built version, receipts included.

The hardware

Two NVIDIA DGX Sparks (GB10, SM121) connected directly for tensor-parallel serving:

• Head: serves the API, holds the serving container.
• Worker: tensor-parallel partner.

The model is GLM-5.3-Flash, quantized for on-disk serving. The envelope is the interesting constraint: 121.7 GiB unified memory, of which 95.37 GiB is frozen PLE embeddings — tight by design, and it shapes every operational decision below.

Pinning the stack

Correction — October 15, 2026: this pair serves one local-model route, not every request from every client. Retained October 13–14 Hermes records also include cloud-backed routes. The recipe pins an image by hash; a locally patched running image can differ, as the bookkeeping gap below explains:

• v1.5 switch (Oct 3): recipe cf28cc4 → 1576746, image v0.6.0-9f73cca659a1 (sha256:ef83797d…), 70 patches, checkpoint 078455ff unchanged. Live identity verified post-switch: running image ID equaled the pinned digest, context 1048576, max streams 4, 3-stream READY smoke all exact in ~1.04 s.
• v1.6 switch (Oct 5): recipe HEAD 22f55f5, image v0.6.0-c4cab25d2d36 (repo digest sha256:b47c19d6…). The watcher ran canonical 12/12 hermetic tests before firing, and the switch receipt records SWITCH COMPLETE: v1.6 live and verified.
• Current state (Oct 13): v1.10 live (switched 2026-10-08) — recipe 549fdc5 (v1.10, 96 patches), published image v0.6.0-a1897d591f70 (digest sha256:bc34d7d6…), health endpoint reporting busy: false, 7,832 lifetime requests, 884.5M prompt tokens served, 2.5 TB disk free.

The discipline that matters: the watcher refuses to switch on a bad prepare. Our first v1.5 arm failed closed — a mis-backgrounded launcher produced no log, so no switch happened. That’s the system working, not failing.

The limits that matter

• Context 1,048,576 (1M). Long-agent workloads stop being a special case.
• Max streams: 4 (the recipe allows up to 8; we run 4 on the pair). Our own clean measurements (qual8b, Oct 7): solo 51.74 tok/s; 3-way concurrent 26–31 tok/s per stream (86.69 aggregate); 4-stream 94.91 aggregate; 63.2 tok/s decode at ~243K context; single-stream prefill ≈ 809 tok/s. Upstream’s own 8-stream figures: prose 130.8 / code 167.0 tok/s.
• max_tokens 16384 per response.
• Scoped egress observation. The cited audit recorded HF_HUB_OFFLINE=1 and no established non-private remotes at that moment. Offline download settings are not a firewall or proof that all future traffic is isolated.

GPU telemetry is not operating cost

Measured, not estimated — spark-doctor under load: GPU 96% util, 52–56 W, 2450–2496 MHz, 64–75 °C. Correction — October 15, 2026: those are GPU-reported readings, not combined wall power for both machines. They do not establish a gaming-PC comparison, electricity cost, cloud savings or break-even. A cost comparison would require wall-meter measurements, tariff, hardware accounting and actual invoices; none is established here.

What broke (the honest section)

1. The contaminated measurement. Our first 3-concurrent qualification was garbage — an external stream ran through it (lane_running_at_start=1), so per-stream medians read 18.66 tok/s. Not a regression; contamination. The fix wasn’t a re-run, it was per-run idle gates in every qualifier since. Measure on a known-idle lane or don’t publish the number.
2. Head memory is the watched resource. 95 GiB of weights + a growing KV pool in 121.7 GiB leaves the host tight: head slipped 1.9 → 1.5 GB MemAvailable, PSI still 0.00 (no actual stalls). The steward cron now alarms at AMBER <1000 MB / RED <500 MB, and MEMORY_RESERVE_GIB is the knob for the next switch.
3. The worker did die once — on purpose. A live-fire drill injected docker stop on the rank-1 worker; the watchdog (3-min cadence, ssh health check, ./start.sh restart) was expected to heal within ~8 minutes. The container restart policy is deliberately no — Docker can’t “self-heal” into a false pass.

Since then: the memory fix (the honest sequel)

The dated October 9–11 repair history below predates this article’s October 13 publication. It is not new work performed after publication.

v1.10 shipped an opt-in kept-state share (TF_GLM_CACHE_SHARE_PCT=50): reuse prompt caches across chats to save prefill. We enabled it. Kept states grew to fill the entire allowance — 70 states / 6.5 GB / 1.92M tokens — pinning 94% of the shrunken pool — and live prefills starved: 3 of 4 streams stuck in prefill with zero progress, ~10 tok/s aggregate vs ~110 at qualification, six requests queued.

The fix wasn’t a tweak; it was a revert with a receipt (2026-10-09 13:23Z: pool back to 2,582,528 tokens, kept prompts 0, solo smoke 84.8 tok/s), followed by the lasting fix — hard caps on kept state (TF_GLM_KEEP_PER_CHAT=2, TF_GLM_KEPT_BYTES_GIB=4, the patch author’s own values), applied and verified: kept_bytes_cap: 4294967296, ok: true. A second opt-in (64 GiB spill to local disk) went live Oct 11.

Post-fix measurements on v1.10: solo 51.89 tok/s, 3-way concurrent 26.24 tok/s per stream, and 71.75 tok/s decode at a 243K-token prompt (TTFT 0.28 s).

One loose end, stated plainly: the running container’s image ID doesn’t match the published v1.10 digest — it’s a locally patched build from the quiet-gated restarts above, and no receipt yet ties that exact image hash to a named patch set. That remains an evidence gap in this article; it is not a verified current-image receipt.

The ops stack around it

Three roles, deliberately split: the rank-1 watchdog heals, the weekly steward trends (GREEN/AMBER/RED), and spark-doctor diagnoses (v0.3.1, read-only, 456 tests, GB10-specific rules). A read-only lane dashboard serves fresh metrics per request on the LAN. Rollback is policy, not hope: the v1.4 image stayed on disk through the v1.5 switch; the v1.5 image and checkpoint stayed through v1.6; ~345 GB of superseded TR3 checkpoint got pruned only after a >24h clean gate.

Verdict

The pair is a useful local route for our long-context serving work, with real memory-pressure failures and repairs rather than zero-maintenance operation. The dated benchmarks above measure serving speed under their own workloads, not model quality or passive fleet throughput. No current all-client coverage, power advantage or financial break-even is established here.

Next in the series: the retirement checklist — how the superseded Qwen lane came out of this fleet with explicit removal checks and a remaining host-verification gap. Published alongside this update.

Public references

NVIDIA DGX Spark user guide describes the platform. It does not independently verify our local model measurements.