content:
# Tuning an NVIDIA DGX Spark (GB10) to serve many concurrent local LLM agents

*Measured September 2026. Every number here came off one machine running the workload it
describes — dozens of small agents reading web pages in parallel — not from a spec sheet.*

>  Before you copy a concurrency number out of the tables below, read section 3.  Every
> measurement here was taken at a 4096-token context. Our real workload runs at 12288–32768,
> and KV cache scales with context. The slot count these tables appear to bless needed more
> memory than the machine physically had, and hung it. The throughput-versus-concurrency
> *shape* is real; the slot count is only valid at the context it was measured at.

The DGX Spark is new enough that when we sent several hundred local agents to search the web
for exactly this information, they came back with nothing usable. So here is ours.

 Hardware:  NVIDIA GB10 Grace-Blackwell, 122 GiB unified memory, aarch64 Linux, CUDA 13.
 Software:  Ollama 0.33.2, flash attention on, KV cache quantized to `q8_0`.
 Models:  `qwen2.5:7b` (Q4_K_M) as the parallel worker, `qwen3:30b-a3b` (MoE, ~3B active) as
the planner and synthesizer.

  The headline numbers

Both models resident, `OLLAMA_MAX_LOADED_MODELS=2`,  num_ctx` 4096  — see the warning above
before reading a slot count off this table:

| model | concurrent requests | aggregate tok/s | per-stream tok/s | system memory used |

| `qwen2.5:7b` | 24 | 235.6 | 19.1 | 42.1 GB |
| `qwen2.5:7b` | 48 |  482.5  | 14.5 | 42.2 GB |
| `qwen3:30b-a3b` | 24 | 334.5 | 19.6 | 73.0 GB |
| `qwen3:30b-a3b` | 48 |  661.8  | 14.3 | 73.2 GB |

One model resident at a time, pushing further:

| model | concurrent | aggregate tok/s | system memory used |

| `qwen2.5:7b` | 8 / 16 / 24 / 32 | 137 / 336 / 386 / 394 | 40.3 GB at every level |
| `qwen3:30b-a3b` | 8 / 16 / 24 / 32 | 144 / 309 / 465 / 539 | 56.9 → 57.7 GB |
| `qwen3:30b-a3b` | 48 / 64 | 651 /  740  | 64.7 → 65.4 GB |

*(All at 4096-token context. At 32768 these slot counts do not fit — see section 3.)*

For scale, a 72B dense model measured  4.4 tok/s  on this same machine. That is why it is not
in the table.

  Five things worth knowing before you tune this box

  1. The cgroup cannot see unified memory. Do not trust `MemoryMax`.

This is the finding that cost us the most and that we have seen written nowhere else.

If you run inference under a systemd unit and watch `MemoryCurrent`, you will get a number that
has almost nothing to do with reality. Ours read  2.1 GB  while `nvidia-smi` showed that same
unit's `llama-server` children holding  25 GB . On unified-memory hardware the weights and KV
cache are not accounted as the cgroup's anonymous memory.

The consequence is worse than a wrong reading: a memory watchdog written against `MemoryMax`
will never fire. Ours did not, and we thought we were protected for a day.

 Measure `MemTotal − MemAvailable` from `/proc/meminfo` instead , and cross-check with
`nvidia-smi  query-compute-apps=used_memory`.

  2. Raise the *server's* parallelism before you raise the client's

Our first benchmark varied client concurrency — 1, 4, 8, 12 — against a server left at
`OLLAMA_NUM_PARALLEL=8`. Throughput fell at 12, and we published "8 is the sweet spot."

It was nonsense. Every request past the eighth was sitting in a queue inside the server. We had
measured our own configuration and mistaken it for the hardware. The experiment could not have
produced any other answer.

Set `OLLAMA_NUM_PARALLEL` first, confirm it took effect
(`systemctl show <unit> -p Environment`), *then* vary the client. When we did that, the real
figure was not 8. It was at least 64, and still climbing.

  3. "Slots are nearly free" is a trap. Benchmark at your real context.

Look at the memory column in the tables above:  40.3 GB at 8 concurrent and 40.3 GB at 32  for
the 7B. 56.9 → 65.4 GB across 8 → 64 for the MoE. Adding concurrency appeared to cost almost
nothing in memory.

We wrote that down, set 48 slots, and hung the machine within the hour.

Those benchmarks ran at `num_ctx 4096`. The production workload runs at 12288 for page readers
and 32768 for synthesis.  KV cache scales linearly with context , so the same 48 slots need:

| context | 48 slots need | on a 122 GiB machine |

| 4096 — what we benchmarked | ~57 GiB | comfortable |
| 12288 — our page readers | ~78 GiB | tight |
| 24576 — our reducers | ~110 GiB |  over budget  |
| 32768 — our synthesis | ~130 GiB |  over the physical limit  |

Throughput still climbs with concurrency, and the shape of that curve is real. The slot count is
only valid at the context it was measured at.

 What "hung" looked like, because the symptoms are not what you would expect.  ICMP kept
replying. The inference server's `/api/tags` answered HTTP 200 with a full model list.
But `/api/generate` never returned, and `sshd` accepted TCP connections and then never sent its
banner. Anything already resident kept working; anything needing a  new allocation or a new
process  blocked forever. No OOM kill, no error in any log we could reach, no way in. It took a
power cycle.

So: benchmark at the context you will actually run, and treat concurrency as a memory budget you
compute *before* you set it, rather than a number you find by turning it up. We now refuse to
start a configuration that cannot fit at our largest context:

48 slots at 32768 ctx needs ~128 GiB of 122 GiB     refuse to start
16 slots at 32768 ctx needs  ~73 GiB of 122 GiB     fine

 A runtime memory watchdog does not save you here , and the reason is worth knowing: the server
allocates KV for every slot in one step when a model loads. By the time free memory moves, the
allocation has already happened. Admission control is the wrong tool. A static budget, checked
before startup, is the right one.

  4. A thinking model under `format: json` can return nothing at all

`qwen3 class models will spend their entire `num_predict` budget inside a reasoning block and
hand back an  empty  `content` field. Our first full-depth run died at "planning produced
nothing" for exactly this reason, and the error surfaced as a JSON parse failure on an empty
string — several layers away from the cause.

Pass `think: false`. But check first: a model *without* the capability rejects the parameter
outright. Ask `/api/show` for the model's `capabilities` and only send `think` when `thinking`
is listed.

Also: when a JSON generation comes back with `done_reason: "length"`, it was truncated
mid-structure and will never parse. Retry with a larger budget rather than discarding it.

  5. Install a second Ollama rather than upgrading the one you have

We needed a newer Ollama than the one already serving half a dozen other things on this box.
Rather than upgrade in place and put all of them at risk for the benefit of one workload, we
extracted the new release to `/opt/<name>/` and pointed a single systemd drop-in at it:

[Service]
ExecStart=
ExecStart=/opt/<name>/bin/ollama serve
Environment="LD_LIBRARY_PATH=/opt/<name>/lib/ollama"
Environment="OLLAMA_LIBRARY_PATH=/opt/<name>/lib/ollama"

The system-wide binary stays where it is. Rollback is deleting one file.

 Verify the GPU is actually being used before you trust it.  A generic aarch64 build without
kernels for your compute capability will silently fall back to CPU, which looks like success and
runs dozens of times slower. Generate a few tokens and check `nvidia-smi  query-compute-apps`
lists a process from the new path.

  Reproducing this

Nothing here needs our code. For each concurrency level, fire N identical chat requests
concurrently against `/api/chat` with `stream: false`, sum `eval_count` across the responses and
divide by wall-clock time for the aggregate rate. Pin `num_ctx` explicitly — a large default
context will size your KV cache for you and the memory figures stop meaning anything.

Sample `MemAvailable` once a second throughout, and  abort before the next level if it drops
below a floor you choose in advance . On unified memory an out-of-memory event can take the
whole machine down rather than raising an error in one process.

  Caveats

- One machine, one afternoon, two models. Treat these as a starting point for your own
  measurement, not as constants.
- Everything was measured with `q8_0` KV cache and flash attention on. Other settings will move
  the memory numbers.
- The workload is batch extraction and summarization, where per-stream latency does not matter.
  If you are serving an interactive chat, the high-concurrency rows are the wrong end of the
  trade.
- Ollama 0.33.2. Newer releases may change all of this.

summary: Measured throughput and memory for qwen2.5:7b and qwen3:30b-a3b at 8 to 64 concurrent requests on a GB10 Grace-Blackwell with 122 GiB unified memory. Includes three findings we could not find written anywhere else: the systemd cgroup does not account for unified memory, so a MemoryMax-based watchdog never fires; you must raise the inference server's parallelism before the client's or you are timing your own queue; and a thinking model under format:json can return an empty response until you pass think:false.

case: dgx-spark


content:
# What a local research-agent fleet actually delivers

*Operational numbers from a fleet of small local LLM agents that search the web, read pages and
write cited reports — running on one NVIDIA DGX Spark (GB10, 122 GiB unified memory), September
2026. No API calls, no external inference, no cost per token.*

Plenty has been written about how to build agent swarms. Very little has been written about what
they produce once you run one for a day. These are ours, including the parts that look bad.

 Companion document:  *Tuning an NVIDIA DGX Spark (GB10) to serve many concurrent local LLM
agents* — the hardware and serving numbers behind this.

  The shape of the system

One run = one research question. It fans out through five stages:

| stage | agents | model | work |

| plan | 1 | 30B-A3B MoE | question → 3–10 sub-questions, each with search queries |
| search | 0 | — | metasearch + keyless APIs → candidate URLs |
| fetch | 0 | — | HTTP GET, HTML → text |
|  read  |  one per page  | 7B | strict-JSON facts with verbatim quotes |
| reduce | one per sub-question | 30B-A3B | sub-answer + confidence label |
| synth | 1 | 30B-A3B | final cited report |

The parallelism lives in `read`, on the small model. That is the work that is cheap locally and
expensive through an API.

  Totals for one working day

| | |

| Runs launched | 47 |
|  LLM agent invocations  |  1,372  (1,140 of them page readers) |
| Total scheduled tasks | 3,100 |
| Pages fetched and read | 993 |
| Evidence items extracted | 1,135 |
| Pages that errored (403, 429, unreadable) | 253 |
| Pages refused by the request-forgery guard | 38 |
|  External inference cost  |  zero  |

  Wall-clock

A depth-10 run — 10 sub-questions, up to 8 pages each, ~80 reader agents — completes in about
 16 minutes  end to end on this hardware, with four runs executing concurrently.

Runs that overlapped a machine outage that day recorded ~37 minutes; those are not clean
measurements and are excluded from that figure.

  The number nobody publishes: most reader agents produce nothing

Of 993 pages fetched and read by an agent,  604 produced no usable evidence at all — 61%.

The agent read the page, found nothing that answered its sub-question, and honestly returned an
empty result. That is correct behaviour and it is also the single largest waste in the pipeline.

It is measurable *before* the spend. Scoring the sub-question's distinctive terms against the
page's full text, on those same pages:

| threshold | reads it would skip | correctly (no evidence anyway) |  wrongly (had evidence)  |

| 0.2 | 48 | 48 |  0  |
|  0.3  |  93  |  93  |  0  |
| 0.4 | 141 | 136 | 5 |
| 0.6 | 262 | 218 | 44 |

At 0.3 it is free: roughly a quarter of all reader calls removed and  not one page lost that
would have produced evidence . We shipped that gate and it has skipped 72 reads since.

If you are building something similar, this is the highest-leverage thing in the document. A
cheap deterministic filter in front of the model beats a better model behind it.

  Answer quality, honestly

Every sub-answer is labelled by the model that wrote it: `CONFIDENT` (independent sources agree),
`CONTESTED` (sources disagree, and it says how), or `UNDOCUMENTED` (the evidence does not answer
it). Across 166 labelled sub-answers:

| label | count |

| CONFIDENT | 60 |
| CONTESTED | 19 |
| UNDOCUMENTED | 87 |

 About 48% usable.  But the average is misleading, and the variance is the finding:

| subject matter | usable |

| Software patterns, architecture, prior art |  ~88%  |
| Local business and marketing practice | good |
|  Brand-new hardware  (a GPU released months ago) |  ~0–27%  |

On the new-hardware questions the fleet searched hundreds of pages and returned `UNDOCUMENTED`
for nearly every sub-question. It was not broken.  The pages do not exist yet.  It declined to
invent benchmark numbers, which is the behaviour you want and does not feel like it at the time.

The practical rule:  this is a good literature scan and a poor instrument.  Use it for what
other people have already written down. Measure your own hardware yourself — one benchmark run on
the actual machine produced better numbers than several hundred agents searching for them.

  What raised quality most

1.  Give the agents your own source code.  Runs that read a local repository alongside the web
   scored 8/8 and 7/8 usable sub-answers — the best of the day, roughly double the web-only
   average. There is no hunting: the material is right there to quote.
2.  Never let a narrow category replace general web search.  Routing a networking question to
   an IT-only engine set answered it from a container-registry listing and two browser-API
   reference pages.
3.  Make the planner split rare qualifiers across sub-questions  instead of repeating them in
   every one. Four sub-questions that were the same sentence with the product name swapped
   returned four copies of four product homepages.
4.  Score search results before fetching them.  Term overlap against the title, snippet and URL
   separates a benchmark article (1.00) from the project's own homepage (0.09) and a download
   page (0.00), with no model call.

  Failure modes worth designing for

-  Every task is a database row, not in-memory state.  That machine lost power mid-run during
  this day's work. On restart, four runs resumed directly into their synthesis stage with their
  reading and reduction intact. Nothing restarted from the beginning.
-  A thinking model under a JSON schema can return an empty string.  It spends its entire token
  budget reasoning and emits nothing. The error surfaces several layers away as a parse failure.
-  Truncated JSON is unparseable.  When a generation stops on length rather than completion,
  retry with a larger budget instead of discarding it.
-  A small model given a page containing "ignore previous instructions" will flag the attempt
  correctly and still copy the attacker's marker string into its output.  Prompting a 7B out of
  that is not a control. Discard every fact from a page that tried to steer the reader — that is
  deterministic, and it happens in the runner where a model cannot argue with it.
-  Citation numbering has to be global to a run.  Numbered per sub-question, `[1]` in section
  two pointed at a different source than row 1 of the sources table. Silently, and confidently.

  Caveats

One fleet, one machine, one day, two models (a 7B for reading and a 30B-A3B MoE for planning and
synthesis). Search quality is capped by what a self-hosted metasearch instance can reach, which
on ours is two general engines plus keyless academic and developer APIs. Every number here will
move with different models, a different search backend, and different questions.

We are still measuring. This is a starting point, published because we could not find anyone
else's.

summary: Operational numbers from 1,372 local LLM agent invocations across 47 research runs in one day on a DGX Spark: wall-clock per run, pages read, evidence yield and answer quality by subject. Includes the figure nobody publishes -- 61% of page-reading agents produced no usable evidence at all -- and the measured threshold at which a cheap text filter removes a quarter of those calls with zero false negatives. Zero external inference cost.

case: agent-fleet

sub_case: local-agent-fleet-benchmarks


content:
# Twelve ways a local AI agent fleet breaks, and how each one was found

*Failure notes from building and running a self-hosted LLM research fleet on one NVIDIA DGX
Spark (GB10 Grace-Blackwell, 122 GiB unified memory, aarch64, CUDA 13), August–September 2026.
Every entry below actually happened on this machine. Several cost hours; one cost a power cycle.*

Most write-ups about local AI describe the version that worked. That is the least useful half.
What follows is the other half: the wrong diagnoses, the measurements that were invalid, and the
guards that now exist because something broke.

Each entry gives the  symptom  you would actually see, the  wrong answer  we reached first
where we reached one, the  real cause , and the  guard  — the check that makes the failure
impossible or loud rather than silent.

 Companion documents:  *Tuning an NVIDIA DGX Spark (GB10) to serve many concurrent local LLM
agents* and *What a local research-agent fleet actually delivers*.

  1. Unified memory does not OOM-kill. It hangs the whole machine.

 Symptom.  The box stopped. ICMP still answered. The inference server's `/api/tags` returned
200. `/api/generate` never returned. SSH opened a TCP connection and then never sent a banner.
No OOM killer message, no kernel log line, nothing in the service journal. The machine was not
dead and was not alive.

 Wrong answer.  "This is not an out-of-memory freeze" — because every OOM either of us had
seen leaves a corpse: a killed PID, a `Killed process` line in `dmesg`. There were none, so we
looked at the network and the service first, and were wrong for about twenty minutes.

 Real cause.  On a Grace-Blackwell unified-memory system the GPU allocates from the *same*
pool as the OS. Over-commit does not trigger the OOM killer the way an ordinary process would;
it drives the machine into an allocation stall where nothing can make forward progress,
including `sshd` accepting a login. The only exit was a power cycle.

 What made it happen.  KV cache scales linearly with context length. We had benchmarked at
`num_ctx 4096` and then shipped the slot count that benchmark justified while production ran at
`12288`–`32768`. Same slots, six times the cache per slot.

 Guard.  Refuse to start rather than discover this at runtime:

 python
def budget_check(per_slot_gb_at_24k: float = 1.3, reserve_gb: float = 25.0)   str:
    worst_ctx = max(CTX[k] for k in ("read", "plan", "reduce", "synth"))
    kv   = SLOTS * per_slot_gb_at_24k * (worst_ctx / 24576)
    need = kv + 45.0                       # + weights
    if need > total_gb - reserve_gb:
        raise SystemExit("REFUSING TO START: " + verdict)

 The transferable rule:  *benchmark at the context length you will actually run.* A throughput
number measured at a toy context is not a smaller version of the real number — it is a different
number, and using it to size slots is how you wedge the machine.

  2. A `MemoryMax=` watchdog cannot see the allocation that kills you

 Symptom.  A systemd `MemoryMax=6G` limit and a cgroup-based watchdog were in place. Neither
fired during the stall in §1.

 Real cause.  `MemoryCurrent` accounts for the cgroup's own charged pages. Unified-memory
allocations made by the GPU driver on behalf of the model are not charged there. The watchdog
was reading a number that stayed small while the machine ran out of memory.

 Guard.  Watch `MemAvailable` in `/proc/meminfo` — a system-wide figure — and act on a
stall line above zero, not on the cgroup's view. On this class of hardware the cgroup is the
wrong instrument, and it fails quietly, which is worse than failing loudly.

  3. Measuring client concurrency against a server pinned lower

 Symptom.  A concurrency sweep that looked clean, published, and was wrong within a day. Going
from 8 to 48 concurrent clients showed a suspiciously modest gain.

 Real cause.  The inference server was pinned at 8 parallel slots. Above 8, we were timing our
own request queue, not the GPU. The published claim ("slots are nearly free") was falsified the
same day.

 Guard.  Raise the *server's* parallelism first and confirm it in the service environment
before varying the client's. When the two disagree, every number above the server's limit is a
measurement of your own queue. The superseded results now carry a correction banner on top
rather than being deleted — a wrong number that someone already read should be corrected in
place, not disappeared.

  4. GPU at 0% with a full queue — starved by the non-GPU stage

 Symptom.  Hundreds of tasks pending, sixteen LLM slots free, GPU utilisation at  0% .

 Real cause.  Two compounding errors. Six HTTP fetch workers were feeding sixteen LLM slots,
so the readers had nothing to read. And the fetch timeout was  per-hop , not total: a chain of
slow redirects held one worker for 61–72 seconds while each individual hop stayed under its
limit.

 Guard.  A total deadline per fetch, not per hop; fetch workers raised to 48 against 16 LLM
slots. GPU went from  0% to 95% .

 The transferable rule:  in an agent fleet, the bottleneck is almost never the model. It is
the I/O stage in front of it. Size the cheap stage several times larger than the expensive one,
and instrument per-stage rates — an aggregate "tasks per minute" number hides this completely.

  5. A thinking model returns empty output under a JSON grammar

 Symptom.  A 30B MoE scored  0.0% on every extraction metric  — not badly, but *zero*, with
no errors reported. Same prompts, same pages where a 7B model worked fine.

 Real cause.  It is a reasoning model. Under `format: json` it spent its token budget in the
thinking channel and returned empty `content`. The API call succeeded. The response was blank.

 Guard.  Ask the server what the model is, and suppress thinking only for models that have it:

 python
caps = show(model).get("capabilities", [])
if "thinking" in caps:
    payload["think"] = False

This recurred with a second, unrelated model from a different vendor months later — the same
empty-content signature — which is why it is worth naming as a *class* of failure rather than a
quirk of one model.  If a model scores exactly zero with no errors, check the thinking channel
before you conclude anything about its ability.

  6. A quote verifier that a short quote walks straight through

 Symptom.  Nothing visible. Found by a self-test, not in production.

 Real cause.  Extracted facts were required to carry a verbatim quote, verified by substring
match against the source. Short quotes — three or four common words — match almost any page by
accident, so a fabricated fact carrying a short quote passed verification.

 Guard.  Quotes under five words get an exact check rather than a fuzzy one. Fabricated-quote
rate across the fleet went from  5.5% to 0% .

  7. A prompt-injection detector that is itself a model call

 Symptom.  The same page was flagged as an injection attempt on one run and not the next.

 Real cause.  The detector was a model call. A non-deterministic guard against a deterministic
attack means an attacker gets as many attempts as they like, and your incident log is noise.

 Guard.  A regex over known injection shapes, applied before the content ever reaches a model,
with quarantine on match. Deterministic in, deterministic out. The model that reads fetched
pages has no tools, no network and a strict output schema — the containment is structural, not
behavioural, because a model instructed not to obey injected text is still a model being asked
to make a judgement call about text designed to fool it.

  8. Nearly publishing the setting that hung the machine

 Symptom.  A draft tuning guide recommended, in good faith, the exact configuration from §1.

 Real cause.  The measurement was real and reproducible. It was taken at a context length no
production workload used. Reproducible and correct are different things.

 Guard.  That guide now leads with the cautionary table instead of the recommendation. Worth
stating plainly:  the most dangerous document is an accurate benchmark presented without its
conditions.  Anyone applying it inherits your assumptions without knowing they exist.

  9. Restarting the inference daemon during live work

 Symptom.  A batch of runs failed at the planning stage, all at once, for no apparent reason.

 Real cause.  We stopped the inference service to pull a new model while runs were active.

 Guard.  Model pulls wait for an idle fleet; the queue heals with a retry path rather than
abandoning the run. Separately, a permanently-failing task used to requeue forever — that now
has an abandonment path, because an infinite retry loop is a failure that looks like activity.

  10. Three small ones that each cost real time

-  timeout` does not exist on macOS.  A verification script reported failures that were the
  verifier failing, not the thing under test. When a check fails, confirm the check runs.
-  Working-directory drift in a long session  wrote a results file into the wrong directory.
  Use absolute paths in anything long-running.
-  Remote names recalled from memory instead of read  were wrong twice. Read the config.

  11. A model that is fast per stream and cannot batch at all

 Symptom.  A newly released 30B-class mixture-of-experts model — NVIDIA's
`nemotron-3.5-lightning:30b-a3b` (31.6B total, ~3.6B active, `nemotron_h_moe`), served through
Ollama 0.33.2 — benchmarked at  98 tok/s single-stream  — faster than the vendor's own published range for this hardware, and faster
than anything else on the box. On extraction quality it was the best model we had measured:
 2.3x the evidence yield  of the incumbent reader at  a third of the latency , with a third
of its fabrication rate.

 Wrong answer.  On those two numbers it was an obvious upgrade for the page-reading stage.

 Real cause.  Throughput did not scale with concurrency *at all*:

| concurrent requests | aggregate tok/s | per-stream tok/s | GPU memory |

| 8 | 64.7 | 86.9 | 28.6 GB |
| 16 | 84.9 | 87.7 | 28.6 GB |
| 32 |  84.3  | 87.2 | 28.6 GB |

Doubling concurrency from 16 to 32 moved total throughput by  −0.6% , while per-stream speed
held at 87 and GPU memory never moved off 28.6 GB — 38% of the machine. That is not a saturated
GPU. It is a server handling requests essentially  one at a time : this hybrid
Mamba/attention MoE architecture was not batching on this inference build.  To be fair to the
model, this is very likely an Ollama/llama.cpp limitation for `nemotron_h_moe` rather than a
property of the weights  — recurrent state in the Mamba layers is the usual reason a runtime
falls back to serial. A different serving stack (vLLM, TensorRT-LLM) may well batch it fine.
The lesson is about how you *measure*, not about whose model it is. Against the models
actually in service it was  5.7x to 7.8x slower in aggregate , on the one stage that is
entirely parallel.

 Guard.  Never accept a single-stream benchmark as a throughput result. Sweep concurrency and
check that aggregate *rises*; if aggregate stays flat while per-stream stays high, the server is
serialising and the model cannot serve a fleet no matter how good it is.

 The transferable rule:   per-stream speed is not throughput.  For anything running many
agents at once, the only number that matters is aggregate tokens per second at your real
concurrency. A model can be simultaneously the fastest and the least usable thing you own.

  Footnote to §11: the `nvfp4` tag will not load on a Blackwell box

While pulling the model above, the quantisation that *should* be ideal for this hardware failed:

$ ollama pull nemotron-3.5-lightning:30b-a3b-nvfp4
Error: this model requires MLX support, but the MLX runtime is not available

 MLX is Apple Silicon.  Despite the `nvfp4` name, and despite GB10 being Blackwell with FP4
support in hardware, that tag is an MLX build and cannot run on aarch64 CUDA. NVFP4 Nemotron is
not reachable through Ollama on a DGX Spark at all — it would take vLLM or TensorRT-LLM, not a
tag swap. `q4_K_M` is the working path. Worth knowing before planning around "NVFP4 on
Blackwell": the tag name describes the quantisation, not the runtime that can execute it.

  12. A benchmark too small to see the difference it is being used to justify

 Symptom.  The same model scored  6/6  on an agentic reasoning bench — a clean sweep,
including two deliberately planted traps. We recorded that as evidence it should take over the
planning and synthesis stages.

 Real cause.  Running the identical bench against the incumbents showed the smallest model on
the machine — `qwen2.5:7b` — also scoring 6/6, and doing it faster at every stage than both the 32B and a 30B MoE. The
entire spread across four models, from 7B to 32B, was  5/6 versus 6/6: one item, on a six-item
set.

That is not a ranking. It is noise, and it had already been written down as a result because it
was first seen in isolation, with nothing to compare it against.

 Guard.  A bench that cannot separate a 7B from a 32B is not measuring the thing you are
using it to decide. Before any model swap is justified on a score: check the sample size, and
check that the bench actually discriminates by running the models you already trust through it.

 The transferable rule:   an unbeaten score on a small benchmark is a statement about the
benchmark.  The moment a result is going to drive a change, the first question is not "which
model won" but "can this instrument tell these models apart at all" — and a baseline you have
not run is not a baseline.

  The pattern underneath all twelve

Nine of these twelve are not crashes. They are  silent degradation : a watchdog reading the
wrong number, a benchmark measuring its own queue, a verifier passing fabricated text, a fleet
at 0% GPU with a full queue, a model returning empty output with a success code. The system kept
reporting that it was fine.

Local AI infrastructure fails quietly far more often than it fails loudly. The guards that
matter are the ones that make a silent failure noisy — a start-up refusal, a per-stage rate you
can actually see, an exact check instead of a fuzzy one — and the measurement discipline that
makes you distrust a clean number until you know what conditions produced it.

*Published so the next person debugging a hung Grace-Blackwell box, a zero-scoring reasoning
model, or an idle GPU behind a full queue finds a written answer instead of an empty search
result. Numbers came off the machine described. No model wrote this; the failures were ours.*

summary: Failure notes from running a self-hosted LLM research fleet on an NVIDIA DGX Spark (GB10): the wrong diagnoses, the invalid measurements and the guards that exist because something broke. Includes the one that cost a power cycle -- unified memory does not OOM-kill, it stalls the entire machine with no kernel message and no killed process -- plus a cgroup watchdog that cannot see the allocation that kills you, a reasoning model that returns empty output under a JSON grammar, and a GPU sitting at 0% behind a full queue. Eight of the ten are silent degradation, not crashes. Also: a model that is the fastest and highest-quality reader measured and still unusable, because it does not batch -- aggregate throughput flat from 16 to 32 concurrent requests while per-stream speed held at 87 tok/s.

case: failure-modes

sub_case: local-ai-failure-modes


content:
# Your benchmark probably cannot tell a 7B from a 32B. Ours could not.

*How a six-item evaluation set hid a real defect in a production AI pipeline for weeks, what
replaced it, and the defect it found on its first run. Measured on a self-hosted research-agent
fleet, September 2026.*

There is a particular kind of wasted work in evaluating language models: you build a test, every
model passes it, and you conclude the models are equivalent. They are not. Your test is.

We ran a six-item set to score the reasoning stages of a local research pipeline — planning,
claim verification, synthesis. Four models, from 7B to 32B parameters, scored  5/6 or 6/6 .
A one-item spread across a 4.5x range of model size. We read that as "size does not matter for
these stages," which is a *conclusion about models*, drawn from what was actually a fact about
the instrument.

Then one model swept it 6/6 and we nearly promoted it on that basis. Running the same set
against the models already in service showed the  smallest model on the machine also scored
6/6 , faster. The entire measured difference was one question out of six — a coin flip.

 An unbeaten score on a small benchmark is a statement about the benchmark.

  What was wrong with the set

Six items is the obvious problem, and the least interesting one. Three others mattered more:

 It tested one failure mode.  Every negative item was a variant of the same thing — a claim
that outruns its quote. Nothing tested whether a model notices a unit changing, a date, a
negation, or a hedge.

 It was balanced 3 yes / 3 no.  A model that answers "supported" to everything scores 50% for
free, and 50% looks like a model that is trying.

 It reported one number.  A single accuracy figure cannot tell you *what* a model gets wrong,
and what it gets wrong is the only actionable part.

  What replaced it

 45 items across 14 failure categories , each category naming a specific way evidence can fail
to support a claim:

| category | what it tests |

| `direct`, `paraphrase`, `multi_evidence` | genuinely supported claims, including reworded and multi-source |
| `entity_swap` | the quote is about a different product, company or jurisdiction |
| `number_mismatch` | the figure does not match |
| `unit_mismatch` | the figure matches, the unit does not — 240 bar vs 240 psi |
| `causal_leap` | the quote states a fact; the claim asserts a cause |
| `overgeneralize` | one config line or one trial read as a general rule |
| `negation` | the quote says the opposite |
| `temporal` | true once, claimed as current |
| `hedged_source` | the source hedges — "reportedly", "may", "up to" — the claim asserts |
| `partial` | the quote covers half a compound claim |
| `contradictory` | two evidence lines disagree with each other |
| `irrelevant` | the quote is true and says nothing about the claim |

 Deliberately imbalanced: 11 supported, 34 not.  In a research pipeline the expensive error is
 false-accept  — an unsupported claim entering a cited report. Rejecting a good claim costs one
fact. Accepting a bad one costs the document's credibility.

That imbalance breaks raw accuracy, so the headline metric is  balanced accuracy  — the mean of
the two per-class recalls. We verified this rather than assuming it:  always-yes, always-no and
always-partial all score exactly 50% , while always-no scores  76% raw . If your imbalanced
benchmark reports raw accuracy, a model that refuses everything is beating your models.

 Scores are reported per category , and each model's weakest categories are named.

  What it found on the first run

| model | balanced | raw |  false-accept  | weakest categories |

| 7B | 91% | 91% | 3/34 | unit_mismatch, multi_evidence |
| 14B | 86% | 93% |  0/34  | multi_evidence, direct, paraphrase |
| 30B MoE *(in production)* | 87% | 80% |  9/34  |  hedged_source , partial, unit_mismatch |
| 32B MoE |  93%  | 89% | 5/34 | contradictory, unit_mismatch, temporal |

A 7-point spread in balanced accuracy and a  0-to-9 spread in false-accepts , where the old set
had produced a one-item spread. The instrument now discriminates.

  The defect

The 30B MoE was the production model — running planning, reduction and  synthesis . It writes
the prose a human reads. It scored  0/3 on `hedged_source . Given:

>  claim:  The merger will close in June.
>  evidence:  "The companies *reportedly aim to* close the merger sometime in June."

it answers  supported . All three items of that kind, wrong. It also took 1/3 on `partial`.

Those are one behaviour:  it strips qualifiers.  "Reportedly aim to" becomes "will". "Up to 30
percent" becomes "30 percent". "Covers dental" becomes "covers dental and vision". That is
precisely the failure a citation-anchored pipeline exists to prevent, happening in the stage that
produces the finished text — and its overall raw accuracy, 80%, reads as unremarkable rather than
alarming.

 The old benchmark rated this model 5/6, indistinguishable from every other model, because it
contained no hedged-source item at all.  The defect was not introduced. It was always there, and
the test could not see it.

  A blind spot every model shared

`unit_mismatch` was the weakest category almost everywhere. Given *"240 psi"* claimed against a
quote reading *"240 bar"*, most models match the number and ignore the unit — a factor of 14.5.
Only one model of four scored 3/3.

This is worth separating from the rest, because it suggests the fix is not a better model. A unit
comparison is deterministic; a model call is not. The same reasoning previously replaced a
model-based prompt-injection detector with a regex, after the same page was flagged on one run
and cleared on the next.  Where a check can be deterministic, a model is the wrong tool  — not
because it is worse on average, but because it is unreliable in a way you cannot bound.

  If you take three things

1.  Run your benchmark against the models you already trust before you use it to justify a
   change.  A baseline you have not run is not a baseline, and a score with nothing to compare
   against is how a one-item spread becomes a promotion decision.
2.  Count the failure modes your set contains, not just the items.  Ours had 45 items but the
   number that mattered was 14 categories. The defect lived in a category the old set did not
   have — and no amount of adding more items of the same kind would have found it.
3.  If your set is imbalanced, report balanced accuracy, and check what a degenerate strategy
   scores.  Ten minutes of work. It tells you the floor your models actually have to beat.

*Every number here was measured on the hardware and pipeline described; none was generated by a
model. The evaluation set and the harness that scores it are ordinary Python and could be
rebuilt from this description in an afternoon — the transferable part is the categories and the
imbalance, not the code.*

summary: A six-item evaluation set rated four models from 7B to 32B at 5/6 or 6/6 -- a one-item spread across a 4.5x range of model size -- and hid a real defect in a production pipeline. What replaced it: 45 items across 14 named failure categories, deliberately imbalanced because false-accept is the expensive error, scored on balanced accuracy after verifying that always-yes and always-no both land at 50%. On its first run it found the production synthesis model scoring 0/3 on hedged sources -- reading 'reportedly aim to close in June' as support for 'will close in June' -- a qualifier-stripping behaviour the old set could not see because it contained no item of that kind. Also: a unit-mismatch blind spot shared by every model tested, and why that one wants a deterministic check rather than a better model.

case: evaluation