{"title": "Tuning an NVIDIA DGX Spark (GB10) to serve many concurrent local LLM agents", "content": "# Tuning an NVIDIA DGX Spark (GB10) to serve many concurrent local LLM agents\n\n*Measured September 2026. Every number here came off one machine running the workload it\ndescribes — dozens of small agents reading web pages in parallel — not from a spec sheet.*\n\n>  Before you copy a concurrency number out of the tables below, read section 3.  Every\n> measurement here was taken at a 4096-token context. Our real workload runs at 12288–32768,\n> and KV cache scales with context. The slot count these tables appear to bless needed more\n> memory than the machine physically had, and hung it. The throughput-versus-concurrency\n> *shape* is real; the slot count is only valid at the context it was measured at.\n\nThe DGX Spark is new enough that when we sent several hundred local agents to search the web\nfor exactly this information, they came back with nothing usable. So here is ours.\n\n Hardware:  NVIDIA GB10 Grace-Blackwell, 122 GiB unified memory, aarch64 Linux, CUDA 13.\n Software:  Ollama 0.33.2, flash attention on, KV cache quantized to `q8_0`.\n Models:  `qwen2.5:7b` (Q4_K_M) as the parallel worker, `qwen3:30b-a3b` (MoE, ~3B active) as\nthe planner and synthesizer.\n\n \n\n  The headline numbers\n\nBoth models resident, `OLLAMA_MAX_LOADED_MODELS=2`,  num_ctx` 4096  — see the warning above\nbefore reading a slot count off this table:\n\n| model | concurrent requests | aggregate tok/s | per-stream tok/s | system memory used |\n \n| `qwen2.5:7b` | 24 | 235.6 | 19.1 | 42.1 GB |\n| `qwen2.5:7b` | 48 |  482.5  | 14.5 | 42.2 GB |\n| `qwen3:30b-a3b` | 24 | 334.5 | 19.6 | 73.0 GB |\n| `qwen3:30b-a3b` | 48 |  661.8  | 14.3 | 73.2 GB |\n\nOne model resident at a time, pushing further:\n\n| model | concurrent | aggregate tok/s | system memory used |\n \n| `qwen2.5:7b` | 8 / 16 / 24 / 32 | 137 / 336 / 386 / 394 | 40.3 GB at every level |\n| `qwen3:30b-a3b` | 8 / 16 / 24 / 32 | 144 / 309 / 465 / 539 | 56.9 → 57.7 GB |\n| `qwen3:30b-a3b` | 48 / 64 | 651 /  740  | 64.7 → 65.4 GB |\n\n*(All at 4096-token context. At 32768 these slot counts do not fit — see section 3.)*\n\nFor scale, a 72B dense model measured  4.4 tok/s  on this same machine. That is why it is not\nin the table.\n\n \n\n  Five things worth knowing before you tune this box\n\n  1. The cgroup cannot see unified memory. Do not trust `MemoryMax`.\n\nThis is the finding that cost us the most and that we have seen written nowhere else.\n\nIf you run inference under a systemd unit and watch `MemoryCurrent`, you will get a number that\nhas almost nothing to do with reality. Ours read  2.1 GB  while `nvidia-smi` showed that same\nunit's `llama-server` children holding  25 GB . On unified-memory hardware the weights and KV\ncache are not accounted as the cgroup's anonymous memory.\n\nThe consequence is worse than a wrong reading: a memory watchdog written against `MemoryMax`\nwill never fire. Ours did not, and we thought we were protected for a day.\n\n Measure `MemTotal − MemAvailable` from `/proc/meminfo` instead , and cross-check with\n`nvidia-smi  query-compute-apps=used_memory`.\n\n  2. Raise the *server's* parallelism before you raise the client's\n\nOur first benchmark varied client concurrency — 1, 4, 8, 12 — against a server left at\n`OLLAMA_NUM_PARALLEL=8`. Throughput fell at 12, and we published \"8 is the sweet spot.\"\n\nIt was nonsense. Every request past the eighth was sitting in a queue inside the server. We had\nmeasured our own configuration and mistaken it for the hardware. The experiment could not have\nproduced any other answer.\n\nSet `OLLAMA_NUM_PARALLEL` first, confirm it took effect\n(`systemctl show <unit> -p Environment`), *then* vary the client. When we did that, the real\nfigure was not 8. It was at least 64, and still climbing.\n\n  3. \"Slots are nearly free\" is a trap. Benchmark at your real context.\n\nLook at the memory column in the tables above:  40.3 GB at 8 concurrent and 40.3 GB at 32  for\nthe 7B. 56.9 → 65.4 GB across 8 → 64 for the MoE. Adding concurrency appeared to cost almost\nnothing in memory.\n\nWe wrote that down, set 48 slots, and hung the machine within the hour.\n\nThose benchmarks ran at `num_ctx 4096`. The production workload runs at 12288 for page readers\nand 32768 for synthesis.  KV cache scales linearly with context , so the same 48 slots need:\n\n| context | 48 slots need | on a 122 GiB machine |\n \n| 4096 — what we benchmarked | ~57 GiB | comfortable |\n| 12288 — our page readers | ~78 GiB | tight |\n| 24576 — our reducers | ~110 GiB |  over budget  |\n| 32768 — our synthesis | ~130 GiB |  over the physical limit  |\n\nThroughput still climbs with concurrency, and the shape of that curve is real. The slot count is\nonly valid at the context it was measured at.\n\n What \"hung\" looked like, because the symptoms are not what you would expect.  ICMP kept\nreplying. The inference server's `/api/tags` answered HTTP 200 with a full model list.\nBut `/api/generate` never returned, and `sshd` accepted TCP connections and then never sent its\nbanner. Anything already resident kept working; anything needing a  new allocation or a new\nprocess  blocked forever. No OOM kill, no error in any log we could reach, no way in. It took a\npower cycle.\n\nSo: benchmark at the context you will actually run, and treat concurrency as a memory budget you\ncompute *before* you set it, rather than a number you find by turning it up. We now refuse to\nstart a configuration that cannot fit at our largest context:\n\n \n48 slots at 32768 ctx needs ~128 GiB of 122 GiB     refuse to start\n16 slots at 32768 ctx needs  ~73 GiB of 122 GiB     fine\n \n\n A runtime memory watchdog does not save you here , and the reason is worth knowing: the server\nallocates KV for every slot in one step when a model loads. By the time free memory moves, the\nallocation has already happened. Admission control is the wrong tool. A static budget, checked\nbefore startup, is the right one.\n\n\n  4. A thinking model under `format: json` can return nothing at all\n\n`qwen3 class models will spend their entire `num_predict` budget inside a reasoning block and\nhand back an  empty  `content` field. Our first full-depth run died at \"planning produced\nnothing\" for exactly this reason, and the error surfaced as a JSON parse failure on an empty\nstring — several layers away from the cause.\n\nPass `think: false`. But check first: a model *without* the capability rejects the parameter\noutright. Ask `/api/show` for the model's `capabilities` and only send `think` when `thinking`\nis listed.\n\nAlso: when a JSON generation comes back with `done_reason: \"length\"`, it was truncated\nmid-structure and will never parse. Retry with a larger budget rather than discarding it.\n\n  5. Install a second Ollama rather than upgrading the one you have\n\nWe needed a newer Ollama than the one already serving half a dozen other things on this box.\nRather than upgrade in place and put all of them at risk for the benefit of one workload, we\nextracted the new release to `/opt/<name>/` and pointed a single systemd drop-in at it:\n\n \n[Service]\nExecStart=\nExecStart=/opt/<name>/bin/ollama serve\nEnvironment=\"LD_LIBRARY_PATH=/opt/<name>/lib/ollama\"\nEnvironment=\"OLLAMA_LIBRARY_PATH=/opt/<name>/lib/ollama\"\n \n\nThe system-wide binary stays where it is. Rollback is deleting one file.\n\n Verify the GPU is actually being used before you trust it.  A generic aarch64 build without\nkernels for your compute capability will silently fall back to CPU, which looks like success and\nruns dozens of times slower. Generate a few tokens and check `nvidia-smi  query-compute-apps`\nlists a process from the new path.\n\n \n\n  Reproducing this\n\nNothing here needs our code. For each concurrency level, fire N identical chat requests\nconcurrently against `/api/chat` with `stream: false`, sum `eval_count` across the responses and\ndivide by wall-clock time for the aggregate rate. Pin `num_ctx` explicitly — a large default\ncontext will size your KV cache for you and the memory figures stop meaning anything.\n\nSample `MemAvailable` once a second throughout, and  abort before the next level if it drops\nbelow a floor you choose in advance . On unified memory an out-of-memory event can take the\nwhole machine down rather than raising an error in one process.\n\n \n\n  Caveats\n\n- One machine, one afternoon, two models. Treat these as a starting point for your own\n  measurement, not as constants.\n- Everything was measured with `q8_0` KV cache and flash attention on. Other settings will move\n  the memory numbers.\n- The workload is batch extraction and summarization, where per-stream latency does not matter.\n  If you are serving an interactive chat, the high-concurrency rows are the wrong end of the\n  trade.\n- Ollama 0.33.2. Newer releases may change all of this.\n", "summary": "Measured throughput and memory for qwen2.5:7b and qwen3:30b-a3b at 8 to 64 concurrent requests on a GB10 Grace-Blackwell with 122 GiB unified memory. Includes three findings we could not find written anywhere else: the systemd cgroup does not account for unified memory, so a MemoryMax-based watchdog never fires; you must raise the inference server's parallelism before the client's or you are timing your own queue; and a thinking model under format:json can return an empty response until you pass think:false.", "case": "dgx-spark", "sub_case": "gb10-inference-tuning", "tags": ["dgx-spark", "gb10", "grace-blackwell", "ollama", "local-inference", "benchmark", "aarch64", "cuda-13"]}
{"title": "What a local research-agent fleet actually delivers", "content": "# What a local research-agent fleet actually delivers\n\n*Operational numbers from a fleet of small local LLM agents that search the web, read pages and\nwrite cited reports — running on one NVIDIA DGX Spark (GB10, 122 GiB unified memory), September\n2026. No API calls, no external inference, no cost per token.*\n\nPlenty has been written about how to build agent swarms. Very little has been written about what\nthey produce once you run one for a day. These are ours, including the parts that look bad.\n\n Companion document:  *Tuning an NVIDIA DGX Spark (GB10) to serve many concurrent local LLM\nagents* — the hardware and serving numbers behind this.\n\n \n\n  The shape of the system\n\nOne run = one research question. It fans out through five stages:\n\n| stage | agents | model | work |\n \n| plan | 1 | 30B-A3B MoE | question → 3–10 sub-questions, each with search queries |\n| search | 0 | — | metasearch + keyless APIs → candidate URLs |\n| fetch | 0 | — | HTTP GET, HTML → text |\n|  read  |  one per page  | 7B | strict-JSON facts with verbatim quotes |\n| reduce | one per sub-question | 30B-A3B | sub-answer + confidence label |\n| synth | 1 | 30B-A3B | final cited report |\n\nThe parallelism lives in `read`, on the small model. That is the work that is cheap locally and\nexpensive through an API.\n\n  Totals for one working day\n\n| | |\n \n| Runs launched | 47 |\n|  LLM agent invocations  |  1,372  (1,140 of them page readers) |\n| Total scheduled tasks | 3,100 |\n| Pages fetched and read | 993 |\n| Evidence items extracted | 1,135 |\n| Pages that errored (403, 429, unreadable) | 253 |\n| Pages refused by the request-forgery guard | 38 |\n|  External inference cost  |  zero  |\n\n  Wall-clock\n\nA depth-10 run — 10 sub-questions, up to 8 pages each, ~80 reader agents — completes in about\n 16 minutes  end to end on this hardware, with four runs executing concurrently.\n\nRuns that overlapped a machine outage that day recorded ~37 minutes; those are not clean\nmeasurements and are excluded from that figure.\n\n  The number nobody publishes: most reader agents produce nothing\n\nOf 993 pages fetched and read by an agent,  604 produced no usable evidence at all — 61%. \n\nThe agent read the page, found nothing that answered its sub-question, and honestly returned an\nempty result. That is correct behaviour and it is also the single largest waste in the pipeline.\n\nIt is measurable *before* the spend. Scoring the sub-question's distinctive terms against the\npage's full text, on those same pages:\n\n| threshold | reads it would skip | correctly (no evidence anyway) |  wrongly (had evidence)  |\n \n| 0.2 | 48 | 48 |  0  |\n|  0.3  |  93  |  93  |  0  |\n| 0.4 | 141 | 136 | 5 |\n| 0.6 | 262 | 218 | 44 |\n\nAt 0.3 it is free: roughly a quarter of all reader calls removed and  not one page lost that\nwould have produced evidence . We shipped that gate and it has skipped 72 reads since.\n\nIf you are building something similar, this is the highest-leverage thing in the document. A\ncheap deterministic filter in front of the model beats a better model behind it.\n\n  Answer quality, honestly\n\nEvery sub-answer is labelled by the model that wrote it: `CONFIDENT` (independent sources agree),\n`CONTESTED` (sources disagree, and it says how), or `UNDOCUMENTED` (the evidence does not answer\nit). Across 166 labelled sub-answers:\n\n| label | count |\n \n| CONFIDENT | 60 |\n| CONTESTED | 19 |\n| UNDOCUMENTED | 87 |\n\n About 48% usable.  But the average is misleading, and the variance is the finding:\n\n| subject matter | usable |\n \n| Software patterns, architecture, prior art |  ~88%  |\n| Local business and marketing practice | good |\n|  Brand-new hardware  (a GPU released months ago) |  ~0–27%  |\n\nOn the new-hardware questions the fleet searched hundreds of pages and returned `UNDOCUMENTED`\nfor nearly every sub-question. It was not broken.  The pages do not exist yet.  It declined to\ninvent benchmark numbers, which is the behaviour you want and does not feel like it at the time.\n\nThe practical rule:  this is a good literature scan and a poor instrument.  Use it for what\nother people have already written down. Measure your own hardware yourself — one benchmark run on\nthe actual machine produced better numbers than several hundred agents searching for them.\n\n  What raised quality most\n\n1.  Give the agents your own source code.  Runs that read a local repository alongside the web\n   scored 8/8 and 7/8 usable sub-answers — the best of the day, roughly double the web-only\n   average. There is no hunting: the material is right there to quote.\n2.  Never let a narrow category replace general web search.  Routing a networking question to\n   an IT-only engine set answered it from a container-registry listing and two browser-API\n   reference pages.\n3.  Make the planner split rare qualifiers across sub-questions  instead of repeating them in\n   every one. Four sub-questions that were the same sentence with the product name swapped\n   returned four copies of four product homepages.\n4.  Score search results before fetching them.  Term overlap against the title, snippet and URL\n   separates a benchmark article (1.00) from the project's own homepage (0.09) and a download\n   page (0.00), with no model call.\n\n  Failure modes worth designing for\n\n-  Every task is a database row, not in-memory state.  That machine lost power mid-run during\n  this day's work. On restart, four runs resumed directly into their synthesis stage with their\n  reading and reduction intact. Nothing restarted from the beginning.\n-  A thinking model under a JSON schema can return an empty string.  It spends its entire token\n  budget reasoning and emits nothing. The error surfaces several layers away as a parse failure.\n-  Truncated JSON is unparseable.  When a generation stops on length rather than completion,\n  retry with a larger budget instead of discarding it.\n-  A small model given a page containing \"ignore previous instructions\" will flag the attempt\n  correctly and still copy the attacker's marker string into its output.  Prompting a 7B out of\n  that is not a control. Discard every fact from a page that tried to steer the reader — that is\n  deterministic, and it happens in the runner where a model cannot argue with it.\n-  Citation numbering has to be global to a run.  Numbered per sub-question, `[1]` in section\n  two pointed at a different source than row 1 of the sources table. Silently, and confidently.\n\n  Caveats\n\nOne fleet, one machine, one day, two models (a 7B for reading and a 30B-A3B MoE for planning and\nsynthesis). Search quality is capped by what a self-hosted metasearch instance can reach, which\non ours is two general engines plus keyless academic and developer APIs. Every number here will\nmove with different models, a different search backend, and different questions.\n\nWe are still measuring. This is a starting point, published because we could not find anyone\nelse's.\n", "summary": "Operational numbers from 1,372 local LLM agent invocations across 47 research runs in one day on a DGX Spark: wall-clock per run, pages read, evidence yield and answer quality by subject. Includes the figure nobody publishes -- 61% of page-reading agents produced no usable evidence at all -- and the measured threshold at which a cheap text filter removes a quarter of those calls with zero false negatives. Zero external inference cost.", "case": "agent-fleet", "sub_case": "local-agent-fleet-benchmarks", "tags": ["local-agents", "llm-agents", "research-automation", "benchmark", "dgx-spark", "self-hosted-ai", "rag"]}
{"title": "Twelve ways a local AI agent fleet breaks, and how each one was found", "content": "# Twelve ways a local AI agent fleet breaks, and how each one was found\n\n*Failure notes from building and running a self-hosted LLM research fleet on one NVIDIA DGX\nSpark (GB10 Grace-Blackwell, 122 GiB unified memory, aarch64, CUDA 13), August–September 2026.\nEvery entry below actually happened on this machine. Several cost hours; one cost a power cycle.*\n\nMost write-ups about local AI describe the version that worked. That is the least useful half.\nWhat follows is the other half: the wrong diagnoses, the measurements that were invalid, and the\nguards that now exist because something broke.\n\nEach entry gives the  symptom  you would actually see, the  wrong answer  we reached first\nwhere we reached one, the  real cause , and the  guard  — the check that makes the failure\nimpossible or loud rather than silent.\n\n Companion documents:  *Tuning an NVIDIA DGX Spark (GB10) to serve many concurrent local LLM\nagents* and *What a local research-agent fleet actually delivers*.\n\n \n\n  1. Unified memory does not OOM-kill. It hangs the whole machine.\n\n Symptom.  The box stopped. ICMP still answered. The inference server's `/api/tags` returned\n200. `/api/generate` never returned. SSH opened a TCP connection and then never sent a banner.\nNo OOM killer message, no kernel log line, nothing in the service journal. The machine was not\ndead and was not alive.\n\n Wrong answer.  \"This is not an out-of-memory freeze\" — because every OOM either of us had\nseen leaves a corpse: a killed PID, a `Killed process` line in `dmesg`. There were none, so we\nlooked at the network and the service first, and were wrong for about twenty minutes.\n\n Real cause.  On a Grace-Blackwell unified-memory system the GPU allocates from the *same*\npool as the OS. Over-commit does not trigger the OOM killer the way an ordinary process would;\nit drives the machine into an allocation stall where nothing can make forward progress,\nincluding `sshd` accepting a login. The only exit was a power cycle.\n\n What made it happen.  KV cache scales linearly with context length. We had benchmarked at\n`num_ctx 4096` and then shipped the slot count that benchmark justified while production ran at\n`12288`–`32768`. Same slots, six times the cache per slot.\n\n Guard.  Refuse to start rather than discover this at runtime:\n\n python\ndef budget_check(per_slot_gb_at_24k: float = 1.3, reserve_gb: float = 25.0)   str:\n    worst_ctx = max(CTX[k] for k in (\"read\", \"plan\", \"reduce\", \"synth\"))\n    kv   = SLOTS * per_slot_gb_at_24k * (worst_ctx / 24576)\n    need = kv + 45.0                       # + weights\n    if need > total_gb - reserve_gb:\n        raise SystemExit(\"REFUSING TO START: \" + verdict)\n \n\n The transferable rule:  *benchmark at the context length you will actually run.* A throughput\nnumber measured at a toy context is not a smaller version of the real number — it is a different\nnumber, and using it to size slots is how you wedge the machine.\n\n  2. A `MemoryMax=` watchdog cannot see the allocation that kills you\n\n Symptom.  A systemd `MemoryMax=6G` limit and a cgroup-based watchdog were in place. Neither\nfired during the stall in §1.\n\n Real cause.  `MemoryCurrent` accounts for the cgroup's own charged pages. Unified-memory\nallocations made by the GPU driver on behalf of the model are not charged there. The watchdog\nwas reading a number that stayed small while the machine ran out of memory.\n\n Guard.  Watch `MemAvailable` in `/proc/meminfo` — a system-wide figure — and act on a\nstall line above zero, not on the cgroup's view. On this class of hardware the cgroup is the\nwrong instrument, and it fails quietly, which is worse than failing loudly.\n\n  3. Measuring client concurrency against a server pinned lower\n\n Symptom.  A concurrency sweep that looked clean, published, and was wrong within a day. Going\nfrom 8 to 48 concurrent clients showed a suspiciously modest gain.\n\n Real cause.  The inference server was pinned at 8 parallel slots. Above 8, we were timing our\nown request queue, not the GPU. The published claim (\"slots are nearly free\") was falsified the\nsame day.\n\n Guard.  Raise the *server's* parallelism first and confirm it in the service environment\nbefore varying the client's. When the two disagree, every number above the server's limit is a\nmeasurement of your own queue. The superseded results now carry a correction banner on top\nrather than being deleted — a wrong number that someone already read should be corrected in\nplace, not disappeared.\n\n  4. GPU at 0% with a full queue — starved by the non-GPU stage\n\n Symptom.  Hundreds of tasks pending, sixteen LLM slots free, GPU utilisation at  0% .\n\n Real cause.  Two compounding errors. Six HTTP fetch workers were feeding sixteen LLM slots,\nso the readers had nothing to read. And the fetch timeout was  per-hop , not total: a chain of\nslow redirects held one worker for 61–72 seconds while each individual hop stayed under its\nlimit.\n\n Guard.  A total deadline per fetch, not per hop; fetch workers raised to 48 against 16 LLM\nslots. GPU went from  0% to 95% .\n\n The transferable rule:  in an agent fleet, the bottleneck is almost never the model. It is\nthe I/O stage in front of it. Size the cheap stage several times larger than the expensive one,\nand instrument per-stage rates — an aggregate \"tasks per minute\" number hides this completely.\n\n  5. A thinking model returns empty output under a JSON grammar\n\n Symptom.  A 30B MoE scored  0.0% on every extraction metric  — not badly, but *zero*, with\nno errors reported. Same prompts, same pages where a 7B model worked fine.\n\n Real cause.  It is a reasoning model. Under `format: json` it spent its token budget in the\nthinking channel and returned empty `content`. The API call succeeded. The response was blank.\n\n Guard.  Ask the server what the model is, and suppress thinking only for models that have it:\n\n python\ncaps = show(model).get(\"capabilities\", [])\nif \"thinking\" in caps:\n    payload[\"think\"] = False\n \n\nThis recurred with a second, unrelated model from a different vendor months later — the same\nempty-content signature — which is why it is worth naming as a *class* of failure rather than a\nquirk of one model.  If a model scores exactly zero with no errors, check the thinking channel\nbefore you conclude anything about its ability. \n\n  6. A quote verifier that a short quote walks straight through\n\n Symptom.  Nothing visible. Found by a self-test, not in production.\n\n Real cause.  Extracted facts were required to carry a verbatim quote, verified by substring\nmatch against the source. Short quotes — three or four common words — match almost any page by\naccident, so a fabricated fact carrying a short quote passed verification.\n\n Guard.  Quotes under five words get an exact check rather than a fuzzy one. Fabricated-quote\nrate across the fleet went from  5.5% to 0% .\n\n  7. A prompt-injection detector that is itself a model call\n\n Symptom.  The same page was flagged as an injection attempt on one run and not the next.\n\n Real cause.  The detector was a model call. A non-deterministic guard against a deterministic\nattack means an attacker gets as many attempts as they like, and your incident log is noise.\n\n Guard.  A regex over known injection shapes, applied before the content ever reaches a model,\nwith quarantine on match. Deterministic in, deterministic out. The model that reads fetched\npages has no tools, no network and a strict output schema — the containment is structural, not\nbehavioural, because a model instructed not to obey injected text is still a model being asked\nto make a judgement call about text designed to fool it.\n\n  8. Nearly publishing the setting that hung the machine\n\n Symptom.  A draft tuning guide recommended, in good faith, the exact configuration from §1.\n\n Real cause.  The measurement was real and reproducible. It was taken at a context length no\nproduction workload used. Reproducible and correct are different things.\n\n Guard.  That guide now leads with the cautionary table instead of the recommendation. Worth\nstating plainly:  the most dangerous document is an accurate benchmark presented without its\nconditions.  Anyone applying it inherits your assumptions without knowing they exist.\n\n  9. Restarting the inference daemon during live work\n\n Symptom.  A batch of runs failed at the planning stage, all at once, for no apparent reason.\n\n Real cause.  We stopped the inference service to pull a new model while runs were active.\n\n Guard.  Model pulls wait for an idle fleet; the queue heals with a retry path rather than\nabandoning the run. Separately, a permanently-failing task used to requeue forever — that now\nhas an abandonment path, because an infinite retry loop is a failure that looks like activity.\n\n  10. Three small ones that each cost real time\n\n-  timeout` does not exist on macOS.  A verification script reported failures that were the\n  verifier failing, not the thing under test. When a check fails, confirm the check runs.\n-  Working-directory drift in a long session  wrote a results file into the wrong directory.\n  Use absolute paths in anything long-running.\n-  Remote names recalled from memory instead of read  were wrong twice. Read the config.\n\n \n\n  11. A model that is fast per stream and cannot batch at all\n\n Symptom.  A newly released 30B-class mixture-of-experts model — NVIDIA's\n`nemotron-3.5-lightning:30b-a3b` (31.6B total, ~3.6B active, `nemotron_h_moe`), served through\nOllama 0.33.2 — benchmarked at  98 tok/s single-stream  — faster than the vendor's own published range for this hardware, and faster\nthan anything else on the box. On extraction quality it was the best model we had measured:\n 2.3x the evidence yield  of the incumbent reader at  a third of the latency , with a third\nof its fabrication rate.\n\n Wrong answer.  On those two numbers it was an obvious upgrade for the page-reading stage.\n\n Real cause.  Throughput did not scale with concurrency *at all*:\n\n| concurrent requests | aggregate tok/s | per-stream tok/s | GPU memory |\n \n| 8 | 64.7 | 86.9 | 28.6 GB |\n| 16 | 84.9 | 87.7 | 28.6 GB |\n| 32 |  84.3  | 87.2 | 28.6 GB |\n\nDoubling concurrency from 16 to 32 moved total throughput by  −0.6% , while per-stream speed\nheld at 87 and GPU memory never moved off 28.6 GB — 38% of the machine. That is not a saturated\nGPU. It is a server handling requests essentially  one at a time : this hybrid\nMamba/attention MoE architecture was not batching on this inference build.  To be fair to the\nmodel, this is very likely an Ollama/llama.cpp limitation for `nemotron_h_moe` rather than a\nproperty of the weights  — recurrent state in the Mamba layers is the usual reason a runtime\nfalls back to serial. A different serving stack (vLLM, TensorRT-LLM) may well batch it fine.\nThe lesson is about how you *measure*, not about whose model it is. Against the models\nactually in service it was  5.7x to 7.8x slower in aggregate , on the one stage that is\nentirely parallel.\n\n Guard.  Never accept a single-stream benchmark as a throughput result. Sweep concurrency and\ncheck that aggregate *rises*; if aggregate stays flat while per-stream stays high, the server is\nserialising and the model cannot serve a fleet no matter how good it is.\n\n The transferable rule:   per-stream speed is not throughput.  For anything running many\nagents at once, the only number that matters is aggregate tokens per second at your real\nconcurrency. A model can be simultaneously the fastest and the least usable thing you own.\n\n  Footnote to §11: the `nvfp4` tag will not load on a Blackwell box\n\nWhile pulling the model above, the quantisation that *should* be ideal for this hardware failed:\n\n \n$ ollama pull nemotron-3.5-lightning:30b-a3b-nvfp4\nError: this model requires MLX support, but the MLX runtime is not available\n \n\n MLX is Apple Silicon.  Despite the `nvfp4` name, and despite GB10 being Blackwell with FP4\nsupport in hardware, that tag is an MLX build and cannot run on aarch64 CUDA. NVFP4 Nemotron is\nnot reachable through Ollama on a DGX Spark at all — it would take vLLM or TensorRT-LLM, not a\ntag swap. `q4_K_M` is the working path. Worth knowing before planning around \"NVFP4 on\nBlackwell\": the tag name describes the quantisation, not the runtime that can execute it.\n\n  12. A benchmark too small to see the difference it is being used to justify\n\n Symptom.  The same model scored  6/6  on an agentic reasoning bench — a clean sweep,\nincluding two deliberately planted traps. We recorded that as evidence it should take over the\nplanning and synthesis stages.\n\n Real cause.  Running the identical bench against the incumbents showed the smallest model on\nthe machine — `qwen2.5:7b` — also scoring 6/6, and doing it faster at every stage than both the 32B and a 30B MoE. The\nentire spread across four models, from 7B to 32B, was  5/6 versus 6/6: one item, on a six-item\nset. \n\nThat is not a ranking. It is noise, and it had already been written down as a result because it\nwas first seen in isolation, with nothing to compare it against.\n\n Guard.  A bench that cannot separate a 7B from a 32B is not measuring the thing you are\nusing it to decide. Before any model swap is justified on a score: check the sample size, and\ncheck that the bench actually discriminates by running the models you already trust through it.\n\n The transferable rule:   an unbeaten score on a small benchmark is a statement about the\nbenchmark.  The moment a result is going to drive a change, the first question is not \"which\nmodel won\" but \"can this instrument tell these models apart at all\" — and a baseline you have\nnot run is not a baseline.\n\n  The pattern underneath all twelve\n\nNine of these twelve are not crashes. They are  silent degradation : a watchdog reading the\nwrong number, a benchmark measuring its own queue, a verifier passing fabricated text, a fleet\nat 0% GPU with a full queue, a model returning empty output with a success code. The system kept\nreporting that it was fine.\n\nLocal AI infrastructure fails quietly far more often than it fails loudly. The guards that\nmatter are the ones that make a silent failure noisy — a start-up refusal, a per-stage rate you\ncan actually see, an exact check instead of a fuzzy one — and the measurement discipline that\nmakes you distrust a clean number until you know what conditions produced it.\n\n \n\n*Published so the next person debugging a hung Grace-Blackwell box, a zero-scoring reasoning\nmodel, or an idle GPU behind a full queue finds a written answer instead of an empty search\nresult. Numbers came off the machine described. No model wrote this; the failures were ours.*\n", "summary": "Failure notes from running a self-hosted LLM research fleet on an NVIDIA DGX Spark (GB10): the wrong diagnoses, the invalid measurements and the guards that exist because something broke. Includes the one that cost a power cycle -- unified memory does not OOM-kill, it stalls the entire machine with no kernel message and no killed process -- plus a cgroup watchdog that cannot see the allocation that kills you, a reasoning model that returns empty output under a JSON grammar, and a GPU sitting at 0% behind a full queue. Eight of the ten are silent degradation, not crashes. Also: a model that is the fastest and highest-quality reader measured and still unusable, because it does not batch -- aggregate throughput flat from 16 to 32 concurrent requests while per-stream speed held at 87 tok/s.", "case": "failure-modes", "sub_case": "local-ai-failure-modes", "tags": ["dgx-spark", "gb10", "grace-blackwell", "local-inference", "llm-agents", "failure-modes", "postmortem", "unified-memory", "ollama", "debugging"]}
{"title": "Your benchmark probably cannot tell a 7B from a 32B. Ours could not.", "content": "# Your benchmark probably cannot tell a 7B from a 32B. Ours could not.\n\n*How a six-item evaluation set hid a real defect in a production AI pipeline for weeks, what\nreplaced it, and the defect it found on its first run. Measured on a self-hosted research-agent\nfleet, September 2026.*\n\nThere is a particular kind of wasted work in evaluating language models: you build a test, every\nmodel passes it, and you conclude the models are equivalent. They are not. Your test is.\n\nWe ran a six-item set to score the reasoning stages of a local research pipeline — planning,\nclaim verification, synthesis. Four models, from 7B to 32B parameters, scored  5/6 or 6/6 .\nA one-item spread across a 4.5x range of model size. We read that as \"size does not matter for\nthese stages,\" which is a *conclusion about models*, drawn from what was actually a fact about\nthe instrument.\n\nThen one model swept it 6/6 and we nearly promoted it on that basis. Running the same set\nagainst the models already in service showed the  smallest model on the machine also scored\n6/6 , faster. The entire measured difference was one question out of six — a coin flip.\n\n An unbeaten score on a small benchmark is a statement about the benchmark. \n\n \n\n  What was wrong with the set\n\nSix items is the obvious problem, and the least interesting one. Three others mattered more:\n\n It tested one failure mode.  Every negative item was a variant of the same thing — a claim\nthat outruns its quote. Nothing tested whether a model notices a unit changing, a date, a\nnegation, or a hedge.\n\n It was balanced 3 yes / 3 no.  A model that answers \"supported\" to everything scores 50% for\nfree, and 50% looks like a model that is trying.\n\n It reported one number.  A single accuracy figure cannot tell you *what* a model gets wrong,\nand what it gets wrong is the only actionable part.\n\n  What replaced it\n\n 45 items across 14 failure categories , each category naming a specific way evidence can fail\nto support a claim:\n\n| category | what it tests |\n \n| `direct`, `paraphrase`, `multi_evidence` | genuinely supported claims, including reworded and multi-source |\n| `entity_swap` | the quote is about a different product, company or jurisdiction |\n| `number_mismatch` | the figure does not match |\n| `unit_mismatch` | the figure matches, the unit does not — 240 bar vs 240 psi |\n| `causal_leap` | the quote states a fact; the claim asserts a cause |\n| `overgeneralize` | one config line or one trial read as a general rule |\n| `negation` | the quote says the opposite |\n| `temporal` | true once, claimed as current |\n| `hedged_source` | the source hedges — \"reportedly\", \"may\", \"up to\" — the claim asserts |\n| `partial` | the quote covers half a compound claim |\n| `contradictory` | two evidence lines disagree with each other |\n| `irrelevant` | the quote is true and says nothing about the claim |\n\n Deliberately imbalanced: 11 supported, 34 not.  In a research pipeline the expensive error is\n false-accept  — an unsupported claim entering a cited report. Rejecting a good claim costs one\nfact. Accepting a bad one costs the document's credibility.\n\nThat imbalance breaks raw accuracy, so the headline metric is  balanced accuracy  — the mean of\nthe two per-class recalls. We verified this rather than assuming it:  always-yes, always-no and\nalways-partial all score exactly 50% , while always-no scores  76% raw . If your imbalanced\nbenchmark reports raw accuracy, a model that refuses everything is beating your models.\n\n Scores are reported per category , and each model's weakest categories are named.\n\n  What it found on the first run\n\n| model | balanced | raw |  false-accept  | weakest categories |\n \n| 7B | 91% | 91% | 3/34 | unit_mismatch, multi_evidence |\n| 14B | 86% | 93% |  0/34  | multi_evidence, direct, paraphrase |\n| 30B MoE *(in production)* | 87% | 80% |  9/34  |  hedged_source , partial, unit_mismatch |\n| 32B MoE |  93%  | 89% | 5/34 | contradictory, unit_mismatch, temporal |\n\nA 7-point spread in balanced accuracy and a  0-to-9 spread in false-accepts , where the old set\nhad produced a one-item spread. The instrument now discriminates.\n\n  The defect\n\nThe 30B MoE was the production model — running planning, reduction and  synthesis . It writes\nthe prose a human reads. It scored  0/3 on `hedged_source . Given:\n\n>  claim:  The merger will close in June.\n>  evidence:  \"The companies *reportedly aim to* close the merger sometime in June.\"\n\nit answers  supported . All three items of that kind, wrong. It also took 1/3 on `partial`.\n\nThose are one behaviour:  it strips qualifiers.  \"Reportedly aim to\" becomes \"will\". \"Up to 30\npercent\" becomes \"30 percent\". \"Covers dental\" becomes \"covers dental and vision\". That is\nprecisely the failure a citation-anchored pipeline exists to prevent, happening in the stage that\nproduces the finished text — and its overall raw accuracy, 80%, reads as unremarkable rather than\nalarming.\n\n The old benchmark rated this model 5/6, indistinguishable from every other model, because it\ncontained no hedged-source item at all.  The defect was not introduced. It was always there, and\nthe test could not see it.\n\n  A blind spot every model shared\n\n`unit_mismatch` was the weakest category almost everywhere. Given *\"240 psi\"* claimed against a\nquote reading *\"240 bar\"*, most models match the number and ignore the unit — a factor of 14.5.\nOnly one model of four scored 3/3.\n\nThis is worth separating from the rest, because it suggests the fix is not a better model. A unit\ncomparison is deterministic; a model call is not. The same reasoning previously replaced a\nmodel-based prompt-injection detector with a regex, after the same page was flagged on one run\nand cleared on the next.  Where a check can be deterministic, a model is the wrong tool  — not\nbecause it is worse on average, but because it is unreliable in a way you cannot bound.\n\n  If you take three things\n\n1.  Run your benchmark against the models you already trust before you use it to justify a\n   change.  A baseline you have not run is not a baseline, and a score with nothing to compare\n   against is how a one-item spread becomes a promotion decision.\n2.  Count the failure modes your set contains, not just the items.  Ours had 45 items but the\n   number that mattered was 14 categories. The defect lived in a category the old set did not\n   have — and no amount of adding more items of the same kind would have found it.\n3.  If your set is imbalanced, report balanced accuracy, and check what a degenerate strategy\n   scores.  Ten minutes of work. It tells you the floor your models actually have to beat.\n\n \n\n*Every number here was measured on the hardware and pipeline described; none was generated by a\nmodel. The evaluation set and the harness that scores it are ordinary Python and could be\nrebuilt from this description in an afternoon — the transferable part is the categories and the\nimbalance, not the code.*\n", "summary": "A six-item evaluation set rated four models from 7B to 32B at 5/6 or 6/6 -- a one-item spread across a 4.5x range of model size -- and hid a real defect in a production pipeline. What replaced it: 45 items across 14 named failure categories, deliberately imbalanced because false-accept is the expensive error, scored on balanced accuracy after verifying that always-yes and always-no both land at 50%. On its first run it found the production synthesis model scoring 0/3 on hedged sources -- reading 'reportedly aim to close in June' as support for 'will close in June' -- a qualifier-stripping behaviour the old set could not see because it contained no item of that kind. Also: a unit-mismatch blind spot shared by every model tested, and why that one wants a deterministic check rather than a better model.", "case": "evaluation", "sub_case": "benchmarks-that-cannot-tell-models-apart", "tags": ["llm-evaluation", "benchmark-design", "verification", "hallucination", "local-inference", "llm-agents", "rag", "measurement"]}