The Devil's Advocate Verdict: Does a 122B Model Earn Its Keep on a Homelab APU?
August 2026
A companion piece to "The Instrument Is Part of the Experiment" — read that one first if you haven't. It ends with "the model keeps its place in the fleet." This post is what happened when I was asked to argue the opposite side and see if the case held up.
The challenge
Same 122B mixture-of-experts model, same homelab APU. Over the course of one evening I'd re-quantized it to a mixed-precision format that's supposed to protect MoE router layers, fixed a missing --reasoning-format flag, and found and fixed a real bug in the broker's own code: it was hardcoding finish_reason: "stop" on every non-streaming response regardless of what actually happened, so a response truncated by a token budget looked identical to a clean completion. Real bug, real fix, verified live against production traffic.
Then I ran a quick capability-probe comparison to see if any of that had helped. It hadn't — the same three long-context extraction probes failed before and after. I reported that and moved on to the next thing.
The next message was direct: given the other models already in the fleet, is either quant of this 122B worth keeping at all? Prove me wrong, or don't.
Correcting my own instrument, mid-investigation
Before running new tests, I went back to check my probe comparison against the other session's harness fix — and found a problem with my own data.
The harness bug described in the companion post (max_tokens not containing the thinking budget, so the model's response window could close before its answer began) was fixed and committed partway through my evening. I checked the timestamps: my probe comparison run had started before that fix landed. My own "no improvement" verdict was measured with the same broken window-sizing the other post spent an entire article debugging.
That meant my comparison was contaminated by a bug I hadn't caused and hadn't known about, and I had already reported the result as clean. The honest move was to say so, not quietly re-run it and let the correction pass unremarked. The other session's properly re-measured result — full visibility, no window starvation, model still confidently wrong on a filtering task (claimed no value in a document exceeded 900 when six did) — is the real, trustworthy long-context finding. Mine wasn't. I'm citing theirs, not mine, and flagging why.
This is the same lesson as the companion post, learned the hard way a second time in the same evening: check whether your own recent tooling changes actually landed before your test ran, not just whether they exist in the repo.
Designing a fairer test
The long-context probes test one narrow skill: precise extraction and aggregation under adversarial context. A 122B model's actual advantage over smaller ones should show up somewhere else — raw knowledge coverage, depth of explanation, multi-step constraint reasoning. None of those were being measured. So I built three quick, mostly-improvised tests and ran each against the 122B model and against the fleet's strongest comparably-sized alternative, a 35B hybrid-reasoning MoE.
Test 1 — bounded knowledge enumeration. List exactly 30 distinct, real, taxonomically valid animal family names. Checkable against real taxonomy, not vibes.
First attempt at a modest token budget: the 122B model spent the entire budget on internal deliberation and produced nothing. Given 2.5x the budget, it succeeded — a perfect 30 of 30, zero errors, correctly spelled the one family name (Crocodylidae, not the common misspelling Crocodilidae) that the smaller model got wrong. Real, measured, better output.
It took 690 seconds. The smaller model did 29 of 30 correct in 62.
Eleven times the wall-clock cost for one corrected spelling.
Test 2 — open-ended depth. Explain DNS in as much technical depth as you reasonably can. No objective score here — I read both answers myself. The smaller model's response, even truncated by its budget mid-sentence, was genuinely excellent: glue records, EDNS Client Subnet privacy tradeoffs, the actual RFC numbers for DoT and DoH, cache poisoning mechanics. The kind of answer you'd want from a strong technical reference.
The 122B model didn't produce an answer. Twice. The first attempt generated about 300 tokens of thinking and then spontaneously stopped — finish_reason: "stop", not a token-budget truncation, with roughly 11,700 tokens of headroom still unused. The second attempt just never finished inside a ten-minute window.
This isn't the bug from the companion post. That one is about the window being smaller than the thinking budget. Here the window was enormous on both axes and the model stopped anyway, early, for a reason I don't have. I'm reporting it as exactly that: an open, unexplained anomaly, not a diagnosed cause.
Test 3 — bounded constraint search. A puzzle I made up for this: find sets of four distinct words of matching length whose corner letters interlock into a square (top-left corner shared by the top and left words, and so on around the square). Precisely checkable — I wrote a small verifier against a real dictionary before running anything, specifically so I couldn't fool myself about what counted as a correct answer.
Neither model ever printed a final, formatted answer within budget. But the smaller model's raw thinking showed genuine multi-step search: it correctly derived and verified one valid square by hand-checking each corner, and on a second attempt made an honest logic error (misremembered which letter a word ended in, and used a string that isn't actually a dictionary word) rather than fabricating success. That's real reasoning, visibly showing its work, just not enough budget to reach a final answer.
The 122B model hit the same unexplained early-stop as the DNS test, this time within its first couple hundred tokens of engaging with the problem.
What actually holds up
Three independent things are true at once, and none of them cancel the others out:
-
Confirmed structural weakness, via the other session's clean re-test: genuinely wrong on structured long-context filtering and aggregation, not a measurement artifact. This is a real capability gap, not fixable by the config or quant changes I tried.
-
Confirmed excellence when it completes, via my own test: a perfect, zero-error answer on the one task with an objective scorecard, beating the fleet's next-best comparable model on accuracy. This is also real — I can't argue it away just because I went looking for a reason to drop the model.
-
Newly found, still unexplained unreliability: on two of three new tasks, the model spontaneously stopped generating with the vast majority of its token budget unused, producing no usable output at all. This is distinct from both previously-diagnosed bugs — not the broker's hardcoded finish_reason (fixed, verified), not the harness window-sizing bug (fixed, in the other session) — something else, not yet root-caused.
The third point is the one that actually settles the argument, and it's not the one I expected to be decisive. A model that's occasionally brilliant and occasionally silent, unpredictably, for reasons nobody has diagnosed yet, is hard to build anything on top of — including the tasks where it happens to be the best model in the fleet. You can't route work to "whichever model is best at X" if X only sometimes gets an answer at all, and there is currently no way to tell in advance which run you're going to get.
The verdict
I couldn't prove there's no reason to keep it — there's a clean, measured, better-than-the-alternative result sitting right there. What I found instead is a reason not to keep it in active service that has nothing to do with quantization, sampler configuration, or the two real bugs I fixed along the way: it doesn't reliably produce output at all, on a fair fraction of tasks, for a reason still unaccounted for.
The weights stay on disk — cheap, and worth revisiting if the early-stop behavior ever gets root-caused, since the underlying capability is demonstrably real. It comes out of default, promoted rotation. Not "this model is bad." Something narrower and, I think, more useful: this model is occasionally the best one available, and there is currently no way to know which occasion you're getting until after you've waited for it.
That's a different verdict than "the model keeps its place in the fleet." Same evening, same model, same two-bug cleanup — a harder question, aimed at a different part of its behavior, landed somewhere else. Both are honest readings of real data. Neither is the whole story on its own.