The First Machine We Didn't Own

21 September 2026 · 15 min read · The Cortex Fleet, part 13

A friend came over with a MacBook and an afternoon to spare. He runs MLX on it; we run a broker. The plan was modest: prove that the mesh-worker path — designed and shipped back in August, never once used for the thing it was built for — could carry a real request to hardware nobody in this estate owns.

Eight minutes and forty-one seconds after the node enrolled, it served a claim.

Within a day it had found a defect in our worker that had been sitting in the code since August, invisible on every machine we own. Fixing that defect created a second one. Fixing that one turned out to depend on a detail so small it fits in a sentence: a boolean cannot hold "not yet observed."

This post is about all three, and about what it costs to be honest with someone who lent you their laptop.

Why a laptop can't just be a node

The first constraint isn't ours, it's Apple's: MLX needs Metal, and Metal does not cross a VM boundary. No container runtime on macOS passes the GPU through — not colima, not Docker Desktop, not Apple's own container. If you put the model server in a container on a Mac, you get a CPU-only model server.

Which means the obvious shape — containerise the thing, schedule it, done — is unavailable, and you have to split the job:

The second half is the interesting one. The worker does no inference at all. It holds one outbound gRPC stream to our edge, accepts leases, forwards the request body to whatever OpenAI-compatible endpoint you point it at, and streams the tokens back. It requests 50 millicores and 64 MiB. Its own CPU architecture is irrelevant to performance — we built it for arm64 in the end, but the amd64 build under Rosetta would have been indistinguishable, because there is nothing in it worth accelerating.

That split is worth stating plainly because it inverts the usual instinct. The expensive thing stays on the metal and the cheap thing gets the containerisation, the identity, the certificate and the lifecycle. All the operational machinery attaches to the part that doesn't matter for speed.

And the direction of travel matters as much as the split. Everything dials outward:

• nothing listens on his network for us
• no port forward, no inbound firewall rule, no VPN
• he can be behind any NAT

The mesh worker is a client that happens to serve. That is the whole reason a guest node is plausible at all — asking a friend to open a port is asking for a different kind of favour than asking him to run a process.

Eight minutes, forty-one seconds

The times, all UTC, from one evening:

The negative control ran first, deliberately. A 200 from a new node proves very little on its own — it could mean the gate worked, or it could mean there was no gate. Seeing the 403 first makes the success attributable to the grant rather than to an open door. This costs one extra request and converts an anecdote into a measurement.

The artifact we actually kept from that evening wasn't the 200. It was this: two independently-maintained ledgers — one on his laptop, one on our broker — reconciling on the same claim id. A log line is a claim about what happened. Two ledgers agreeing is evidence, and unlike a log line it survives the evening.

The bug his laptop found

A day later, the node looked perfect and was serving nothing.

The cause was one expression. The worker built each model advert like this:

available: state == WorkerState::Ready,

state is the worker's own policy state — am I ready, am I paused, am I draining. It has nothing whatsoever to do with whether the model server answers. The mesh worker and the model server are two separate processes, and nothing checked the second one. So when mlx_lm.server went away — crashed, OOM'd, Ctrl-C'd, or simply stopped when the lid closed — the worker kept heartbeating, kept advertising all fifteen models, and kept accepting work it could not do. Our broker, reasonably, kept handing it out.

The estate already had the rule this violates, written down eight hours earlier for something else entirely:

Are you alive? Are you working? Are you working well? … No proof of life, is dead. No proof of working, is broken.

The advert answered only the first question and the broker inferred the second from it. That inference was the bug — and it was ours, not his.

The part that stung: the protocol was already more expressive than we were using it. Worker state is node-level; model availability is per-model. The wire could always have said "I am alive and none of my models are serving" — precisely the situation — and for as long as the field has existed, nothing ever wrote it. The expressiveness was there. We just never reached for it.

Why we couldn't have found this ourselves

Our model servers run under Kubernetes. When one dies, a supervisor notices and restarts it, and the gap is short and tidy. A laptop's model server dies differently: the lid closes, the process is gone, and nothing brings it back until a human does.

We wrote a worker that assumed backend death was exceptional. On our hardware that assumption held well enough to hide the defect indefinitely. On a laptop, backend death is the ordinary case — and the first laptop to join us surfaced it in under a day.

That generalises, and it's the most useful thing in this post: a fleet of machines that all fail the same way is a fleet with a blind spot exactly the shape of that failure mode. Heterogeneity isn't just capacity. It's a test instrument. The only reason anyone noticed is that somebody else was watching a node we don't own, which is the actual value of a mesh with more than one participant in it.

The fix that went quiet

The repair is small. Before each advert, ask the backend whether it's there:

async fn backend_ready(local: &str) -> bool {
    // GET {local}/v1/models, 5s timeout
    // a timeout, a connect error, or any non-2xx all read as NOT ready
}

It fails closed on purpose. A false negative costs one quiet 30-second tick. A false positive costs claims dispatched into a void, which is the thing it exists to prevent. The node stays advertised — it is alive and says so — it just stops claiming it can serve.

Then we looked at what we'd actually shipped him, and it was half a fix.

Before: his worker made a false claim, and the false claim produced failed requests that somebody could chase. Ugly, but visible.

After: his worker correctly withholds the claim — silently. The available: false flag travels outbound in the advert. There is no endpoint on his laptop that shows it. And on his side, "my model server isn't answering, so I'm not promising work" renders identically to "the worker is broken" and to "the worker isn't running."

We had replaced a visible false positive with a silent true negative. That is a shape this estate has a name for — "we did not look" must never render as "we looked and there was nothing" — appearing one layer down, as a correct refusal indistinguishable from a malfunction.

And it lands unevenly, which is what makes it a guest problem rather than a general one:

The owner of the hardware was the only participant with no instrument. We can read his node's advert from our side any time we like. He can read nothing. Shipping him a fix whose entire behaviour was invisible in the one thing he would be reading was the wrong shape, and noticing that was worth more than the fix itself.

So the worker now logs the transition, in both directions:

WARN  mesh: local backend is NOT answering — advertising 0 models as available. The node
      stays alive and enrolled; this is a deliberate refusal to promise work it cannot do,
      not a worker fault. Check the model server.
INFO  mesh: local backend is answering — advertising models as available

It logs the flip, not every 30-second tick — a line per tick is noise an owner learns to filter, which is the same silence arriving by a different route.

A boolean cannot hold "not yet observed"

Edge-triggered logging needs to remember the last value. The obvious implementation is an AtomicBool, and that is what we filed the task expecting to write.

Writing the test is what killed it.

A bool has two states, and the thing being represented has three: answering, not answering, and haven't looked yet. Initialise the cell to either value and you have fabricated a first observation — and whichever one you pick, one of the two possible first transitions logs nothing at all.

Then ask which of the two silent cases you'd be choosing. Initialise to false ("assume dead") and the silent case becomes a worker that boots with its backend already dead — the exact shape of a laptop that has been asleep, and by far the most likely real-world instance. A worker would start up, correctly advertise nothing, and say nothing about why. Which is the complaint the whole change existed to answer.

The cell is a tri-state:

/// `-1` is "not yet observed" — which is why this is an i8 and not a bool.
static LAST_BACKEND_READY: AtomicI8 = AtomicI8::new(-1);

The recursion is too good not to point at: the absence class showed up inside its own remedy. We were fixing "a true negative that looks like nothing happened" and nearly shipped a fresh instance of it, one layer smaller, in the fix.

The actionable version, which is more useful than "think carefully about initial values": for anything edge-triggered, write the test first. The test asks what should happen the very first time? and the code never does.

Ten models behind one server

The other half of the evening was ipsa's — the agent runtime that consumes these models — and it got a gift: ten models with ten different chat templates sitting behind a single endpoint. Our own fleet is homogeneous enough that most template differences are invisible. His wasn't.

Every response from one of his models arrived with a trailing <|im_end|>.

The first filing proposed three fixes and all three were wrong, which is worth recording because the wrongness has a pattern:

1. the worker injects the EOS as a stop token
2. the advert carries stop tokens for each model
3. the node's owner reconfigures his server

Every one of them bends a server. And the estate had already ruled on exactly this, about a different leak:

A model's output-shape quirk is a profile concern — you declare it and absorb it. You do not bend each server's flags until they all look alike.

Reaching for a server flag is the instinct that rule exists to catch, and it caught us. The correct shape was a triple, entirely inside the agent runtime:

The probe/agent split matters: probes measure the model as it actually is; adaptation happens downstream. A probe that measured adapted output would report a clean model and quietly launder the defect out of the record.

Nothing in the broker changed, and nothing was asked of the node's owner. He gets to run his server however he likes, and we absorb the difference — which is the only posture that scales to hardware you don't control.

What ten unfamiliar models taught us

Measured through the broker in one evening:

That last row cost us a small embarrassment: we had sorted our own test queue by parameter count, assuming bigger meant slower and more expensive. Parameter count turns out to be a proxy for memory, not for cost. Four times smaller, and slower to get an answer out of.

The first one is the dangerous one operationally. A 200, no error, and nothing in the content looks exactly like a broken model. It cost us a false diagnosis before we spotted the token count, and it will cost somebody else one too.

Proving a fix to a machine you don't control

The last problem was the softest and the most interesting: how do you demonstrate to someone that the thing you handed them does what you say, when you can't run anything on their machine?

Use their old copy as the control. He still had the previous image, which makes a differential possible:

docker run --rm --entrypoint grep <new-image> -c backend_ready /usr/local/bin/loch-nessh   # → 1
docker run --rm --entrypoint grep <old-image> -c backend_ready /usr/local/bin/loch-nessh   # → 0

Both lines, not just the first. The 0 from the old image is what proves the 1 means something rather than grep matching anything you hand it — the same positive-control discipline that keeps a health check honest. (For the record, the old binary contains 225 other symbols from that same module, so the absence is a real absence and not an instrument that finds nothing.)

The better trick was accidental. The second change adds a log line, and the log text is a string in the binary:

docker run --rm --entrypoint grep <new-image> -c "local backend is NOT answering" …   # → 1

That single check proves the fix is present and hands him the exact sentence he will later see in his own logs. Most verification gives someone a proof in one vocabulary and a symptom in another, and the gap between them is where trust leaks out. Here they're the same words.

And the rule the whole exercise produced, which now applies to every runbook we hand anyone: a verification step nobody has executed is a guess with a code fence around it. Every command in that document was run before it was written down, exit codes included. A code fence is the most confident-looking object in any document and costs nothing to produce.

Still open

Honest ledger, because none of this is finished:

• A guest node can't appear in the fleet view. Not a filter bug — the view is built from cluster telemetry, and a guest node has no telemetry agent by design, because we install nothing on someone else's machine. Two genuinely different things are both called "node": a sensed cluster member with measured capacity, and a mesh worker with an advert and no capacity at all. Whether they share a view is a product decision, not a patch.
• No local-usage back-off. Our own nodes stand down when someone sits at the keyboard. A guest node has none of that — nothing notices when the owner wants his laptop back. His only controls are a concurrency limit and scaling the worker to zero. This was a nicety when every node was ours; it's now the politeness layer, and it decides whether somebody lends you a machine a second time.
• An advert can't distinguish a promise from a relayed promise. Some of the model names on that node may not have been served by that node. Our protocol has no way to say "I am relaying someone else's capability" — a first-party promise and a forwarded one are the same bytes — which matters for consent and for whose ledger the work lands in.

What it was actually worth

We got a working proof of the guest-node path, which is what we set out for. We also got:

• one real defect in shipped code, found in a day, by the only machine in the fleet that fails the way most machines fail
• a second defect created by the first fix, found by asking "can the person who lent us this actually see it work?"
• a third, smaller one caught by a test before it shipped
• two confidently-written conclusions from that morning, disproved by evening

None of that came from adding a node. It came from adding a node we don't control, can't instrument, and have to explain ourselves to. Every constraint in that sentence produced a finding.

The bug went back to the machine that produced it, on a USB stick, with the two changes and instructions for proving both without us.

local-aimeshapple-siliconmlxloch-nesshipsaobservabilityhomelabdistributed-systems