The last post ended with flux2-dev generating 1024×1024 images after a two-day chase through Vulkan heap limits, GTT pools, and a GPU memory advisory call that made buffers CPU-inaccessible at exactly the wrong moment. The fix — amd_iommu=off amdgpu.gttsize=126976, giving the Ryzen AI Max 395+'s ROCm runtime access to the full 124 GB unified pool instead of a 24 GB slice of it — read at the time like a Stable Diffusion story.
It wasn't. Six weeks later, that same fix is why the LLM fleet can run larger KV caches, why two models are multimodal for the first time, why the broker can accept a webcam frame as an inference input, and indirectly why there are now four new agent binaries watching the cluster instead of one. This post is the accumulated backlog — sixteen commits, three subsystems, committed in one sitting after three weeks of work that outran the commit habit. Consider it the "everything since June 1" post.
The flux2-dev fix solved a Stable Diffusion problem, but the root cause — Mesa RADV's Vulkan backend reporting only ~4 GB of "device-local" heap on an APU, versus ROCm's ability to reach the full GTT pool once IOMMU stops enforcing the BIOS carveout boundary — applies to any workload with large compute buffers. wan22-video and ltx-video needed it in June (a 30 GB compute buffer doesn't fit in 4 GB either). This time it was the LLM side's turn.
A new Dockerfile.llama-rocm builds llama.cpp with GGML_HIP=ON targeting gfx1151 natively — ROCm 7.2.4 knows the chip by name now, so there's no HSA_OVERRIDE_GFX_VERSION shim pretending it's a discrete gfx1100 card (the same override, recall, that caused the SIGSEGV in the flux2-dev saga by making hipMemAdviseSetCoarseGrain succeed when it should have failed harmlessly). Three models moved from Vulkan to ROCm: gemma4-31b, devstral-small-beast, and deepseek-r1-70b.
With the GTT ceiling gone, the KV cache math changes. loch-nessh's configmap total_vram_gb went from 96 to 124 to reflect the actual pool. Some models spent that headroom on quality — gemma4-31b and deepseek-r1-70b went from q4_0 to q8_0 KV quantization. Others spent it on efficiency instead: hermes-beast had never had explicit -ctk/-ctv flags, silently defaulting to full f16 KV at 17 GB for 128K context. Setting it to q8_0 explicitly dropped that to 9 GB — same context, half the memory, freed up for co-loading. Not every KV change in this pass moved the same direction; the point was making the quantization a deliberate choice everywhere instead of an accident on some models and a default on others.
The manifests also picked up readinessProbe blocks and explicit memory requests on the ROCm deployments — small, but it means Kubernetes now knows when a ROCm pod is actually serving versus still loading a 47 GB model into unified memory, instead of routing traffic the instant the container starts.
Once gemma4-31b was on ROCm, it picked up --mmproj and jumped from 128K to 512K context. The context jump isn't just "more headroom" — Gemma4 uses sliding-window attention (SWA): 50 of its 62 layers are fixed at a 1536-cell window regardless of context length, so only the remaining 10 full-attention layers actually cost more memory as context grows. That's why 4× the context "only" costs the difference between 44 GB and 60-odd GB of KV, not four times as much.
A smaller sibling, gemma4-12b-rocm, is a new deployment feeding loch-nessh's new gemma4-12b-unified model — described in the configmap as "encoder-free," meaning it takes image patches and audio waveforms directly rather than through a bolted-on projector. This is the model backing the broker's new Sense feature (below).
ltx-2.3, the joint audio-video generator, also got finalized this pass. It had been running with an estimated 28 GB footprint and a T5/CLIP text encoder; that's now replaced by Gemma-3-12B — the same pattern already used for flux2-dev's Mistral-Small swap — and the real footprint is closer to 40 GB once you add the DiT (22.8 GB), the text encoder (7.3 GB), the video+audio VAEs (1.8 GB), and connector weights (2.3 GB). Audio generation is also no longer opt-in: it used to need a --av flag per request, and now the audio VAE loads at pod startup and every generation carries sound.
Every previous media feature in loch-nessh was output: text in, image or video out. The new /ui/api/sense/{models,query} endpoints (src/api/ui/sense.rs, new) invert that. A dashboard "Query" tab lets you attach a webcam frame or a mic clip to a text prompt and send it to any Chat model — the multimodal-input counterpart to the multimodal-output work the Media tab has done since June 1st.
The interesting part is what happens to the media once it's uploaded. A new artifacts.rs store POSTs it to /ui/api/artifacts, stashes it in Valkey under artifact:{uuid} with a 60-minute TTL, and hands back an ID. Every subsequent turn in that conversation references the artifact by ID instead of re-sending the base64 payload — the difference between a multi-turn conversation staying lightweight and one where every turn drags a growing pile of images along with it.
Getting arbitrary media through the request path required loosening a few things that had assumed text-only content for two years:
ContentPart (api/shared.rs) now preserves image_url/input_audio fields via #[serde(flatten)] instead of dropping anything it doesn't recognizebroker/executor.rs::normalize_messages no longer collapses array-typed message content down to a plain string if any part of it is non-text — that collapse step made sense when everything actually was text, and silently ate images otherwiseDefaultBodyLimit went from whatever Axum's default is up to 64 MB, because a base64-encoded webcam frame is not a small request bodyNone of this would be safe to ship without knowing whether a given model can actually make sense of what's sent to it, which is where the second half of this post's throughline comes in.
ipsa-probe (the capability-testing harness behind the fleet audit two posts ago) picked up a matching multimodal wing. ipsa-core::capability.rs now has VisionCapability, AcousticCapability, and MultimodalCapability structs, tiered the same way the text probes are: geometric grounding first (can it read a bounding box), then relational/spatial reasoning, then affective/semantic judgment (emotion, tone, identity) at the top.
A few of the new probes are worth calling out specifically:
image_scale.rs — a resolution ladder probe. It feeds a model steadily larger images until something breaks (wrong answer, timeout, OOM), and records where. This is the multimodal equivalent of the context-length probes: "how big a picture can this model actually handle" turns out to be exactly the kind of question that doesn't have a spec-sheet answer.visual_search.rs — a "find Waldo" probe against a new 3600×2544 fixture image. It's a blunt but effective test of whether a model is actually looking at the full image or attending mostly to whatever got downsampled into the first few patches.acoustic.rs and multimodal.rs — parallel suites for audio, and for prompts that require combining both modalities with text.Two new probe-scored profiles (gemma4-12b.toml, gemma4-12b-unified.toml) are the first capability data for the models backing Sense. The research behind how to structure these probes — tiering, avoiding compression-artifact confounds, adding an explicit affective/demographic classification layer rather than bolting it onto the grounding tier — is now written up in agents/knowledge/multimodal-probing/, in case future-me needs to remember why the tiers are ordered the way they are.
Multimodal input isn't just useful for a dashboard chat box. Four new standalone binaries in ipsa-runner exist specifically to consume it:
oracle — points a webcam at something, asks a vision model about itauditor — records from ALSA, asks an audio model about itsentinel — not multimodal, but new: a Kubernetes/VictoriaMetrics observability agent that can answer "why is this pod pending" or "what's the VRAM situation on ness-linux3" by actually calling kubectl and querying metrics, built on a new reusable tool_loop primitivesecuritas — the largest of the four: a continuous audio/video surveillance pipeline (capture → ingest → processor → archiver) with trash/keep/archive retention tiers and a reaper task cleaning up what's no longer worth keepingRunning four more LLM-backed agents raised an obvious question: not everything needs a full agent loop and a model call. ipsa (the CLI, src/ipsa.rs + ipsa_cli/) is the answer — a quick-access front door with a zero-LLM fast path for read-only operations (ls, grep, find, sandboxed to a Workspace), a tool-calling ipsa ask mode against a small "ipsa-quick" model for anything that does need reasoning, and a guided ipsa commit (status → diff → sanity check → generated message → explicit y/N confirm) that is, not coincidentally, roughly what I do by hand every time I've put off committing for three weeks.
Underneath all of this, a new skills system (src/skills/ + top-level skills/) lets an agent load a SKILL.md manifest plus a folder of scripts and get them exposed as callable tools without writing Rust. Two packages exist so far: cortex-ops (VRAM checks, pod health, restart-deployment remediation) and ansible-root-ops (wraps the existing per-distro update playbooks in ~/git/lab/ansible-root). It's the difference between "the agent needs a new Rust tool for every new operational task" and "the agent needs a markdown file and a bash script."
More agents making tool calls against real infrastructure means more chances to do something dumb at 2am. Two changes this pass are specifically about that risk:
atomus (the hierarchical task-decomposition engine behind mimir) gained a rules.rs validator. The first rule, no_raw_shell_effector, rejects any plan where a mutating step shells out directly instead of going through a typed tool: resolver — which matters more now that mimir's default_tools() wiring means git/build/container operations have a typed path available, so there's no excuse for a raw shell escape hatch except the explicit --unsafe-shell flag added alongside it. Atomus also gained two new primitive kinds: DerivedFile (skip a plan step if its output is already newer than its inputs — a make-style freshness check for LLM-driven pipelines) and Reconcile (when an atom's actual state doesn't match the plan's expectation, ask an LLM to reconcile it, and escalate to a human if that fails repeatedly rather than looping forever).
On the Discord side, sainer — the safety gate in front of the bothy agents — went from stateless keyword matching to a stateful reputation gate: per-sender severity scoring (0–3) with TTL'd violation counts, judging a pattern of behavior rather than reacting to any single message in isolation. grieve, the moderation monitor, now round-robins across a pool of models instead of depending on one (GRIEVE_MODEL is comma-separated now), and "warn" verdicts post to the channel instead of disappearing into a log file — a moderation action nobody can see is closer to a moderation intention.
loch-nessh picked up its own version of "don't let one bad request wreck things": a new preempt: none|hol|cancel|hybrid setting per model. The hybrid mode is the interesting one — it keeps a rolling history of how long a model's jobs typically take, and uses that to decide whether an incoming higher-priority request should wait head-of-line or actually cancel-and-requeue the in-flight one. A RETRY_SENTINEL written into the outbox stream lets any consumer — streaming, non-streaming, or a /v1/wait poller — discover a preemption happened and transparently pick up the retry rather than returning a half-finished response. Only Chat executions are preemptible; image and video jobs poll an external service and can't be safely interrupted mid-flight.
Also new: a related grouping feature, ModelGroup, lets equivalent deployments across nodes (say, phi4-mini on both ness-linux3 and ness-legion1) be addressed by a single name, with the broker picking whichever idle member has VRAM and no lock held. Combined with preemption, the scheduler is doing meaningfully more judgment than "is there a free slot" now.
Two things from earlier in the window that never got their own writeup:
en_wiki was OOMKilling every four hours. Not a leak — glibc malloc fragmentation, where freed pages from the 4 GB article cache churn never made it back to the OS, so RSS climbed until the pod hit its limit and died. Swapping in tikv-jemallocator as the global allocator fixed it outright (jemalloc actually returns freed pages via madvise); the cache size was also halved to leave more headroom for RocksDB, and the RocksDB block cache/bloom filters/write buffer got explicit bounds instead of defaults.
Video generation stopped blocking the executor for hours. Image and video jobs used to hold an executor thread for the full generation time. They're fire-and-forget now: submit, persist the poll URL to the claim, spawn a detached polling task, return immediately. On a loch-nessh restart, in-flight video claims reconnect to their poll URL instead of being re-queued or lost. This is also what made ogma-discord's image generation reliable — the old synchronous poll pattern was getting killed by kube-proxy's idle TCP timeout on anything that took more than a few minutes, which flux2-dev generations reliably do.
deepseek-r1-70b's emission=None issue from the May fleet audit is still open — still under investigation, still not blocking the model's continued presence in the fleet.
The vision/acoustic probe suites are new enough that only two models have real profiles (gemma4-12b, gemma4-12b-unified). Whether gemma4-31b's freshly-added --mmproj support actually performs well, or just loads without crashing, is an open question the resolution-ladder and visual-search probes exist to answer — next up.
securitas's retention tiers (trash/keep/archive) are implemented but I haven't run it continuously long enough to know if the reaper task's cleanup timing is right, or whether "keep" is going to quietly fill a disk somewhere. That's this month's problem.
And the obvious one: three weeks passed between the last commit and this one. The ipsa commit skill exists specifically because I know exactly how that happens, and the fact that I built a guided commit tool and then still needed a marathon session to catch up on sixteen commits' worth of work is not lost on me.