June 2026 · Jon Harkness
I spent two days debugging why Flux2-dev would generate a perfectly good 512×512 image and then silently fail at 1024×1024. No error message. No crash log. Just nothing. The journey took me through Vulkan heap limits, GTT pools, IOMMU, hipMallocManaged, and a particularly devious GPU memory advisory call that makes memory CPU-inaccessible at exactly the wrong moment. Along the way I learned why the Ryzen AI Max 395+ can put 70 GB of unified memory behind a single 1-megapixel image — and why that matters for quality.
This is the full story.
The machine is a Ryzen AI Max 395+ — AMD's Strix Halo platform. It has 128 GB of LPDDR5-8000 unified memory shared between the CPU and an integrated Radeon 8060S GPU (RDNA 3.5, gfx1151). There's no discrete GPU, no GDDR. The GPU and CPU share the same physical DRAM.
The inference stack is stable-diffusion.cpp running as a Kubernetes pod, wrapped behind a custom VRAM-aware broker called loch-nessh. The model is Flux2-dev at Q8_0 quantization: a joint DiT architecture with a 33 GB diffusion backbone and a 24 GB Mistral-Small text encoder.
When it works, it works beautifully. The problem was getting there.
The symptom was clean and reproducible: submit a 512×512 generation, get an image. Submit 1024×1024, get nothing. No pod crash. No OOM event. The generation just never completed.
The math tells you why before you even look at the code.
Flux uses a joint DiT with 2×2 latent patches. The patch count scales with resolution:
But attention is quadratic in patch count. The self-attention matrices scale as patches², so going from 512 to 1024 doesn't cost 4× more memory — it costs 16× more. The intermediate tensors for a 1024×1024 image are sixteen times the size of the 512×512 case.
With the Vulkan backend, those intermediate tensors have to fit inside the device-local Vulkan heap. On this machine, Mesa RADV reports that heap as approximately 4 GB.
4 GB / 16 = 256 KB of headroom per tensor step. Of course it fails.
The machine has 128 GB of unified memory. Why is the Vulkan heap only 4 GB?
This is a Mesa RADV limitation on APU configurations. On a discrete GPU, the device-local heap is the GDDR; Mesa reports all of it. On an integrated GPU sharing system memory, Mesa conservatively reports only a small fragment as "device-local" to avoid starving the CPU. For gfx1151 (Strix Halo), that fragment is around 4 GB — enough for the model weights that were pre-loaded into a BIOS carveout, but nowhere near enough for the quadratic attention tensors at full resolution.
The fix is to stop using Vulkan for this model and use ROCm/HIP instead. ROCm can address the full physical memory pool.
Switching to a ROCm-based container immediately revealed the next problem. ggml reported success finding the GPU, then failed on the very first allocation:
ggml_cuda_init: found 1 ROCm devices (Total VRAM: 98304 MiB):
Device 0: AMD Radeon Graphics, gfx1100, VMM: no, VRAM: 98304 MiB
ggml_backend_cuda_buffer_type_alloc_buffer: allocating 33381.06 MiB on device 0:
cudaMalloc failed: out of memory
The device reports 96 GB of VRAM. The allocation needs 33 GB. 33 < 96. And it fails.
The confusion resolves when you understand what those numbers actually mean.
The BIOS on this machine is configured to carve out ~96 GB as "GPU VRAM" — a fixed reservation from the physical DRAM. The other ~32 GB is left as CPU-accessible system RAM. ROCm correctly reports the carveout as VRAM. But hipMalloc doesn't allocate from the carveout. On an APU without IOMMU disabled, it allocates from the GTT pool — the portion of system RAM that the GPU can access via address translation.
The GTT pool is drawn from the CPU-visible RAM, not the carveout. And its default size is:
mem_info_gtt_total: 24576 MB (24 GB)
The amdgpu driver defaults to making ~75% of available system RAM available as GTT. With 32 GB of CPU-visible RAM (the rest is the carveout), that gives 24 GB. The diffusion model alone needs 33 GB. There's no way it fits.
The root cause is AMD's IOMMU. With IOMMU active, GPU memory access goes through an address translation layer that enforces the BIOS carveout boundary. The GPU can't reach past the 32 GB of CPU-visible RAM without the IOMMU getting in the way, so the GTT pool is necessarily small.
The fix, documented by the Strix Halo homelab community, is three GRUB kernel parameters:
amd_iommu=off amdgpu.gttsize=126976 ttm.pages_limit=32505856
amd_iommu=off disables the IOMMU entirely. The GPU can now address all 128 GB of physical memory. amdgpu.gttsize=126976 raises the GTT pool cap to 124 GB. ttm.pages_limit=32505856 raises the TTM pinned-memory page limit to match (32,505,856 × 4 KiB ≈ 124 GB).
After reboot:
mem_info_gtt_total: 126976 MB (124 GB)
The 33 GB allocation for the DiT model now has 91 GB of headroom. The 24 GB text encoder fits too. We're in business.
There's also a BIOS setting worth considering: reducing the GPU VRAM carveout from ~96 GB to 512 MB. The community canonical configuration skips the large BIOS carveout entirely and relies on GTT for everything. That's what I did.
After the GTT fix, the model loaded cleanly — all 909 tensors, 57 GB into GPU memory, server listening on :8080. Then the first generation request came in, the Euler sampler started, and at about 55 seconds: exit code 139. SIGSEGV. Every time.
This one took a while to trace.
The crash was happening inside the GPU compute path, not during weight loading. To isolate it, I ran the model on the CPU backend. CPU loaded all 909 tensors to 300+ without any crash — confirming the issue was specific to the GPU path. (Side note: --backend cpu loads the full 57 GB into system RAM. We burned through 120 GB before stopping that test. Don't do this casually.)
The actual cause is subtle. When GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 is set, ggml's ggml_cuda_device_malloc() calls hipMallocManaged instead of hipMalloc. That allocates memory from the unified GTT pool, accessible to both CPU and GPU. So far so good.
But immediately after every hipMallocManaged, ggml calls:
(void)cudaMemAdvise(*ptr, size, hipMemAdviseSetCoarseGrain, device);
(void)hipGetLastError(); // clear any error
hipMemAdviseSetCoarseGrain marks a managed allocation as "coarse-grained" — meaning the memory is migrated to GPU-local storage and the CPU coherency guarantee is dropped. The CPU can no longer safely read or write it.
On a discrete GPU (gfx1100, Navi 31), this call fails with hipErrorInvalidValue because GDDR6 doesn't support the coarse-grain hint. The error is swallowed by the hipGetLastError() call immediately after, and memory stays in unified mode. Fine.
On our APU — with the device being reported by ggml as gfx1100 — the same call succeeds. The managed allocation migrates to the BIOS carveout, becomes GPU-local, and the CPU can no longer touch it.
When ggml then runs an Euler denoising step and any code path tries to synchronize results or read an intermediate tensor from the CPU side: SIGSEGV. Always around 55 seconds in, always exit 139.
The original plan was to use ROCm 7.2.4 with native gfx1151 support instead of the HSA_OVERRIDE_GFX_VERSION=11.0.0 shim. In theory, ROCm 7.2.4 knows gfx1151 is an APU and hipMemAdviseSetCoarseGrain would correctly fail — no SIGSEGV.
In practice, ggml was still reporting gfx1100 for the device name (via hipDeviceProp_t.gcnArchName). The ROCm runtime maps gfx1151 to the gfx1100 ISA family for compute code compatibility — which is correct from a shader execution standpoint but wrong from a memory advisory standpoint. The CoarseGrain call kept succeeding.
The fix turned out to be simpler: don't use GGML_CUDA_ENABLE_UNIFIED_MEMORY at all.
This env var switches ggml from hipMalloc to hipMallocManaged. We needed it originally because hipMalloc was capped at the 24 GB GTT ceiling. But with the GRUB params applied and GTT at 124 GB, hipMalloc now allocates from the full pool without any issues. No managed memory, no coarse-grain advisory, no SIGSEGV. Just a regular GPU allocation that fits.
Remove one environment variable. Everything works.
Here's what the running system looks like during a 1024×1024 Flux2-dev generation:
total params memory size = 56748 MB (VRAM 56748 MB, RAM 0 MB):
text_encoders 23206 MB (VRAM)
diffusion_model 33381 MB (VRAM)
vae 160 MB (VRAM)
That's 57 GB just for the model weights, all resident in GPU memory. The attention compute buffers for 1024×1024 (4096 patches, quadratic attention) push the total memory usage to roughly 70 GB during inference.
On a discrete GPU — even a flagship RTX 4090 with 24 GB VRAM — this model doesn't fit. The entire class of consumer GPU hardware is below the threshold. The Ryzen AI Max 395+ with its unified 128 GB pool runs it fully resident, with zero swapping, zero quality compromise.
The difference in output quality is visible. When you have to offload attention tensors to CPU or truncate context, the model has to approximate. When everything fits in one contiguous pool and the GPU can address all of it directly, the model computes exactly what it was trained to compute. At 1 MP with full-precision attention across 4096 patches, the results look like it.
| Layer | Root Cause | Fix |
|---|---|---|
| Vulkan heap | Mesa RADV reports ~4 GB device-local heap on gfx1151 APU | Switch to ROCm/HIP |
| GTT ceiling | IOMMU enforces carveout boundary; GTT pool = 24 GB | amd_iommu=off amdgpu.gttsize=126976 ttm.pages_limit=32505856 |
| hipMallocManaged + CoarseGrain | gfx1100 ISA mapping causes hipMemAdviseSetCoarseGrain to succeed, making compute buffers CPU-inaccessible → SIGSEGV | Remove GGML_CUDA_ENABLE_UNIFIED_MEMORY; let regular hipMalloc use the expanded GTT pool |
| Container OOMKill | hipMallocManaged maps into process address space, cgroup counts it against the memory limit | Remove container memory limit in K8s manifest |
A few other things that helped or were necessary to rule out:
linux-firmware-20251125 critically breaks ROCm on Strix Halo. Stable is 20260110 or later. We were on 20260519 — fine.GGML_HIP_UMA cmake flag: exists in llama.cpp's ggml fork but not in stable-diffusion.cpp's vendored ggml. Compiling with it does nothing — it's silently ignored.HSA_XNACK=1: Irrelevant here. XNACK is page-fault retry on discrete GPUs. Not the problem.--backend cpu as a test: Confirms crash is GPU-path specific, but uses 57 GB of system RAM. Dangerous on a machine with 32 GB of CPU-visible RAM.None of the GRUB parameter combination was something I would have found without the Strix Halo homelab Discord and kyuz0/amd-strix-halo-toolboxes. That community has done the work of documenting what the BIOS settings should be, what kernel flags are required, and which firmware versions to avoid. If you're running AMD Strix Halo for AI inference, start there.
The inference stack described here is loch-nessh + stable-diffusion.cpp, running on K3s in the cortex namespace. All model traffic routes through loch-nessh for VRAM-aware scheduling and GPU lock serialization.