ROCM / COMFYUI

ComfyUI Black Images on AMD ROCm: The bf16 Bug and How to Fix It

Your AMD GPU generates black images. The sampler runs, the progress bar completes, the VAE decodes — and what lands in SaveImage is a featureless black rectangle. If your console shows RuntimeWarning: invalid value encountered in cast, the culprit almost certainly isn't your model, your prompt, or your workflow. It's bfloat16.

The symptom

This shows up in a very specific and very frustrating way. Everything looks like it worked. There is no traceback, no failed node, no red border in the graph. ComfyUI happily reports a finished job. But the output is black, or occasionally noise, and the only clue anywhere in the log is a single warning from numpy:

RuntimeWarning: invalid value encountered in cast

The result is black images or NaN values propagating through the pipeline. Because the warning is non-fatal, most people never connect it to the black output at all — they go and try a different checkpoint, a different sampler, a different scheduler, and burn an afternoon on it.

The cause: numpy does not support bfloat16

This is the elephant in the room, and it is worth stating plainly because it explains why the failure is silent.

numpy does not support bfloat16. When PyTorch tensors in bf16 format get converted to numpy arrays for image saving, the cast fails silently and produces garbage. There is no exception to catch. The bytes come out the other side and get written into a PNG as pixel values, and those values are meaningless.

AMD Strix Halo APUs — and potentially other ROCm devices — have issues with bfloat16 precision, so bf16 tensors reaching the numpy boundary is a much more common event on ROCm than it is elsewhere. But the underlying numpy gap is not an AMD problem. It affects everyone: AMD ROCm, older NVIDIA cards, Apple Silicon, and any unified memory architecture where bf16 is common. If numpy added native bf16 support, a significant portion of these "black image" issues across the ML ecosystem would simply disappear. Until then, the fix is to make sure nothing in bf16 ever reaches that cast.

Why "just switch to fp32" isn't the whole answer

You can force the entire pipeline to float32 and the black images will usually go away — but you have then given up the memory headroom that made large diffusion models runnable on the hardware in the first place, and you still don't know which stage was actually broken. There are three distinct failure modes that all present as a black image:

The VAE decodes correctly but the bf16→numpy cast destroys the result on the way out.
The diffusion model produces NaN or an all-zero latent, and the VAE is faithfully decoding garbage.
The text encoder produces NaN conditioning, so the sampler had nothing coherent to work with from step one.

Only the first one is the numpy bug. The other two need different fixes. So the productive order of operations is: find out where the NaN first appears, then apply the narrowest fix that addresses it.

The diagnostic tools

The HALO Debug Pack is a small set of ComfyUI custom nodes built for exactly this triage. It contains two things: debug nodes to identify exactly where NaN/precision issues occur in your workflow, and FP32 VAE nodes to force VAE encode/decode to use float32, which fixes the numpy bf16 bug.

Install it into your ComfyUI custom nodes directory:

cd /path/to/ComfyUI/custom_nodes

git clone https://github.com/bkpaine1/halo_pack.git

Restart ComfyUI, and the nodes appear under the HALO category.

Five nodes ship in the pack:

HALO VAE Decode (FP32) — Force VAE decode to FP32. Fixes the numpy bf16 conversion bug.
HALO VAE Encode (FP32) — Force VAE encode to FP32. Use for img2img workflows.
HALO Latent Debug — Check sampler output for NaN/dead latents. Put between sampler and VAE.
HALO Conditioning Debug — Check text encoder output for NaN. Put after CLIP Text Encode.
HALO Model Debug — Check model dtype (bf16/fp16/fp32). Put after model loader.

Wiring up a diagnostic pass

Drop the debug nodes inline at each boundary. They are pass-through — they return their input unchanged and only print to the console, so you can leave them in a working graph without altering results:

CLIPLoader → CLIPTextEncode → HALO Conditioning Debug
                                      ↓
UNETLoader → Sampler → HALO Latent Debug → HALO VAE Decode → SaveImage

A healthy latent reads like this in the console:

============================================================
  HALO LATENT DEBUG
============================================================
  dtype:  torch.float32
  shape:  torch.Size([1, 16, 64, 64])
  device: cuda:0
  min:    -4.234521
  max:    3.891234
  mean:   0.012345
  std:    1.234567

  ✓ Latent looks valid!
============================================================

And a healthy FP32 VAE decode, showing the dtype being forced before the numpy boundary:

[HALO-VAE] === DECODE START ===
[HALO-VAE] Latent: dtype=torch.float32, shape=torch.Size([1, 16, 64, 64])
[HALO-VAE] Latent stats: min=-3.2451, max=4.1234
[HALO-VAE] Forced VAE model to fp32 (was torch.bfloat16)
[HALO-VAE] Output stats: min=0.0000, max=1.0000
[HALO-VAE] ✓ Output looks valid!
[HALO-VAE] === DECODE COMPLETE ===

Reading the failure output

Three console messages tell you which of the three failure modes you're in.

If the sampler produced nothing at all, the VAE node says so before it even tries to decode:

[HALO-VAE] ⚠️ LATENT IS DEAD (all zeros) - problem is upstream!

If the diffusion model is producing NaN, the latent debug node counts them:

  ❌ 1234 NaN values!
      Model is producing garbage - precision issue.

And if the problem started even earlier, at text encoding:

    ❌ 1234 NaN values in conditioning!
       TEXT ENCODER is outputting garbage!

The HALO Model Debug node adds the other half of the picture by printing the loaded model's dtype and device. If it reports bf16, it flags it as a likely cause on ROCm and suggests loading with weight_dtype=fp32 or using --force-fp32.

The fix, in order

If you're getting black images from a bf16 VAE, start narrow:

Replace VAE Decode with HALO VAE Decode (FP32).
If you're still getting black or NaN output, add HALO Latent Debug before the VAE to check whether the problem is upstream.
Add HALO Conditioning Debug after text encode to check conditioning.
Add HALO Model Debug after the model loader to see the dtype.

Interpreting the results: if NaN appears in the latent, the problem is in the diffusion model — try the --force-fp32 flag. If NaN appears in the conditioning, the problem is in the text encoder. The HALO VAE nodes fix the specific bf16→numpy issue, but they cannot fix upstream NaN; nothing downstream can rescue a latent that was already garbage when it arrived.

Internally, the FP32 decode node saves the VAE's original dtype, casts vae.first_stage_model to torch.float32, converts the incoming latent to fp32, decodes under torch.no_grad(), and restores the original dtype in a finally block so the rest of your session is unaffected. Before returning, it explicitly ensures the pixel tensor is torch.float32 — which is the actual point of the whole exercise, since that's the tensor numpy is about to touch.

Tested on

The pack was developed and tested on an AMD Strix Halo APU (Ryzen AI Max+ 395 w/ Radeon 8060S) with 128GB Unified Memory, running ROCm 6.x on Ubuntu Linux. It was written while debugging Z-Image on Strix Halo hardware. The name comes from AMD Strix Halo — the first APU with enough unified memory to run large diffusion models locally, which is precisely why so many people are hitting bf16 edge cases on it now.

It is MIT licensed. If your symptoms match — black output, no traceback, and an invalid value encountered in cast warning buried in the log — the FP32 VAE decode node is a two-minute test that will tell you a great deal.

Source and full code: github.com/bkpaine1/halo_pack