COMFYUI / AMD

ComfyUI Lip Sync on AMD: No Subprocesses, No Ghost Files

Most ComfyUI lip sync nodes were written against an NVIDIA workstation and a scratch directory. On an AMD APU with unified memory, both of those assumptions turn into crashes. Halo-Lipsy is a Wav2Lip node that runs entirely inside the ComfyUI process — no shelling out, no temp files, tensors in and tensors out.

The failure mode nobody writes down

If you have tried to get lip sync working in ComfyUI on a Ryzen AI Max+ 395 or any other ROCm box, you have probably seen the same sequence: the node installs cleanly, the graph validates, and then it dies somewhere in the middle of execution with an error that has nothing to do with lip sync. The reason is architectural rather than model-specific. Halo-Lipsy's README lists the four ways the older nodes break:

Why subprocesses are the root of it

The original Wav2Lip release is a command-line tool. The path of least resistance when wrapping it as a ComfyUI node is to keep it a command-line tool: dump the frames to disk, dump the audio to disk, call the inference script with subprocess.run(), then read the resulting video back in. That works on a machine where the system Python and the ComfyUI Python are effectively the same environment.

They are not the same environment on a ROCm install. ComfyUI runs from a virtualenv with a ROCm-flavored PyTorch build; the interpreter the subprocess reaches for may resolve a different torch, a different NumPy ABI, or no GPU runtime at all. The child process is not a stack frame you can debug — it is a separate program with its own import graph, and the exception you get back is whatever it happened to print before it exited.

The disk round-trip is the second half of the problem. Every intermediate written to a temp directory is a file that has to be created, flushed, closed, and re-read — and if the writing process dies partway, you are left with a zero-byte artifact that the next stage happily opens and fails on. That is the "ghost file": OpenCV returns nothing readable, and the traceback points at your image loader instead of at the process that never finished writing. Orphaned children and a slowly filling temp directory are the ambient tax on top.

The in-process design

Halo-Lipsy's fix is to stop leaving the process at all. The README states the approach plainly:

The Wav2Lip architecture is reimplemented as plain torch.nn modules inside halo_lipsync.py, so loading a checkpoint is a torch.load and a load_state_dict rather than a shell invocation. The node's inputs are ComfyUI's native IMAGE and AUDIO types and its outputs are ("IMAGE", "AUDIO") — the frames go straight to VHS Video Combine or any other IMAGE consumer without ever becoming a file.

The .float().cpu().numpy() rule is worth dwelling on, because it is the one-line answer to the half-precision failure. On a unified-memory APU the GPU allocation is carved out of the same physical RAM as the host, and a half-precision tensor handed to NumPy is a type NumPy has no business touching. Casting to float32 and explicitly landing on CPU before the NumPy conversion removes the ambiguity everywhere it occurs — the audio waveform, the incoming image batch, and the model's predictions on the way back out.

Model instances are cached in a class-level dictionary keyed by checkpoint path, device, and precision, so a second run of the graph reuses the loaded weights instead of re-reading the .pth. FP16 is enabled when a GPU is in play and skipped under force_cpu.

Installing it

Via ComfyUI Manager, search for "Halo-Lipsy" and click Install. Manually:

cd ComfyUI/custom_nodes
git clone https://github.com/YOUR_USERNAME/Halo-Lipsy.git
cd Halo-Lipsy
pip install -r requirements.txt

The dependency list is deliberately ordinary — librosa, opencv-python, scipy, numpy, torchaudio, tqdm, and mediapipe. Nothing that assumes a CUDA toolchain.

Then fetch wav2lip_gan.pth from the original Wav2Lip repo and put it in Halo-Lipsy/checkpoints/wav2lip_gan.pth (recommended) or ComfyUI/models/wav2lip/wav2lip_gan.pth. The node auto-detects either location; if it finds neither it raises a FileNotFoundError that tells you exactly which directories it searched instead of failing deeper in.

In the graph, find Halo-Lipsy in the node menu under the category Halo-Lipsy, wire your video frames into images and your audio into audio, and send the output to VHS Video Combine.

Face detection on the CPU, on purpose

Putting the detector on the GPU alongside Wav2Lip is the obvious optimization and the wrong one on unified memory — two allocators competing for the same pool while a video model may already be resident. Halo-Lipsy pins detection to CPU and tries three backends in order of quality: S3FD first, then MediaPipe (which the source notes works well on AI-generated faces), then OpenCV Haar cascades as a last resort, logged as "least reliable." Whichever it picks, it prints the choice at load time.

Detected boxes are smoothed across frames with an adaptive window that shrinks or disables itself when the face is actually moving, so tracking stays stable on a still head without smearing on a moving one. Frames with no detection inherit the nearest valid box rather than dropping out. Silent audio short-circuits the whole pass and returns the original video untouched, and if no face is found in any frame the node says so and returns the input rather than producing garbage.

Tuning

The controls that matter most in practice are sync and mouth speed. sync_offset shifts the audio by up to ±10 frames — set it to -2 or -3 if the lips lag the audio, +2 or +3 if they lead it. mel_step_multiplier scales mouth speed: 1.1 if the mouth is too slow, 0.9 if it is too fast. Set fps to match your source video, because mel chunking is derived from it.

For throughput, inference_batch (default 64) governs the GPU pass and face_detect_batch (default 4) the CPU pass; if you run out of VRAM, lower inference_batch or enable force_cpu, which runs everything on CPU with zero VRAM. Jittery face boxes call for a larger smooth_box_frames. Composite quality is handled by a tight lips-only gradient mask with color matching against the original face and a mild unsharp pass to recover the detail lost coming back up from Wav2Lip's 96×96 working resolution — so only the mouth region is replaced and the rest of the frame stays exactly as you rendered it.

What it was tested on

The README reports a Ryzen AI Max+ 395 (Strix Halo) with 128GB unified memory split 64/64 RAM/VRAM, ROCm 7.11, and ComfyUI with HunyuanVideo loaded taking 15GB+ — which is the interesting part, since the whole point of not fighting over memory is being able to keep a video model resident while you sync. It should work on any AMD APU or GPU with ROCm, and on NVIDIA too. MIT licensed.

The general lesson outlives this one node: if a ComfyUI extension shells out to a CLI or stages work through a temp directory, it has taken on a second environment and a second failure surface that the graph engine cannot see. Keeping inference in-process is not just tidier — on unified memory it is the difference between working and not.

Source and full code: github.com/bkpaine1/Halo-Lipsy