
Every architecture diagram survives until you actually build it. Last post I drew the topology for the three-box lab: the XTX holds the resident LLM, the 7900 XT on OCuLink handles image and music generation, everything stays out of everything else’s VRAM. Clean lines on the diagram. This post is what happened when I sat down to stand up the Stable Diffusion half of that on skuld, which is now running Ubuntu 26.04 LTS — and 26.04 turns out to be just new enough to break things in interesting ways.
Spoiler: it works, the XT does exactly what the diagram promised, and the reason it took an evening instead of an hour is a lesson worth writing down.
The kernel is newer than AMD’s driver
First move on any ROCm build:
sudo amdgpu-install --usecase=rocm,graphics
That died in dpkg configuring amdgpu-dkms. The DKMS make.log had the actual story:
error: conflicting types for 'zone_device_page_init'; have 'void(struct page *)'
note: previous declaration of 'zone_device_page_init' with type
'void(struct page *, struct dev_pagemap *, unsigned int)'
Ubuntu 26.04 ships kernel 7.0.x and GCC 15. AMD’s out-of-tree DKMS module carries a kernel-compat shim layer (amdkcl) that still declares the old one-argument zone_device_page_init, and kernel 7.0 changed the signature. The module hasn’t caught up to the API churn. This is a known open issue upstream — reinstalling headers won’t fix it, and neither will anything else you do locally.
The fix is to stop needing the module. The amdgpu driver has been in the mainline kernel for years, and gfx1100 is a first-class citizen in the inbox driver 26.04 already ships. The DKMS package only exists to give you a newer driver than your kernel carries, and on a kernel this new, there’s nothing newer to want:
sudo apt remove --purge amdgpu-dkms amdgpu-dkms-firmware
sudo dpkg --configure -a
sudo apt --fix-broken install
sudo amdgpu-install -y --usecase=rocm --no-dkms
sudo reboot
ROCm 7.2.4 userspace on the inbox kernel driver. The bonus nobody advertises: an in-tree driver can never version-drift from the kernel it ships inside. One less moving part, permanently.
Verification looks exactly like it did in the last post — rocm_agent_enumerator shows two gfx1100 (the XT and XTX are the same Navi 31 die, identical ISA string) plus the gfx1036 Raphael iGPU. rocm-smi --showproductname is what actually tells the discrete cards apart: GPU[0] was the XTX, GPU[1] the XT. Write those indices down. They run this whole story.
Ubuntu 26.04’s Python is too new for the entire ecosystem
Here’s the wall every 26.04 user doing AI work is going to hit: the distro ships Python 3.14. Only 3.14. There is no python3.12 package in the archive at all — no, you’re not missing a repo — and essentially nothing in the Stable Diffusion world supports 3.14 yet.
The clean answer is uv, which manages standalone CPython builds without touching the OS:
uv python install 3.12
ln -sf "$(uv python find 3.12)" ~/.local/bin/python3.12
Two traps in this neighborhood, both of which got me:
- uv venvs ship without a
pipbinary. Activate one, typepip install, and the command silently falls through to system Python 3.14 — which slams into PEP 668‘sexternally-managed-environmentrefusal — the same wall that’s been breaking people’s setups since 24.04. The error namespython3.14in its path, which is your tell. Inside uv venvs, it’suv pip install, full stop. - Do not reach for
--break-system-packages. It’s the wrong answer every time. It doesn’t solve the problem; it relocates the mess into your OS.
The Forge dead end
I wanted a simple form-based UI rather than ComfyUI‘s node graph, so I tried Stable Diffusion WebUI Forge first. Its requirements pin scikit-image==0.21.0 — a mid-2023 release with prebuilt wheels only up to Python 3.11. On 3.12 it builds from source, its bundled pythran headers use C++17 constructs like std::is_integral_v, the build passes -std=c++14, and GCC 15 — unlike its more forgiving ancestors — refuses. A few hundred lines of template errors later, the message was clear.
You can dodge it by dropping to Python 3.11. I chose not to, because the failure told me something more useful than the workaround: Forge’s dependency pins are two years behind my toolchain, and this would not be the last collision. On a bleeding-edge distro, run actively maintained software. ComfyUI had installed clean on this same box an hour earlier…
SwarmUI: form UI on top, maintained engine underneath
SwarmUI is the answer to wanting prompt boxes and sliders without giving up ComfyUI: a .NET web frontend that installs and manages ComfyUI as its backend. Notes from installing it on 26.04:
- The archive only has
dotnet-sdk-10.0and Swarm targets net8.0. Doesn’t matter — .NET rolled forward and the build succeeded first try. - Swarm’s installer hard-requires Python 3.11/3.12 on the PATH. The uv-managed 3.12 plus the
~/.local/binsymlink satisfied it. - Its ComfyUI installer pulls PyTorch from the
rocm7.1nightly index rather than detecting my 7.2.4. That’s fine: torch’s ROCm wheels bundle their own runtime libraries and share only the kernel driver with the system, and the user/kernel compatibility window is wide.
Install clean, models downloaded, backend up, first image generated. On the XTX. The one card in this house that already had a job.
The visibility war
This is the part that earned the post.
ROCm gives you three overlapping ways to control which GPUs a process sees: ROCR_VISIBLE_DEVICES (filters at the ROCr runtime, lowest level), HIP_VISIBLE_DEVICES (the HIP layer), and CUDA_VISIBLE_DEVICES (honored for CUDA-compat). Any one of them does the job. Two at once is where the pain lives.
I put HIP_VISIBLE_DEVICES=1 in Swarm’s systemd unit — the exact mechanism that pins Ollama to the XTX today, and the one that worked every time I launched ComfyUI by hand. Under Swarm, the backend died on startup with RuntimeError: No CUDA GPUs are available. Because Swarm also sets CUDA_VISIBLE_DEVICES itself, driven by its per-backend GPU_ID setting. ROCm found itself holding two conflicting device lists and said so, in the single most useful log line of the night:
W agent.cpp:554] Conflicting visibility of agent-1 between HIP_VISIBLE_DEVICES
and CUDA_VISIBLE_DEVICES. Assuming HIP_VISIBLE_DEVICES supersedes
W agent.cpp:461] Attempt to enable hip visibility for agent-2 which is not
visible to HSA (ROCR)
The intersection of the two filters was the empty set. Zero visible GPUs, dead backend. I burned several rounds trying to make the two mechanisms agree before the obvious observation ended it: standalone ComfyUI, launched by hand with one variable and a clean environment, came up correctly every single time:
Total VRAM 20464 MB
Device: cuda:0 Radeon RX 7900 XT : native
The backend was never broken. The launcher was the variable. So take the launcher out of the launch path.
(Related housekeeping from the same evening: an old config on this box was still exporting HSA_OVERRIDE_GFX_VERSION=11.0.0. As I said last post — if you’re reaching for that override on a 7900-series card, something else is wrong. It got deleted. Nothing noticed.)
The fix: own the backend yourself
The working architecture is a clean split of responsibilities. ComfyUI runs as its own systemd service with exactly one visibility variable and nothing to argue with it. SwarmUI stops self-starting anything and connects to that backend over its API.
One flag matters that cost me a confused ten minutes: Swarm keeps models under ~/SwarmUI/Models/ and maps them into ComfyUI with --extra-model-paths-config. Launch ComfyUI without it and your model list is empty. Substitute your own username in the paths before applying this:
# /etc/systemd/system/comfyui.service
[Unit]
Description=ComfyUI (7900 XT)
After=network-online.target
[Service]
User=jeff
WorkingDirectory=/home/jeff/SwarmUI/dlbackend/ComfyUI
Environment=HIP_VISIBLE_DEVICES=1
ExecStart=/home/jeff/SwarmUI/dlbackend/ComfyUI/venv/bin/python main.py \
--extra-model-paths-config /home/jeff/SwarmUI/Data/comfy-auto-model.yaml \
--listen 127.0.0.1 --port 7821
Restart=on-failure
[Install]
WantedBy=multi-user.target
Swarm gets its own unit — the PATH entry is load-bearing, since systemd’s minimal default PATH doesn’t include ~/.local/bin where the Python 3.12 symlink lives:
# /etc/systemd/system/swarmui.service
[Unit]
Description=SwarmUI
After=network-online.target
Wants=network-online.target
[Service]
User=jeff
WorkingDirectory=/home/jeff/SwarmUI
Environment=PATH=/home/jeff/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ExecStart=/home/jeff/SwarmUI/launch-linux.sh --launch_mode none --host 0.0.0.0 --port 7801
Restart=on-failure
RestartSec=10
[Install]
WantedBy=multi-user.target
Then in Swarm’s UI: Server → Backends, disable the “ComfyUI Self-Starting” backend, add a ComfyUI API By URL backend pointed at http://127.0.0.1:7821. It goes green in seconds, because the backend it’s looking for is already up and already on the right card.
The verification I actually trust: journalctl -u comfyui shows the XT banner above, and watch -n1 rocm-smi during a generation shows GPU[1] spiking while GPU[0] sits untouched, LLM still resident. Ollama keeps ROCR_VISIBLE_DEVICES=0 in its unit and the XTX to itself. Two cards, two workloads, zero contention — the diagram from last post, now with a service behind every box.
The one trade: Swarm no longer manages the backend’s lifecycle, so ComfyUI updates are mine — an occasional git pull in the backend directory and a service restart. I’ll take a deterministic GPU pin over automated updates every day of the week.
What I’d tell you at hour zero
- Read the log, not the summary. The DKMS failure, the Forge explosion, and the visibility conflict were each fully explained in their own output. Every wrong turn I took came from acting on an assumption; every right turn came from a log line.
- Era-match your software to your distro. 26.04’s kernel 7.0, GCC 15, and Python 3.14 are ahead of half the AI ecosystem’s dependency pins. Actively maintained projects sailed through. Projects pinned to 2023 are a compiler upgrade away from detonating.
- Pick exactly one GPU visibility mechanism per process.
ROCR_,HIP_, orCUDA_VISIBLE_DEVICES— any of them alone is fine. Stack two and the intersection can silently be nothing. If an app manages one internally, either use only the app’s knob or take the app out of the launch path. - Works standalone, fails under the orchestrator? Debug the orchestrator. The most clarifying test of the night was launching the backend by hand with a clean environment. The moment it printed the right card, the search space collapsed to “whatever the launcher does differently.”
--no-dkmsisn’t a compromise on RDNA3. The inbox driver is mature, and it can’t drift from its own kernel.
That’s the recurring lesson with this lab: the hard part is almost never the workload. The models ran fine the moment they had a card. The evening went to the plumbing between a process and its GPU — and the fix, as usual, was fewer layers, not more.
Jeff
Leave a Reply