Building a Local Jarvis Across Three Boxes I Already Owned

Every home-lab project starts the same way for me: a reasonable-sounding idea that quietly metastasizes into a hardware audit. This one began as “can I run a local model that generates images and music, plus a thinking model to act as an assistant?” and ended with me flattening three machines and redrawing how they talk to each other. That’s the natural life cycle of these things.

Regular readers will recognize some of this hardware. My earlier headless inference lab ran everything on a single Dell Precision with an Oculink-attached RTX 3090. That 3090 is gone — I sold it to fund a pair of Radeons — which forced a rethink. The result is better: three machines, each doing the one thing it’s actually good at.

The three boxes

skuld — AM5, Radeon RX 7900 XT (20GB) and RX 7900 XTX (24GB), 32GB DDR5, NVMe, Ubuntu Server LTS. This is my old gaming machine, repurposed as the GPU compute node. Two gfx1100 cards, 44GB of VRAM.

precision — the Dell Precision from the last build, dual Xeon, 60GB RAM, three NVMe drives on NVMe2PCI adapters in a ZFS pool (fastpool). No GPU anymore. It stops doing inference and becomes the storage and data anchor.

wyrd — a Dell OptiPlex 5070 SFF, i7-8700T (6C/12T, 35W, AVX2), 32GB DDR4. Low-power, always-on. The orchestration and voice layer.

What “one or two models” actually means

I went in thinking I wanted “a couple of models.” That framing is wrong, and it’s worth saying why, because it’s the mistake that shapes everything downstream.

An LLM, an image generator, and a music generator are three different runtimes, not three model files you load into one program. llama.cpp runs the LLM. It does not run diffusion or audio models — those are PyTorch + ROCm stacks, which in practice means ComfyUI. So the real inventory is llama-server for the thinking model, ComfyUI for image generation, and ComfyUI again for music via ACE-Step 1.5 (MIT-licensed, official ROCm support, slots in as a custom node). Image and music share the ComfyUI backend; the LLM is its own thing. So it’s really two runtimes, and they want to live on different cards.

Carving up skuld’s GPUs

The 20/24 asymmetry turns out to be a gift. Instead of splitting one big model across both cards — which costs PCIe bandwidth on every token and eats system RAM during load — I pin each card to a job:

• XTX (24GB) runs the LLM. A 32B at Q4/Q5 fits comfortably and stays resident. I gave up the option of a split 70B to get this, and it’s the right trade: a warm, instant assistant matters more than a bigger model that’s slower and blocks the second card.
• XT (20GB) runs ComfyUI. Image and music models load and unload here via ROCR_VISIBLE_DEVICES in the service environment, never touching the LLM.

No cross-card splitting, no contention, each runtime has a card to itself. On a single-user box you’re almost never generating music and chatting in the same instant anyway.

A note on the AMD situation, since last time I looked into this it was grim: ROCm tooling shipped real RDNA3 support recently, and gfx1100 is natively supported now. If you find yourself reaching for HSA_OVERRIDE_GFX_VERSION, that’s a sign something else is wrong — you shouldn’t need it on a 7900-series card anymore. Anything you read from 2025 about xformers blocking consumer Radeon cards is stale.

Confirming the cards enumerate correctly is one command:

$ rocm_agent_enumerator
gfx1100
gfx1100
gfx1036

Two gfx1100 is exactly right — the XT and the XTX seen independently. The gfx1036 is the CPU’s integrated Raphael graphics; harmless, but some frameworks will happily assign it device 0, so pin the discrete cards explicitly.

The RAM wall, and why precision solves it

Here’s the constraint that shaped the whole design. skuld has 32GB of system RAM against 44GB of VRAM — an awkward ratio. The reason I can’t just add DDR5 is that DRAM prices are absurd right now, and I’m not paying current prices to solve a problem I can solve for free.

And it is solvable, because the RAM pressure was never really about VRAM. llama.cpp mmaps its GGUF, so most of what it “uses” is evictable page cache, not a hard allocation. The genuinely hard allocation is ComfyUI’s PyTorch process during model load and VAE decode. What 32GB costs you is margin: cold loads re-read from NVMe because nothing stays cached.

This is exactly where precision earns its keep. A dual-Xeon box with 60GB of RAM and a three-NVMe ZFS pool is a far better home for the models than skuld’s local disk:

• One canonical model library. The Unsloth dynamic GGUFs, SDXL checkpoints, ACE-Step weights, LoRAs — all live once on fastpool instead of scattered across three machines. ZFS gives you checksumming and snapshots on the collection, which matters when a 40GB download corrupts silently.
• The 60GB of Xeon RAM becomes ARC — ZFS’s cache. Frequently-loaded models get served from precision’s RAM over the network, which for skuld can be faster than its own cold NVMe read. I’ve effectively relocated the page cache skuld doesn’t have room for onto a box with plenty.
• The RAG vector DB and document corpus live here too, snapshotted and safe, not on wyrd’s local disk.

No GPU needed; this box does storage and CPU-side data work.

wyrd: the cheap box that runs the show

The OptiPlex can’t do inference of any consequence — its 200W PSU caps you at 75W slot-powered low-profile GPUs — but that’s not what I need from it. What a cheap, always-on, low-power box is good for is orchestration. Open WebUI (the frontend everything points at), Whisper for speech-to-text, TTS, Home Assistant, and whatever glue accretes over time. All CPU/RAM work, none of it competing with the GPUs.

The i7-8700T being the 35W “T” part is a feature here, not a compromise. The services are bursty, not pegged flat, and for a 24/7 box low heat and low idle draw are exactly right. It has AVX2, so a base/small Whisper model transcribes faster than real-time.

Why three boxes beats one

Beyond the RAM math, separating the layers buys resilience the all-on-one-box version never had. I can reboot skuld to swap models or fight with ROCm without taking down Open WebUI, my chat history, the vector DB, or Home Assistant — those live on wyrd and precision and just wait for the backend to return. On machines I’m actively tinkering with, that isolation is worth a lot.

The assistant loop ends up clean: talk to wyrd, Whisper transcribes locally, Open WebUI routes the text to llama-server on skuld, skuld pulls the model from fastpool on precision (ARC-cached, often straight from Xeon RAM), the reply comes back, and TTS on wyrd speaks it. Image and music requests route from Open WebUI to ComfyUI on skuld’s XT. Every model file lives once, on precision.

Wiring notes

A few things that matter when you connect it up:

• llama-server and ComfyUI bind to 0.0.0.0, or wyrd can’t reach them. Default loopback binding means “this host only,” which isn’t what you want across three machines.
• Open WebUI is just a frontend. It needs no GPU passthrough, no /dev/kfd, no video group. It talks HTTP to the backends and mounts fastpool over NFS for its data.
• Skip Docker for the ROCm backends. Device passthrough into containers is the fiddly part; running llama-server and ComfyUI on the host keeps ROCm out of containers entirely. Containerize the stateless frontend, not the GPU work.
• fastpool exports over NFS to both skuld and wyrd, so there’s exactly one copy of every model and the ARC cache does real work.

The parts list, and the non-purchase

The satisfying thing about this build is the bill of materials:

• Old gaming PC (skuld) → already owned
• Dell Precision (precision) → already owned, GPU sold off to fund the Radeons
• Forgotten OptiPlex (wyrd) → already owned
• The DDR5 kit I decided not to buy → $0

The best upgrade in this whole project was the one I talked myself out of. When DRAM prices come back to earth I’ll put skuld back to 64GB dual-channel and the margin gets comfortable instead of merely sufficient. Until then, the architecture — GPUs do GPU work, Xeons hold the data, the cheap box runs the plumbing — means I don’t need to.

That’s the lesson I keep relearning in the home lab: the constraint is rarely the thing you think you need to buy. It’s usually the way you’ve arranged what you already have.

Jeff