A black workstation chassis on the left with a glowing red GPU inside; an amber racing-car silhouette sits on a clean black track to the right. A red axis with a thin triangular arrowhead connects them. Above, a smaller simulator screen version of the same car floats inside a broken cream frame to show the abandoned virtualised path. Warm cream canvas.
[ Virtualisation ]7 min read

Bare Metal vs. Virtual Machines: The AI Dilemma

Phase 3 · The AI Behemoth

Two weeks of GPU passthrough failures, and the moment I accepted that the hypervisor was the wrong tool for the job.

Proxmox VE 8.1IOMMUVFIOUbuntu 22.04 Server

By the time the new AI workstation was on the bench with the GPU driver loaded and nvidia-smi happy, I had a decision to make that was not obvious until I had made the wrong choice once. The whole lab to that point ran on Proxmox. Virtualisation had paid off a dozen times over: clean rollbacks, declarative machines, fault isolation. The instinct was to install Proxmox on the new box, hand the GPU through to a VM, and keep the architecture uniform. That instinct was wrong, and it cost two weekends to prove.

GPU passthrough from a hypervisor is a real, supported configuration. It works beautifully when the hardware and firmware cooperate, and the documentation on it is extensive. What the docs understate is the fragility: a BIOS update rearranges IOMMU groups and your carefully isolated PCIe device now shares a group with the USB controller you need for the keyboard; a kernel upgrade changes the vfio-pci binding order; an Nvidia driver point-release changes the minimum QEMU version. The setup is correct on Tuesday and broken on Thursday, and the only signal you get is a VM that boots to a black screen.

At some point the economics change. The effort to keep the hypervisor-plus-passthrough path working is not zero. The flexibility it delivers is real but finite. For a node whose only job is to load models and run inference, the question is not “can I virtualise this?” but “what am I actually buying by virtualising this?” I wrote the costs and the benefits on a sticky note on the monitor. The note settled the argument.

The Analogy

A Formula One race team would not put their car inside a simulator for race day. The simulator is invaluable for training: safe, resettable, cheap, able to reproduce any track. But on race day, the car has to be on the tarmac. The simulator adds a millisecond of input lag that does not matter to a commuter driver and absolutely matters to a machine designed to scream through a turn at 320 km/h.

A virtual machine is a simulator. The inference GPU is the race car. For most of the lab, the simulator is the right tool: ten VMs running web services, a wiki, a backup appliance, a small personal CI runner. For the one node whose entire purpose is to get every cycle out of a piece of silicon, the simulator is a liability.

The Tech Translated

GPU passthrough works by binding the physical card to the VFIO driver on the host instead of to the host’s own Nvidia driver. The host treats the card as an opaque device; the hypervisor (QEMU/KVM on Proxmox) maps it into the guest, where the guest installs its own Nvidia driver and uses the card directly. The process depends on four things being simultaneously correct:

  1. IOMMU enabled in BIOS, and the CPU/chipset support IOMMU groups fine-grained enough that the GPU does not share a group with other devices you need.
  2. The host does not touch the GPU first. If the host’s kernel probes the card on boot, the Nvidia host driver claims it and vfio-pci finds nothing to bind. You solve this with early-boot kernel parameters (vfio-pci.ids=) and a modprobe blacklist for the Nvidia modules on the host.
  3. The guest sees a “whole” device. For Nvidia consumer cards in particular, you may need the kvm=off hypervisor flag in the VM config because consumer drivers historically refused to run under a detected hypervisor.
  4. The firmware is cooperative. BIOS options like Above 4G Decoding, Resizable BAR, and Secure Boot each have a story, and changes to any of them can break a working passthrough overnight.

A single missed detail produces the failure mode I spent two weekends in: the VM starts, the Nvidia driver loads inside the guest, and the whole guest freezes the first time a CUDA workload tries to touch the card. No useful log. No clean failure. Just silence.

The honest summary is that GPU passthrough is a supported configuration, not a robust one, on consumer hardware. Enterprise cards (NVIDIA A-series, H-series) are engineered for this path and Just Work. Consumer cards are not, and treating them like they are is how weekends disappear.

The Action

I wiped the hypervisor install. On the same hardware, I installed Ubuntu 22.04 Server directly, installed the Nvidia proprietary driver, and put the Ollama binary on it. Every line of the install was prosaic. Every piece of the passthrough headache vanished.

# Fresh Ubuntu 22.04 Server on the AI node.
sudo apt update && sudo apt -y full-upgrade

# Nvidia driver, matching the recommended branch for the card.
sudo ubuntu-drivers autoinstall
sudo reboot

# Verify the card is usable from the host.
nvidia-smi

# CUDA toolkit — only needed for building from source; skip if you
# only plan to run prebuilt inference engines like Ollama or llama.cpp.
sudo apt -y install nvidia-cuda-toolkit

# Lock the driver version so a surprise kernel upgrade does not
# break the module this weekend.
sudo apt-mark hold nvidia-driver-550 nvidia-dkms-550

Four commands. The card is visible to the host. Any inference engine on the system talks to it with zero overhead, because there is nothing between the engine and the driver.

What broke

Before I accepted the loss of virtualisation, I tried three different passthrough recipes across five days. The one that consumed the most time is worth telling. I had the card isolated, bound to vfio-pci on host boot, and the VM would boot. The Nvidia driver loaded inside the guest. nvidia-smi in the guest printed the card’s name, memory, driver version. It looked perfect. The moment I ran the simplest possible inference (ollama run phi3), the guest froze hard. SSH into the guest went unresponsive; the Proxmox console showed a locked frame. Host was unaffected.

After a day of log-reading, the cause was a power-state mismatch: the host’s ACPI subsystem was still managing the card’s D-states (power saving levels) while the guest was trying to drive it, and the first context switch into a deep D-state locked the card. The fix was two lines: disable the host’s PCI power management for the GPU, and set pm_disabled=1 in the guest’s Nvidia driver options. With both, the freeze vanished.

The fact that I could find the fix did not persuade me to keep the architecture. It persuaded me of the opposite. If I needed two lines of obscure kernel parameters to keep inference from locking the machine, and a BIOS update could undo either of them silently, I was buying ten days of my own time per year to preserve an abstraction that was not earning its keep on this node.

The principle

The lab still runs Proxmox on the general-purpose hypervisor. Every container, VM, service, and network appliance lives there. Virtualisation is the right default for almost everything I do. The AI node is the exception, and it is the exception for a specific reason: its workload is a single process that wants 100% of a specific piece of silicon, and nothing else on that machine needs to be isolated from it.

The principle I now apply across the lab: virtualise by default; go bare-metal when the workload is a single process with exclusive hardware requirements and the VM layer adds no isolation value. AI inference meets that test. Web servers do not. Backup appliances do not. Almost nothing else does. That is why this node is the only bare-metal box in the rack, and why it is likely to stay that way.

The racing car went back on the tarmac. The simulator is still in the garage for everything else.

Footnotes

  1. The canonical reference is the Proxmox VE wiki’s “PCI(e) Passthrough” page, maintained since 2016 and updated regularly. Linux kernel docs cover the VFIO subsystem at kernel.org/doc/Documentation/vfio.txt. ↩

Table of Contents