The Dustbin

FreeBSD USB-IP and Qemu

Using QEMU to do USB-passthrough to Bhyve on FreeBSD

I want to run the an ADS-B software stack using a RTL-SDR USB receiver on my FreeBSD-based home server. There are many excellent recipies for getting this running under Linux (for example here), but some important software will not run at all under FreeBSD (such as the closed-source FlightRadar24 feeder). Normally, I'd just spin up a bhyve-based Linux virtual machine, and be done, but I need to pass thorugh the RTL-SDR USB device. Bhyve unfortunately does not offer USB-passthrough for a single device — I'd need to pass through the entire PCI device, which would put all of my USB devices inside the VM. Unfortunately my server only has a single PCI USB controller, so that option is not workable.

FreeBSD does support QEMU, and qemu does support USB-passthrough. Unfortunately, qemu runs quite slowly under FreeBSD. I did get the whole ADS-B stack with docker containers, etc. running just fine in qemu, but the system was consuming 150% CPU at rest. There's a better approach, using both qemu and bhyve. Linux supports USB over IP (see USB/IP Project), as does Windows, but sadly FreeBSD does not. Passing the USB device through to a lightweight qemu-hosted Linux VM, and then sharing that device using USB-IP to another bhyve-hosted Linux VM (which runs very efficiently) results in a working system, with a bit more complexity, but much less overall system resource use and load.

The FreeBSD documentation was as always very helpful to get qemu up and running:

First I created an alpine-linux system under qemu:

# mkdir -p /root/qemu/iso /root/qemu/iso /root/qemu/logs /root/qemu/vm
# pkg install qemu
# qemu-img create -f qcow2 -o preallocation=full,cluster_size=512K,lazy_refcounts=on /root/qemu/vm/alpine.qcow2 5G`
# fetch https://dl-cdn.alpinelinux.org/alpine/v3.21/releases/x86_64/alpine-extended-3.21.0-x86_64.iso

Create boot script, passing through the USB device I need, use a serial console.

#!/bin/sh
/usr/local/bin/qemu-system-x86_64 \
  -serial telnet:localhost:4410,server=on,wait=off \
  -monitor none \
  -machine pc \
  -cpu Skylake-Client-IBRS \
  -smp 1 \
  -accel tcg \
  -m 1024 \
  -nographic \
  -boot order=cd,menu=on \
  -usb \
  -device qemu-xhci \
  -device usb-host,vendorid=0x0bda,productid=0x2838,id=rtlsdr \
  -cdrom /root/qemu/iso/alpine-extended-3.21.0-x86_64.iso \
  -drive if=none,id=drive0,cache=writeback,aio=threads,format=qcow2,discard=unmap,file=/root/qemu/vm/alpine.qcow2 \
  -device virtio-blk-pci,drive=drive0,bootindex=1  \
  -netdev tap,id=nd0,ifname=tap0,script=no,downscript=no,br=bridge0 \
  -device e1000,netdev=nd0,mac=02:20:6c:65:66:74 \
  -name \"alpine\"

I then logged into alpine (with telnet localhost 4410) and configured it as a traditional on-disk system. When that completed, I removed the -cdrom ... line from the script above and rebooted alpine.

Inside of alpine linux, install the needed tools for USB-IP:

# apk add linux-tools-usbip-openrc linux-tools-usbip findutils hwdata-usb

There's a bug in the /etc/init.d/usbip file — make the following edits:

# /etc/init.d/usbip # fixed find command to look for ".ko.gz" instead of ".ko"
# /etc/conf.d/usbip # turn on daemon

Set usbip to start by default and check local devices:

# ln -s /etc/init.d/usbip /etc/default/runlevels/usbpip
# service usbip restart
# usbip list -l
 - busid 2-1 (0bda:2838)
   Realtek Semiconductor Corp. : RTL2838 DVB-T (0bda:2838)

Here I see the RTL-SDR device I want to pass through to another VM. It has a busid of 2-1. Edit /etc/conf.d/usbip to add device "2-1" to the bind list. Restart the service and check

adsb:/etc# usbip list -r localhost
Exportable USB devices
======================
 - localhost
        2-1: Realtek Semiconductor Corp. : RTL2838 DVB-T (0bda:2838)
           : /sys/devices/pci0000:00/0000:00:03.0/usb2/2-1
           : (Defined at Interface level) (00/00/00)

I then installed a stock Debian 12 Linux VM using bhyve. In that VM, I configured the client side of USB-IP:

# apt install usb usbutils
# modprobe usbip_core
# echo usbip_core >> /etc/modules
# modprobe vhci_hcd
# echo vhci_hcd >> /etc/modules
# lsusb
# usbip list -r 192.168.1.6
# usbip attach -r 192.168.1.6 -b 2-1
# lsusb

(Of coure use the correct IP address of the Alpine linux VM). I then proceeded to install Docker, Portainer, and the ADS-B docker stack on the bhyve-VM. Total system load and resource usage is less than half of the qemu only solution. If more USB devices are needed (for example, to pass through to a HomeAssistant VM), I can use the same pattern to share from the source Alpine Linux qemu VM to other bhyve VMs.

#software