FreeBSD/Ubuntu Dual-boot testbed using Desktop Hardware

Tony needed his computers back, he is a great friend and so he offered me some replacements. This means that we get to make version 2 of the "The Bedroom by the bed testbed" .

The first version of the testbed was built to answer questions of the form:

"How does stock Ubuntu compare to FreeBSD?"

Version 2 of the testbed continues this approach with an addition:

How does stock Ubuntu compare to FreeBSD, and what if we try emulating a satellite network?"

Tony and I spoke for a while about my needs from a testbed, the machines he lent me before were part of a set, they were two boxes in a 6 node Bioinfromatics cluster. He needed them back to start doing shake out tests of the cluster so he can start selling time on it ( check out his excellent company ).

While I have been using his machines all of the work so far has been developing experiment tooling. For me the time to replace the machines was actually quite fortunate, I have managed to get automation working, but I don't yet have any finished results. This means that I can move to new machines and apart from a short down time reconfiguring things there shouldn't be any disruption.

To replace the Opteron 6380 systems he offered me 2 Threadripper 1950X systems.

-left , -right

  • Threadripper 1950X
  • Asrock X399M Taichi motherboard
  • 32GB RAM @2666MHz ( -right has 64GB, -left will get more once it arrives in the mail)
  • SSD storage

The Opteron machines were server motherboards, in massive whale sized cases and they made whale sized sounds when they were running. The Threadrippers are in lovely little mini-ATX cases and when they are running the make a lovely little hum that easily vanished into the background when I type on my cherry blue keyboard.

The smaller quieter form factor comes from the use of desktop hardware, sadly this also means the loss of lights out management.

When Tony listened to my needs he also offered me a third system (so I can answer the questions the addition raises), but I turned him down. For longer term plans I need to own machines and I am grudgingly happy to buy hardware to get experiments running.

I wanted to lean hard into using desktop hardware for non desktop tasks (my computers don't have an SLA to fulfil). With this goal in mind I speced out a Ryzen 5950X system that was a bit of a monster. I got cold feet at the price and decided to build a compatible system using a lower end Ryzen 7 processor. Gazing into the future I think this might be a safe bet, if I need more compute I should be able to pick up a 5950X on ebay for ~50% of the list price.

For 10GbE network I put a single port Mellanox Connect-X 3 Pro interface in each.

pokeitwithastick

  • Ryzen 3700x
  • Asrock X570 PRO4
  • 32GB RAM @2666MHz
  • NVME storage

From the 'big machine' spec I culled this down to using less and slower RAM and lower end processor. I think I should be able to push up this rig with more faster RAM and a bigger processor, but I get the flexibility to try doing this on the cheap first.

The Ryzen system has a Mellanox Connect-X 3 Pro interface with Dual ports. It is going to be routing packets.

This machine is able to be a more moving target so I installed FreeBSD-14-CURRENT on it from a recent snapshot. The use of stock CURRENT is worth noting when you think about the performance of the Ryzen system compared to the others.

I only know 1 functional test that I really care about and that is building FreeBSD. I pulled the pairs of drives from the testbed v1 machines and transferred them over (the lack of 2.5" drive hot swap on the Threadrippers was annoying). Once I got the drives on the correct SATA cables the boxes came up and I was able to see how all three machines did:

host            processor       RAM time buildworl buildkernel

freebsd-left        Opteron 6380        128GB   58:00
freebsd-left            Threadripper 19050X  32GB   30:45
freebsd-right           Threadripper 1950X   64GB   30:06
pokeitwithastick        Ryzen 3700X      32GB   33:09

Network Setup

With a third testbed machine the network diagram from before changes slightly. Rather than the interfaces of the two machines being connected back to back, they are now connected to pokeitwithastick which is acting as a router.

The network now looks like this:

              -left                pokeitwithastick        -right

                          10.0.10.x               10.0.20.x
           +-------------+         +-------------+         +-------------+
           |           .2|         |.1         .1|         |.2           |
           |             |         |             |         |             |
           |       mlxen0+<------->+mlxen0 mlxen1+<------->+mlxen0       |
           |             |         |             |         |             |
           |     igb0    |         |    igb0     |         |     igb0    |
           +------+------+         +-----+-------+         +------+------+
                  |                      |                        |
                  |               freebsd|192.168.100.50          |
                  ._______               |                    ____.
                          \________.     |    .______________/
freebsd 192.168.100.10             V     V    V              freebsd 192.168.100.20
linux   192.168.100.11           +--------------+             linux   192.168.100.21
                                 |    switch    |
                                 |   (openwrt)  |
                                 +--------------+
                                        ^
                                        |
                                        | freebsd 192.168.100.2
                                   +---------+
                                   | control |
                                   +---------+

This is a pretty standard dumbell network and is good set up for performance work when you need a bottleneck.

Remote Power On

There were two features of the previous hardware that I really liked. Both machines had serial ports, which gave me a last ditch management interface option if I completley hosed the network while I wasn't at home. And the machines support lights out management with IPMI. In a sheer irony, even if the boxes hadn't had serial broken out on the motherboard, IPMI would have given me access.

I was using serial and IPMI to allow me to power on the machines remotely and control which Operating System they booted into. IPMI allowed power on, grub was configured to output to serial and video and that gave me boot control.

Tony doesn't have the same requirements as me for machines so while my Asrock X570 motherboard has a COM Port header, the Taichi motherboards don't.

Wake on LAN (WOL) is a poor replacement for IPMI power control. It is sort of famously badly implemented and it is sort of clear why. It is a packet with the MAC addresses repeated a bunch of times that turns on the system by magic. No matter your opinion of WOL the Intel network interfaces on all three machines seem to be very good at booting with WOL when they get the packets.

WOL had to be configured in the BIOS before it could be used:

In the X570 PRO4 bios configure:        
    Advanced->ACPI Configuration->PCIE Devices Power On
        "Allow the system to be wakeed up by a
        PCIE device and enable wake on LAN"

In the Taichi bios configure:           
    Advanced->ACPI Configuration->PCIE Devices Power On
        "Allow the system to be wakeed up by a
        PCIE device and enable wake on LAN"

With the BIOS set up remote power on requires using a WOL tool to send a magic packet, the control host is well placed on the network to do this with the wol command:

control $ wol a8:a1:59:95:87:60

After running the command I got nothing.

This is fine, I am a network engineer and a hacker(!), I can debug this sort of issue. Some time with tcpdump showed that I wasn't getting broadcast traffic through at all.

I tested this assertion by using a host directed WOL packet:

control $ wol a8:a1:59:95:77:ab -i 192.168.100.50

These packets appear on the host in tcpdump and after a power off are able to wake the machines up. Well at first, if I waited a while then the machine was still not responding to the WOL packet.

The diagram in the v1 network and the diagram I would have drawn for the v2 network at first was a lie. control is connected to the switch on the back of an OpenWRT router that is acting as a WiFi client to the network in the house. That switch is in turn connected to an unmanaged Netgear switch that all the rest of the network is connected to (IPMI and useful interfaces).

One of these switches was not forwarding broadcast traffic and it was only forwarding unicast traffic when the host was 'alive' enough. Not having IPMI anymore I was able to remove the Netgear switch, my needs now fit onto the four port switch on the OpenWRT router. Removing the Netgear switch didn't solve the problem and I can't remove the OpenWRT router so a different solution is required.

OpenWRT has a tool called etherwake to support Wake On LAN, I installed this on the router and I immediately got consistently working WOL:

root@OpenWrt:~# etherwake a8:a1:59:95:77:ab

I have to ssh to the router to run power on commands, but that is enough for the testbed to be useful now.

Controlling the booted OS

Having a serial interface to access the grub menu and select the booted OS was great. But I have to wipe away my tears and accept that this isn't possible with this hardware.

Grub supports something called grub-reboot , normally grub tries very hard to not write anything to disk in normal operation. You can however configure grub to use a scratch space and remember which operating system was booted before (and maybe other things).

grub-reboot uses this mechanism from the Operating System to control which menu item grub uses as default when it boots. This is part of the grub environment and is documented here .

/etc/default/grub

# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=saved
GRUB_TIMEOUT_STYLE=menu
GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="console=ttyS0,115200 console=tty0"
GRUB_CMDLINE_LINUX=""

# Uncomment to disable graphical terminal (grub-pc only)
GRUB_TERMINAL="console serial"
GRUB_SERIAL_COMMAND="serial --speed=115200"

To use grub-reboot you need to set the default menu entry to 'saved'. If you changed grub to enable grub-reboot make sure to update grub:

# sudo update-grub

With grub configured with a default menu entry of saved , we can configure grub to boot the next time from a different menu entry other than the first one.

# sudo grub-reboot 4
# sudo reboot

On these systems this allows me to boot from the FreeBSD menu entry (it is fifth in the list). Without serial to boot into FreeBSD I have to do a round trip into Linux, but this is a lot better than having to sit near the hot loud computers.

Performance

The generation change in AMD hardware had a huge improvement in processing speed, going from 58 minutes for a FreeBSD build to 30 minutes is amazing. These machines do networking stuff so it is good to look at network benchmarks for a baseline.

On Ubuntu and FreeBSD, the Threadripper machines are able to saturate 10GbE with TCP and get a about 6Gbit/s of UDP traffic. They can generate enough UDP to saturate the link so this is a huge step forward.

iperf3 benchmarks where the Ryzen system is the receiver manage half the traffic that the Threadripper systems do, capping out at about 4.5Gbit/s, the Ryzen can however send enough to saturate the link.

Base forwarding tests show no change in the throughput of the Threadripper systems. I had never considered that receive could be harder than transmit, but these baselines seem (and chatting to FreeBSD developers) seem to suggest that this isn't uncommon. For now this isn't a problem, but later I might need the Ryzen system to have more head room when running tests. If that happens I'll have a good reason to get a faster processor :D