FreeBSD/Ubuntu Dual-boot testbed using Desktop Hardware
Tony needed his computers back, he is a great friend and so he offered me some replacements. This means that we get to make version 2 of the "The Bedroom by the bed testbed" .
The first version of the testbed was built to answer questions of the form:
"How does stock Ubuntu compare to FreeBSD?"
Version 2 of the testbed continues this approach with an addition:
How does stock Ubuntu compare to FreeBSD, and what if we try emulating a satellite network?"
Tony and I spoke for a while about my needs from a testbed, the machines he lent me before were part of a set, they were two boxes in a 6 node Bioinfromatics cluster. He needed them back to start doing shake out tests of the cluster so he can start selling time on it ( check out his excellent company ).
While I have been using his machines all of the work so far has been developing experiment tooling. For me the time to replace the machines was actually quite fortunate, I have managed to get automation working, but I don't yet have any finished results. This means that I can move to new machines and apart from a short down time reconfiguring things there shouldn't be any disruption.
To replace the Opteron 6380 systems he offered me 2 Threadripper 1950X systems.
-left , -right
- Threadripper 1950X
- Asrock X399M Taichi motherboard
- 32GB RAM @2666MHz ( -right has 64GB, -left will get more once it arrives in the mail)
- SSD storage
The Opteron machines were server motherboards, in massive whale sized cases and they made whale sized sounds when they were running. The Threadrippers are in lovely little mini-ATX cases and when they are running the make a lovely little hum that easily vanished into the background when I type on my cherry blue keyboard.
The smaller quieter form factor comes from the use of desktop hardware, sadly this also means the loss of lights out management.
When Tony listened to my needs he also offered me a third system (so I can answer the questions the addition raises), but I turned him down. For longer term plans I need to own machines and I am grudgingly happy to buy hardware to get experiments running.
I wanted to lean hard into using desktop hardware for non desktop tasks (my computers don't have an SLA to fulfil). With this goal in mind I speced out a Ryzen 5950X system that was a bit of a monster. I got cold feet at the price and decided to build a compatible system using a lower end Ryzen 7 processor. Gazing into the future I think this might be a safe bet, if I need more compute I should be able to pick up a 5950X on ebay for ~50% of the list price.
For 10GbE network I put a single port Mellanox Connect-X 3 Pro interface in each.
pokeitwithastick
- Ryzen 3700x
- Asrock X570 PRO4
- 32GB RAM @2666MHz
- NVME storage
From the 'big machine' spec I culled this down to using less and slower RAM and lower end processor. I think I should be able to push up this rig with more faster RAM and a bigger processor, but I get the flexibility to try doing this on the cheap first.
The Ryzen system has a Mellanox Connect-X 3 Pro interface with Dual ports. It is going to be routing packets.
This machine is able to be a more moving target so I installed FreeBSD-14-CURRENT on it from a recent snapshot. The use of stock CURRENT is worth noting when you think about the performance of the Ryzen system compared to the others.
I only know 1 functional test that I really care about and that is building FreeBSD. I pulled the pairs of drives from the testbed v1 machines and transferred them over (the lack of 2.5" drive hot swap on the Threadrippers was annoying). Once I got the drives on the correct SATA cables the boxes came up and I was able to see how all three machines did:
host processor RAM time buildworl buildkernel
freebsd-left Opteron 6380 128GB 58:00
freebsd-left Threadripper 19050X 32GB 30:45
freebsd-right Threadripper 1950X 64GB 30:06
pokeitwithastick Ryzen 3700X 32GB 33:09
Network Setup
With a third testbed machine the network diagram from before changes slightly. Rather than the interfaces of the two machines being connected back to back, they are now connected to pokeitwithastick which is acting as a router.
The network now looks like this:
-left pokeitwithastick -right
10.0.10.x 10.0.20.x
+-------------+ +-------------+ +-------------+
| .2| |.1 .1| |.2 |
| | | | | |
| mlxen0+<------->+mlxen0 mlxen1+<------->+mlxen0 |
| | | | | |
| igb0 | | igb0 | | igb0 |
+------+------+ +-----+-------+ +------+------+
| | |
| freebsd|192.168.100.50 |
._______ | ____.
\________. | .______________/
freebsd 192.168.100.10 V V V freebsd 192.168.100.20
linux 192.168.100.11 +--------------+ linux 192.168.100.21
| switch |
| (openwrt) |
+--------------+
^
|
| freebsd 192.168.100.2
+---------+
| control |
+---------+
This is a pretty standard dumbell network and is good set up for performance work when you need a bottleneck.
Remote Power On
There were two features of the previous hardware that I really liked. Both machines had serial ports, which gave me a last ditch management interface option if I completley hosed the network while I wasn't at home. And the machines support lights out management with IPMI. In a sheer irony, even if the boxes hadn't had serial broken out on the motherboard, IPMI would have given me access.
I was using serial and IPMI to allow me to power on the machines remotely and control which Operating System they booted into. IPMI allowed power on, grub was configured to output to serial and video and that gave me boot control.
Tony doesn't have the same requirements as me for machines so while my Asrock X570 motherboard has a COM Port header, the Taichi motherboards don't.
Wake on LAN (WOL) is a poor replacement for IPMI power control. It is sort of famously badly implemented and it is sort of clear why. It is a packet with the MAC addresses repeated a bunch of times that turns on the system by magic. No matter your opinion of WOL the Intel network interfaces on all three machines seem to be very good at booting with WOL when they get the packets.
WOL had to be configured in the BIOS before it could be used:
In the X570 PRO4 bios configure:
Advanced->ACPI Configuration->PCIE Devices Power On
"Allow the system to be wakeed up by a
PCIE device and enable wake on LAN"
In the Taichi bios configure:
Advanced->ACPI Configuration->PCIE Devices Power On
"Allow the system to be wakeed up by a
PCIE device and enable wake on LAN"
With the BIOS set up remote power on requires using a WOL tool to send a magic
packet, the control host is well placed on the network to do this with the
wol
command:
control $ wol a8:a1:59:95:87:60
After running the command I got nothing.
This is fine, I am a network engineer and a hacker(!), I can debug this sort of
issue. Some time with
tcpdump
showed that I wasn't getting broadcast traffic
through at all.
I tested this assertion by using a host directed WOL packet:
control $ wol a8:a1:59:95:77:ab -i 192.168.100.50
These packets appear on the host in
tcpdump
and after a power off are able to
wake the machines up. Well at first, if I waited a while then the machine was
still not responding to the WOL packet.
The diagram in the v1 network and the diagram I would have drawn for the v2 network at first was a lie. control is connected to the switch on the back of an OpenWRT router that is acting as a WiFi client to the network in the house. That switch is in turn connected to an unmanaged Netgear switch that all the rest of the network is connected to (IPMI and useful interfaces).
One of these switches was not forwarding broadcast traffic and it was only forwarding unicast traffic when the host was 'alive' enough. Not having IPMI anymore I was able to remove the Netgear switch, my needs now fit onto the four port switch on the OpenWRT router. Removing the Netgear switch didn't solve the problem and I can't remove the OpenWRT router so a different solution is required.
OpenWRT has a tool called
etherwake
to support Wake On LAN, I installed this
on the router and I immediately got consistently working WOL:
root@OpenWrt:~# etherwake a8:a1:59:95:77:ab
I have to ssh to the router to run power on commands, but that is enough for the testbed to be useful now.
Controlling the booted OS
Having a serial interface to access the grub menu and select the booted OS was great. But I have to wipe away my tears and accept that this isn't possible with this hardware.
Grub supports something called
grub-reboot
, normally grub tries very hard to
not write anything to disk in normal operation. You can however configure grub
to use a scratch space and remember which operating system was booted before
(and maybe other things).
grub-reboot
uses this mechanism from the Operating System to control which
menu item grub uses as default when it boots. This is part of the
grub
environment and is documented here
.
/etc/default/grub
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
# info -f grub -n 'Simple configuration'
GRUB_DEFAULT=saved
GRUB_TIMEOUT_STYLE=menu
GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="console=ttyS0,115200 console=tty0"
GRUB_CMDLINE_LINUX=""
# Uncomment to disable graphical terminal (grub-pc only)
GRUB_TERMINAL="console serial"
GRUB_SERIAL_COMMAND="serial --speed=115200"
To use
grub-reboot
you need to set the default menu entry to 'saved'. If you
changed grub to enable
grub-reboot
make sure to update grub:
# sudo update-grub
With grub configured with a default menu entry of
saved
, we can configure
grub to boot the next time from a different menu entry other than the first
one.
# sudo grub-reboot 4
# sudo reboot
On these systems this allows me to boot from the FreeBSD menu entry (it is fifth in the list). Without serial to boot into FreeBSD I have to do a round trip into Linux, but this is a lot better than having to sit near the hot loud computers.
Performance
The generation change in AMD hardware had a huge improvement in processing speed, going from 58 minutes for a FreeBSD build to 30 minutes is amazing. These machines do networking stuff so it is good to look at network benchmarks for a baseline.
On Ubuntu and FreeBSD, the Threadripper machines are able to saturate 10GbE with TCP and get a about 6Gbit/s of UDP traffic. They can generate enough UDP to saturate the link so this is a huge step forward.
iperf3
benchmarks where the Ryzen system is the receiver manage half the
traffic that the Threadripper systems do, capping out at about 4.5Gbit/s, the
Ryzen can however send enough to saturate the link.
Base forwarding tests show no change in the throughput of the Threadripper systems. I had never considered that receive could be harder than transmit, but these baselines seem (and chatting to FreeBSD developers) seem to suggest that this isn't uncommon. For now this isn't a problem, but later I might need the Ryzen system to have more head room when running tests. If that happens I'll have a good reason to get a faster processor :D