FreeBSD Network Status Week 49 2024

Goings on

BSD Devroom at FOSDEM 2024

The CFP for the BSD Devroom at FOSDEM closed over the weekend, submitters have been told to expect responses on the 15th of December. I'll provide a run down of upcomming talks in the first status update of 2025.

FreeBSD 14.2 was released

Congratulations to everyone involved and a big thanks to the RE team for getting FreeBSD 14.2 out. The release notes are here

CFT for rtwn, ath and iwn

hi!

I've been working on some net80211 clean-ups and rtwn bugfixes in
preparation for getting the 11ac support for rtwn landed into the tree.

If you're using rtwn, ath or iwn then the latest stuff I've done in -HEAD
may affect you. Please update to the latest -HEAD and let me know if
anything has regressed! (or if it still works fine, that'd be nice to know
too!)

thanks,

-adrian

adrian@ posted a cft for users of rtwn, ath and iwn after recent CURRENT changes

Network Stack

Lots of tidying in netlink.

Tidying in in6_pcbbind . This is a very difficult part of the network stack to work in and tidying here makes things much easier.

Transport

Tidy ups in tcp_hpts .

Netdev

Tie receive packet headers to the correct NUMA domain, this sort of change can help a lot in systems with multiple NUMA domains to reduce cross domain traffic. This happening in iflib makes it 'free' for all iflib drivers.

Add a big list of skus to igc

adrian@ landed some macros that help with handling ht rates, this makes drivers cleaner:

and he landed a bunch of improvements to rtwn from debugging. Chat in the wifi irc channel suggest that rtwn might get a lot faster soon. If you use rtwn you should test HEAD now to help catch any regressions.

bz@ has landed HT and VHT improvements for the Linux KPI.

Firewalls

Following up on the pf EIM NAT support Damjan Jovanovic has implemented support for EIM NAT for libalias users. If you use ipfw, ppp or netgraph you can now experiment with a more transparent NAT mapping.

Some stability fixes in pf and potential panic fixes.

Other stuff

First up, documentation of a wonderful hack from phk@ to show network traffic in bits per second by running with an 8 second interval. This had a few confused follow ups, it is so simple so people thought it couldn't possibly work.

Following on from the excellent Fall Summit talk by Ian from Metify on rural broadband, do a better job of documenting the cellular hardware we support.

I like the commit message on this one:

Make things easier for users to understand, but also preserve the past.

Please Send Feedback

That is all for this week. Next week will be the last report of the year, I'll be travelling (please say hi if you are at 38c3) and I think a report break might be nice.

The first report will probably be on the 10th of January, look for it where you found this one.

I would love to know if this summary was any help, if it was, or if you think I should cover other thing please let me know (thj@freebsd.org).

If you find a typo or have a correct let me know and I'll thank you at the end here.

You can see all prior posts here. ( rss )


My work on FreeBSD is supported by the FreeBSD Foundation , you can contribute to improving FreeBSD with code, documentation or financially by donating to the FreeBSD Foundation .

Everyone gets a WiFi

Building on my earlier blog post on setting up PCIe pass through with bhyve , I've found a little time to move things around in my test bed and try something stupid.

Bitcoin is certainly a terrible thing, but it has made some useful components super cheap (though they are getting rare again). One feature of bitcoin mining is that it is super low usage on the PCIe bus, you feed your GPU something to do, it does it, and if you find a result you send a couple of hundred bytes back. That leaves a lot of bandwidth left over.

Miners knew what to do with this took advantage of PCIe topology and added switches, expanders and risers to get as many GPUs per motherboard as they could.

We can apply their tools for other things like driver development.

The idea is this, get a bunch of devices onto a VM host through the use of cheap downstream PCIe switches, have the host ignore them and make them available to guests on the host.

This makes it possible to do a lot of concentrated testing. You can for example stick 4 different revisions of a card on host and simultaneously test them all on the latest current, or stick 3 versions of the same card on the bus and test them on CURRENT, STABLE and STABLE-1 at the same time to unsure there aren't regressions.

For porting you could run guests for FreeBSD, Linux and OpenBSD and compare functionality and commands to the card for debugging.

Proof of Concept

I had this idea early this year and managed to get a POC together in a short 12 months of forgetting about it and an afternoon of moving machines around in the attic.

I picked up two types of PCIe expander, PCIe x1 -> x16 risers (which aren't used right now) and an ASMedia ASM1184e 4-Port PCIe x1 Gen2 Packet Switch'. Later I got a cheap 12 GPU frame, adding a final piece of structure needed to make the set up practical.

The PCIe expander uses a USB3-A cable to carry the PCIe bus out an adapter card connected to the motherboard out to the switch board. The switch board has 4 PCIe x1 slots for cards.

I mounted all this on the frame with 2 Intel AX210 WiFi cards for a POC. Pulling the card to check functionality the switch appears in pciconf -lf like so:

pcib6@pci0:5:0:0:       class=0x060400 rev=0x00 hdr=0x01 vendor=0x1b21 device=0x1184 subvendor=0x1b21 subdevice=0x118f
    vendor     = 'ASMedia Technology Inc.'
    device     = 'ASM1184e 4-Port PCIe x1 Gen2 Packet Switch'
    class      = bridge
    subclass   = PCI-PCI
pcib7@pci0:6:1:0:       class=0x060400 rev=0x00 hdr=0x01 vendor=0x1b21 device=0x1184 subvendor=0x1b21 subdevice=0x118f
    vendor     = 'ASMedia Technology Inc.'
    device     = 'ASM1184e 4-Port PCIe x1 Gen2 Packet Switch'
    class      = bridge
    subclass   = PCI-PCI
pcib8@pci0:6:3:0:       class=0x060400 rev=0x00 hdr=0x01 vendor=0x1b21 device=0x1184 subvendor=0x1b21 subdevice=0x118f
    vendor     = 'ASMedia Technology Inc.'
    device     = 'ASM1184e 4-Port PCIe x1 Gen2 Packet Switch'
    class      = bridge
    subclass   = PCI-PCI
pcib9@pci0:6:5:0:       class=0x060400 rev=0x00 hdr=0x01 vendor=0x1b21 device=0x1184 subvendor=0x1b21 subdevice=0x118f
    vendor     = 'ASMedia Technology Inc.'
    device     = 'ASM1184e 4-Port PCIe x1 Gen2 Packet Switch'
    class      = bridge
    subclass   = PCI-PCI
pcib10@pci0:6:7:0:      class=0x060400 rev=0x00 hdr=0x01 vendor=0x1b21 device=0x1184 subvendor=0x1b21 subdevice=0x118f
    vendor     = 'ASMedia Technology Inc.'
    device     = 'ASM1184e 4-Port PCIe x1 Gen2 Packet Switch'
    class      = bridge
    subclass   = PCI-PCI

With the switch there I added the two PCIe WiFi cards and rebooted:

pcib6@pci0:5:0:0:       class=0x060400 rev=0x00 hdr=0x01 vendor=0x1b21 device=0x1184 subvendor=0x1b21 subdevice=0x118f
    vendor     = 'ASMedia Technology Inc.'
    device     = 'ASM1184e 4-Port PCIe x1 Gen2 Packet Switch'
    class      = bridge
    subclass   = PCI-PCI
pcib7@pci0:6:1:0:       class=0x060400 rev=0x00 hdr=0x01 vendor=0x1b21 device=0x1184 subvendor=0x1b21 subdevice=0x118f
    vendor     = 'ASMedia Technology Inc.'
    device     = 'ASM1184e 4-Port PCIe x1 Gen2 Packet Switch'
    class      = bridge
    subclass   = PCI-PCI
pcib8@pci0:6:3:0:       class=0x060400 rev=0x00 hdr=0x01 vendor=0x1b21 device=0x1184 subvendor=0x1b21 subdevice=0x118f
    vendor     = 'ASMedia Technology Inc.'
    device     = 'ASM1184e 4-Port PCIe x1 Gen2 Packet Switch'
    class      = bridge
    subclass   = PCI-PCI
pcib9@pci0:6:5:0:       class=0x060400 rev=0x00 hdr=0x01 vendor=0x1b21 device=0x1184 subvendor=0x1b21 subdevice=0x118f
    vendor     = 'ASMedia Technology Inc.'
    device     = 'ASM1184e 4-Port PCIe x1 Gen2 Packet Switch'
    class      = bridge
    subclass   = PCI-PCI
pcib10@pci0:6:7:0:      class=0x060400 rev=0x00 hdr=0x01 vendor=0x1b21 device=0x1184 subvendor=0x1b21 subdevice=0x118f
    vendor     = 'ASMedia Technology Inc.'
    device     = 'ASM1184e 4-Port PCIe x1 Gen2 Packet Switch'
    class      = bridge
    subclass   = PCI-PCI
iwlwifi0@pci0:7:0:0:    class=0x028000 rev=0x1a hdr=0x00 vendor=0x8086 device=0x2725 subvendor=0x8086 subdevice=0x0024
    vendor     = 'Intel Corporation'
    device     = 'Wi-Fi 6E(802.11ax) AX210/AX1675* 2x2 [Typhoon Peak]'
    class      = network
iwlwifi1@pci0:10:0:0:   class=0x028000 rev=0x1a hdr=0x00 vendor=0x8086 device=0x2725 subvendor=0x8086 subdevice=0x0024
    vendor     = 'Intel Corporation'
    device     = 'Wi-Fi 6E(802.11ax) AX210/AX1675* 2x2 [Typhoon Peak]'
    class      = network

Each card can be used on the host simultaneously. I took the cards from the host and enabled PCIe pass through for AMD in /boot/loader.conf :

pptdevs="7/0/0 10/0/0"          # intel ax210 wifi
hw.vmm.amdvi.enable=1

Finally I used the script from the previous post to run two FreeBSD CURRENT VMs, giving each one of the cards and the successfully came up and could do the wireless.

The cards can scan from here, harder workloads are a future problem. I want to install 2 more expanders on this machine and more cards to see how things work when heavily loaded. This 12 GPU bracket has 35 holes so I could add a lot of stuff.

There are PCIe bus overheads here, I'm reducing down a PCIe4 bus to PCIe2 for the Wifi cards, but they aren't exactly high performance devices in PCIe terms. I might also hit power limits, I didn't really want to ask about doing this POC, I figured I wouldn't get good answers based on similar questions on the FreeBSD mailing lists in the last year.

This seems to work, scaling is to be seen.


My work on FreeBSD is supported by the FreeBSD Foundation , you can contribute to improving FreeBSD with code, documentation or financially by donating to the FreeBSD Foundation .

PCIe pass through for Wifi Dev

For the last little while I have been working on porting a WiFi driver. I initially planned to use one of my testbed machines, but the card I had wasn't supported by the original driver and locked up the system.

While waiting for another card I spun up development on the Morefine M6 which had failed complete at its proposed role as a desktop mirrored storage array.

The Morefine M6 has an Intel N200 SoC and supported Intel wifi.

I always set out to try PCIe pass through and the FreeBSD wiki page provided most of what I needed .

On the host you need to exclude the PCIe devices you wish to give to a guest. You need to find the PCIe address for the device to exclude, pciconf -lv is good for this:

# pciconf -lv 
...
none2@pci0:0:20:2:      class=0x050000 rev=0x00 hdr=0x00 vendor=0x8086 device=0x54ef subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'                        
    device     = 'Alder Lake-N PCH Shared SRAM'                 
    class      = memory                                         
    subclass   = RAM                                            
ppt0@pci0:0:20:3:       class=0x028000 rev=0x00 hdr=0x00 vendor=0x8086 device=0x54f0 subvendor=0x8086 subdevice=0x0070
    vendor     = 'Intel Corporation'
    device     = 'CNVi: Wi-Fi'                                  
    class      = network                                        
ig4iic0@pci0:0:21:0:    class=0x0c8000 rev=0x00 hdr=0x00 vendor=0x8086 device=0x54e8 subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    class      = serial bus                                     
ig4iic1@pci0:0:21:1:    class=0x0c8000 rev=0x00 hdr=0x00 vendor=0x8086 device=0x54e9 subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    class      = serial bus 
...

I have passed PCIe device 0:20:3 to my guests, in the pciconf list you will see the driver assigned to the device followed by the bus and the address. So the ig4 i2c driver has devices ig4iic0@pci0:0:21:0 and ig4iic1@pci0:0:21:1 and the 'Wi-Fi' device has been taken by ppt0.

This is configured by adding the PCIe address to /boot/loader.conf :

pptdevs="0/20/3"                # intel wifi

We need to give our PCIe device to our VM, I am simple person and do all my bhyve through vmrun.sh and wrapper shell scripts. For FreeBSD the script is this:

#!/bin/sh                                                                   

diskimage="not set"                                                         
vmname="not set"                                                            
pcidev="0/20/3"                                                             

diskimage="/home/tj/vms/fbsd-iwx.raw"                                       
vmname=$(basename $diskimage .raw)                                          

if [ $# -ge 1 ]                                                             
then                                                                        
        for x in $@                                                         
        do                                                                  
                interfaces="$interfaces -t $x"                              
        done                                                                
else                                                                        
        echo 'usage: launchfreebsd.sh tapDev [tapDev tapDev ...]'           
        exit                                                                
fi                                                                          

echo starting vm $vmname from image $diskimage with interfaces ${interfaces}

sh /usr/share/examples/bhyve/vmrun.sh \                                     
        -c $(nproc) \                                                       
        -m 8G \                                                             
        -p ${pcidev} \                                                      
        ${interfaces} \                                                     
        -d $diskimage \                                                     
        $vmname

And for OpenBSD:

#!/bin/sh                                                                   

diskimage="not set"                                                         
vmname="not set"                                                            
pcidev="0/20/3"                                                             

diskimage="/home/tj/vms/obsd.raw"                                          
vmname=$(basename $diskimage .raw)                                          

#setup="true"                                                               

if [ $# -ge 1 ]                                                             
then                                                                        
        for x in $@                                                         
        do                                                                  
                interfaces="$interfaces -t $x"                              
        done                                                                
else                                                                        
        echo 'usage: $0 tapDev [tapDev tapDev ...]'                         
        exit                                                                
fi                                                                          

echo starting vm $vmname from image $diskimage with interfaces ${interfaces}

if [ -z $setup ]                                                            
then                                                                        
        bhyve -A -D -H -P -S -u -w -c 4 -m 8G                           \   
                -s 0,amd_hostbridge                                     \   
                -s 3,virtio-blk,${diskimage}                            \   
                -s 5,passthru,${pcidev}                                 \   
                -s 10,virtio-net,tap0                                   \   
                -s 20,virtio-rnd                                        \   
                -s 31,lpc -l com1,stdio                                 \   
                -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \   
                $vmname                                                     
else                                                                        

        vmname=openbsd-installer                                            
        echo "Trying to run installer, remember to configure serial"        
        echo "          set tty com0"                                       
        bhyve -A -D -H -P -S -u -w -c 4 -m 8G                           \   
                -s 0,amd_hostbridge                                     \   
                -s 3,virtio-blk,${diskimage}                            \   
                -s 4,ahci-hd,/home/tj/miniroot76.img                    \   
                -s 5,passthru,${pcidev}                                 \   
                -s 10,virtio-net,tap0                                   \   
                -s 20,virtio-rnd                                        \   
                -s 31,lpc -l com1,stdio                                 \   
                -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \   
                $vmname                                                     
fi                                                                          

bhyvectl --destroy --vm=$vmname

In the FreeBSD VM I want to override the default driver so I can test my port this is done by adding a block line to /etc/rc.conf :

devmatch_blocklist="if_iwlwifi"

My work on FreeBSD is supported by the FreeBSD Foundation , you can contribute to improving FreeBSD with code, documentation or financially by donating to the FreeBSD Foundation .

FreeBSD Network Status Week 48 2024

Goings on

14.2 Release Builds should have started today (or be starting?). If all goes to schedule the release will be out the door and announced on Monday the 3th of December 2024.

BSD Devroom at FOSDEM 2024

The BSD Devroom is back again at FOSDEM this year. The CFP closes on Sunday so this is your last chance to submit something. I hear The FreeBSD Foundation has a stand again this year, so if you want to come and see me in person you could find me there.

Submit early submit often! Now is a bit late, but better than not submitting.

Welcome to the BSD Devroom Call For Participation. The BSD Devroom aims
to provide a dedicated space for presentations covering BSD operating
system family.

Key dates

Proposals can be submitted by October the 30th, 2024
Submission deadline : 1st December 2024 Brussels time
Announcement of selected talks : 15th of December 2024
Conference dates : 1 & 2 February 2025
BSD devroom date : Saturday February 1st, 2025 afternoon (second half)

CFP is here

It was stab week:

On Mon, Nov 25, 2024 at 01:01:05AM -0800, Gleb Smirnoff wrote:
T> This is an automated email to inform you that the November 2024 stabilization week
T> started with FreeBSD/main at main-n273822-ff4c19bb5427, which was tagged as
T> main-stabweek-2024-Nov.

At Netflix testing we didn't discover any new regressions comparing to the
October stabweek.  My personal machines on the new stanpshot are also doing
well.  I didn't receive any emails reporting regressions through the last days,
hence releasing the advisory freeze.

P.S. We are aware of regression in ZFS, that happened between September and
October stabweeks and are working on a reliable reproducer.  A panic happens
when using md(4) device backed by a file on ZFS.

Sounds like there wasn't anything big. This is a reminder that you can directly test your workloads against upcoming releases at a suspected stable point of the tree.

Early testing helps avoid late surprises.

Transport

This is a collision of transport and firewalls ( and I guess packet forwarding). Using the new __tcp_get_flags call teach ppp, pf and ipf about the Accurate ECN AE flag. ECN is one way for the network to communicate with a flow about network conditions.

Small change to memory copying for udp_input , memcpy doesn't need concern itself with overlapping segments and so can be faster.

Network Stack

Fix setting the Don't Fragment bit when tunneling IPv6 over IPv4.

Netdev

Some e1000 changes, the main one here is a further attempt to better implement auto negotiation according to the standard. The commit message has a lot of details and is worth a click through.

More names for T6 cards:

Change the default mode of igc in promisc mode to not show bad packets.

Improvements to mlx5 and family.

Firewalls

Mostly fixes in pf, the first commit here is an improvement to IPv6 fragment handling which just sort of makes me sad. Nothing to do with the change and everything to do with networking.

We have other firewalls too!

User Tooling

With the -n flag the any addr ( 0.0.0.0/0 ) is now printed as default rather than the string default .

Align domain entry so all upper case domains should work. I was expecting this to be a hangover from the past, but RFC 8881 which specifies this behaviour is from 2023. I bet this has confused a lot of people in the past as we are generally pretty loose with capitalization and domain names.

Other stuff

I'm putting this here to highlight it:

Kernel TLS is now enabled by default in kernels including KTLS
support.  KTLS is included in GENERIC kernels for aarch64,
amd64, powerpc64, and powerpc64le.

Please Send Feedback

I would love to know if this summary was any help, if it was, or if you think I should cover other thing please let me know (thj@freebsd.org).

emaste@ would like to know how people are finding these updates. Is it from the mailing list emails, frequent reader of my blog, somewhere else? Please, let me know.

If you find a typo or have a correct let me know and I'll thank you at the end here.

Thanks to:

  • emaste@ for correcting the order of change inthe netstat -n commit.

You can see all prior posts here. ( rss )


My work on FreeBSD is supported by the FreeBSD Foundation , you can contribute to improving FreeBSD with code, documentation or financially by donating to the FreeBSD Foundation .

FreeBSD Network Status Week 47 2024

A cold snap has hit my part of Europe hard and it is feeling very seasonal. The rate of commits seems down this week, that is probably partially the 14.2 release and if I project a bit the weather.

Some nice changes coming in the WiFi stack and to other tools. In other stuff that I'm not covering here there is some good progress on risc-v support.

Goings on

Fall 2024 FreeBSD Summit

The videos from the Fall FreeBSD summit have been parted out and are available in this playlist:

BSD Devroom at FOSDEM 2024

Welcome to the BSD Devroom Call For Participation. The BSD Devroom aims
to provide a dedicated space for presentations covering BSD operating
system family.

Key dates

Proposals can be submitted by October the 30th, 2024
Submission deadline : 1st December 2024 Brussels time
Announcement of selected talks : 15th of December 2024
Conference dates : 1 & 2 February 2025
BSD devroom date : Saturday February 1st, 2025 afternoon (second half)

CFP is here

FreeBSD 14.2 Code Slush in Effet

THe 14.2 Code Slush is running and final (hopefully changes are landing) RC1 build should have started and we might make it without an RC2 build.

The 14.2-RELEASE cycle schedule as it stands now is:

 ports quarterly branch:  October 1, 2024
 reminder email:          October 7, 2024
 stable/14 slush:         October 18, 2024
 doc/ tree slush:         October 25, 2024
 releng/14.2 branch:      November 1, 2024
 BETA1 build starts:      November 1, 2024
 BETA2 build starts:      November 8, 2024
 doc/ tree tag:           November 8, 2024
 BETA3 build starts:      November 16, 2024
 ports package builds:    November 16-18, 2024
    <<< WE ARE HERE >>>
 RC1 build starts:        November 22, 2024
 RELEASE build starts:    November 29, 2024
 RELEASE announcement:    December 3, 2024
 14.1 EoL:                March 31, 2025
 14.2 EoL:                September 30, 2025
 14.* EoL:                November 30, 2028

https://wiki.freebsd.org/Releng/14.2ISSUES

Transport

I think this is mostly tidying things up, the rename is an adventure in sed. Writing these weekly and sitting on the regular transport calls it is sometimes hard to point at new features. I think the consistent development is a feature.

Network Stack

A nice optimization in forwarding code when we detect a blackholed route.

Netdev

Driver improvements around TSO to support Accurate ECN in the future in igc, em and ixgbe.

Lots of movement in WiFi world. adrian@ has popped back up and started working out a queue of stuff for rtwn. There should be some good improvements to throughput and consistency in HEAD if you want to grab that and try.

We are seeing some steps towards higher rates in rtwn as well.

bz@ landed changes to move iwlwifi from the source tree (and therefore shipped kernel modules) out of the tree. This includes support for adding the needed firmware from the installer, assuming you have internet access in the installer.

Firewalls

Netlink API improvements for pf

Fixes to tests and the build. Lots of tests use the packet accounting in FreeBSD which when we started we quickly learned was missing a lot of packets.

Please Send Feedback

I would love to know if this summary was any help, if it was, or if you think I should cover other thing please let me know (thj@freebsd.org).

If you find a typo or have a correct let me know and I'll thank you at the end here.

You can see all prior posts here. ( rss )

This is the 9th edition of the Weekly Network Status report, currently I plan for the last report of the year to be written on the 13th of December.

I have received a lot of very positive feedback which helps me to write these. Right now I intend to start writing these again in the new year, I might do a summary for the Solstice month all at once, or I'll just do a week. I'm not sure yet.


My work on FreeBSD is supported by the FreeBSD Foundation , you can contribute to improving FreeBSD with code, documentation or financially by donating to the FreeBSD Foundation .