FreeBSD Network Status Week 50 2024

Pretty quiet week, driver changes normally come as large series and there aren't a lot this week. This week isn't an outlier.

Goings on

BSD Devroom at FOSDEM 2024

The schedule for the FOSDEM is online . The BSD Devroom is only a half day with the following sessions:

How FreeBSD security audits have improved our security culture
Wake up, FreeBSD! Implementing Modern Standby with S0ix
Tracking bulk builds in pkgsrc - from Cloud to NetBSD Native
High Performance Packet filtering in BSD. A holistic review
A packet's journey through pf
Making NetBSD as a fast(er) booting microvm
Writing about FreeBSD
FreeBSD audit source and other syslog-ng news

I will be talking about writing about FreeBSD, which means right now I am writing about talking about writing. I'm going to cover avenues in, how and where to publish stuff and the process and idea behind this series of reports.

Network Stack

Swap a tab for a space so in6_ifaddr can be grepped for.

a6a0b8d50e87 in6_var.h: make struct in6_ifaddr declaration searchable with grep(1)

ICMP responses are rate limited, but there might be issues if an attacker could discover the rate limit so a jitter value is applied to the rate limit. This improves the description:

eba715c544ea icmp.4: improve icmplim and add icmplim_jitter description

Netdev

More progress in cxgbe to make TLS offload available by default.

Tidy up a module ordering issue that leads to a hang during module load for iwlwifi.

87e140a5c6f8 iwlwifi: avoid (hard) hang on loading module

API improvements in net80211.

Lot of updates to rtwn moving towards HT (80211n) and VHT (80211ac), adrian@ is talking about doing 50Mbit right now from rtwn which is great progress.

Firewalls

Add a new flag to pfctl to only reset counters for table entries which items logged against them. This preserves the zeroed timestamp on entires with nothing logged.

5b59b0c61e29 pfctl: add -T reset to touch pfras_tzero only for non-zero entries

Catch a state tracking issue after the TCP_AE introductions in the last few weeks in ipfw.

9ea8d692f4cb ipfw: use only needed TCP flags for state tracking

Tidying in netlink.

User Tooling

Add option to allow setting of mount port number for nfs.

0e8a36a2ab12 mount_nfs.c: Add an NFS mount option to set a port# for Mount

Other stuff

Please Send Feedback

This is the 12th Network status report and the final one for 2024.I have enjoyed writing these and plan to pick them up again in the New Year.

I would love to know if this summary was any help, if it was, or if you think I should cover other thing please let me know (thj@freebsd.org).

If you find a typo or have a correct let me know and I'll thank you at the end here.

You can see all prior posts here. ( rss )

My work on FreeBSD is supported by the FreeBSD Foundation , you can contribute to improving FreeBSD with code, documentation or financially by donating to the FreeBSD Foundation .

FreeBSD Network Status Week 49 2024

Goings on

BSD Devroom at FOSDEM 2024

The CFP for the BSD Devroom at FOSDEM closed over the weekend, submitters have been told to expect responses on the 15th of December. I'll provide a run down of upcomming talks in the first status update of 2025.

FreeBSD 14.2 was released

Congratulations to everyone involved and a big thanks to the RE team for getting FreeBSD 14.2 out. The release notes are here

CFT for rtwn, ath and iwn

hi!

I've been working on some net80211 clean-ups and rtwn bugfixes in
preparation for getting the 11ac support for rtwn landed into the tree.

If you're using rtwn, ath or iwn then the latest stuff I've done in -HEAD
may affect you. Please update to the latest -HEAD and let me know if
anything has regressed! (or if it still works fine, that'd be nice to know
too!)

thanks,

-adrian

adrian@ posted a cft for users of rtwn, ath and iwn after recent CURRENT changes

Network Stack

Lots of tidying in netlink.

Tidying in in6_pcbbind . This is a very difficult part of the network stack to work in and tidying here makes things much easier.

Transport

Tidy ups in tcp_hpts .

Netdev

Tie receive packet headers to the correct NUMA domain, this sort of change can help a lot in systems with multiple NUMA domains to reduce cross domain traffic. This happening in iflib makes it 'free' for all iflib drivers.

3d642b0f71c5 iflib: Set the NUMA domain in receive packet headers

Add a big list of skus to igc

850f78d5a191 igc.4: Add I226 and other additions to supported list

adrian@ landed some macros that help with handling ht rates, this makes drivers cleaner:

and he landed a bunch of improvements to rtwn from debugging. Chat in the wifi irc channel suggest that rtwn might get a lot faster soon. If you use rtwn you should test HEAD now to help catch any regressions.

bz@ has landed HT and VHT improvements for the Linux KPI.

Firewalls

Following up on the pf EIM NAT support Damjan Jovanovic has implemented support for EIM NAT for libalias users. If you use ipfw, ppp or netgraph you can now experiment with a more transparent NAT mapping.

Some stability fixes in pf and potential panic fixes.

Other stuff

First up, documentation of a wonderful hack from phk@ to show network traffic in bits per second by running with an 8 second interval. This had a few confused follow ups, it is so simple so people thought it couldn't possibly work.

b5a8abe9502e How to show interface traffic in bits per second

Following on from the excellent Fall Summit talk by Ian from Metify on rural broadband, do a better job of documenting the cellular hardware we support.

53484f516ac4 u3g.4: Fall cleaning

I like the commit message on this one:

4d12c7b9df66 cu.1: describe better + tag spdx

cu.1: describe better + tag spdx Preserve over 40 years of "call UNIX" BSD heritage while answering "one line about what it does" e.g.

"how do I get a serial console?" "% apropos serial"

Make things easier for users to understand, but also preserve the past.

Please Send Feedback

That is all for this week. Next week will be the last report of the year, I'll be travelling (please say hi if you are at 38c3) and I think a report break might be nice.

The first report will probably be on the 10th of January, look for it where you found this one.

I would love to know if this summary was any help, if it was, or if you think I should cover other thing please let me know (thj@freebsd.org).

If you find a typo or have a correct let me know and I'll thank you at the end here.

You can see all prior posts here. ( rss )

My work on FreeBSD is supported by the FreeBSD Foundation , you can contribute to improving FreeBSD with code, documentation or financially by donating to the FreeBSD Foundation .

Everyone gets a WiFi

Building on my earlier blog post on setting up PCIe pass through with bhyve , I've found a little time to move things around in my test bed and try something stupid.

Bitcoin is certainly a terrible thing, but it has made some useful components super cheap (though they are getting rare again). One feature of bitcoin mining is that it is super low usage on the PCIe bus, you feed your GPU something to do, it does it, and if you find a result you send a couple of hundred bytes back. That leaves a lot of bandwidth left over.

Miners knew what to do with this took advantage of PCIe topology and added switches, expanders and risers to get as many GPUs per motherboard as they could.

We can apply their tools for other things like driver development.

The idea is this, get a bunch of devices onto a VM host through the use of cheap downstream PCIe switches, have the host ignore them and make them available to guests on the host.

This makes it possible to do a lot of concentrated testing. You can for example stick 4 different revisions of a card on host and simultaneously test them all on the latest current, or stick 3 versions of the same card on the bus and test them on CURRENT, STABLE and STABLE-1 at the same time to unsure there aren't regressions.

For porting you could run guests for FreeBSD, Linux and OpenBSD and compare functionality and commands to the card for debugging.

Proof of Concept

I had this idea early this year and managed to get a POC together in a short 12 months of forgetting about it and an afternoon of moving machines around in the attic.

I picked up two types of PCIe expander, PCIe x1 -> x16 risers (which aren't used right now) and an ASMedia ASM1184e 4-Port PCIe x1 Gen2 Packet Switch'. Later I got a cheap 12 GPU frame, adding a final piece of structure needed to make the set up practical.

The PCIe expander uses a USB3-A cable to carry the PCIe bus out an adapter card connected to the motherboard out to the switch board. The switch board has 4 PCIe x1 slots for cards.

I mounted all this on the frame with 2 Intel AX210 WiFi cards for a POC. Pulling the card to check functionality the switch appears in pciconf -lf like so:

pcib6@pci0:5:0:0:       class=0x060400 rev=0x00 hdr=0x01 vendor=0x1b21 device=0x1184 subvendor=0x1b21 subdevice=0x118f
    vendor     = 'ASMedia Technology Inc.'
    device     = 'ASM1184e 4-Port PCIe x1 Gen2 Packet Switch'
    class      = bridge
    subclass   = PCI-PCI
pcib7@pci0:6:1:0:       class=0x060400 rev=0x00 hdr=0x01 vendor=0x1b21 device=0x1184 subvendor=0x1b21 subdevice=0x118f
    vendor     = 'ASMedia Technology Inc.'
    device     = 'ASM1184e 4-Port PCIe x1 Gen2 Packet Switch'
    class      = bridge
    subclass   = PCI-PCI
pcib8@pci0:6:3:0:       class=0x060400 rev=0x00 hdr=0x01 vendor=0x1b21 device=0x1184 subvendor=0x1b21 subdevice=0x118f
    vendor     = 'ASMedia Technology Inc.'
    device     = 'ASM1184e 4-Port PCIe x1 Gen2 Packet Switch'
    class      = bridge
    subclass   = PCI-PCI
pcib9@pci0:6:5:0:       class=0x060400 rev=0x00 hdr=0x01 vendor=0x1b21 device=0x1184 subvendor=0x1b21 subdevice=0x118f
    vendor     = 'ASMedia Technology Inc.'
    device     = 'ASM1184e 4-Port PCIe x1 Gen2 Packet Switch'
    class      = bridge
    subclass   = PCI-PCI
pcib10@pci0:6:7:0:      class=0x060400 rev=0x00 hdr=0x01 vendor=0x1b21 device=0x1184 subvendor=0x1b21 subdevice=0x118f
    vendor     = 'ASMedia Technology Inc.'
    device     = 'ASM1184e 4-Port PCIe x1 Gen2 Packet Switch'
    class      = bridge
    subclass   = PCI-PCI

With the switch there I added the two PCIe WiFi cards and rebooted:

pcib6@pci0:5:0:0:       class=0x060400 rev=0x00 hdr=0x01 vendor=0x1b21 device=0x1184 subvendor=0x1b21 subdevice=0x118f
    vendor     = 'ASMedia Technology Inc.'
    device     = 'ASM1184e 4-Port PCIe x1 Gen2 Packet Switch'
    class      = bridge
    subclass   = PCI-PCI
pcib7@pci0:6:1:0:       class=0x060400 rev=0x00 hdr=0x01 vendor=0x1b21 device=0x1184 subvendor=0x1b21 subdevice=0x118f
    vendor     = 'ASMedia Technology Inc.'
    device     = 'ASM1184e 4-Port PCIe x1 Gen2 Packet Switch'
    class      = bridge
    subclass   = PCI-PCI
pcib8@pci0:6:3:0:       class=0x060400 rev=0x00 hdr=0x01 vendor=0x1b21 device=0x1184 subvendor=0x1b21 subdevice=0x118f
    vendor     = 'ASMedia Technology Inc.'
    device     = 'ASM1184e 4-Port PCIe x1 Gen2 Packet Switch'
    class      = bridge
    subclass   = PCI-PCI
pcib9@pci0:6:5:0:       class=0x060400 rev=0x00 hdr=0x01 vendor=0x1b21 device=0x1184 subvendor=0x1b21 subdevice=0x118f
    vendor     = 'ASMedia Technology Inc.'
    device     = 'ASM1184e 4-Port PCIe x1 Gen2 Packet Switch'
    class      = bridge
    subclass   = PCI-PCI
pcib10@pci0:6:7:0:      class=0x060400 rev=0x00 hdr=0x01 vendor=0x1b21 device=0x1184 subvendor=0x1b21 subdevice=0x118f
    vendor     = 'ASMedia Technology Inc.'
    device     = 'ASM1184e 4-Port PCIe x1 Gen2 Packet Switch'
    class      = bridge
    subclass   = PCI-PCI
iwlwifi0@pci0:7:0:0:    class=0x028000 rev=0x1a hdr=0x00 vendor=0x8086 device=0x2725 subvendor=0x8086 subdevice=0x0024
    vendor     = 'Intel Corporation'
    device     = 'Wi-Fi 6E(802.11ax) AX210/AX1675* 2x2 [Typhoon Peak]'
    class      = network
iwlwifi1@pci0:10:0:0:   class=0x028000 rev=0x1a hdr=0x00 vendor=0x8086 device=0x2725 subvendor=0x8086 subdevice=0x0024
    vendor     = 'Intel Corporation'
    device     = 'Wi-Fi 6E(802.11ax) AX210/AX1675* 2x2 [Typhoon Peak]'
    class      = network

Each card can be used on the host simultaneously. I took the cards from the host and enabled PCIe pass through for AMD in /boot/loader.conf :

pptdevs="7/0/0 10/0/0"          # intel ax210 wifi
hw.vmm.amdvi.enable=1

Finally I used the script from the previous post to run two FreeBSD CURRENT VMs, giving each one of the cards and the successfully came up and could do the wireless.

The cards can scan from here, harder workloads are a future problem. I want to install 2 more expanders on this machine and more cards to see how things work when heavily loaded. This 12 GPU bracket has 35 holes so I could add a lot of stuff.

There are PCIe bus overheads here, I'm reducing down a PCIe4 bus to PCIe2 for the Wifi cards, but they aren't exactly high performance devices in PCIe terms. I might also hit power limits, I didn't really want to ask about doing this POC, I figured I wouldn't get good answers based on similar questions on the FreeBSD mailing lists in the last year.

This seems to work, scaling is to be seen.

My work on FreeBSD is supported by the FreeBSD Foundation , you can contribute to improving FreeBSD with code, documentation or financially by donating to the FreeBSD Foundation .

PCIe pass through for Wifi Dev

For the last little while I have been working on porting a WiFi driver. I initially planned to use one of my testbed machines, but the card I had wasn't supported by the original driver and locked up the system.

While waiting for another card I spun up development on the Morefine M6 which had failed complete at its proposed role as a desktop mirrored storage array.

The Morefine M6 has an Intel N200 SoC and supported Intel wifi.

I always set out to try PCIe pass through and the FreeBSD wiki page provided most of what I needed .

On the host you need to exclude the PCIe devices you wish to give to a guest. You need to find the PCIe address for the device to exclude, pciconf -lv is good for this:

# pciconf -lv 
...
none2@pci0:0:20:2:      class=0x050000 rev=0x00 hdr=0x00 vendor=0x8086 device=0x54ef subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'                        
    device     = 'Alder Lake-N PCH Shared SRAM'                 
    class      = memory                                         
    subclass   = RAM                                            
ppt0@pci0:0:20:3:       class=0x028000 rev=0x00 hdr=0x00 vendor=0x8086 device=0x54f0 subvendor=0x8086 subdevice=0x0070
    vendor     = 'Intel Corporation'
    device     = 'CNVi: Wi-Fi'                                  
    class      = network                                        
ig4iic0@pci0:0:21:0:    class=0x0c8000 rev=0x00 hdr=0x00 vendor=0x8086 device=0x54e8 subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    class      = serial bus                                     
ig4iic1@pci0:0:21:1:    class=0x0c8000 rev=0x00 hdr=0x00 vendor=0x8086 device=0x54e9 subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    class      = serial bus 
...

I have passed PCIe device 0:20:3 to my guests, in the pciconf list you will see the driver assigned to the device followed by the bus and the address. So the ig4 i2c driver has devices ig4iic0@pci0:0:21:0 and ig4iic1@pci0:0:21:1 and the 'Wi-Fi' device has been taken by ppt0.

This is configured by adding the PCIe address to /boot/loader.conf :

pptdevs="0/20/3"                # intel wifi

We need to give our PCIe device to our VM, I am simple person and do all my bhyve through vmrun.sh and wrapper shell scripts. For FreeBSD the script is this:

#!/bin/sh                                                                   

diskimage="not set"                                                         
vmname="not set"                                                            
pcidev="0/20/3"                                                             

diskimage="/home/tj/vms/fbsd-iwx.raw"                                       
vmname=$(basename $diskimage .raw)                                          

if [ $# -ge 1 ]                                                             
then                                                                        
        for x in $@                                                         
        do                                                                  
                interfaces="$interfaces -t $x"                              
        done                                                                
else                                                                        
        echo 'usage: launchfreebsd.sh tapDev [tapDev tapDev ...]'           
        exit                                                                
fi                                                                          

echo starting vm $vmname from image $diskimage with interfaces ${interfaces}

sh /usr/share/examples/bhyve/vmrun.sh \                                     
        -c $(nproc) \                                                       
        -m 8G \                                                             
        -p ${pcidev} \                                                      
        ${interfaces} \                                                     
        -d $diskimage \                                                     
        $vmname

And for OpenBSD:

#!/bin/sh                                                                   

diskimage="not set"                                                         
vmname="not set"                                                            
pcidev="0/20/3"                                                             

diskimage="/home/tj/vms/obsd.raw"                                          
vmname=$(basename $diskimage .raw)                                          

#setup="true"                                                               

if [ $# -ge 1 ]                                                             
then                                                                        
        for x in $@                                                         
        do                                                                  
                interfaces="$interfaces -t $x"                              
        done                                                                
else                                                                        
        echo 'usage: $0 tapDev [tapDev tapDev ...]'                         
        exit                                                                
fi                                                                          

echo starting vm $vmname from image $diskimage with interfaces ${interfaces}

if [ -z $setup ]                                                            
then                                                                        
        bhyve -A -D -H -P -S -u -w -c 4 -m 8G                           \   
                -s 0,amd_hostbridge                                     \   
                -s 3,virtio-blk,${diskimage}                            \   
                -s 5,passthru,${pcidev}                                 \   
                -s 10,virtio-net,tap0                                   \   
                -s 20,virtio-rnd                                        \   
                -s 31,lpc -l com1,stdio                                 \   
                -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \   
                $vmname                                                     
else                                                                        

        vmname=openbsd-installer                                            
        echo "Trying to run installer, remember to configure serial"        
        echo "          set tty com0"                                       
        bhyve -A -D -H -P -S -u -w -c 4 -m 8G                           \   
                -s 0,amd_hostbridge                                     \   
                -s 3,virtio-blk,${diskimage}                            \   
                -s 4,ahci-hd,/home/tj/miniroot76.img                    \   
                -s 5,passthru,${pcidev}                                 \   
                -s 10,virtio-net,tap0                                   \   
                -s 20,virtio-rnd                                        \   
                -s 31,lpc -l com1,stdio                                 \   
                -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \   
                $vmname                                                     
fi                                                                          

bhyvectl --destroy --vm=$vmname

In the FreeBSD VM I want to override the default driver so I can test my port this is done by adding a block line to /etc/rc.conf :

devmatch_blocklist="if_iwlwifi"

My work on FreeBSD is supported by the FreeBSD Foundation , you can contribute to improving FreeBSD with code, documentation or financially by donating to the FreeBSD Foundation .

FreeBSD Network Status Week 48 2024

Goings on

14.2 Release Builds should have started today (or be starting?). If all goes to schedule the release will be out the door and announced on Monday the 3th of December 2024.

BSD Devroom at FOSDEM 2024

The BSD Devroom is back again at FOSDEM this year. The CFP closes on Sunday so this is your last chance to submit something. I hear The FreeBSD Foundation has a stand again this year, so if you want to come and see me in person you could find me there.

Submit early submit often! Now is a bit late, but better than not submitting.

Welcome to the BSD Devroom Call For Participation. The BSD Devroom aims
to provide a dedicated space for presentations covering BSD operating
system family.

Key dates

Proposals can be submitted by October the 30th, 2024
Submission deadline : 1st December 2024 Brussels time
Announcement of selected talks : 15th of December 2024
Conference dates : 1 & 2 February 2025
BSD devroom date : Saturday February 1st, 2025 afternoon (second half)

CFP is here

It was stab week:

On Mon, Nov 25, 2024 at 01:01:05AM -0800, Gleb Smirnoff wrote:
T> This is an automated email to inform you that the November 2024 stabilization week
T> started with FreeBSD/main at main-n273822-ff4c19bb5427, which was tagged as
T> main-stabweek-2024-Nov.

At Netflix testing we didn't discover any new regressions comparing to the
October stabweek.  My personal machines on the new stanpshot are also doing
well.  I didn't receive any emails reporting regressions through the last days,
hence releasing the advisory freeze.

P.S. We are aware of regression in ZFS, that happened between September and
October stabweeks and are working on a reliable reproducer.  A panic happens
when using md(4) device backed by a file on ZFS.

Sounds like there wasn't anything big. This is a reminder that you can directly test your workloads against upcoming releases at a suspected stable point of the tree.

Early testing helps avoid late surprises.

Transport

This is a collision of transport and firewalls ( and I guess packet forwarding). Using the new __tcp_get_flags call teach ppp, pf and ipf about the Accurate ECN AE flag. ECN is one way for the network to communicate with a flow about network conditions.

Small change to memory copying for udp_input , memcpy doesn't need concern itself with overlapping segments and so can be faster.

949190c5afbc udp: Prefer memcpy() over bcopy()

Network Stack

Fix setting the Don't Fragment bit when tunneling IPv6 over IPv4.

c94d6389e428 ipsec: fix IPv6 over IPv4 tunneling.

Netdev

Some e1000 changes, the main one here is a further attempt to better implement auto negotiation according to the standard. The commit message has a lot of details and is worth a click through.

More names for T6 cards:

5c15094916bf cxgbe(4): Update the board names of the T6 OCP cards.

Change the default mode of igc in promisc mode to not show bad packets.

Improvements to mlx5 and family.

Firewalls

Mostly fixes in pf, the first commit here is an improvement to IPv6 fragment handling which just sort of makes me sad. Nothing to do with the change and everything to do with networking.

We have other firewalls too!

User Tooling

With the -n flag the any addr ( 0.0.0.0/0 ) is now printed as default rather than the string default .

9206c7996198 usr.bin/netstat: -n should not print symbolic names

Align domain entry so all upper case domains should work. I was expecting this to be a hangover from the past, but RFC 8881 which specifies this behaviour is from 2023. I bet this has confused a lot of people in the past as we are generally pretty loose with capitalization and domain names.

0347ddf41f42 nfs_commonsubs.c: Make all upper case user domain work

Other stuff

I'm putting this here to highlight it:

Kernel TLS is now enabled by default in kernels including KTLS
support.  KTLS is included in GENERIC kernels for aarch64,
amd64, powerpc64, and powerpc64le.

Please Send Feedback

I would love to know if this summary was any help, if it was, or if you think I should cover other thing please let me know (thj@freebsd.org).

emaste@ would like to know how people are finding these updates. Is it from the mailing list emails, frequent reader of my blog, somewhere else? Please, let me know.

If you find a typo or have a correct let me know and I'll thank you at the end here.

Thanks to:

emaste@ for correcting the order of change inthe netstat -n commit.

You can see all prior posts here. ( rss )

My work on FreeBSD is supported by the FreeBSD Foundation , you can contribute to improving FreeBSD with code, documentation or financially by donating to the FreeBSD Foundation .