FreeBSD Network Status Week 41 2024

Hey folks, here we are rounding out another week of development. This Network Status Report is an experiment I am making on documenting what has been happening in the FreeBSD network stack by generating reports with the help of some simple tooling. This is the third such report, but the first one I've really told anyone about.

The previous reports are available here , some context and goals are available in the first weeks report .

A big change for this week is streaming of the report writing process. I'm hoping that by being more open about this there will be a weekly chance for community engagement - at least for people that like the network stack.

Goings on

Since last week I have integrated collating notes into my tooling (and if you consumed the stream broke the script a little). This means I can capture things going by on the mailing lists more easily for discussion later.

The bugzilla storm continues, when it starts to slow down I'll review pulling in interesting bugs.

What I want from reviews and bugs is a list of interesting things in the last week. That might be new items, but it is also likely to be items that have had a change in the last week, lots of comments, or have finally closed. Landing commits aren't so interesting. I think I have the bugzilla query sorted out, but I cannot for the life of me get sense from the phabricator API.

If you can generate queries that sort of match what I want AND they will give me plain text summaries as helpful as a git --oneline I'd love to see them.

Fall 2024 FreeBSD Summit

On the 7th and 8th of November 2024 there will be a FreeBSD Summit kindly hosted by NetApp in their San Jose campus.

So far the program includes:

  • Pawel Dawidek, Fudo Security on "FreeBSD Security Improvements"
  • Dorr Clark, NetScaler on “Using FreeBSD in Products"
  • George Neville-Neil on "OSDB: Turning the Tables on Kernel Data"
  • Dr. Marshall Kirk McKusick on “History of the BSD Daemon”
  • And more!

The summit is open to the public, with a registration fee of US $150.

Registration and event information is available here:

axgbe CFT

zlei@ has an open call for testing for come changes to the axgbe driver. This changes how the axgbe driver handles the promisc flag, zlei@ doesn't have hardware available to test. If you use axgbe then you should test and report results on the phabricator review.

https://reviews.freebsd.org/D46794

Transport

Oddly not TCP caught in my filter this week, but there have been some improvements around the SCTP API.

tuexen@ has been doing some review of locking and socket options. Generally the socket layer is quite complex, getting this right is difficult.

Netdev

kbowling@ MFC'd a lot of stuff from the Intel driver changes we covered the past two weeks. That is great news if you are on a stable branch of FreeBSD.

A big change is the re-addition of Adaptive Interrupt Mode for the e1000 series NICS (including lem , em and igb ). AIM gives a balance between latency when there are relatively low packet rates and performance when the link is very busy.

In most cases kbowling@ says: 

this might be worth a few sys% on common CPUs, but may be meaningful when
multiplied such as if_lagg, if_bridge and forwarding setups.

In WiFi land bz@ landed a nice rtw89 panic fix:

And we see some other bits of tidying up in cxgbe , mlx5 and iflib .

Firewalls

A mixture of tidy ups with several changes coming through from OpenBSD. If I were to guess (and I am!) many of these are from presentations and conversations at EuroBSDCon. If I were to ask kp@ he would tell me this was part of an ongoing continuous maintenance project sponsored by Netgate.

And the continued netlinkification of pf .

igoro@ made some tidy up commits to dummymbuf. While I have seen commits go by I hadn't looked into dummymbuf(4) yet. This is test kernel module for unusual mbuf layouts which hooks into the pfil (firewall) layer.

For continued compatibility with libpcap some struct definitions for pflogd were moved out of the header file, preventing others from using them.

User Tooling

Fix stopping sendmail during shutdown.

And finally a big change to kyua, skipped tests no longer report as passing.

Please Send Feedback

As with last week are are at ~50 minutes as I get to this part of the report.

I am going to disseminate this one much further, probably to the freebsd-net and current mailing lists.

I would love to know if this summary was any help, if it was, or if you think I should cover other thing please let me know (thj@freebsd.org).

If you find a typo or have a correct let me know and I'll thank you at the end here.

  • Boris asked for there to be an rss feed, so there is now one here
  • Graham Perrin hight lighted a typo in the tags ( tags->tag ) link.
  • Jim Thompson told me off for guessing.

You can see all prior posts here. ( rss )

My work on FreeBSD is supported by the FreeBSD Foundation , you can contribute to improving FreeBSD with code, documentation or financially by donating to the FreeBSD Foundation .

FreeBSD Network Status Week 40 2024

Here we are for round 2, last weeks report can be and a little context found here: FreeBSD Network Status Week 39 2024 .

All of the reports so far are available via the networkstatus tag on my blog.

Goings on

Since last week I haven't touched any of the tooling, I have been busy doing something else and it is a good exercise in making these reports "minimal touch". There are some manual edits in the generated template I have had to make today which I will roll into the generation script.

I do read my email (no matter what I might claim) so most commits going by aren't completely new to me.

I am yet to add tooling for getting reviews and interesting bugs, reviews I'm not sure how to figure out as phabricator doesn't have a friendly API. My email contains everything in #transport , #network and #wireless so worst case I just read from email.

Bugzilla has a useable API, Mark Linimon (linimon@) is doing a heroic review of old bugs and squashing those which are out of date. That might be important for you for two reasons, if you have an old bug you care about you should respond to the bug email or it'll be closed and it makes a search of the form "what changed in the last week" far too noisy to interpret right now.

I still plan to include bugs and reviews though.

One point of discussion this week was support of NOINET6 in the kernel build. Users reported that NOINET6 build has been failing due to some changes in pf and two questions were raised. Why are you disabling IPv6 is left for a higher power to explain. Support for non-standard kernel builds is going to get a statement in the handbook or similar to clarify how what is expected of users and developers when running something other than GENERIC.

Anyway, what has been going on in the network stack this week?

My log command 

git log --format="%h" --after="last week" --before="now"

found 133 commits from which I thought 39 were relevant for this update.

Transport

The regular #transport meeting occured yesterday . I think a good summary of the meeting and the last week of commits is covered by the second commit in my list 'small cleanup'. The #transport group make continual progress and this call is tracking a few longer running reviews.

I took the call as an opportunity to ask about netdump performance having done one for the first time. teuxen@ made some debugging suggestions and in email after the call we think performance should be ok on 'data center' links. Certainly better than the 5Mbit/s I saw in my test. I will do some more investigation once my current project starts generating enough crash dump to be a problem.

Netdev

Last week saw a flurry of driver updates, this week looks to just be follow ups to that work. Some commits to the Intel driver which were updated last week:

And some tidy ups in other places:

dougm@ has been doing some on the the rounding logic in libkern which hit mlx:

For WiFI there is some organising in LinuxKPI 80211.

Firewalls

Lots of change in pf, I would say this falls into the category of "continuous improvement". pf changes are about half of the network stack stuff I pulled out this week and while there are no major features. Small steps are how things get really good.

User Tooling

Bug fixes in nfs and a robustness change in the dhclient code.

Please Send Feedback

End of writing this week is at ~50 minutes this week, which makes two reports done at an hour each. I'll use those 10 minutes to fix the generation stuff I hand wrote today. Or make coffee.

Fewer tent pole features coming through, but those can't land everyday. I am in two minds about how frequently this should run, a monthly report would be more likely to always carry some meat, but then I would loose an entire Friday once a month to writing about work rather than doing it and I think that would make me a bit sad.

I would love to know if this summary was any help, if it was, or if you think I should cover other thing please let me know (thj@freebsd.org).

If you find a typo or have a correct let me know and I'll thank you at the end here.

Thanks to: - mgdm

You can see all prior posts here. ( rss )

My work on FreeBSD is supported by the FreeBSD Foundation , you can contribute to improving FreeBSD with code, documentation or financially by donating to the FreeBSD Foundation .

FreeBSD Network Status Week 39 2024

This is an experiment in creating tooling driven reports from the FreeBSD development ecosystem, git, code review and bug tracking. The goal is to provide an insight into what has happened in FreeBSD in the last little while, but with limited time to gather, assess and write up what is going on. I am only going to focus on "networking", which for this week was about a third of all commits to main (last week it was about half).

It is a process that will lead to things being missed, don't feel bad, let me know and I'll add the stuff you think is important.

This report was generated on Friday 27th September 2024 for the period between "last week" and "now" as git understand them. The git log command was: 

git log --format="%h" --after="last week" --before="now"

With this report the tooling lets me take or leave commits and generates urls and such, but only on the FreeBSD main branch. I'll add stable branches for future reports, but first I want to see what can be done in the allotted time.

I also have searches and a little tooling for dealing with bugs (bugzilla) and reviews (phabricator), but neither of these platforms are as friendly to program as git. That should tell you what they are like.

I plan to write a few of these and if there is demand decided if the time investment is worth it. I'm only going to continue if I think it is worth the work.

Goings on

There are two important events invisible to the development system in the last week, EuroBSDcon and the accompanying DevSummit . While they could show up, it doesn't look like anyone used the "Event:" tag on a commit to main in apart from one GSoc commit earlier in the month.

Stab week ran without much happening, there is 1 reported issue discovered with ixl(4) which has been reverted .

170 commits matched my log command and I thought 64 of them were "networking" based on a cursory glance.

Transport

The regular #transport meeting coincided with the FreeBSD DevSummit. The next meeting is on the 3rd of October 2024, if you have issues with transport protocols (TCP, UDP, SCTP) or the socket layer please join, there is a public meeting link on each agenda page ( accessible via this wiki page ). It is more business like than chat ops, which helps us mostly keep the meetings to less than the planned hour.

Two sets of changes landed in my filter, the MAC change is the first in some further work tidying up the MAC SYN code.

The second are tidying up stuff in the TCP stack. The last remanent of a broken attempt to control burstiness of the TCP stack 20 years ago. We turned it back on a couple of years ago and I'm pretty sure it deadlocked connections immediately. TCP is very difficult.

Netdev

Most of the changes this week land in device drivers, it is most of the kernel after all.

bz@ did some git to add missing tracking for vendored Linux WiFi drivers. The commits are 'empty', but contain evil: 

This was done using what I'd call "git magic" provided by Ed no one
else will hopefully ever need again.

Improvements and updates to e1000 (igb, em) have been coming in from kbowling@, with igb updated to 2.25.28-fbsd:

ix/ixgbe has been updated to the latest release (ix-3.3.38) from Intel .

The ixl revert mentioned in the stab week report went in:

On top of this there has been a bunch of tidying up of old code in bfp, iflib and other places by zlei@.

Firewalls

Work has continued to keep FreeBSD pf the best tested firewall:

And some general tidying also landed.

User tooling

tcpdump has been updated to 4.99.5

There have been further commits to nuageinit, which I'm not sure is 100% on topic for a network status report. I do think it is one of the most interesting new tools to be added to FreeBSD for a long time and I'm hoping it'll get a raft of documentation and features to remove the need for cloud-init cloud-init at all.

netstat saw some small updates to libxo output and the ntp man page got some attention.

Please send feedback

As I write this my timer is at 58 minutes , so the 1 hour goal seems doable.

I plan to add interesting bugs and reviews, once I figure out how to get them from the relevant places. Once the tooling is more than sparkling git log I'll put it up on a repo somewhere.

I would love to know if this summary was any help, if it was, or if you think I should cover other thing please let me know (thj@freebsd.org).

If you find a typo or have a correct let me know and I'll thank you at the end here.

Thanks to typos and feedback from: - mgdm - hibby - brd@

You can see all prior posts here. ( rss )

My work on FreeBSD is supported by the FreeBSD Foundation , you can contribute to improving FreeBSD with code, documentation or financially by donating to the Foundation .

acpidumping

Sometimes you need to play with ACPI tables, there are even rumours in the man pages of people patching up their ACPI tables to make them work better. This is not ground I have trod.

FreeBSD has a tool in base called acpidump , but it is lacking in comparison to the full upstream tools you might use on Linux. The full Intel ACPI tools are available in the acpica-tools package on FreeBSD.

With those you can then use the acpidump commands you might be asked to on Linux. If asked what you ACPI tables contain using these tools for a dump looks like this: 

# /usr/local/bin/acpidump -b 
# ls
apic.dat    ecdt.dat    msdm.dat    ssdt11.dat  ssdt5.dat   wsmt.dat
asf!.dat    facp.dat    nhlt.dat    ssdt12.dat  ssdt6.dat   xsdt.dat
bgrt.dat    facs.dat    phat.dat    ssdt13.dat  ssdt7.dat
dbg2.dat    fpdt.dat    sdev.dat    ssdt14.dat  ssdt8.dat
dbgp.dat    hpet.dat    ssdt.dat    ssdt2.dat   ssdt9.dat
dmar.dat    lpit.dat    ssdt1.dat   ssdt3.dat   tpm2.dat
dsdt.dat    mcfg.dat    ssdt10.dat  ssdt4.dat   uefi.dat

These binary files can be decompiled using the iasl tool: 

# iasl *.dat

This gives you a dsl file (source?) for each dat file which might be helpful if you want to check what support is on your motherboard. 

$ head -n 20 lpit.dsl
/*
 * Intel ACPI Component Architecture
 * AML/ASL+ Disassembler version 20230628 (64-bit version)
 * Copyright (c) 2000 - 2023 Intel Corporation
 *
 * Disassembly of lpit.dat, Wed Sep 25 09:42:49 2024
 *
 * ACPI Data Table [LPIT]
 *
 * Format: [HexOffset DecimalOffset ByteLength]  FieldName : FieldValue (in hex)
 */

[000h 0000 004h]                   Signature : "LPIT"    [Low Power Idle Table]
[004h 0004 004h]                Table Length : 000000CC
[008h 0008 001h]                    Revision : 01
[009h 0009 001h]                    Checksum : 05
[00Ah 0010 006h]                      Oem ID : "INSYDE"
[010h 0016 008h]                Oem Table ID : "ADL-P-M"
[018h 0024 004h]                Oem Revision : 00000002
[01Ch 0028 004h]             Asl Compiler ID : "ACPI"

My work on FreeBSD is supported by the FreeBSD Foundation , you can contribute to improving FreeBSD with code, documentation or financially by donating to the Foundation .

Smol KVM

I'm not really sure if I want to own so many computers, but if you want to perform anything like science you need a lot of them. I have a small testbed which I like to say is an experiment in using gaming hardware for doing network science (I can report early that I wish I had enterprise motherboards with lights off management).

Having more machines that I have monitors means that I need some way to manage them independent of switching my monitor to their output. For this reason and so I could work from either home or elsewhere I made sure to buy the only motherboards I could find that had serial ports (thanks Asrock).

I have gotten by sort of coarsely using the serial outputs and the bios feature of power up on AC reset coupled with smart power outlets to give me remote management and a control interface.

This doesn't work very well. I think if the capacitors on the motherboard hold any charge the reset isn't detected, meaning if I forget to power off the smart plug when I shut a machine down I have to do an on/off dance through the smart plug until the machine eventually boots.

The motherboards do not seem to offer the option of sending the bios menu out over serial which I know other machines can do as I have used them. This isn't too much hassle until you add temperamental network cards that require very specific bios configuration to power up. Debugging these issues required me to find a monitor, more keyboards and do a near continuous pull machine rebuild dance for what felt like a year but probably spanned a month or two of 2023.

In a short 300 words, I feel I have a need for a KVM and the spirit of the testbed requires it be cheap.

Options

I know the PiKVM exists, but I can't help think of doing it all myself with FreeBSD every time I consider setting it up and the really surplus Pis I have don't have USB OTG. I have considered building my own USB keyboard connector, but some things I am able to resist doing through sheer will and procrastination.

Recently the Openterface Mini-KVM (oh god is that really the name? I'm going to continue calling it the 'OpenKVM') Crowdsupply crowd funder came by and while I back it, I know I won't get anything for longer than can be planned in. I'm sure this will deliver, Crowdsupply have an exceptional success rate, but hardware is hard and time is sand.

This environment made the super cheap NanoKVM preorder very interesting. Interesting enough to buy one to try out.

NanoKVM

It appeared as a complete surprise this week in quite nice packaging. The device is absolutely tiny with an OLED screen, two buttons and quite a lot of ports. There is Ethernet for network, HDMI for video from the host and 3 USB-C ports; Power, HID and KVM-B (motherboard button header). There are also pin headers to connect up two serial ports for managing things other than PCs.

It is very small and the is completely overwhelmed when you connect all the cables you actually need for it to be useful.

The KVM-B cable connects from USB-C to a break out board, this is then connected to a PCs button headers with jumper cables.

The device powered up on my desk without any trouble, it comes up really quickly. But I haven't run Ethernet yet in the new house and I have no sensible way to offer it a DHCP lease so for testing it is to the attic I must ascend.

Up to the attic I go, plug in the nightmare which is motherboard pin headers in an installed system and power on the device.

With DHCP the little screen shows an acquired IP address matching the alert from my Wifi controller, connecting to this on my phone gives me a black screen.

I climb down the ladder and try from my laptop-desktop, also a black screen, but because computers aren't completely useless like phones is I can inspect the page and see that I have html, but something isn't work.

Up the ladder, poke things, turn off computer, turn on computer using the "Pwr" button on the NanoKVM and it turns on. Try phone again, nothing.

Disconnect NanoKVM from everything, climb down ladder, look at computer and see a login prompt. Look up default creds (admin, admin) and try to login which doesn't work.

Realise the NanoKWM is on my desk disconnected from power.

Climb back up the ladder, reconnect NanoKVM, try phone, get nothing, climb down ladder return to desk. Get a login prompt, log in woo!

My testbed machines boot to Grub and wait for input, this makes dual booting Linux and FreeBSD manageable and you don't have to panic hit a prompt before it times out.

On the KVM I can see the grub menu.

Click on the window and pressing the arrows keys does nothing, bring up the on screen keyboard and using its arrow keys does nothing.

1 frame per connection seems a bit weak, there is an "Update firmware" option in the menu, clicking it offers me to update from 2.0.4 to 2.0.5 I agree. It spins for a while, goes away and now trying to connect gives me just a black screen.

I leave it there for the duration of a meeting about TCP changes in FreeBSD, but nothing.

Pull down attic ladder, climb ladder, unplug NanoKVM power, check the OLED is alive replug power, climb down ladder, stow ladder, reload the page on my laptop-desktop and the NanoKVM is sitting at the login prompt.

Go and buy stuff for supper while the bios times out a check for a peer interface on the 100G card which isn't turned on.

Post update and power cycle and I have enough KVM to watch a boot, log in and type. There is very much latency.

the planet rotates

The next morning I turn my laptop, mount it to the wall in this ridiculous dock I have forced my self to use and try to power on the machine I need to do some work.

The NanoKVM web interface offers three options for button presses, Reset, Power (short click), Power (long click) 8s .

None of these do anything.

I watch the power usage on my energy monitor - nothing. I try to boot the old way, toggling power remotely and nothing. It is 5am, I don't think I can open the attic, pull down the ladder and climb the ladder without waking up the sleeping members of my family.

I click on Terminal->NanoKVM Terminal and get a new window that drops me to a root shell on the KVM itself. 

# cat /proc/cpuinfo
processor       : 0
hart            : 0
isa             : rv64imafdvcsu
mmu             : sv39

# uname -a
Linux kvm-c12c 5.10.4-tag- #59 PREEMPT Tue Jul 23 20:13:44 PDT 2024 riscv64 GNU/Linux

As a KVM, with an update it seemed okay, not good, not much worse that IPMI interfaces I've used from across the Atlantic. As a device to save me trips to the attic it has failed miserably.

I will probably be better in the future.

previous next