Advanced Documentation Retrieval on FreeBSD 

On Fri, Jul 16, 2021 at 04:23:20PM -0400, ░▒▓░░ ░▐░▒ wrote:
> Hi Tom.
> I just not realized you've not have your inbox assaulted by our listener
> feedback. Is there a specific address you'd like that routed to?
> 
> Anyway, here is a choice one.
> 

*sigh*

░▒▓░░ I don't know what we should do about Michael . I spoke to a priest
about an exorcism and he said "I'm not going near that monster, not on your
life", which I thought was pretty alarmist for a priest.

Follows is a rough markdown draft of the article "Advanced Documentation
Retrieval on FreeBSD" as we discussed

- Tom

-----------

Advanced Documentation Retrieval on FreeBSD

FreeBSD is renowned for its very high quality documentation. For many queries the man pages have a wealth of accurate and up to date documentation that is frequently a surprise to uses of other operating systems. It is not uncommon to hear from new FreeBSD users that they have to relearn to try the man pages before searching on the web.

Beyond man pages FreeBSD also has very high quality documentation in the form of the FreeBSD handbook. Only talking about the FreeBSD Handbook actually cuts short the range of really high quality documentation that the FreeBSD project offers.

The FreeBSD Handbook covers installation and day to day usage of a FreeBSD system and it is kept reasonably up to date by the FreeBSD documentation team. The handbook is an amazing document that comes from a time before blog posts and wikis and it contains a mixture of official project direction and tutorial style walkthroughs on how to use different FreeBSD subsystems and third party software. Not everything is covered by the handbook and many times only one path of a piece of software use is covered, but it is an excellent resource to get started using and configuring a FreeBSD system.

Deeper into FreeBSD there are several other 'books' that the FreeBSD project maintains. The full list of books is available from https://docs.freebsd.org/en/books/ , it includes technical information about how the FreeBSD kernel works in the forms of the Design and Implementation of the 4.4BSD Operating System and the Architecture handbook. Documentation on how to contribute to different parts of the operating system as the porters-handbook, the fdp-primer and the developers-handbook.

The FreeBSD project also hosts a wiki which contains less formal and in progress documentation, written by users and developers. The wiki can sometimes be much more like a temporary source of information, but it does contain valuable guides. It is the right place for pages such as the laptop compatibility matrix . The wiki is a unique FreeBSD project resource in that users are also able to have edit access.

What else is there?

Beyond the documentation the project provides there are outside sources of information on how to use and configure a FreeBSD system. Searching the web will bring up a lot of Technical information in the form of blog posts and articles.

Searching the web is not the only way to get more information on FreeBSD systems. We can use external 'daemonised' resources by using the FreeBSD base system tool invoke . This tool is a little esoteric to use and sadly it is one of the excellent FreeBSD tools written by developers that just quite haven't seen the light of day.

invoke ships in the FreeBSD source tree and is in src/tools/tools/invoke . invoke isn't built by default, but it is easy to build on a system using its Makefile: 

# cd /usr/src/tools/tools/invoke
# make
# make install

invoke requires quite a lot of information to be useful, annoying the author ���������@ was in the process of writing a book on invoke when they went missing travelling in rural Romania. However from reading the source we can see the list of information or 'principals' required to correctly invoke documentation.

Principals are information locators which are tied closely to the source of the information. Personal web pages of authors work well and are easy to obtain, more potent sources such as hand written notes or the authors blood work the best, but we can substitute social media accounts to get a similar level familiar information about the author.

XXX more on principals XXX

For our example I have collected the personal web page, blog and twitter account of the source we want to use, we can pass them to invoke as arguments or in a configuration file.

In addition to 'principals' the invoke tool needs to be run from a special environment. A comment in the source code describes this, but it took the author some trial an error to figure it out in practice. 

/*
 * invoke must be run from either a larger or lesser circle. These
 * can be ancient such as the very high quality circle at Midmar,
 * however, if you are unable to travel a Ars Theurgia Goetia will
 * suffice. You must interface the machine running invoke via a 
 * galvanic isolator. The transformer in an Ethernet connector with
 * magnetics works great, if you only have an SBC you'll have to 
 * figure out something with transformers
 */

From this comment we can see that we need to create a substitute circle to use with invoke and connect it to our computer, but isolate it from the machine. If we fail to isolate properly we can damage the machine and likely destroy it.

We can create the substitute circle by using our preferred ethereal bonding fluid. Blood works very well, but collection can be legally tricky. Luckily we can use a vegan alternative in the form of beeted oat milk. If your local shops don't carry beeted oat milk you can make your own by mixing oat milk with beetroot juice during a full moon.

With the substitute bonding fluid we need to draw our circle for summoning and holding triangle (you can see a good example here ). In this example we are going to use an Ethernet interface with magnetics, all you really need to do is to plug it into the circle.

With the interface connected to the circle we can check for state using ifconfig, we need to look for the ETHER flag in the list of options. 

igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500                            
options=e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,ETHER>
ether ac:1f:6b:46:9e:da                                                                               
inet 11.14.17.13 netmask 0xffffff00 broadcast 137.50.17.255 
inet6 fe80::ae1f:6bff:fe46:9eda%igb0 prefixlen 64 scopeid 0x1                                         
media: Ethernet autoselect (1000baseT <full-duplex>)                                                  
status: active                                                                                        
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

Running Invoke

With our principals collected, and our link established to the circle we can run the invoke command as follows: 

# invoke -p [principals] [-i interface] [question]

We pass in the principals we have gather and tell invoke the interface name. If you hacked together a device you'll need to pass in the gpio controller and pin using -d, for the gpio controller and -P for the pin number. 

# invoke -p [principals] -d /dev/gpioc0 -P 1 [question]

The final argument to invoke is a quoted string that contains the question.

If you are all set up then you can use invoke to summon your documentation author of choice, for this example I am using Michael W. Lucas : 

# invoke -p https://mwl.io https://twitter.com/mwlauthor -i igb0 "How do I configure dummynet with weighted fair queues?"

If all goes well you should hear (an normally quite grumpy) disembodied voice bark the answer to your question back to you.

As great as this is for single questions sometimes you want the author to hang around and help with protracted debugging sessions. We can achieve this with the manifest option to invoke: 

# invoke -p https://mwl.io https://twitter.com/mwlauthor -i igb0 manifest

With this option you will get the author directly summoned into the triangle attached to your summoning circle. Care should be taken with manifestations, the circle and triangle must remain intact for the duration of the session. If you are using a laptop (which I recommend and you need to use with an existing circle). Then make sure to watch the battery life careful.

If you don't properly shutdown the session then you risk getting the spirit of the author trapped in your machine and no one wants that.

Conclusion

FreeBSD has a great range of available documentation, but sometimes you hit the limits of information that is readily available online. Here we discussed the invoke command and the ways it can be used to get direct help from the authors of high quality documentation that are a little too public.

Blog more in 2020

In June I tried to write 4 blog posts and I elicited help from some of my friends to do this. I managed to write 5 posts beyond the announcement I would blog:

Of course it wasn't just me, I asked other people to blog to help me stay on track. The idea here was that seeing other peoples blog posts would inspire and force me to keep going. This worked reasonably well. The pressure to write the blog posts was there, but publishing was harder. This ended up with me pushing several posts in the final few days of June.

The pressure didn't really show up either, I know that the others wrote blog posts, but they didn't tell me!

They were great sports to get involved and help me with this, you should look up their blogs and drop them into your rss reader.

Because this wort of worked I think we should aim to keep doing this. Now 4 posts a month is a lot (maybe even too much) and so I thought that 8 more this year would be good. That is about 1.3333333... a month and seems entirely achievable.

I am going to try and blow this number out the water, but even if I fail completely and only manage one or two more post that will still be great.

Posted on

Simple ipfw NAT for bhyve virtual machines and vnet jails

Most of the time, I want to do some throw away networking temporally to play with something or to try something out. I really don't like changing all the config on a machine just to try something. The FreeBSD documentation leans the other way first showing you what to edit in rc.conf before maybe mentioning that actual commands to run.

The ipfw documentation has a different problem. The example in the handbook and online are both very verbose and very complicated. Because ipfw is normally configured with a shell script the authors go absolutely wild with all the features they can.

I had a hard time figuring out ipfw in-kernel NAT from these guides. Instead here I present the simplest set of commands I could find to set up a NAT and a little explanation to help you debug when it doesn't work.

This is based on a great email from Allan Jude on the freebsd-virtualization list from 2014 that laid out the basics of this setup.

Set up Overview

For testing I want to run virtual machines and vnet jails on my laptop and give them have access to the internet. I want a throw away NAT setup that is ready to go quickly.

My laptop connects to my home network (and eventually the internet) over wifi. The wifi network offers me an address in the 192.168.1.0/24 subnet. On my laptop I want to have multiple guests. To do this we are going to use ipfw NAT and a bridge interface. It will look something like this: 

         TO INTERNET
          ^
          |
          |
          v
          +-------+  192.168.1.x
+-------------| wlan0 |---------------+
|             +-------+               |
|                 ^                   |
|                 |                   |
|              ipfw nat               |
|                 |                   |
|                 V                   |
|            +---------+              |
| 10.0.4.1   | bridge0 |              |
|            +----+----+              |
|                 ^                   |
|                 | 10.0.4.0/24       |
|      ___________+_______________    |
|      |       |        |        |    |
|      v       v        v        v    |
|  +---+--+ +--+---+ +--+---+ +--+---+|
|  | jail | |  vm  | | jail | | ...  ||
|  +------+ +------+ +------+ +------+|
+-------------- laptop ---------------+

The interfaces in the jails (the b half of the epair) and the virtual machines (the vtnet in the V) won't be visible to ipfw, but will exist in their own world. To work around this we will use a bridge with the epairs and tap interfaces.

Setting up ipfw NAT

We need to load the kernel modules for ipfw and the ipfw in kernel NAT. ipfw has the frustrating default (and annoyingly different to ipf and pf) of to dening all traffic. This default has the great property of locking you out of a machine you are setting up remotely.

This is control by a sysctl that cannot be changed at run time, but we can change the default behaviour with kenv before we load the module: 

# kenv net.inet.ip.fw.default_to_accept=1

Now we can safely load ipfw and the in-kernel NAT. 

# kldload ipfw ipfw_nat

ipfw should load enabled, if you are having trouble later on double check that the firewall is actually enabled. 

# sysctl net.inet.ip.fw.enable
net.inet.ip.fw.enable: 1

When we do NAT we are acting as a gateway between the traffic on the NATd interface and the real interface. For any packets to be passed we need to enable forwarding. 

# sysctl net.inet.ip.forwarding=1
# sysctl net.inet6.ip6.forwarding=1

ipfw rule set

We need to create an IPFW NAT instance configured with the interface we want to NAT (wlan0 in this case) and configure rules to pass all traffic from the bridge through the NAT. 

# ipfw nat 1 config if wlan0
# ipfw add 101 nat 1 ip from 10.0.4.0/24 to any out via wlan0
# ipfw add 103 nat 1 ip from any to any in via wlan0

I like to leave a gap between rules like this so I can insert an ipfw log command for the eventual case that nothing makes sense and everything is broken.

set up interfaces

A bridge is the center of our guest network, we will give it the default root address that all of our guests will speak to. 

# ifconfig bridge create
bridge0
# ifconfig bridge0 inet 10.0.4.1/24 up

Our jail will use an epair interface to speak to the outside world. They come as an a and a b part, ifconfig only tells us about the a part when it clones the interface. When we give a vnet jail an interface it is no longer visible to the host system. An epair gives us two interfaces that act like a virtual ethernet cable, we stick one end into the jail and the other is connected to the bridge. 

# ifconfig epair create
epair0a

Our virtual machine will use a tap interface to access the world. The tap interface needs to be brought up. There is a helpful sysctl that is off by default which will trigger the interface to be brought up when it is first opened. I like to set this to one, otherwise I find myself debugging networking inside the VM alot with little success. 

# ifconfig tap create
tap0
# sysctl net.link.tap.up_on_open=1

With all the interfaces set up we need to add them to our bridge. 

# ifconfig bridge0 addm epair0a addm tap0

Create jail

Never spoken about is the bsdinstall jail command. It takes a directory and installs a jail into it. This command will ask you some questions, it would be cool if it didn't, that would make automating jail creation in scripts much easier for me. 

# mkdir testjail
# bsdinstall jail testjail

We make our jail persist so it will stick around as we experiment. The following command creates the jail on the host: 

# jail -c name=testjail persist vnet path=testjail vnet.interface=epair0b

Now we can jexec into the jail and configure the epair. When you bring one end of an epair up, the other end comes up, when it goes down the other end goes down. We just need to configure an address and a default route in our jail. 

# jexec testjail sh
[testjail] # ifconfig epair0b inet 10.0.4.4/24 up
[testjail] # route add default 10.0.4.1
[testjail] # ping -c 1 10.0.4.1
[testjail] # ping -c 1 192.168.1.1
[testjail] # ping -c 1 8.8.8.8

With this setup the jail can speak to our bridge, the local network and the wider Internet.

Create and config a VM

The FreeBSD offers prebuilt virtual machine images, The latest current one is available from a url like this: 

# fetch ftp://ftp.freebsd.org/pub/FreeBSD/snapshots/VM-IMAGES/13.0-CURRENT/amd64/Latest/FreeBSD-13.0-CURRENT-amd64.raw.xz

It would be cool if there was a latest symlink that gave you a new head VM from one static place. The image comes xz compressed, we need to unpack it and I like to move it to a consistent place: 

# xz -d FreeBSD-13.0-CURRENT-amd64.raw.xz
# mv FreeBSD-13.0-CURRENT-amd64.raw /vms/freebsd-current

bhyve requires we load the vmm kernel module, with that we can use the excellent vmrun.sh script to launch our vm. 

# kldload vmm
# sh /usr/share/examples/bhyve/vmrun.sh -c 4 -m 1024 -t tap0 -d /vms/freebsd-current freebsd-current

Once that comes up you can log in and do some manual config. 

[vm] # ifconfig vtnet0 inet 10.0.4.5/24 up
[vm] # route add default 10.0.4.1
[vm] # ping 8.8.8.8

For DNS in both the jail and the virtual machines I have to manually set up the name server local from my network.

/etc/resolv.conf 

search lan
nameserver 192.168.1.1

This won't be valid as I move to other networks, but I am sure I will remember after only a little confusion and debugging.

Conclusion

That is all it takes. The NAT configuration is 3 firewall rules and enabling forwarding. None of this is persistent and that isn't great practice for a production environment, but it you just want to experiment with ipfw and NAT, or spin up a VM for today knowing how to do this in a non-persistent way is really helpful.

Quick and Dirty Network Scanning

Ever want to scan a subnet in the nosiest, least reliable way and generate too many processes while doing so? Yes? Well do I have a script for you: 

#!/bin/sh

default=172.20.10

if [ -z $1 ]
then
        prefix=$default
else
        prefix=$1
fi

pinghost ()
{
        ping -t 1 -c 1 $1 > /dev/null
        if [ $? -eq 0 ]
        then
                echo hit $1
        fi
}

for x in `jot 254`
do
        pinghost $prefix.$x &
done

I wrote this while I was doing hack the box challenges and it was a fun and quick way to look to actually find things on my test network. I do not recommend using this. Some operating systems won't let you run it twice in succession as it generates a lot of processes.

My Streaming Setup

In April and May I did some streaming on twitch of hardware hacking projects . I started this as a way to work through the material for my cancelled BSDCan hardware hacking tutorial. With the tutorial cancelled I have been left with quite a few NanoPi Neo LTS and I was thinking about doing the tutorial as a series of videos with the idea of selling intro kits with the boards.

So far I have done four streams aiming for about an hour for each. I will say now that I haven't streamed again in June, I am not saying it is the end, but I (and I bet you too) need to control my commitments or I just never get anything done.

I have also been asked to write up my streaming set up so other people can use it. This vanity pic I tooted seems to include most of it.

Equipment

I have a bunch of equipment because I want to stream stuff in the real world. If you just wanted to share some windows and your webcam as Kristof Provost does then you can get by just with your laptop.

This is quite a lot of stuff, but other than the Aputure light I had all of it already from other projects. I would like to be able to capture the display of my work oscilloscope, thankfully the scope has VGA out so it is just a matter of figuring out a way to capture.

Software

I am using Open Broadcast Studio (OBS) on macOS to mix video and feed it to twitch.

OBS supports a rich variety of input sources that can mixed to make a scene. I have been using three scenes to make it easy to setup what is streamed in advance:

  1. Everything
  2. Just the camera
  3. Just the presentation

Everything is the mixed view of my terminal, the camera output and the slides in a web browser. My tutorial slides are derived from my EuroBSDCan Tutorial and render in a web browser natively. Having a browser in the video mix quickly turned out to be really helpful.

OBS has been rock solid and I have had no problems with stability while streaming. The interface is a bit of a maze, but I suspect that is a natural result of the power it offers.

Canon Camera

For most of the top down shots I used a canon 600D with a 18-55mm kit lens. The battery in this thing is quite old and only manages about an hour and a half of video output. I have since gotten a USB powered battery insert that should allow me to run forever, but am yet to try streaming with it.

My canon camera does not support acting as a webcam, but gphoto2 does support grabbing live video from it. This should have been easy to feed into OBS, but I couldn't get it working and ended up instead using ffplay to render the video and grabbed that with a video capture. To get video from gphoto2 into ffplay I ran: 

gphoto2 --stdout --capture-movie | ffplay -

This turned out to be very stable and easy to set up. There is quite high latency between the capture output and the gphoto2 capture, but it worked fine if I didn't have to move the camera much.

Nikon Camera

I also have a Nikon J1 and while the camera is awful to use I do have a 10mm lens for it with a macro extender. This allows me to take very high detail pictures of PCBs. I wanted to add this to the streaming set up too.

The Nikon firmware is garbage and while gphoto2 does support the camera I found I could only get a single image from the camera before having to reset it and was completely unable to get video from it. Never mind, my 2018 EuroBSDCon Talk used a HDMI network extender as a capture device. In the stream Crimes Against Computers3 I did something awful with a BeagleBone Black and got video from the Nikon through the HDMI extender into OBS.

This was a lot of faff to setup (though I since did get a USB-C Ethernet for the MacBook) and I am still kind of dubious of the stability of the LKV373 so I have only used this in the stream where I figured it out. I might try this again later.

Since doing that stream cheap USB2 and USB3 HDMI capture devices have appeared. These are supposed to appear as UVC webcams to the system. I suspect these will be a better method than the LKV373, but I want to try before recommending them.

Audio

For audio I am using a Zoom H3 dictaphone thing. I really like the audio from the Zoom, but I seem to be in the minority. In the end it is the microphone I have.

It is stereo which should be mixed down or you will drift across the channels when you move your head. OBS supports this so it wasn't a problem, but I did have to be told about it.

The firmware for this thing isn't so great either, it cannot record to the Micro SD card while acting as a USB interface and Annoyingly there is a bug where it defaults to 44KHz for the audio. The USB driver doesn't seem to advertise this to the properly and if you continue with the default you get weird audio. I dealt with this before in FreeeBSD , but was surprised to see it appear in macOS. You should always double check your audio before starting a stream.

Why not stream from FreeBSD?

I wanted a pain free approach where I could set up and go for my first streams so while OBS is ported to FreeBSD I expected it to be a lot more work than the much more common macOS/OBS combination. I expect FreeBSD/OBS support to get better if people continue streaming FreeBSD stuff.

The MacBook is similar specs to my Thinkpad, but the wireless card in the Thinkpad doesn't have great support and only manages 80211g rates. That might just be enough to stream out the video, but it seems very risky to me.

I will look at streaming with OBS from FreeBSD in the future.

The streams have done okay so far, there is a big social media boost network of FreeBSD users and developers and that has helped people find the content. I am sure the next stream will have fewer viewers as there has been a gap in my streaming.

You can support this

Finally, I set up a ko-fi page to support me writing and streaming. I plan to write more this year and if there is interest I will do more streams too. The streams so far have mostly been using equipment and parts I already had. Doing more is going to require me spending money.

If you have enjoyed my streams, or if you just want me to do more a very solid signal of your support would be tipping me a cup of coffee or something through ko-fi. You can support my hardware habbit here .

previous next