Simple ipfw NAT for bhyve virtual machines and vnet jails

Most of the time, I want to do some throw away networking temporally to play with something or to try something out. I really don't like changing all the config on a machine just to try something. The FreeBSD documentation leans the other way first showing you what to edit in rc.conf before maybe mentioning that actual commands to run.

The ipfw documentation has a different problem. The example in the handbook and online are both very verbose and very complicated. Because ipfw is normally configured with a shell script the authors go absolutely wild with all the features they can.

I had a hard time figuring out ipfw in-kernel NAT from these guides. Instead here I present the simplest set of commands I could find to set up a NAT and a little explanation to help you debug when it doesn't work.

This is based on a great email from Allan Jude on the freebsd-virtualization list from 2014 that laid out the basics of this setup.

Set up Overview

For testing I want to run virtual machines and vnet jails on my laptop and give them have access to the internet. I want a throw away NAT setup that is ready to go quickly.

My laptop connects to my home network (and eventually the internet) over wifi. The wifi network offers me an address in the 192.168.1.0/24 subnet. On my laptop I want to have multiple guests. To do this we are going to use ipfw NAT and a bridge interface. It will look something like this:

         TO INTERNET
          ^
          |
          |
          v
          +-------+  192.168.1.x
+-------------| wlan0 |---------------+
|             +-------+               |
|                 ^                   |
|                 |                   |
|              ipfw nat               |
|                 |                   |
|                 V                   |
|            +---------+              |
| 10.0.4.1   | bridge0 |              |
|            +----+----+              |
|                 ^                   |
|                 | 10.0.4.0/24       |
|      ___________+_______________    |
|      |       |        |        |    |
|      v       v        v        v    |
|  +---+--+ +--+---+ +--+---+ +--+---+|
|  | jail | |  vm  | | jail | | ...  ||
|  +------+ +------+ +------+ +------+|
+-------------- laptop ---------------+

The interfaces in the jails (the b half of the epair) and the virtual machines (the vtnet in the V) won't be visible to ipfw, but will exist in their own world. To work around this we will use a bridge with the epairs and tap interfaces.

Setting up ipfw NAT

We need to load the kernel modules for ipfw and the ipfw in kernel NAT. ipfw has the frustrating default (and annoyingly different to ipf and pf) of to dening all traffic. This default has the great property of locking you out of a machine you are setting up remotely.

This is control by a sysctl that cannot be changed at run time, but we can change the default behaviour with kenv before we load the module:

# kenv net.inet.ip.fw.default_to_accept=1

Now we can safely load ipfw and the in-kernel NAT.

# kldload ipfw ipfw_nat

ipfw should load enabled, if you are having trouble later on double check that the firewall is actually enabled.

# sysctl net.inet.ip.fw.enable
net.inet.ip.fw.enable: 1

When we do NAT we are acting as a gateway between the traffic on the NATd interface and the real interface. For any packets to be passed we need to enable forwarding.

# sysctl net.inet.ip.forwarding=1
# sysctl net.inet6.ip6.forwarding=1

ipfw rule set

We need to create an IPFW NAT instance configured with the interface we want to NAT (wlan0 in this case) and configure rules to pass all traffic from the bridge through the NAT.

# ipfw nat 1 config if wlan0
# ipfw add 101 nat 1 ip from 10.0.4.0/24 to any out via wlan0
# ipfw add 103 nat 1 ip from any to any in via wlan0

I like to leave a gap between rules like this so I can insert an ipfw log command for the eventual case that nothing makes sense and everything is broken.

set up interfaces

A bridge is the center of our guest network, we will give it the default root address that all of our guests will speak to.

# ifconfig bridge create
bridge0
# ifconfig bridge0 inet 10.0.4.1/24 up

Our jail will use an epair interface to speak to the outside world. They come as an a and a b part, ifconfig only tells us about the a part when it clones the interface. When we give a vnet jail an interface it is no longer visible to the host system. An epair gives us two interfaces that act like a virtual ethernet cable, we stick one end into the jail and the other is connected to the bridge.

# ifconfig epair create
epair0a

Our virtual machine will use a tap interface to access the world. The tap interface needs to be brought up. There is a helpful sysctl that is off by default which will trigger the interface to be brought up when it is first opened. I like to set this to one, otherwise I find myself debugging networking inside the VM alot with little success.

# ifconfig tap create
tap0
# sysctl net.link.tap.up_on_open=1

With all the interfaces set up we need to add them to our bridge.

# ifconfig bridge0 addm epair0a addm tap0

Create jail

Never spoken about is the bsdinstall jail command. It takes a directory and installs a jail into it. This command will ask you some questions, it would be cool if it didn't, that would make automating jail creation in scripts much easier for me.

# mkdir testjail
# bsdinstall jail testjail

We make our jail persist so it will stick around as we experiment. The following command creates the jail on the host:

# jail -c name=testjail persist vnet path=testjail vnet.interface=epair0b

Now we can jexec into the jail and configure the epair. When you bring one end of an epair up, the other end comes up, when it goes down the other end goes down. We just need to configure an address and a default route in our jail.

# jexec testjail sh
[testjail] # ifconfig epair0b inet 10.0.4.4/24 up
[testjail] # route add default 10.0.4.1
[testjail] # ping -c 1 10.0.4.1
[testjail] # ping -c 1 192.168.1.1
[testjail] # ping -c 1 8.8.8.8

With this setup the jail can speak to our bridge, the local network and the wider Internet.

Create and config a VM

The FreeBSD offers prebuilt virtual machine images, The latest current one is available from a url like this:

# fetch ftp://ftp.freebsd.org/pub/FreeBSD/snapshots/VM-IMAGES/13.0-CURRENT/amd64/Latest/FreeBSD-13.0-CURRENT-amd64.raw.xz

It would be cool if there was a latest symlink that gave you a new head VM from one static place. The image comes xz compressed, we need to unpack it and I like to move it to a consistent place:

# xz -d FreeBSD-13.0-CURRENT-amd64.raw.xz
# mv FreeBSD-13.0-CURRENT-amd64.raw /vms/freebsd-current

bhyve requires we load the vmm kernel module, with that we can use the excellent vmrun.sh script to launch our vm.

# kldload vmm
# sh /usr/share/examples/bhyve/vmrun.sh -c 4 -m 1024 -t tap0 -d /vms/freebsd-current freebsd-current

Once that comes up you can log in and do some manual config.

[vm] # ifconfig vtnet0 inet 10.0.4.5/24 up
[vm] # route add default 10.0.4.1
[vm] # ping 8.8.8.8

For DNS in both the jail and the virtual machines I have to manually set up the name server local from my network.

/etc/resolv.conf

search lan
nameserver 192.168.1.1

This won't be valid as I move to other networks, but I am sure I will remember after only a little confusion and debugging.

Conclusion

That is all it takes. The NAT configuration is 3 firewall rules and enabling forwarding. None of this is persistent and that isn't great practice for a production environment, but it you just want to experiment with ipfw and NAT, or spin up a VM for today knowing how to do this in a non-persistent way is really helpful.