FreeBSD Network Status Week 39 2024

This is an experiment in creating tooling driven reports from the FreeBSD development ecosystem, git, code review and bug tracking. The goal is to provide an insight into what has happened in FreeBSD in the last little while, but with limited time to gather, assess and write up what is going on. I am only going to focus on "networking", which for this week was about a third of all commits to main (last week it was about half).

It is a process that will lead to things being missed, don't feel bad, let me know and I'll add the stuff you think is important.

This report was generated on Friday 27th September 2024 for the period between "last week" and "now" as git understand them. The git log command was:

git log --format="%h" --after="last week" --before="now"

With this report the tooling lets me take or leave commits and generates urls and such, but only on the FreeBSD main branch. I'll add stable branches for future reports, but first I want to see what can be done in the allotted time.

I also have searches and a little tooling for dealing with bugs (bugzilla) and reviews (phabricator), but neither of these platforms are as friendly to program as git. That should tell you what they are like.

I plan to write a few of these and if there is demand decided if the time investment is worth it. I'm only going to continue if I think it is worth the work.

Goings on

There are two important events invisible to the development system in the last week, EuroBSDcon and the accompanying DevSummit . While they could show up, it doesn't look like anyone used the "Event:" tag on a commit to main in apart from one GSoc commit earlier in the month.

Stab week ran without much happening, there is 1 reported issue discovered with ixl(4) which has been reverted .

170 commits matched my log command and I thought 64 of them were "networking" based on a cursory glance.

Transport

The regular #transport meeting coincided with the FreeBSD DevSummit. The next meeting is on the 3rd of October 2024, if you have issues with transport protocols (TCP, UDP, SCTP) or the socket layer please join, there is a public meeting link on each agenda page ( accessible via this wiki page ). It is more business like than chat ops, which helps us mostly keep the meetings to less than the planned hour.

Two sets of changes landed in my filter, the MAC change is the first in some further work tidying up the MAC SYN code.

The second are tidying up stuff in the TCP stack. The last remanent of a broken attempt to control burstiness of the TCP stack 20 years ago. We turned it back on a couple of years ago and I'm pretty sure it deadlocked connections immediately. TCP is very difficult.

Netdev

Most of the changes this week land in device drivers, it is most of the kernel after all.

bz@ did some git to add missing tracking for vendored Linux WiFi drivers. The commits are 'empty', but contain evil:

This was done using what I'd call "git magic" provided by Ed no one
else will hopefully ever need again.

Improvements and updates to e1000 (igb, em) have been coming in from kbowling@, with igb updated to 2.25.28-fbsd:

ix/ixgbe has been updated to the latest release (ix-3.3.38) from Intel .

The ixl revert mentioned in the stab week report went in:

On top of this there has been a bunch of tidying up of old code in bfp, iflib and other places by zlei@.

Firewalls

Work has continued to keep FreeBSD pf the best tested firewall:

And some general tidying also landed.

User tooling

tcpdump has been updated to 4.99.5

There have been further commits to nuageinit, which I'm not sure is 100% on topic for a network status report. I do think it is one of the most interesting new tools to be added to FreeBSD for a long time and I'm hoping it'll get a raft of documentation and features to remove the need for cloud-init cloud-init at all.

netstat saw some small updates to libxo output and the ntp man page got some attention.

Please send feedback

As I write this my timer is at 58 minutes , so the 1 hour goal seems doable.

I plan to add interesting bugs and reviews, once I figure out how to get them from the relevant places. Once the tooling is more than sparkling git log I'll put it up on a repo somewhere.

I would love to know if this summary was any help, if it was, or if you think I should cover other thing please let me know (thj@freebsd.org).

If you find a typo or have a correct let me know and I'll thank you at the end here.

Thanks to typos and feedback from: - mgdm - hibby - brd@

You can see all prior posts here. ( rss )


My work on FreeBSD is supported by the FreeBSD Foundation , you can contribute to improving FreeBSD with code, documentation or financially by donating to the Foundation .

acpidumping

Sometimes you need to play with ACPI tables, there are even rumours in the man pages of people patching up their ACPI tables to make them work better. This is not ground I have trod.

FreeBSD has a tool in base called acpidump , but it is lacking in comparison to the full upstream tools you might use on Linux. The full Intel ACPI tools are available in the acpica-tools package on FreeBSD.

With those you can then use the acpidump commands you might be asked to on Linux. If asked what you ACPI tables contain using these tools for a dump looks like this:

# /usr/local/bin/acpidump -b 
# ls
apic.dat    ecdt.dat    msdm.dat    ssdt11.dat  ssdt5.dat   wsmt.dat
asf!.dat    facp.dat    nhlt.dat    ssdt12.dat  ssdt6.dat   xsdt.dat
bgrt.dat    facs.dat    phat.dat    ssdt13.dat  ssdt7.dat
dbg2.dat    fpdt.dat    sdev.dat    ssdt14.dat  ssdt8.dat
dbgp.dat    hpet.dat    ssdt.dat    ssdt2.dat   ssdt9.dat
dmar.dat    lpit.dat    ssdt1.dat   ssdt3.dat   tpm2.dat
dsdt.dat    mcfg.dat    ssdt10.dat  ssdt4.dat   uefi.dat

These binary files can be decompiled using the iasl tool:

# iasl *.dat

This gives you a dsl file (source?) for each dat file which might be helpful if you want to check what support is on your motherboard.

$ head -n 20 lpit.dsl
/*
 * Intel ACPI Component Architecture
 * AML/ASL+ Disassembler version 20230628 (64-bit version)
 * Copyright (c) 2000 - 2023 Intel Corporation
 *
 * Disassembly of lpit.dat, Wed Sep 25 09:42:49 2024
 *
 * ACPI Data Table [LPIT]
 *
 * Format: [HexOffset DecimalOffset ByteLength]  FieldName : FieldValue (in hex)
 */

[000h 0000 004h]                   Signature : "LPIT"    [Low Power Idle Table]
[004h 0004 004h]                Table Length : 000000CC
[008h 0008 001h]                    Revision : 01
[009h 0009 001h]                    Checksum : 05
[00Ah 0010 006h]                      Oem ID : "INSYDE"
[010h 0016 008h]                Oem Table ID : "ADL-P-M"
[018h 0024 004h]                Oem Revision : 00000002
[01Ch 0028 004h]             Asl Compiler ID : "ACPI"

My work on FreeBSD is supported by the FreeBSD Foundation , you can contribute to improving FreeBSD with code, documentation or financially by donating to the Foundation .

Smol KVM

I'm not really sure if I want to own so many computers, but if you want to perform anything like science you need a lot of them. I have a small testbed which I like to say is an experiment in using gaming hardware for doing network science (I can report early that I wish I had enterprise motherboards with lights off management).

Having more machines that I have monitors means that I need some way to manage them independent of switching my monitor to their output. For this reason and so I could work from either home or elsewhere I made sure to buy the only motherboards I could find that had serial ports (thanks Asrock).

I have gotten by sort of coarsely using the serial outputs and the bios feature of power up on AC reset coupled with smart power outlets to give me remote management and a control interface.

This doesn't work very well. I think if the capacitors on the motherboard hold any charge the reset isn't detected, meaning if I forget to power off the smart plug when I shut a machine down I have to do an on/off dance through the smart plug until the machine eventually boots.

The motherboards do not seem to offer the option of sending the bios menu out over serial which I know other machines can do as I have used them. This isn't too much hassle until you add temperamental network cards that require very specific bios configuration to power up. Debugging these issues required me to find a monitor, more keyboards and do a near continuous pull machine rebuild dance for what felt like a year but probably spanned a month or two of 2023.

In a short 300 words, I feel I have a need for a KVM and the spirit of the testbed requires it be cheap.

Options

I know the PiKVM exists, but I can't help think of doing it all myself with FreeBSD every time I consider setting it up and the really surplus Pis I have don't have USB OTG. I have considered building my own USB keyboard connector, but some things I am able to resist doing through sheer will and procrastination.

Recently the Openterface Mini-KVM (oh god is that really the name? I'm going to continue calling it the 'OpenKVM') Crowdsupply crowd funder came by and while I back it, I know I won't get anything for longer than can be planned in. I'm sure this will deliver, Crowdsupply have an exceptional success rate, but hardware is hard and time is sand.

This environment made the super cheap NanoKVM preorder very interesting. Interesting enough to buy one to try out.

NanoKVM

It appeared as a complete surprise this week in quite nice packaging. The device is absolutely tiny with an OLED screen, two buttons and quite a lot of ports. There is Ethernet for network, HDMI for video from the host and 3 USB-C ports; Power, HID and KVM-B (motherboard button header). There are also pin headers to connect up two serial ports for managing things other than PCs.

It is very small and the is completely overwhelmed when you connect all the cables you actually need for it to be useful.

The KVM-B cable connects from USB-C to a break out board, this is then connected to a PCs button headers with jumper cables.

The device powered up on my desk without any trouble, it comes up really quickly. But I haven't run Ethernet yet in the new house and I have no sensible way to offer it a DHCP lease so for testing it is to the attic I must ascend.

Up to the attic I go, plug in the nightmare which is motherboard pin headers in an installed system and power on the device.

With DHCP the little screen shows an acquired IP address matching the alert from my Wifi controller, connecting to this on my phone gives me a black screen.

I climb down the ladder and try from my laptop-desktop, also a black screen, but because computers aren't completely useless like phones is I can inspect the page and see that I have html, but something isn't work.

Up the ladder, poke things, turn off computer, turn on computer using the "Pwr" button on the NanoKVM and it turns on. Try phone again, nothing.

Disconnect NanoKVM from everything, climb down ladder, look at computer and see a login prompt. Look up default creds (admin, admin) and try to login which doesn't work.

Realise the NanoKWM is on my desk disconnected from power.

Climb back up the ladder, reconnect NanoKVM, try phone, get nothing, climb down ladder return to desk. Get a login prompt, log in woo!

My testbed machines boot to Grub and wait for input, this makes dual booting Linux and FreeBSD manageable and you don't have to panic hit a prompt before it times out.

On the KVM I can see the grub menu.

Click on the window and pressing the arrows keys does nothing, bring up the on screen keyboard and using its arrow keys does nothing.

1 frame per connection seems a bit weak, there is an "Update firmware" option in the menu, clicking it offers me to update from 2.0.4 to 2.0.5 I agree. It spins for a while, goes away and now trying to connect gives me just a black screen.

I leave it there for the duration of a meeting about TCP changes in FreeBSD, but nothing.

Pull down attic ladder, climb ladder, unplug NanoKVM power, check the OLED is alive replug power, climb down ladder, stow ladder, reload the page on my laptop-desktop and the NanoKVM is sitting at the login prompt.

Go and buy stuff for supper while the bios times out a check for a peer interface on the 100G card which isn't turned on.

Post update and power cycle and I have enough KVM to watch a boot, log in and type. There is very much latency.

the planet rotates

The next morning I turn my laptop, mount it to the wall in this ridiculous dock I have forced my self to use and try to power on the machine I need to do some work.

The NanoKVM web interface offers three options for button presses, Reset, Power (short click), Power (long click) 8s .

None of these do anything.

I watch the power usage on my energy monitor - nothing. I try to boot the old way, toggling power remotely and nothing. It is 5am, I don't think I can open the attic, pull down the ladder and climb the ladder without waking up the sleeping members of my family.

I click on Terminal->NanoKVM Terminal and get a new window that drops me to a root shell on the KVM itself.

# cat /proc/cpuinfo
processor       : 0
hart            : 0
isa             : rv64imafdvcsu
mmu             : sv39

# uname -a
Linux kvm-c12c 5.10.4-tag- #59 PREEMPT Tue Jul 23 20:13:44 PDT 2024 riscv64 GNU/Linux

As a KVM, with an update it seemed okay, not good, not much worse that IPMI interfaces I've used from across the Atlantic. As a device to save me trips to the attic it has failed miserably.

I will probably be better in the future.

Ploopy Adept

Ploopy:

Ploop Ploopy Adept ploop plop ploop ploop, ploopy plop plop ploopy plop. Ploop ploop ploop ploop plop plop an excessive plop (ploopy ploopy plop plop plop 61 plop) ploopidy plop plop ploop ploop.

Ploop the ploop ploop, ploop ploopidy ploop. Ploop plop ploop ploopy ploop, ploop ploop ploop ploopy ploop plop plop and plop plop adorable ploopy bearings ploopy plop plop.

Ploop plop plop trackball plop ploopy ploopy ploop plop plop fuzzy lop. That said it seems great.

Ploopy the ploopy ploop plop plop ploopy plop plop ploop ploop ploop ploop ploopy plop ploop ploop. Plop the ploopy binaries in the plopy ploopy ploop.

Ploopy ploop ploopy plop? Ploopy ploopy ploopy ploopy plop plop.

English:

The Ploopy Adept is an open hardware track ball, available in a selection of whimsical colours. I got mine as a kit and after an excessive shipping window (who knew 30-90 day shipping could actually take 61 days) mine appeared and I set it up.

There is a little soldering to the kit, to mount the sensor. I guess everything else was placed by PCBA, then you need to assemble the 3D mount case and mount some adorable tiny bearings which support the ball.

This is my first trackball since they were a feature on Android phones so I am no expert. That said it seems great.

I wanted to fix a USB descriptor issue which was causing the ums driver to not load on FreeBSD when I attached the trackball and while the device is open hardware I couldn't find any source for the QMK based firmware. There are just binaries in the git repo.

Maybe it is all upstream? The problem went away and I am amazing finding side quests in my side quests so I didn't feel the need to pursue this one.


On FreeBSD you might need to load the ums kernel module before attaching the Ploopy Adept to your USB bus. I did for a while, when I didn't use the hub on the kvm which has a mouse as well. The need for this change went away, but it might help you if you run a FreeBSD release rather than CURRENT pkg base.

# kldload ums

# # plug in trackball, roll the ball, have a great time.

Realmode bhyve

I have been poking around bhyve, seeing what is up and I came across this article about writing a Linux kvm driver from scratch . In the article is an example of minimal program to run as a first test in the kvm driver:

; Output to port 0x3f8
mov dx, 0x3f8

; Store the address of the message in bx, so we can increment it
mov bx, message

loop:
    ; Load a byte from `bx` into the `al` register
    mov al, [bx]

    ; Jump to the `hlt` instruction if we encountered the NUL terminator
    cmp al, 0
    je end

    ; Output to the serial port
    out dx, al
    ; Increment `bx` by one byte to point to the next character
    inc bx

    jmp loop

end:
    hlt

message:
    db "Hello, KVM!", 0

That seems fun, a nice small example of getting some code running. I don't really want to write my own bhyve, I like the one we have, but it might be nice to try and get this running.

I assembled the example:

nasm -fbin nello.S nello

And looked around to see how to load a bios in bhyve. bhyve(8) has some examples at the end, it looks like the -l flag can be used to set a bootrom (bios) like so:

$ sudo bhyve -l bootrom,./nello nello

vm exit[0]
        reason          VMX
        rip             0x000000000000fff0
        inst_length     3
        status          0
        exit_reason     48 (EPT violation)
        qualification   0x0000000000000784
        inst_type               0
        inst_error              0

Well that didn't work. I poked a bit in bhyve, but it wasn't clear what to do about an EPT violation. The examples also mentioned using /usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd , I opted for the CSM version:

$ sudo bhyve -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CSM.fd hello

I had a poke around the CSM bootrom and while it is always fun to use hexdump, it really didn't help me understand what was wrong with my example assembly.

I tried with BHYVE_UEFI_CSM.fd and guess what I got:

vm exit[0]
        reason          VMX
        rip             0x000000000000fff0
        inst_length     3
        status          0
        exit_reason     48 (EPT violation)
        qualification   0x0000000000000784
        inst_type               0
        inst_error              0

The same trap!

I think that means I need to figure out the minimal viable bhyve command that will run known good bootrom before I try running that example. The last example in bhyve(8) is:

Run a UEFI virtual machine with a VARS file to save EFI variables.  Note
that bhyve will write guest modifications to the given VARS file.  Be
sure to create a per-guest copy of the template VARS file from /usr.

      bhyve -c 2 -m 4g -w -H \
        -s 0,hostbridge \
        -s 31,lpc -l com1,stdio \
        -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd,BHYVE_UEFI_VARS.fd
         uefivm

-w waits for the debugger and -H emulates halt to save power, no need for those. So I tried:

bhyve -s 0,hostbridge -s 31,lpc -l com1,stdio -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CSM.fd hello

And that worked:

Boot Failed. CDROM 0
Boot Failed. Harddisk 1
UEFI Interactive Shell v2.1
EDK II
UEFI v2.40 (BHYVE, 0x00010000)
Error. No mapping found
Press ESC in 1 seconds to skip startup.nsh or any other key to continue.

Now to try my bios:

$ sudo bhyve -s 31,lpc -l com1,stdio  -l bootrom,./nello hello
bhyve: ROM size 65552 is not a multiple of the page size
Device emulation initialization error: No such file or directory

32 (the raw unpadded 16 bit program size) is also not a multiple of the page size, I padded out the example using TIMES 4096 - ($ - $$) db 0 from a bootsector nasm example

This has not succeeded.

Fine, whatever, I will use gdb to look at what is going on. bhyve supports the -G flag to integrate with gdb. I added

-G wlocalhost:1234

to the bhyve command asking bhyve to wait for gdb to attach and continue listening on localhost port 1234.

(gdb) target remote localhost:1234
Remote debugging using localhost:1234
warning: No executable has been specified and target does not support
determining executable automatically.  Try using the "file" command.
0x000000000000fff0 in ?? ()
(gdb) x/32i 0x000000000000fff0
=> 0xfff0:      add    %al,(%rax)
   0xfff2:      add    %al,(%rax)
   ...
--Type <RET> for more, q to quit, c to continue without paging--q
Quit
(gdb) x/32x 0x000000000000fff0
0xfff0: 0x00000000      0x00000000      0x00000000      0x00000000
0x10000:        0x00000000      0x00000000      0x00000000      0x00000000
...
0x10060:        0x00000000      0x00000000      0x00000000      0x00000000
(gdb) x/32x 0x0
0x0:    0x00000000      0x00000000      0x00000000      0x00000000
0x10:   0x00000000      0x00000000      0x00000000      0x00000000
0x20:   0x00000000      0x00000000      0x00000000      0x00000000
0x30:   0x00000000      0x00000000      0x00000000      0x00000000

Connecting and poking around shows the obvious places are all zeros (or sometimes all 1s).

gdb has a 'find' command for searching memory, our example is pretty distinctive so it should find it.

Didn't work for me this time

Stepping immediately just starts the program, for nello we are stopped with rip as 0x000000000000ffef.

0x000000000000ffef in ?? ()
(gdb) x/64x $rip
0xffef: 0x960000ff      0x00ffff00      0x00000200      0x46f00000
0xffff: 0x00000000      0x00000000      0x00000000      0x00000000

disassembly time, FreeBSD's llvm-objdump doesn't have support for 16 bit x86 (fair), so I grabbed binutils and used a command like this:

x86_64-unknown-freebsd15.0-objdump -b binary -m i386 -D -Maddr16,data16 -Mintel nello

Working from objdump I tweaked some offsets to get bytes into the correct places with padding, but there wasn't an obvious clue what was up. I couldn't associate the memory I could read in gdb to anything from my binary.

$ hexdump -C nello
00000000  ba f8 03 bb 11 00 8a 07  3c 00 74 04 ee 43 eb f6  |........<.t..C..|
00000010  f4 48 65 6c 6c 6f 2c 20  62 68 79 76 65 21 00 90  |.Hello, bhyve!..|
00000020  90 90 90 90 90 90 90 90  90 90 90 90 90 90 90 90  |................|
*
0000fff0  e9 0d 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00010000

I turned to qemu to see if that helped:

$ qemu-system-i386 -bios nello -S -s -nographic

(gdb) target remote localhost:1234
Remote debugging using localhost:1234
warning: No executable has been specified and target does not support
determining executable automatically.  Try using the "file" command.
0x0000fff0 in ?? ()
(gdb) x/32xb 0xffff0000
0xffff0000:     0xba    0xf8    0x03    0xbb    0x11    0x00    0x8a    0x07
0xffff0008:     0x3c    0x00    0x74    0x04    0xee    0x43    0xeb    0xf6
0xffff0010:     0xf4    0x48    0x65    0x6c    0x6c    0x6f    0x2c    0x20
0xffff0018:     0x62    0x68    0x79    0x76    0x65    0x21    0x00    0x90
(gdb) x/16xb 0xfffffff0
0xfffffff0:     0xe9    0x0d    0x00    0x00    0x00    0x00    0x00    0x00
0xfffffff8:     0x00    0x00    0x00    0x00    0x00    0x00    0x00    0x00
(gdb) c
Continuing.

That all looks good, it matches up with our hexdump of the bios example. If I hit ^C then we stop at 0x00000011 .

^C
Program received signal SIGINT, Interrupt.
0x00000011 in ?? ()

If we recall that we are running in 16 bit mode in the last sector and convert that off set into the memory dumps we find the byte value 0xf4 an x86 halt instruction.

"HLT causes the 80386 to stop execution. Following a halt, execution can
only be resumed by the receipt of an enabled interrupt or by a reset of
the computer."

- Programming the 80386

So we did what we wanted to and stopped, but qemu gave us no output. I think that has confirmed that the bios image is now correct if not functional. So either we are running fine in bhyve and just not getting output, or there is something else up.

In the example minimal Linux hypervisor they just did a straight printf for an IO vmexit. Lets catch the vmexit handlers in bhyve and see what is up:

diff --git a/usr.sbin/bhyve/amd64/vmexit.c b/usr.sbin/bhyve/amd64/vmexit.c         
index e0b9aec2d17a..e1669c2b5051 100644                                            
--- a/usr.sbin/bhyve/amd64/vmexit.c                                                
+++ b/usr.sbin/bhyve/amd64/vmexit.c                                                
@@ -72,6 +72,7 @@ vm_inject_fault(struct vcpu *vcpu, int vector, int errcode_valid,
 static int                                                                        
 vmexit_inout(struct vmctx *ctx, struct vcpu *vcpu, struct vm_run *vmrun)          
 {                                                                                 
+fprintf(stderr, "%s:%d\n", __func__, __LINE__)                                    
        struct vm_exit *vme;                                                       
        int error;                                                                 
        int bytes, port, in;

I reconfigured my test script to output serial to /dev/nmdm0A so I would get printfs from bhyve, but nothing.

Our assembly doesn't do what we think it should does.

Adding port configuration from this so and osdev wiki led my modified bhyve to print on calls to vmexit_inout .

$ sudo sh ./run.sh nello
outputting serial to /dev/nmdm0B
waiting for gdb
vmexit_inout:75
vmexit_inout:75
vmexit_inout:75
vmexit_inout:75
vmexit_inout:75
vmexit_inout:75
vmexit_inout:75
vmexit_inout:75
vmexit_inout:75
vmexit_inout:75

Those 10 vmexit_input lines match up perfectly with the configuration and test example. This is an excellent debugging sign.

With an extreme amount of further faffing I discovered that the loop in the example I started from was not making it to the first print statement. I confirmed this by stripping away all of the configuation and just spat out some characters explicitly.

In the hexdump nasm was loading the wrong address into bx , but even with the correct address in bx I got no output. As I only wanted to say hello from real mode, I'm done. Debugging segments (segments, not even once) in a pre bios environment where you can't single step just isn't my idea of fun.

The example I started from was run from the base address, by writing their own kvm driver they were able to configure the instruction pointer and segments to look sensible. Me - an idiot, decided to work with the brain melting x86 hardware as it is.

Most of my fighting here was because gdb connecting to the bhyve stub isn't able to read guest memory in the bios region. Neither qemu or bhyve let me single step instructions, which just makes debugging here tedious.

OS Dev wiki is a great resource, but it is very annoying to have lots of "you shouldn't do this" everywhere when you push their 'perfect path'. I just want to know what I need to know.

If you want to play with real mode in bhyve you can start from this, minimal, working example:

; A 64k bios for bhyve which does nothing at all
bits 16
equ PORT 0x3f8

%macro outb 1
        mov al, %1
        out dx, al
%endmacro

start:
        mov dx, PORT                    ; store the port

        outb 0x0a                       ; print a message
        outb 'b'
        outb 'h'
        outb 'y'
        outb 'v'
        outb 'e'
        outb '!'
        outb 0x0a
end:
    hlt                                 ; hang around

TIMES 0xFFF0 - ($ - $$) db 0            ; pad out to reset vector
; cpu is going to start from 0xFFF0, with CS set to 0xF000 basically we are
; going to start at 0xFFFFFFF0, with only 16 bytes to play with, but we can
; just to start of the 64k segment reasonably easily.
jmp start                               

TIMES 0x10000 - ($ - $$) db 0           ; padd out to 64k

Hopefully that end isn't too negative, I had a lot of fun doing this, I just don't want to do anymore of it.