Adventurist.me: Letters into the void

Presentations with mdp

Sat, 03 Oct 2020 00:00:00 +0000

It feels like work is just a constant stream of preparing, travelling for and giving presentations. Brief words and pictures is an excellent for conveying information between small groups of humans. All of these presentations I write in keynote , keynote manages to be light weight, powerful and not horrific to use. As a bonus, my boss feels at home in keynote and is happy to make edits there.

The keynote workflow does not match well to how I think. When dreaming up a presentation I want to shit of a stream of conciousness and have it magically become slides in the right shape.

I might write a series of headings like:

# intro
# who
# meat 
# details
# questions?

I will iterate on these to add bodies, details and more slides.

For quite a while I have wanted a system where I could write plain text and have it become slides. I wrote about the sent tool from suckless, but in the end I found it wanting. I have also considered just showing screens of text, but a nightmare DEFCON wireess village talk by Hak5 scared me away. They attempted to just present using just a plain text file and less, but the window size got out of whack and it all fell apart.

Enter mdp

mdp is a terminal presentation program, it takes slides it approximately markdown and takes over the entire terminal as its presentation surface.

Intrigued I used an opportunity to speak at a local tech event to try out mdp . The slides from that presentation can be found on my talks page and overall I thought mdp worked quite well.

I was able to draft in the stream of conciousness style I want, getting the bulk of the slides written very quickly. Adding diagrams required resorting to ASCII art which isn't so bad, I like ascii art . mdp worked great in practice, I had to find readable dimensions for the text by trial and error, but overall it went well.

Plain text as a format does have some major downsides, mdp has a way to encode builds for text (see below), but I couldn't use it with my tools. ASCII art diagrams also meant that the builds I did use were eggregious to maintain, any modification required manual propigation through the build chain.

mdp does not support a portable output format. You may say the source markdown is an excellent format for portability, but I find it lacks the crispness of having a single slide in view at once.

I wanted to be able to point at a viewable copy of my slides and so I hacked together some tools to export the mdp presentation to html, but for this I had to sacrifice the built in build mechanism of mdp

Finally there was no way to include images in the mdp presentation let alone the sacride gif format required to correctly convey nyan cat. I played with some terminal graphics viewers, but none of them worked well and after a while I started to think 'what is the point of reinventing everything'.

Drafting the presentation in markdown fit very well with my work flow, but the difficulties in getting a complete presentation with mdp meant that I didn't want to use it for future presentations.

Exporting to html

Getting html of the mdp presentation hinged on a complete hack. There is a tool I had seen in the past that can output a html dump of a tmux session unsurprisingly called tmux2html . With some playing around I was able to automate a tmux session to work through the slides and use tmux2html to grab each slide as a frame.

Finding the number of slides in the deck required splitting on the slide seperator from the markdown, this ruled out using the built in build mechanism as I would end up with the wrong number of slides.

The output script runs through the markdown to find the number of slides then uses tmux send-keys to control moving through the deck.

#!/bin/sh

set -e 

command -v tmux >/dev/null 2>&1 || { echo >&2 "I require tmux but it's not installed.  Aborting."; exit 1; }
command -v tmux2html >/dev/null 2>&1 || { echo >&2 "I require tmux2html but it's not installed.  Aborting."; exit 1; }
command -v mdp >/dev/null 2>&1 || { echo >&2 "I require mdp but it's not installed.  Aborting."; exit 1; }

if [ -z "$1" ]
  then
    echo "tohtml presentatin.md [outfile.html]"
    exit
fi

file=$1
outfile=outfile.html

if [ ! -z "$2" ]
  then
  outfile=$2
fi

javascript="<script>function page(){var e=!1,n=document.getElementsByClassName('tmux-html'),l=0; document.onkeydown=function(t){if(t=t||window.event,key=t.keyCode,e)if(13==key){e=!1,l=0;for(var i=0;i<n.length;i++)n[i].style.display='inline'}else{37==key&&--l<0&&(l=0),39==key&&++l>=n.length&&(l=n.length-1);for(i=0;i<n.length;i++)n[i].style.display='none';n[l].style.display='inline'}else if(13==key){e=!0,n[0].style.display='inline',l=0;for(i=1;i<n.length;i++)n[i].style.display='none'}}}window.onload=function(){page()};</script>"

tmpfile=tmpfilenamefilething
tmux='mdptohtmlconverstionsession'

slides=`grep -e "^---" $file | wc -l`

tmux new-session -s $tmux -d -x 96 -y 25

tmux send-keys -t $tmux "mdp $file"
tmux send-keys -t $tmux "Enter"

tmux send-keys -t $tmux 'g'
tmux2html -o $tmpfile $tmux 1>/dev/null

# insert javascript
lines=`cat $tmpfile | wc -l`
styleend=`cat -n $tmpfile | grep -e "</style>" | awk '{print \$1}'`
head -n $styleend $tmpfile > $outfile
echo $javascript >> $outfile
tail -n $((lines-styleend)) $tmpfile >> $outfile
mv $outfile $tmpfile

# remove closing tag
lines=`cat $tmpfile | wc -l `
end=`tail -n 1 $tmpfile`
head -n $((lines-1)) $tmpfile > $outfile

echo turning $file into $((slides+1)) slides 

i=1
while [ $i -lt $((slides+1)) ]
do
    printf "\rSlide $i"
    tmux send-keys -t $tmux 'j'

    tmux2html -o $tmpfile $tmux 1>/dev/null 
    grep -e "^<div" $tmpfile >> $outfile
    (( i++ ))
done

echo $end >> $outfile
tmux kill-session -t $tmux 
rm $tmpfile
printf "\rwritten to $outfile \n"

If you view the presentation page you will see the entire slide deck, this was the first output I got from this script. All the slides in a nice order. After a little pondering I wrote up some javascript to give controls, if you hit enter it will go from all slides to single slide. Arrow keys in single slide mode will allow you to move through the slide deck. The unminified javascript for this is below.

function page() 
{
    var presenting = false
    var elements = document.getElementsByClassName('tmux-html');
    var current = 0;

    document.onkeydown = function(evt) {
        evt = evt || window.event;
        key = evt.keyCode 

        if (presenting) {
            if (key == 13) {
                presenting = false;
                current = 0;
                for (var i = 0; i < elements.length;i++)
                    elements[i].style.display='inline'
            } else {
                if (key == 37) {    //left
                    current--;
                    if (current < 0)
                        current = 0;
                }
                if (key == 39) {    //right
                    current++;
                    if (current >= elements.length)
                        current = elements.length-1;
                }
                for (var i = 0; i < elements.length;i++)
                    elements[i].style.display='none'
                elements[current].style.display='inline'
            }
        } else {
            if (key == 13) {
                presenting = true;
                elements[0].style.display='inline'

                current = 0;
                for (var i = 1; i < elements.length;i++)
                    elements[i].style.display='none'
            }
        }
    };
}

window.onload = function () {
   page();
}

Presentations with remarkjs

Sat, 03 Oct 2020 00:00:00 +0000

I enjoyed using mdp to write slides, being able to hammer in markdown gave a satisfying sense of flow and I felt like I was able to get the slides out of my head in a straightforward manner. But I knew for my eurobsdcon presentation I was going to have to include photos of equipment and maybe even demo videos.

Shelling out to vlc or feh for pictures and video wouldn't do, it would throw off both me and the audience. That ruled out using mdp for making slides and it also ruled out using sent from suckless

I canvassed around on mastodon and tried out a bunch of other tools, the main factor in ruling out most of the tools was there handling of very long titles. Something I couldn't avoid when the title of my talk was 84 charactars.

remarkjs was the tool I settled on.

remarkjs can take slides either as an external markdown file if you have a way to serve them to the js, or embedded into a html file. I ended up embedded the slides into the markdown as this was the fastest way to get from nothing to having some slides appearing. remarkjs has a boat of documentation, which I thourghouly ignored until after the presentation, in fact in the days after when I was toying with implementing a presentation view I found remarkjs already has one built in!

remarkjs was great for authoring into, the ability to add style to documents was a big bonus for me too. The fact there was style did mean I had to write some css to get videos into the right place in the slide was annoying, but it worked out well.

Integrating diagrams

My mdp slides included diagrams as most slide decks do, I wanted to add diagrams to this slide deck. The mdp diagrams are just ASCII art, showing ASCII art in a web page is fine, that is show I made a sharable version of the page, but I felt I could do better.

goat can render ascii art diagrams in a restricted set into svg diagrams.

example example example

Gives an svg diagram like:

svg

The svg output is very verbode and really not something you would want to embed in the middle of a slide deck.

svg quoted cut off

For this to be managable I wrote a python script to 'render' the document. The script searches the input for lines starting with 'diagram:' and takes the remainder of the line as a file name to render and substitute.

import sys
import subprocess

filename = sys.argv[1]
infile = open(filename, 'r')
outfile = open('out.html', 'w')

cmd = "cat"
cmd = "goat"

for l in infile:
    if l.startswith('diagram:'):
        if len(l.split(' ')) != 2:
            print('bad line {}'.format(l))
        diagram = 'diagrams/{}'.format(l.split(' ')[1].strip())

        result = subprocess.run([cmd, diagram], stdout=subprocess.PIPE, encoding='utf-8')
        if result.returncode == 0:
            count = 0
            outfile.write('.center[\n')
            for o in result.stdout.split('\n'):
            #    print('    ' + o)
                outfile.write(o + '\n')
            outfile.write(']\n')
        else:
            for o in result.stdout:
                print(o, end='')
            outfile.write(l)
    else:
        outfile.write(l)

infile.close()
outfile.close()

I really like remarkjs

I was happy enough using remarkjs that I was considering adding a presentation mode. However there are some downsides, firefox really struggled when rendering slides, when I had 40MB mp4 video files firefox would peg all cpus, as the slides were just a page the autoplaying video pulled firefox down all the time.

remarkjs "supports" exporting to pdf via chromes print preview, but all I could get chrome to do was hang. Someone else managed to get an export from safari, overall not the best.

Joining two sets with LINQ

Sun, 02 Feb 2014 00:00:00 +0000

Dealing with a horrible database this last week, I found the need to combine things in a reasonable way. It took a lot of searching to find out how to query on multiple sets. So thought I would put it here.

var roles = (from x in userRoles
                from y in editUser.UserRoles
                where y.SOXRole && y.Id == x.RoleId
                select x).ToList<DBModel.UserToRoles>();

I was pretty happy with it.

FreeBSD USB Installer

Wed, 12 Feb 2014 00:00:00 +0000

Years ago I got a copy of Designing BSD RootKits by Joseph Kong. A combination of lack of hardware and probably my own ability has stopped me from working through the book so far. But now with 57 North up and running and an influx of free machines I have everything I need.

The machine I have been given is part of an old biomed cluster and is really over powered for what I need. As a 2U server it doesn't have a floppy or CD drive to easily install an OS, but it does have the ability to boot off of a USB stick.

The first thing I tried to get a FreeBSD installer running was burning an ISO image to a USB stick with UNetBootin. I think the project might actually be dead as the newest version of FreeBSD it supports is 8.0. UNetBootin takes forever to set up the USB stick and after the second failed attempt I couldn't stomach another.

I dug around the FreeBSD install guides for a while and then found something that should have been really obvious. FreeBSD supports installation from USB and provides a pre packaged .IMG file to dd to the USB.

All the information is here with the USB stuff near the bottom. FreeBSD is nice enough to include simple instructions that work even from windows. This meant I could test the new media from work and all seems good.

Wot Happened; The break in

Sun, 09 Mar 2014 00:00:00 +0000

by Tom Jones age 23 and 1/4

On Saturday the 8th march 2014 we did a run through of the MakeIt-Glo workshop. Afterwards I went to the pub, leaving my bag(laptop and camera) in the space. Ed and Calum stayed in the space.

Charlene text me and awoke the hangover at 0500 on the 9th. Unable to sleep I headed into the space to get my laptop and bag and shit. The time lock was disabled at 0657 when I came in and the main door was open for the world.

I came up the stairs and saw that the door to the kitchen area had been pried open and all round damaged. I saw a guy that I thought was a lock smith(hungover head is optimistic), he pointed at our door and said something like "It is locked". I unlocked the door and walked up to him, I fumbled questions about his name and what was going on. He went to leave, but I saw my (United Pixel Workers) laptop sticker sticking out of the bag.

I said it was my laptop, he put the bag down and I grabbed it, my camera and laptop charger. He placed both the bags he was holding on the floor. I walked across the lab and put my stuff in my bag then pulled out my phone. He said "I've called the police already" my witty retort was "Well I'm doing it again". As the police call center answered he disappeared down the stairs.

Police came and took immediate details and put out a bulletin. Anther robbery happened on king st while the officers where talking with my. Logic ties the two together to both me and the officers. A crime scene officer came and printed the broken door and the items we were sure he had touched.

This bloke didn't wear gloves, tried to break through an unlocked door and didn't manage to grab our beer money jar. He broke into a dentist, I have no idea what he was expecting to steal. The bastard tried to steal our drinks cupboard.

Atari used to 'make' printers

Wed, 26 Mar 2014 00:00:00 +0000

It is a rare day that I remember markdown link syntax.

These guys are lucky they can walk.

Thu, 27 Mar 2014 00:00:00 +0000

The decapping of the 3DS chip is the sort of reverse engineer that just amazes me. The author mentions Bunnie's Book as a source of inspiration and I have to agree. Bunnie's book operates well above the level of chip decapping, but it gives you a window into an entire world of engineering that is usually hidden. Last year Bunnie released his book for free in memory of Aaron Swartz.

Bunnie is really cool guy and a hero of hackers around the world. Amoung other projects he is making the ultimate engineers laptop .

Stripe now takes Bitcoins

Thu, 27 Mar 2014 00:00:00 +0000

I am a big fan of stripe, recently using them for 57north's MakeIt-Glo workshop . The payment was smooth and easy to use, we didn't hear about any issues with stripe from any of the attendee's either.

Bitcoin I am still unsure of. I would love to make £1000's from idle speculation, but I haven't been able to buy it from anywhere other than people in the real world.

Being able to use both together can only be seen as a good thing, the more services that start to take bitcoin the better. Services like stripe that are genuinely legitimate and have good standing go a way to remove a lot of the alarmism around the currency.

Tarsnap is one of my favourite services on the internet. If you are looking for secure small scale off site backup I can't think of anything better.

Printing code with Enscript

Fri, 28 Mar 2014 00:00:00 +0000

I need to read some complicated tcp code. Being able to scribble all over code makes it a lot easier for me to follow the flow of what is happening.

enscript -2 -r -Ec -o - tcp_cong.c | ps2pdf - tcp_cong.pdf

I tried, but I couldn't get colour to work. In the end I don't really care if it is on paper.

Computer are very complex

Mon, 31 Mar 2014 00:00:00 +0000

Maps are cool.

Computers are now very complex, that Anandtech article really blew my mind. Mike Ash's article on 64bit arm has this the same insane sort of detail that the Anand Tech article does. Computers are cool.

This is the worst day of the year

Tue, 01 Apr 2014 00:00:00 +0000

This is the worst day of the year for the internet. Terrible internet holiday you can't really trust anything you read and most of it is really just quite annoying. A part from Google Pokemon, that was cool.

There is a lot of misinformed dialog about TCP and UDP for gaming. The 1024 Monkey's article covers a lot of real issue with using TCP and doesn't fall on the normal argument "Well I am pretty much reimplementing TCP". If that was the case you probably wouldn't get much of your game done.

Imprecision

Wed, 02 Apr 2014 00:00:00 +0000

Silly time changes

Sat, 05 Apr 2014 00:00:00 +0000

For some reason we change the time zone throughout the year. It doesn't make any sense to me and it makes it hard when I am using applications on my vps.

To avoid confusion it is nice to have irrsi running at local time.

$ TZ="UTC-1" irssi

Raspberry Pi Compute board

Mon, 07 Apr 2014 00:00:00 +0000

It took me ages to find Dan's box puzzle, the puzzle box was an awsome marketing tool that really shows how Nokia was.

Resize VDI

Wed, 16 Apr 2014 00:00:00 +0000

# /Applications/VirtualBox.app/Contents/MacOS/VBoxManage modifyhd YOUR_HARD_DISK.vdi --resize SIZE_IN_MB

Where SIZE IN MB is the new size for the drive

Use tcpdump to save wireless bridge

Sat, 03 May 2014 00:00:00 +0000

For campGND we need to extend a wireless network about 500m from the farm down to the site. We have been trying to salvage some equipment but where having trouble getting control of a pair of Senao wireless bridges (Senao Long Rage Multi-Client Bridge).

The devices has previously been configured by someone else to bridge a network between two buildings. Problem being we have no idea how these boxes have been setup. Looking online there was nothing helpful about factory resetting these boxes unless you already had access.

I decided to put a box on our ethernet and use tcpdump to scan for any traffic coming from the MAC Address on the bottom of the bridge.

# tcpdump -e -i en0 ether src 00:02:6F:45:C9:83

After a reboot of bridge the following appeared in my terminal.

115:48:15.741750 00:02:6f:45:c9:83 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 60: Request who-has 10.0.2.66 tell 10.0.2.1, length 46

Bingo, exactly what I was looking for. That arp request tells us where the bridge thinks it is 10.0.2.1 .

Now I could navigate to the bridges web interface, but I was still locked out. I read through the manufactures guide for the bridge, but I still couldn't see anything that looked like a factory reset. The guide did mention that the default ip for the bridge was 192.168.1.1 and it used a admin:admin as the login.

I decided to try powering on the bridge with the hardware button held down. I left tcpdump running so if there was any change on the bridges interface. I held down the reset switch and powered the bridge on, counting to 30 seconds. I then toggled the power and finally saw

15:48:17.750222 00:02:6f:45:c9:83 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 60: Request who-has 192.168.1.66 tell 192.168.1.1, length 46

The bridge had reset to the factory default.

Videos I have watched from BSDCan 2014

Wed, 04 Jun 2014 00:00:00 +0000

Videos from BSDCan that have jumped out to me so far.

Announcing Builds

Fri, 06 Jun 2014 00:00:00 +0000

I have been building kernels on my imac in virtual machines. This can take a while and I wanted notifications when the build had finished running.

On my imac

$ while true 
    nc -l 4000 | say
done

This makes netcat(1) wait in a loop for any connections then pipes the output into say(1).

On the vm

$ make buildkernel; echo "Build Complete" | nc -N imac 4000

Replace imac with the hostname or ip of your machine.

I have seen the build side hang and not close the connection until killed, I am not sure why.

Minimal R plots

Fri, 13 Jun 2014 00:00:00 +0000

For my new business cards I wanted to impose data from an experiment onto the background of an image. For the best results I wanted to render the plot of data onto a transparent png without any axis, values or the standard box.

After a while I got to

> png(filename="transplot.png",width=900,height=400,bg="transparent")
> plot(timestamp[0:1000],snd_cwnd[0:1000],type="h",yaxt="n",xaxt="n",ann="F",frame.plot="F")
> dev.off()

Which generates the following image.

It is actually

Tue, 17 Jun 2014 00:00:00 +0000

campGND is coming up and it is time to start talking about my projects for the weekend. With our remote location I thought it would be fun to play with something flaming and dangerous.

Rockets were the first thing that came to mind, I haven't done much with rockets beyond launching fireworks a couple of times. Doing my first launches at campGND would probably slow everything down somewhat. I got myself a starter kit from Model Rocket Shop and some extra motors, for a bigger bang.

Iain and myself went out to Balmedie Beach to have a test run with my new toy. We got a couple of videos of the rockets going up, excuse the portrait slow-mo.

On the first launch the recovery canopy got slightly melted by the rocket motor. This meant we didn't really have any recovery mechanism for the rest of the launches. The beach was pretty deserted in the dunes so this wasn't a big deal. At campGND loosing recovery could make things a little tricky.

For campGND I am planning on adding some telemetric data to the rockets, using an Arduino and some sensors. I also want to try adding a camera to the nose cone on a rocket.

campGND network

Thu, 19 Jun 2014 00:00:00 +0000

One of the facilities at campGND is going to be a wireless network. The hope is to have the network running for the majority of the time. I have built a wireless network at a campsite before, that was made easier by having guaranteed bandwidth from a satellite terminal.

The plan is to have a wireless network for the campsite served by a MikroTik . Using a wireless bridge to reach to the farmhouse. The farmhouse is out of site of the fields we are planning to use. Instead of having wifi doing the full jump I am going to run ethernet as far as possible.

At campGND we are depending on a few things that could be fickle.

BT Home Broadband
A long run of ethernet
Solar Cells and a battery for network power.

Our final back haul is the BT network the site is pretty off the grid for phone reception so we are stuck with BT. We have to be able to make a long hop form the farmhouse before we can do a wireless link down to the site. The solar cells will provide enough to run wireless access points during the day. I think at night we might be a little drunk to care.

I still need to do some testing of the wireless hardware but the plan is to use the following.

100M run of Ethernet.
Injected POE, then split POE.
2x WRT54G's.
A MikroTik access point.
Solar Cells for with battery backing for night time.

pv

Mon, 14 Jul 2014 00:00:00 +0000

I found a tool called pv via a Hacker News thread. pv or pipe viewer allows you to view data as it is pulled out of a Unix pipe. This is really helpful when dealing with long running commands. I used it today to check on the progress of encrypting a large tar archive.

$ pv archive.tar.xz | gpg --sign --symmetric - > archive.tar.xz.gpg
6.51GiB 0:08:52 [6.23MiB/s] [======================>       ] 70% ETA 0:03:47

While pv is running you get a progress, time elapsed, speed, a progress bar, 70% complete and an estimation of time until complete.

GPGME with Mutt on OS X

Wed, 30 Jul 2014 00:00:00 +0000

I found it quite difficult to get GPGME working with Mutt in OS X, I was using Homebrew to install mutt. I could see the option in the brew build file to use GPGME, but it was set as an optional dependency. I fought with it for a while then jumped across to #homebrew on freenode to get an answer.

I had to force brew to build mutt from source to get the dependency included. You will have to uninstall mutt if you have already installed it.

brew install mutt --build-from-source --with-gpgme

You will need to add the correct bits to your .muttrc to get mutt to use.

set crypt_use_gpgme = yes
set crypt_autosign = yes
set pgp_sign_as = 0xYOURGPGKEYGOESHERE*****

urtwn on FreeBSD ARM

Wed, 24 Sep 2014 00:00:00 +0000

This weekend I got FreeBSD on my Chromebook Snow in a usable state. Getting wifi going was a bit of a bother. I have an Edimax Wifi Adapter , but the default kernel config builds out support for wifi and the urtwn device driver.

The Beaglebone Black page on the FreeBSD wiki has a kernel config that includes the drivers I need. I took the wifi config and added them to a CHROMEBOOK-WIFI config so I could build a kernel for the Chromebook with support.

#USB WiFi
# Wireless NIC cards
device          wlan            # 802.11 support
options         IEEE80211_DEBUG
device          wlan_wep        # 802.11 WEP support
device          wlan_ccmp       # 802.11 CCMP support
device          wlan_tkip       # 802.11 TKIP support
device          wlan_xauth

device          firmware        # Required to load firmware
device          urtwnfw         # Firmware for RTL driver below
device          urtwn           # Realtek RTL8188CU/RTL8192CU

After building the new kernel and moving it over to the USB stick I use for the Chromebook I needed tell FreeBSD to accept the license terms for the wifi firmware.

Add to loader.conf
legal.realtek.license_ack=1

After that it was pretty norm wifi setup.

# ifconfig wlan0 create wlandev urtwn0
# wpa_supplicant -i wlan0 -c /etc/wpa_supplicant.conf -B
# dhclient wlan0

Strange Mast

Wed, 26 Nov 2014 00:00:00 +0000

Walking home from the hackerspace last night I came across this interesting mast behind a car parked on the pavement. I had to grab a picture of this strange thing on Union Street.

The guy operating the mast spotted me taking the picture and came down for a chat. This mast was acting as a 4G base station, there was a second vehicle driving around the city, listening for this mast to map 4G propagation. It turns out that Aberdeen city council are planning to roll out 4G across the entire city, with free access. The council want to use this for fleet management and I think it is probably part of their initiative to improve bandwidth in the city.

According to the operator of this mast the 4G won't just cover the city center, they have been mapping industrial estates in Altens and out towards the edges of the Bridge of Don

31c3

Tue, 20 Jan 2015 00:00:00 +0000

31c3 Finished 21 days, probably enough for me to get past the conference high, but also enough time for me to catch up on almost all of the talks. All of the talks from 31c3 and most of the other congresses have been put online at media.ccc.de .

I took notes during Congress, but I had a hard time turning them into a post. If anyone wants to see my notes I am sure they could appear. Instead here are some thoughts.

Congress is Europe's largest temporary art installation. The CCH is a simply massive building, after 5 days of wandering around the I am still not sure I saw everything. It wasn't until day 4 of Congress that I realised the floor numbers were not floor numbers, but actually the Saal you were closest to. The ones and twos I could see around me were not helpful directions.

The building was augmented by the CCC to make it a home for hackers. The lights were dimmed, there were blinkenlights in every corner and just to make things even better there seemed to be a pop up interactive installation at random intervals. A pneumatic tube system was run around the ground floor of the building.

Of course there were talks 12 hours of the day, but the talks were streamed later. There were too many things that could only be found in the 4 days of Congress to sit in a full lecture theatre.

Instead we held court at our table in the international hackerspace village. Hung around with the crazy cooks in the Food Hacking Base , argued politics in Noisy Square and wandered the cavern is a daze.

At night(due to the lighting it was hard to tell when that was) we would be at our table hacking on something super cool, or hiding in the amazing nightclub.

It is probably impossible to describe congress, there is just too much happening. It is probably unfair to try and give someone else a picture, the only real way to know what it is like is to experience it.

Join us next year.

User SPI Adventures

Sat, 28 Feb 2015 00:00:00 +0000

Last year I started working on a project in the space that I thought would be pretty cool, a Mystery Box. I made a box out of foam board, mounted a servo as a catch. Inside I wired up an Arduino to control the servo. The Arduino connected to a Raspberry Pi over SPI, I used SPI because it was a nice simple protocol to implement on both the Arduino and the Raspbery Pi.

The Mystery Box was going to run a BBS which had control over the servo. My idea was to use the box as a simple CTF target, with the servo giving instant and substantial feedback for success.

Around this time I had been asked to port NewCWV to FreeBSD, we wanted to have more than one implementation of the proposed standard available. Doing some development in the FreeBSD kernel made me want to look at using the operating system in other places.

Up to this point I had been using Linux on the Pi in the Mystery box, but I thought it would be fun to try FreeBSD. FreeBSD on the Pi was in a reasonable state, I didn't have trouble getting the Pi to boot. When it came to controlling the Arduino over SPI I hit a snag.

There wasn't (and still isn't) user space SPI support in FreeBSD. This means that I can control devices to connected to SPI from a kernel driver, but I can't do so from user space. Kernel code is harder to write, not portable and means I can't reuse Linux code. User space code is easier to write and if the interface is similar to existing ones I can reuse a lot of other code.

I was enjoying writing the NewCWV port and I thought to myself: "I should make the world a better place and write a user space SPI layer". And that is exactly what I set out to do over the next few months.

This week, well over 9 months later I finally have a working SPI layer. I can issue, read(2), write(2) and ioctl(2) commands and see the bus burst into life with data flowing across.

On the other end of the bus I have a Trinket Pro (Adafruit arduino clone). The Trinket acts as a SPI slave, when there is activity on the bus it spits on the values from the master over UART and writes the SPI values back onto the bus with 10 added.

The Arduino seems to struggle at the default bus speed of 500KHz, but runs fine when I lower the speed with a sysctl down to 50KHz.

Speaking to the Arduino is okay, but it doesn't make a very exciting demo. I had a couple of devices that can be controlled over SPI, a SSD1306 OLED Screen and a PN532 NFC Reader .

The NFC Reader is supported by libnfc , a quick look shows that libnfc is available in the FreeBSD ports tree. I looked at the libnfc code while writing the user space layer and it seemed pretty straight forward. libnfc opens the SPI device and calls the SPI IOC MESSAGE ioctl to send spi ioc transfer structs to the driver.

The interface I have written is a little different, using an object to describe the transfer similar to the way iic(4) works. To add FreeBSD support I need to create the struct and swap the ioctl(2) call, this should be straight forward.

Using the OLED is a little different. Adafruit have provided a python library to speak to the screen using either i2c or SPI. The python library imports a module to speak to SPI and seems to mostly use read(2) to control the screen. This is going to be harder to port across and get working.

There is also an Adafruit Arduino library for the SSD1306 and Arduino compatible boards. This is C++ that has been written to run on an Arduino rather than on a unix machine. This code could probably be slimmed down, with the calls to fastspiwrite swapped out to calls to the kernel interface.

My implementation is still a little rough around the edges, the code needs to be tidied up, moved into the kernel directly and tested on the latest head. I think that getting user space code working with it will show up any bugs, it will certainly make for more meaningful test cases.

My next step is to get the NFC reader working with libnfc and the raspberry pi, then I can start work on the screen. Once I have some SPI examples working I might even get back to setting up the Mystery Box for a CTF in the space.

libnfc i2c usage

Sun, 01 Mar 2015 00:00:00 +0000

I have been working on applications using the SPI user space api. One of the devices I have been playing with is a PN523 NFC reader. The reader is supported by libnfc and can communicate using serial, SPI, i2c or usb depending on device support.

I wanted to get the nfc reader working over i2c, to form a baseline to compare it against SPI. I couldn't get the nfc-list to show any NFC devices connected to the Pi. I then tried to use the i2c -s command to scan the i2c bus, but instead of device detection the command threw an error.

It turns out that the iic driver for the Pi only supports one ioctl, I2CRDWR. That neuters most of the FreeBSD i2c tools as they use other ioctl's and error out on failure.

Learning that I looked at the Makefile for libnfc, this time realising that the i2c and the SPI device options are both commented out. I missed this the first time I looked at the FreeBSD port, there is still the option to use a serial device once I dig out a usb serial adapter.

It is looking a lot harder than I thought to get devices working with user space SPI on FreeBSD. It doesn't help that Linux has been the only operating system with user space SPI support for quite a long time.

msp430 hello world

Mon, 02 Mar 2015 00:00:00 +0000

A couple of years ago the TI Launchpad made quite a splash when TI released the boards for just $5 each. The Launchpad uses the msp430 low power microcontroller from TI, these microcontrollers don't have the same pretty face as the avr microcontrollers in the AVR world.

This means the code is a little harder to read and write. It is a lot closer to assembly language than the high level Arduino C/C++. While it looks worse I find it more fun to write and it will leave you with a much better understanding of how the controller is working.

So lets load up a simple blinking light program(no more of that sketch nonesense), fire up the debugger and stop the code as it is executing. Grab the following code and make file and compile up the main.elf target. If you are using a different microcontroller then you should change the g2553 to the microcontroller you are using.

Blink Program

#include <msp430g2553.h>

int i = 0;
int j = 0;
int
main(void)
{
    WDTCTL = WDTPW + WDTHOLD;   //Stop the watch dog timer
    P1DIR |= 0x41;              //Set the direction bit of P1(0x01) as an output

    P1OUT = 0x40;               //Set P1.6 high, P1.0 low

    for(;;) {
        P1OUT ^= 0x41;          //Toggle both P1.1 and P1.6

        for(i = 0; i < 20000;i++){  //Loop for a while to block
            nop();
        }   
    }   
    return 0;
}

Makefile

CC=msp430-gcc
CFLAGS=-Os -Wall -g -mmcu=msp430g2231

OBJS=main.o

all: $(OBJS)
    $(CC) $(CFLAGS) -o main.elf $(OBJS)

%.o: %.c 
    $(CC) $(CFLAGS) -c $<

clean:
    rm -fr main.elf $(OBJS)

We use the mspdebug program to flash the program to the msp430 like so.

$ mspdebug -q rf2500
Trying to open interface 1 on 006
rf2500: warning: can't detach kernel driver: No data available
fet: FET returned error code 4 (Could not find device or device not supported)
fet: command C_IDENT1 failed
Device: MSP430G2xx3
fet: FET returned NAK
warning: device does not support power profiling
(mspdebug) prog main.elf
Erasing...
Programming...
Done, 152 bytes total
(mspdebug) run
Running. Press Ctrl+C to interrupt...
^C
    ( PC: 0c068)  ( R4: 0dff7)  ( R8: 0ffbb)  (R12: 0fdf7)  
    ( SP: 00400)  ( R5: 05a08)  ( R9: 0dcff)  (R13: 0fd67)  
    ( SR: 00004)  ( R6: 0b775)  (R10: 0dddf)  (R14: 0dfff)  
    ( R3: 00000)  ( R7: 0ff1f)  (R11: 0dfff)  (R15: 00000)  
main+0x2a:
    0c068: f9 3b                     JL      0xc05c
    0c06a: f2 3f                     JMP     0xc050
    0c06c: 32 d0 f0 00               BIS     #0x00f0, SR
    0c070: fd 3f                     JMP     0xc06c
    0c072: 30 40 76 c0               BR      #0xc076
    0c076: 00 13                     RETI    
(mspdebug) exit

There you have it, your first hello world on the msp430.

Tech seen debugging bus sign

Tue, 03 Mar 2015 00:00:00 +0000

In Aberdeen we have digital displays mounted in most of the bus stops, in fact most major cities in the world probably have similar signs. The signs get their data via radio broadcasts, these broadcasts have in fact been captured before and reverse engineered.

For a long time I have been thinking about doing a similar thing and figuring out the bus information that is in the air. I am sure I will get to it one day.

Well this morning as I headed off to work I caught a technician in the act of debugging one of these signs. I grabbed a quick picture of the guy working, but I didn't want to bother him.

It looked like the tech was using a serial cable from his laptop up to the display. The antenna on the bus shelter looked much larger than the normal ones.

Attaching a debugger to mspdebug

Wed, 04 Mar 2015 00:00:00 +0000

The cool thing we get with the msp430 and the Launchpad is on chip debugging. This goes a long way to pave over the warts of writing straight C for the 430. We will load up the blink program from last time , run it with the debugger and pause execution.

Start up mspdebug as before, this time we pass a command for mspdebug to run directly. The "gdb" command will cause mspdebug to listen on port 2000, gdb can then connect and control the debugger.

$ mspdebug rf2500 "gdb"

Next we are going to start up msp430-gdb, load the program from before and start it running.

$ msp430-gdb
(gdb) target remote localhost:2000
(gdb) file led.elf
(gdb) load led.elf
(gdb) continue
^C
(gdb) break main.c:14
(gdb) c                 #We can shorthand commands
Now between each continue we will see the LED's toggle, red then green.
(gdb) continue

From gdb we can send mspdebug directly with the monitor command.

Reading analog values on the msp430

Thu, 05 Mar 2015 00:00:00 +0000

I finally got all the components to use the bag of electret microphones I got last year. Using a LM36 audio amp I set up a simple circuit to read values from the microphone and set LED's corresponding to the value read.

I had trouble finding a solid example of doing analog reads in a loop for the msp430. This program, sets up LED's(On P1.1 and P1.6), sets up the ADC, then sits in a loop. If the value on the ADC is grater than 512 the LED will go red.

I tested this by shouting at the micro controller, something I have done many times before, but never with such satisfying results.

#include <msp430g2553.h>

unsigned int adcvalue = 0;

void configureADC(void);

int
main(void)
{
    WDTCTL = WDTPW + WDTHOLD;       //Stop the Watch dog

    P1DIR |= 0x41;                  //Enabled P1.0 and P1.6
    P1OUT = 0x41;

    BCSCTL1 = CALBC1_1MHZ;          //Set range
    BCSCTL2 &= CALBC1_1MHZ;         //SMCLK = DCO = 1MHz

    P1SEL |= BIT3;

    configureADC();
    __enable_interrupt();

    while(1) {
        __delay_cycles(20000);
        ADC10CTL0 |= ENC + ADC10SC;         //Sampling and conversion start
        adcvalue = ADC10MEM;                //Read from the ADC

        if(adcvalue > 512)
            P1OUT = 0x40;
        else
            P1OUT = 0x01;

    }

    return 0;
}

void
configureADC(void)
{
    ADC10CTL1 = INCH_3 + ADC10DIV_3;
    ADC10CTL0 = SREF_0 + ADC10SHT_3 + ADC10ON + ADC10IE;
    ADC10AE0 |= BIT3;
}

Using portmaster to bulk install ports

Fri, 06 Mar 2015 00:00:00 +0000

As I write this there are not any packages available for FreeBSD arm. That means on the Raspberry Pi I have to build the things I need from ports. Ports gives a lot of control about how the software is built, but right now I just want the tools I need installed.

It is hard to set up ports to install a collection of tools in a oner, but we can use portmaster to do this for us. The Pi is quite slow and will take a long time to build a small number of tools and their dependencies.

Normally portmaster will prompt for configuration for each package as it builds, it will then build a list of packages and prompt again to install these. The following line will install the listed tools without any prompting.

$ portmaster -mBATCH=yes --no-confirm -y sysutils/tmux editors/vim

The -m flag will pass options to make
--no-confirm will disable the 'install' prompt
-y will answer yes to any prompts

Controlling keyboard leds

Mon, 09 Mar 2015 00:00:00 +0000

I have just finished reading Cryptonomicon , without any spoilers I can say that at a certain point a character controls the led's on his keyboard. This morning I found the kbdled utility via a forum post. It looks like a nice simple way to make the keyboard do something useful.

wait_on

Mon, 16 Mar 2015 00:00:00 +0000

I have been working on stuff in latex recently and wanted something to trigger regeneration of pdfs without manual intervention. At first I thought about doing so in a loop every so often, but it didn't seem like the best approach.

Knowing about the cool kqueue framework I looked to see if there was a utility to watch files for me. I found the wait_on command via a FreeBSD forums post.

The wait on command will wait until the watched file has been changed and then exit. It is meant to be used within a loop. I through this together in zsh. Now when I :wq in vim wait on exists and my document rebuilds, evinces picks up on this and refreshes the display.

$ while true; do wait_on pres.tex; xelatex pres.tex; sleep 5; done

I have the sleep at the end to stop the document being built too often.

Adding a TrueType font

Thu, 19 Mar 2015 00:00:00 +0000

I needed to add a .ttf font for a presentation I was working on, it turned out to be a lot more hassle than it needed to be.

$ mkdir ~/.font  
$ cp font.ttf ~/.font/
$ mkfontdir ~/.font  
$ fc-cache

I read a lot about editing xorg.conf to change font paths, in the end it was a matter of refreshing the font cache and restarting evince.

Screenshotting in i3

Tue, 31 Mar 2015 00:00:00 +0000

It took me a while to find a screenshot tool as useful as the built in screenshotting tool in OS X. I looked again today and found the import tool that comes as part of ImageMagick.

$ import screenshot.png

You can use it to capture an area on the screen with the above command or you can capture a whole window.

$ import -window root screenshot.png

I will probably throw this into a script and bind it to a key for ease of use.

rtl_sdr and OpenWRT

Fri, 03 Apr 2015 00:00:00 +0000

Using a TP-Link WR703N and a RTLSDR I decided to make a small dedicate SDR box. Using rtl_tcp I can set up the box and a suitable antenna and use it to receive IQ values over a wifi or ethernet link. Using the wifi means I can do this without pluggin a ton of crap into my laptop.

WR703N

The WR703N has been really well documented, with a full section of mods on its wiki page. I have add serial console headers and a rp-sma antenna connector on the box I used for this project.

These were fun to do, the serial connector makes debricking the WR703N a lot easer, the rp-sma connector allows different antennas to be used with the router. With some more gain behind it, I should be able to place the sdr box somewhere high and out of the way and still be able to connect to it.

For getting OpenWRT onto the WR703N you can follow the generic flashing instructions. Make sure to install Barrier Breaker or later, BB has a prebuilt package for rtl_sdr.

I installed the rtl_sdr software via the web interface, but it can be done from the command line with something like the following.

# opkg update
# opkg install rtl_sdr

rtl_tcp

Once you have the rtl sdr packages installed, connect your rtl sdr dongle to the usb port then run the following.

$ rtl_tcp -a 192.168.1.1 -n 8 -b 8

This command will start rtl_tcp and have it listen on the 192.168.1.1 address for external connections, without this it will only listen on localhost. If you have configured your network differently you will want to change the listen address.

I had a lot of trouble running rtl tcp for more than a few seconds with a client connected, this was fixed by configuring the buffer options. The -n option configures the number of linked lists rtl tcp will use, -b configures the number of buffers it will use. I have had a quick look at the rtl_tcp source, but I couldn't really figure out why this helped so much.

Viewing the data

The last thing to do to test this is connect a client. The rtl sdr tools can't connect to a rtl tcp source, but we can connect and grab some data using netcat.

$ nc 192.168.1.1 > capture.iq

This might be enough if you have a process for dealing with iq data, but I like to look at things. The GrOsmoSDR package comes with a couple of tools for viewing ffts and waterfalls using GNURadio.

$ osmocom_fft -W -s 2000000 -f 144000000 -a 'rtl_tcp=192.168.4.1:1234'

        Without any of the following it will show an fft
-W      Show a waterfall
-S      Show a scope
-F      Show the cool fosphor display

Setting serial baud rate on FreeBSD

Tue, 07 Apr 2015 00:00:00 +0000

I have a navspark gps microcontroller board I backed on indiegogo last year. The board has been sat in my desk for a year so I decided to just use it as a dumb gps and not bother with the microcontroller part of the board.

The default firmware sends nmea strings over a usb serial controller at 115200 baud, this was easy to test with cu. I wanted to use gpsd with the gps, I am planning to integrate it into a wardriving box in the next few weeks.

cu -l /dev/ttyU0 -s 115200

gpsd is unable to accept baud rate changes, instead there is workaround in the faq. The faq is probably wildly out of date, I couldn't get stty to change the baud rate on FreeBSD. I found that FreeBSD offers .init files for each of the serial devices and they should be used for configuring the serial device.

Using the following command worked for me and allowed gpsd to speak to the navspark.

# stty -f /dev/ttyU0.init speed 115200
# gpsd

I could then connect to the gpsd and make sure it is working with cgps.

$ cgps -s -u m

I am not really happy with the navspark, the indiegogo made the board look really cool, but so far there no community has formed around the board. This has led to a lack of approachable documentation and an ide only available with Linux and Windows bulds.

I would love to find a cheap gps that emits data over serial. The closest thing is the Adafruit Ultimate gps , but it is far too expensive for what it is. I have a pair of U-Blox PCI GPS cards , so far I haven't been able to get them working with anything.

Talks from BSDCan2015

Wed, 24 Jun 2015 00:00:00 +0000

Like last year , here are from BSDCan that have stood out to me. I don't think all of the videos have been posted yet so there are probably some gems left to watch. All of the videos are here

The laddie and the Tramp

Sun, 28 Jun 2015 00:00:00 +0000

The best way to get around Paris is to use the metro, if you are coming into CGD you can take the train to Gard du Nord then hop onto the metro from there. Metro stations seem to be dense enough that there will be one near to your destination, I didn't see more than a 10 minute walk.

Using the metro fulfilled every Parisian stereotype I had, lovers kissing, gypsies begging, men busking with accordions. The metro was a brilliant way to get around very entertaining.

Just as entertaining for me(though some might not enjoy it) was my pre metro knowledge walk across Paris to reach my hotel. On the map before traveling the walk didn't look every long. I didn't have any frame of reference for Paris, but a similar distance around the Thames in London would be a reasonable walk. Well reasonable to people that like to walk through cities.

With the 30°C heat at 1800 it was probably a little long for a 6Km walk through the city. But the walk was very fortuitous if I had been down in the metro I wouldn't have seen the stunning sights of Paris, large buildings, street gangs, passed out tramps that have pissed them selves and the myriad of cheap suit shops. Shiny silver suits are a steal at 50€.

After a couple of bouts of despair I reached my hotel in once piece, only loosing about 5 kilos in water.

The fox in the city

Mon, 29 Jun 2015 00:00:00 +0000

The Mega Charity Mozilla keeps offices for their staff in many major cities. I think most of their staff work from home, but some must visit offices and the require space to hold meetings. Hopefully Mozilla Space Paris is the most decant of them all.

The space has all the trappings you would expect from a hip and trendy startup, mozilla sort of is. They have a big airy space, a fancy cateries kitchen and the most insane meeting room I have ever seen. You can see from the pictures why the French Revolution started.

I should probably apoligize to anyone that has donated to mozilla in the past. I took full use of their stocked kitchen and to avoid the ridiculous parisian beer prices drank more than my share of mozilla beer. Yum yum.

Paris loves the theatre

Tue, 30 Jun 2015 00:00:00 +0000

Paris loves the theatre, they have world renowned plays enjoyed by douchy teenage girls the world around. They love no theatre more than Security Theatre. To transit through Paris CDG and make it to the departure lounge you need to show your passport twice and your boarding pass at least six times.

In fact one agent of the airport was enjoying her role more than anyone I have seen at work. She scanned my boarding pass and scrutinized my passport before sending me through the metal discoverer.

At the other side I waited and waited expecting my bag. Instead I hear a shriek! 'You did not show me your pass'. I am dragged back through the magnetic arch to show my passport once again. This this with the agent shouting as if I had stripped half naked.

Oh fun.

BHX is a strange airport, security are trying to stay in business and keep manning up. They manage this by directing transfer passengers back through security to redo the dance, I did get set on the priority track though.

Of course, with a flights worth of passengers transferring this wasn't quick.

Past security and a worm whole takes you to a mall in the center of the city. A shopping horror exists until you overcome the forces of capitalism and resign your self to sit in the uncomfortable long.

Radio day to Balmedie Beach

Fri, 03 Jul 2015 00:00:00 +0000

Planning a radio field day was all it took to ruin a week of perfect weather. Instead of the glorious sunshine and high temperatures of the previous days the North Sea took revenge and summoned an mighty Haar to punish us for our hubris.

We hit the beach with a bbq, food and a couple of radios. Hibby had his new toy, a clansman set including a 5m mast. The mast was light to carry, easy to slot together actually really easy to put up. I think with some practice it could be erected by one person by pegging in the guy lines first.

The bands were relatively quiet considering it was a Friday afternoon, but Hibby and Derecho had a couple of good contacts from across Europe. This radio nonsense was what interested me though, I was more interested in the playing in the dune system.

Open Screenshotting

Mon, 13 Jul 2015 00:00:00 +0000

There are a number of services out there that allow you to take a screenshot and upload it to a website. All of these tools that I have seen(I didn't look, at all) used have involved a proprietary service and uploading your images to someone else's hosting.

That isn't good enough for me, I needed an open tool I could use anywhere (FreeBSD support) with the ability to drop the resulting png into a directory on a webserver I control.

Here is my tool to solve this problem, screenshot. Screenshot can capture either the entire window or offer a picker to grab a certain area. I used import from ImageMagick to handle the capturing and some glue to upload the image. There is another option to open the image with feh if required.

$ screenshot open
$ screenshot upload
$ screenshot pick upload

The script also dumps file names and url into a log file, this makes it easy to track down the last taken screen shots. I have some awk magic to pull out the last url and throw it onto my clipboard.

#!/bin/sh

shotdir=$HOME/screenshots
site="mysite.me"
uploaddir="webdir/screenshots/"

if [ ! -d $shotdir ]; then
    mkdir $shotdir
fi

one=`word`
two=`word`

word=$one-$two.png
file=$shotdir/$word
name=`basename $file`
url=$site/screenshots/$name

pick=false
open=false
upload=false

for var in "$@"
do
    if [ "$var" = "pick" ]; then
        pick=true
        continue;
    fi

    if [ "$var" = "upload" ]; then
        upload=true
        continue;
    fi

    if [ "$var" = "open" ]; then
        open=true
        continue;
    fi
    file=$var
done

echo "File:" $file

echo $file "http://"$url >> $shotdir/screenshot.log

if $pick; then
    import $file;
else
    import -window root $file;
fi

if $upload; then
    scp $file $site:$uploaddir/$name
fi

if $open; then
    feh $file
fi

I use another shell script to generate a random word. This script uses my local system dictionary, /dev/random and some glue to get a random word. The glue uses three bytes read from /dev/random and uses od to format those bytes into something useful. I then use sed to seek to the line in the dictionary to get the word.

#!/bin/sh

words="/usr/share/dict/words"

num=`od -An -N3 -i /dev/random`

line=$(($num % `wc -l < $words`))

word=`sed -n "$line"p $words`

echo -n $word

Touch Screen and Tablet on x220 Tablet and FreeBSD

Mon, 19 Oct 2015 00:00:00 +0000

My main laptop is a Lenovo x220 Tablet with an an awesome swivel screen. The screen on the laptop is a touch screen and wacom tablet which uses a pen that hides in the side of the laptop.

I had quite a bit of trouble getting this all setup. Wacom touch and pen devices are supported by webcamd in FreeBSD. I set up webcamd as documented elsewhere on the internet and while I could see webcamd grabbing the input devices the touch screen or pen didn't work at all under X.

Eventually I figured out the problem was xorg not detecting the hid device nodes. To solve this I had to manually create an xorg.conf and the following sections.

Section "ServerLayout"
    Identifier     "X.org Configured"
    Screen      0  "Screen0" 0 0
    Screen      1  "Screen1" RightOf "Screen0"
    InputDevice    "Mouse0" "CorePointer"
    InputDevice    "Keyboard0" "CoreKeyboard"
    InputDevice    "stylus" "SendCoreEvents"
    InputDevice    "touch" "SendCoreEvents" 
EndSection

...

Section "InputDevice"
    Driver        "wacom"
    Identifier    "stylus"
    Option        "Device"       "/dev/input/event0"
    Option        "Type"         "stylus"
    Option        "USB"          "on"                 # USB ONLY
    Option        "Mode"         "Absolute"           # other option: "Absolute"
    Option        "Vendor"       "WACOM"
    Option        "tilt"         "off"  # add this if your tablet supports tilt
    Option        "Threshold"    "5"   # the official linuxwacom howto advises this line
EndSection

Section "InputDevice"
    Driver        "wacom"
    Identifier    "touch"
    Option        "Device"       "/dev/input/event1"
    Option        "Type"         "touch"
    Option        "USB"          "on"                  # USB ONLY
    Option        "Mode"         "Absolute"            # other option: "Absolute"
    Option        "Vendor"       "WACOM"
    Option        "tilt"         "off"  # add this if your tablet supports tilt
    Option        "Threshold"    "5"   # the official linuxwacom howto advises this line
EndSection

With the now xorg.conf dropped into /etc I could now restart the server and boom, touch screen and tablet working quite well.

When I sniveled the screen I wanted to to be able to rotate my display and input devices to the correct orientation. I wrote a little shell script that can either advance the screen rotation by 90 degrees or set it back to the default orientation.

#!/bin/sh

output=LVDS1
rotation="normal";

stylus="stylus"
touch="touch"

if [ "normal" == "$1" ]; then
    rotation="left";
else
    rotation=`xrandr --query --verbose | grep $output | awk '{print $5}'`
fi

case $rotation in
    normal)
        xrandr --output $output --rotation right
        xsetwacom --set "$stylus" Rotate cw
        xsetwacom --set "$touch"  Rotate cw
    ;;
    right)
        xrandr --output $output --rotation inverted
        xsetwacom --set "$stylus" Rotate half
        xsetwacom --set "$touch"  Rotate half
    ;;
    inverted)
        xrandr --output $output --rotation left
        xsetwacom --set "$stylus" Rotate ccw
        xsetwacom --set "$touch"  Rotate ccw
    ;;
    left)
        xrandr --output $output --rotation normal
        xsetwacom --set "$stylus" Rotate none
        xsetwacom --set "$touch"  Rotate none
    ;;
esac

I bound the script in my .i3/config to the two screen rotation buttons on the fron of the bezzel. I found the keycodes by using xev.

bindcode 198 exec rotate normal
bindcode 204 exec rotate

Overall the touch screen and tablet work quite well. When webcamd starts it doesn't always detect both the touch screen and tablet and sometimes in places them at different event points. If I could figure out a way to make these predictable or if a later xorg detects the input devices correctly then this setup would be perfect.

gimme pcbs

Tue, 20 Oct 2015 00:00:00 +0000

With the launch of the yardstick one I remembered the im-me I bought earlier this year. Not wanting to risk destroying one of the last available im-me's in the world I decided to get pcbs made of Michael Ossmann's gimme .

I found a link to the OSH Park board page an ordered a small batch(3 boards) for less than £10. They came in about 3 weeks and seem to be reasonable quality, I will try them when my goodfet appears this week.

rfcat on FreeBSD

Wed, 21 Oct 2015 00:00:00 +0000

My Yardstick One appeared yesterday, time to set up RFCat.

RFCat has not yet been packaged on FreeBSD so I had to install it manually. I pulled the RFCat source from bitbucket which includes both the firmware and the client tools. To play with the stock firmware on the YSO I just had to install the client tools.

The client tools depends on libusb-1.0 , which ships in FreeBSD and on pyusb . Pyusb is offered by the py27-usb port.

$ sudo pkg install py27-usb

Then I built the rfcat client tools:

$ cd code
$ hg clone ssh://hg@bitbucket.org/atlas0fd00m/rfcat
$ cd rfcat
$ sudo python setup.py install

I had to set up devfs rules to access the usb devices, with my account in the usb group I have the following:

# /etc/devfs.rules
[localrules=10]
add path 'usb/*' mode 0660 group usb 

#/etc/rc.conf
devfs_system_ruleset="localrules"
devd_enable="YES"

With that all set up I can now try the rfcat tools

$ rfcat -r
'RfCat, the greatest thing since Frequency Hopping!'

Research Mode: enjoy the raw power of rflib

currently your environment has an object called "d" for dongle.  this is how 
you interact with the rfcat dongle:
    >>> d.ping()
    >>> d.setFreq(433000000)
    >>> d.setMdmModulation(MOD_ASK_OOK)
    >>> d.makePktFLEN(250)
    >>> d.RFxmit("HALLO")
    >>> d.RFrecv()
    >>> print d.reprRadioConfig()

The r flag tells the client to throw me into the research prompt and I get left in something that looks sufficiently like ipython. To test that everything was working I decided to transmit some bytes in a loop in the ism 433 band.

In [1]: d.setFreq(433920000)

In [2]: d.setMdmModulation(MOD_ASK_OOK)

In [3]: d.makePktFLEN(4)

In [4]: d.setMdmDRate(4800)

In [5]: for i in range(0,15):d.RFxmit('\xDE\xAD\xBE\xEF');

In [6]: for i in range(0,15):d.RFxmit('\xDE\xAD\xBE\xEF');

In [7]: quit()

I used an rtlsdr dongle and sdrtouch on my phone to get a quick demod of the spectrum and to see a waterfall. I tried this a few times, but I wasn't seeing the expected signal. Right off to the far right edge of the screen I was seeing a jump in strength, tuning around a bit while transmitting I eventually caught my burst packet. It seems that my rtl dongle is about 400KHz off the actual observed frequency.

IM-ME Specan

Thu, 22 Oct 2015 00:00:00 +0000

Just before I left work yesterday I built one of the gimme boards I got earlier this week and connected it up to a goodfet . I had to do a little source editing to let the goodfet run and connect to the correct serial port. If you need to change the serial port from the default it is a quick grep through the source tree to find literal string "/dev/ttyU0" to change.

I followed the instructions on the git repo for the specan code . The first time I ran the flasher the IM-ME booted into the stock firmware again. I erased the flash, tried again and it all worked. I am not sure how long the flashing took, but if you will be holding gimmme expect it to be a few minutes.

To flash the IM-ME I did:

$ goodfet.cc erase
$ goodfet.cc flash specan.hex

This turned out to be a lot easier than I expected, everything seems to be well documented. If you can get an IM-ME and want to flash it with a goodfet and a gimme, send me an email and I will send you one of my spare(partially assembled) boards.

57N Stupid Shit No One Needs Hackathon

Fri, 23 Oct 2015 00:00:00 +0000

Hibby and I were happy to announce the first 57N Stupid Shit No One Needs Hackathon this week. It isn't often that you come across a strange link in your search history and it turns into an awesome event, we seem to have beaten the odds.

The first Stupid hackathon I read about produced some of the coolest ideas for pointless things I have ever seen. The best one to make the event clear has to be endless.horse .

So what are you going to do Tom?

Of the many terrible ideas I have each day, only a few are worth spending 48 hours polishing to death. This coming weekend I have decided to take two technologies I have been gradually learning, mirco controllers and DSP, and build the most terrifyingly bad things I can think of.

So tomorrow prepare yourself to see the start of a paper cup and string telegraph being forged in 57North Hacklab.

57N Stupid Shit No One Needs Hackathon - Results

Mon, 26 Oct 2015 00:00:00 +0000

This weekend was the first 57N Stupid Shit No One Needs Hackathon . I tried this weekend to perform serial comms over a string and cup using the msp430 based TI Launchpad.

I had the tone generation working really quickly and then spent 15 hours trying to demod and tones and find a byte stream using a microphone. I had no chance, it didn't work at all.

I was able to transmit the tone a long the string over 3 meters of room. So the core idea does work. I think I will try this project again after reading some more dsp.

FreeBSD on a pi with a small screen

Tue, 27 Oct 2015 00:00:00 +0000

For the past month or two the uboot on the FreeBSD RPI-B images has been unable to boot on most sd cards. This weekend a new version of uboot was released and new images were created. The new images boot no problem and I am finally able to try this cheap 5 inch screen I got on ebay.

Super simple presentations

Wed, 25 Nov 2015 00:00:00 +0000

This weekends In The Other BSD's section had a link to a nycbug thread about presentation software. That was strangely apropos, last week I made slides for a lightning talk using my own template and beamer just exploded. I fixed the issue with beamer, I was pretty upset, upset enough to try looking for other software to use when I can.

At the start of the nycbug thread suckless sent is mentioned. sent is a really simple presentation tool that, it takes some input files and shows them as a slideshow. No pdf output, no templates, just a presentation.

sent isn't packaged in FreeBSD, suckless make it rather easy to build their tools(you normally edit a header and rebuild to do config) so I grabbed the source and built it.

I had to add some search paths to get it to build:

$ git diff
diff --git a/config.mk b/config.mk
index ed08199..6f5f3e4 100644
--- a/config.mk
+++ b/config.mk
@@ -11,8 +11,9 @@ X11INC = /usr/X11R6/include
 X11LIB = /usr/X11R6/lib

 # includes and libs
-INCS = -I. -I/usr/include -I/usr/include/freetype2 -I${X11INC}
-LIBS = -L/usr/lib -lc -lm -L${X11LIB} -lXft -lfontconfig -lX11 -lpng
+INCS = -I. -I/usr/include -I/usr/include/freetype2 -I${X11INC} -I/usr/local/include -I/usr/local/include/freetype2
+
+LIBS = -L/usr/lib -L/usr/local/lib -lc -lm -L${X11LIB} -lXft -lfontconfig -lX11 -lpng

# flags
CPPFLAGS = -DVERSION=\"${VERSION}\" -D_XOPEN_SOURCE=600

Sent just expects paragraphs of text.

Quick gifs

Thu, 26 Nov 2015 00:00:00 +0000

In the last post I showed an animated gif of the of the post source run through sent.

This gif was super easy to make manually, I ran sent on the post source file, then I ran my screenshot tool from dmenu on each slide. I stepped through each slide manually.

For a long presentation, or if I might do this more often I would probably automate this in some way.

I was left with a directory of files call 1.png, 2.png, for each of the slides. I used the convert tool from imagemagick to turn these into an animated gif.

$ convert -delay 100 -loop 0 *.png sent.gif

Animated gifs can be played with the animated tool from imagemagick to see how the delay is working.

Tamogotchi Hive

Fri, 27 Nov 2015 00:00:00 +0000

spritesmod has returned, this time making the matrix for tamogotchis

Happy Hackmas from 57North

Thu, 17 Dec 2015 00:00:00 +0000

We have two old Black and White CRT monitors in the hackerspace, they look really cool. I put together a Card and gif for the holidays:

I used a raspberry pi running FreeBSD for each monitor. The bottom monitor is running aafire which gives a nice fireplace effect. I did some big text in figlet for the message.

I also put together a paper card by using the gcard package in latex to put together some cards. This relies on double sided printing to get the message inside the cards. It was a bit of trouble(I had to trim the cards down), but the cards came out quite well for an hours work.

Card Outside

\documentclass[]{article}
\usepackage{gcard}
\usepackage{fontspec}

\setmainfont{Ubuntu Light}

\begin{document}

    \begin{frontcover}
        \centering
        \makebox[0pt]{\includegraphics[width = .5\paperwidth, height=13.9cm]{card.jpg}}
    \end{frontcover}

    \begin{insideright}
        \centering
        \makebox[0pt]{\includegraphics[width = .5\paperwidth, height=13.9cm]{card.jpg}}
    \end{insideright}

    \begin{insideleft}
    \end{insideleft}

    \begin{backcover}
    \end{backcover}

\end{document}

Card Inside

\documentclass[]{article}
\usepackage{gcard}
\usepackage{fontspec}

\setmainfont{Ubuntu Light}

\begin{document}
    \begin{frontcover}
        \centering{
            {\large{Happy Hacking}}

            {\large{From Everyone at 57North Hacklab}}
        }
    \end{frontcover}

    \begin{insideright}
        \centering{
            {\large{Happy Hacking}}

            {\large{From Everyone at 57North Hacklab}}
        }
    \end{insideright}

    \begin{insideleft}
    \end{insideleft}

    \begin{backcover}
    \end{backcover}

\end{document}

This generates two pdf files, I used pdfjoin to join them together into one file:

pdfjoin inside.pdf outside.pdf -o hackmascard.pdf

pdf is here

DSO138 Kit

Tue, 05 Jan 2016 00:00:00 +0000

Around the December holidays I received three sets of the jyetech lcd scope kit. This cheap kit (~£10) builds a small low frequency (1Msps) oscilloscope.

In all it took me about 2 hours to solder everything together, that includes me misplacing a resistor and a capacitor. I wish I had sorted the resistors with a multimeter that scales automatically before starting.

I am planning to build these kits into some audio projects later in the year, getting three of them was great luck. The kit was really straightforward to build and didn't take too long, there are serial logging features on the board as well. This kit could be built into a portable work bench without much thought.

The Thirty Second Chaos Communication Congress

Thu, 07 Jan 2016 00:00:00 +0000

I still can't describe CCC, you have to go.

I wrote a post about how to survive congress, but didn't publish it. It contained a list a little like this:

All of the talks are recorded, streamed and put online at media.ccc.de.
The self organised sessions are not recorded.
The most interesting things are happening at the assemblies.

These points hold true, my original suggestion because the talks are available after the fact there isn't much point sitting in the lectures. At 32c3 I didn't attend any of the talks, this was a mistake. I really regret not going to any talks.

Going to the talks gives you something to talk about with the people at CCC.

The self organised sessions I went to were great and hanging out with people at their assemblies and at the Scottish Consulate was great. If I had been in a lecture instead of at our table I definitely would have missed #toiletparty . But I think if I had gone to some of the talks early each day I would have gotten much more out of the event.

Next CCC I will head to the event with more of a plan. I don't think there is a right way to do congress, it is just too insane, but I will try to go to each one in a different way.

Making GIFS with FFMPEG

Tue, 12 Jan 2016 00:00:00 +0000

ffmpeg can now make gifs in a single step, no longer do you have to generate frames then pass them into ImageMagick. For most of the videos I have tried the initial gif from ffmpeg hasn't been very good.

I found a stackoverflow post that describes a two step process for generating gifs with ffmpeg that has great results. The first step generates a palette from the source video, then this palette is used as a filter when converting the video into a gif.

ffmpeg -i input.mov -vf fps=10,scale=320:-1:flags=lanczos,palettegen palette.png

Output the GIF using the palette:

ffmpeg  -i input.mov -i palette.png -filter_complex "fps=10,scale=320:-1:flags=lanczos[x];[x][1:v]paletteuse" output.gif

The improvement is more evident if you click and watch the full size gifs side by side. The stackoverflow post links a blog post with even more information on generating high quality gifs from video .

#!/bin/sh

if [ "$#" -ne 1 ]; then
    echo "usage: makegif filename.mp4"
    exit 1
fi

input=$1
filename="${input%.*}"

ffmpeg -y -i $input -vf fps=10,scale=320:-1:flags=lanczos,palettegen palette.png
ffmpeg -y -i $input -i palette.png -filter_complex "fps=10,scale=0:-1:flags=lanczos[x];[x][1:v]paletteuse" $filename.gif

Setting up xorg on the pi

Thu, 14 Jan 2016 00:00:00 +0000

The Raspberry Pi page on the FreeBSD Wiki links to a blogpost about setting up xorg on the Pi. That post was written back in 2013 and most of the information there seems to be out of date.

I set up X on a Pi at the end of December 2015, this information is up to date for r292413. pkg is now available on arm images so there is no need to build everything from ports, considering tools like tmux could take 6 hours to build on the pi itself this is a huge improvement. I installed the following packages to get X up and running on the Pi:

# pkg install xorg xf86-video-scfb i3

The Pi isn't able to auto detect the X configuration, I looked for a while for a config that would work. Eventually I dug the following one from a mailing list post. Place the following into /etc/xorg.conf:

Section "Device"
    Identifier "Generic FB"
    Driver "scfb"
EndSection

Section "Screen"
    Identifier "Screen"
    Device "Generic FB"
    Monitor "Monitor"
    SubSection "Display"
        Depth 16 #24 32
    EndSubsection
EndSection

With a minimal .xinitrc I was then able to start an X server with i3:

exec i3

Glitch Cards

Tue, 19 Jan 2016 00:00:00 +0000

Great news today about David Miranda's Case , but I can't help but feel down with the direction of the country. I can see British law being deemed incompatible with the ECHR being used to strengthen arguments against being a signatory to ECHR and part of the EU.

At home we have GCHQ dismantling secure communications at every turn. The low price of oil is causing a down turn up here and it doesn't look like there is bright future. Sometimes it is hard to stay positive when you let the real world seep in.

While I sit numbly at my desk I like to restlessly fumble with anything at hand. This week it has been this awesome mind bending deck of cards . I have already had many visitors complain my cards are misprinted and hurt their head, this real world glitch is doing well. The glitch_art sub reddit contains many more examples of images like these. None quite as satisfying as holding these 'broken' playing cards.

Unreasonable Podcast

Thu, 21 Jan 2016 00:00:00 +0000

Yakamo and I have started a podcast , the first episode was released yesterday. The website is still very simple and I don't think there is an rss feed setup yet. But, we have managed to put out the first episode and the second episode is lined up to be released on Monday.

Give it a listen if you like podcasts, any feed back should be directed to stuff@yakamo.org or /dev/null. Thats where I send your emails anyway.

Recording Audio on FreeBSD

Tue, 26 Jan 2016 00:00:00 +0000

For some reason I have been recording a lot of audio on my desktop recently. I also saw a conversation in irc about how to simply record audio from a microphone on FreeBSD.

I hoped I was going to find a super simple OpenBSD style solution to capturing samples, but I wasn't able to dig anything out. I did play with cat for a little while, but nothing useful came from it.

Audacity is the tool I have been using to record long sessions the most. Audacity is now probably the foss standard for doing audio editing/production and it has been really stable for me. On FreeBSD it has been rock solid so far if a little heavy weight.

ffmpeg is an audio and video swiss army knife and can be used to capture video from webcams and audio from capture devices. The only issue I have had with ffmpeg on FreeBSD is that lame support is not built into the default packages.

ffmpeg can be used to caputure audio from a source:

ffmpeg -f oss -i /dev/dsp -vn -ab 128k test.wav

Sox is the ultimate tool for handling audio, a long with the two front ends play and rec you can do most operations on an audio stream. Sox can built with codec support for a ton of formats. It is quite simple to use sox to convert different bit formats of sdr capture files with sox.

Rec can be used to caputure audio from a source:

rec -c 2 test.wav

How to do(bad) encryption in vim

Thu, 28 Jan 2016 00:00:00 +0000

Via the ReverseEngineering subreddit I found that vim's built in :X encryption mode can be pretty easily broken. I didn't know that vim had anything built in to encrypt files, in hindsight I should have expected some functionality.

Looking into the vim documentation on on Encryption shows that most of these methods aren't recommended for use. It also looks really easy to accidentally destroy a file using vim. If you do not decrypt the correctly you get a vim buffer filed with encrypted noise, if you save that buffer you destroy the original file.

I have been using vimwiki in a git repo since August last year. Vimwiki is a really simple markdown style wiki, the features are really limited. There is some markup, links and that is all. It has been filling all of my needs perfectly. I would like to be able to encrypt the wiki files so I could have a little more peace of mind, but with a little searching I haven't found anything that has the utility I need.

I could write something myself that worked well with both git and vimwiki, but I don't really want to subject my personal files to my own bugs. If you know of a solution to encrypting files in git repo or integrating with vimwiki that would be really helpful.

Command line notifications

Tue, 02 Feb 2016 00:00:00 +0000

JCS was interviewed on the latest episode of Garbage , he spoke about his app pushover . Pushover is an Android, iOS and mac app that works with a service backend. You can send notifications to pushover via simple api (you can just use curl) and the notifications are delivered to your devices.

This is awesome, I can set up pushover on my phone, a client on my build machine and get alerts when builds are complete. No more checking while a build finishes, instead I can get notifications directly on my pebble via pushover and the pebble app.

Looking through the app directory I found the command line tool ntfy . ntfy is really easy to set up and use, for pushover you need a simple .ntfy.json (with a real user_key) like:

{
    "backends": ["pushover"],
    "pushover": { "user_key": "fjaudfaufjkjdufdaskufdaskfjads"}
}

You can then send messages with ntfy or send the result of a command:

$ ntfy -t "Test" send "This is a test message"
$ ntfy done false

By default ntfy will set the message title to the user@host, but the -t flag can override this. ntfy supports other backend services and a 'linux' backend. I though the linux backend would tie into the same thing as notify-send, but that wasn't the case. I need to figure out how those tie in.

I have to run FreeBSD builds with root privileges, I didn't want to give a tool like ntfy root access. I wrote a small alias to send the result of the previous command.

alias buildres="if [ $? -eq 0 ]; then ntfy send 'Build passed'; else ntfy send 'Build failed'; fi"

Unreasonable Podcast Episode 0x02

Wed, 03 Feb 2016 00:00:00 +0000

So far yakamo and I have been consistently able to record our unreasonable podcast , the third episode, 0x02 came out on Monday.

Recording for the show has been okay so far, we have been using mumble to run our call while producing local recordings with audacity and a backup recording with mumble itself.

The back up recording has already been useful, I selected the wrong channel in audacity and didn't notice the flat waveform coming out of my mic. There has been a lot of trouble with the audio streams going out of sync making it bothersome to edit in audacity. I hope that is related to my own rate issues on my local recordings. I will see when I get to editing 0x03.

I think the show will probably get closer in structure to other shows as we go, I have already given in and you will hear intro noise in episode 0x02 . You will probably also hear us saying the name of the podcast a lot, that should remind people what they are listening to.

Fixing rate issues with audio on FreeBSD

Thu, 04 Feb 2016 00:00:00 +0000

It has been mentioned by a friend that my voice in the recent unreasonable podcast episodes is much higher than it is in reality. Of course for the first few episodes he just said 'your audio is fucked' which didn't help me resolve the issue at all.

With the detail that pitch was off I knew where to start looking, the pitch issue was present on both the audacity recording and the mumble back up. The audio rate for the microphone and audacity where both the same, 44.1kHz.

The last thing to check was the audio sub system on FreeBSD. I read the snd man page and it pointed me to a few sysctl knobs that I might be able to tweak. I also checked the man page for usb audio and found this little notice in the bugs section:

BUGS The PCM framework in FreeBSD only supports synchronous device detach. That means all mixer and DSP character devices belonging to a given USB audio device must be closed when receiving an error on a DSP read, a DSP write or a DSP IOCTL request. Else the USB audio driver will wait for this to happen, preventing enumeration of new devices on the parenting USB controller.

 Some USB audio devices might refuse to work properly unless the sample
 rate is configured the same for both recording and playback, even if only
 simplex is used.  See the dev.pcm.%d.[play|rec].vchanrate sysctls.

 The PCM framework in FreeBSD currently doesn't support the full set of
 USB audio mixer controls.  Some mixer controls are only available as
 dev.pcm.%d.mixer sysctls.

vchanrate is a per device sample rate that can be controlled by a sysctl, toggling the value showed me the problem. With the rate at the correct 44100 my deep voice poured out of my microphone and into a file.

$ sudo sysctl dev.pcm.4.rec.vchanrate=44100
dev.pcm.4.rec.vchanrate: 48000 -> 44100

Unreasonable Podcast Episode 0x03

Tue, 09 Feb 2016 00:00:00 +0000

So yakamo and I have been managed to do it again, the most recent episode of the unreasonable podcast , episode 0x03 appeared yesterday.

This episode should now be missing large chunks of silence and features a much deeper sexier voice for me. Turns out there were some config issues with my microphone. There were also some observation issues, I didn't actually listen to the whole show or check the waveform before doing the renders.

Should all be fixed now and it will sound silky smooth.

Or something.

More Unreasonable Podcast Stuff

Tue, 23 Feb 2016 00:00:00 +0000

I was hit pretty hard the last few weeks with the standard issue winter cold. I was quite surprised to see how little energy I had for working on anything after work. I certainly see the value in taking time of work even if it just means more energy for side projects.

Yakamo and I have continued to push out our podcast , with another episode appearing today. Our audio is getting better though you won't hear it in the most recent show as I did all of the editing with my backup audio.

Editing the backup audio showed some of the issues with mumbles recorder. It was a lot of trouble to keep my levels balanced, there seems to be gain correction on the mumble feed so my starting audio is super loud and it just tampers off.

Finding package build options

Thu, 25 Feb 2016 00:00:00 +0000

I am making my email setup better. I have moved hosts to Fastmail (referrer link) and I am setting up mutt to work with folders correctly. There is a patch for mutt called mutt-sidebar which gives a list of folders in a side pane in mutt, much like the interface would be on the web.

Looking at Freshports it looks like the sidebar patch is part of the FreeBSD port, but it isn't setup by default. I wanted to check this and looked for the option in pkg to list the build options. There is the pkg query command that will show information about installed packages, but it is a little mental to use.

I found how to use pkg query to list the build options for a pkg in the wiki .

$ pkg query "%n is compiled with option %Ok set to %Ov" mutt
mutt is compiled with option ASPELL set to off
mutt is compiled with option COMPRESSED_FOLDERS set to on
mutt is compiled with option DEBUG set to off
...
mutt is compiled with option SIDEBAR_PATCH set to off
...

That tells me that I new to build mutt from the port and turn sidebar patch on.

Ubuntu Touch, Nowhere near ready yet

Thu, 24 Mar 2016 00:00:00 +0000

I dropped my android phone at congress, I was trying to figure out which direction to walk in the compass was spinning wildly being absolutely no help. I slide the phone into my pocket missed and the screen cracked on contact with the ground. The phone had a terminal diagnosis then, but I soldered on for as long as I could.

Eventually the nexus 5 dropped the cellular network and I travelled 40 minutes to a vet when I didn't need to. The next day I jumped on to the bq site and ordered my ubuntu edition E5.

The BQ sales process was just terrible, it didn't help that my email provider decided I didn't need to receive email that weekend. The BQ site is really badly laid out, confusing and proactively difficult to login to and view your orders. They don't offer tracking number, so I don't know where the extra €10 'international' shipping went.

The ubuntu touch os out of the box is annoying as any other smart phone, you are run through the standard dialog that asks mostly pointless questions. The version of ubuntu touch that ships on the E5 is "really old" (with really old being may 2015, 9 months prior). I was unable to install anything from the ubuntu app store (well I installed a hex colour picker, but that doesn't count).

I tried to check for updates, but the phone was happy believing it was all up to date. I searched for a while, but any search term with 'ubuntu touch' results in ubuntu users having issues with their touch screen laptops. Eventually I fell into the #ubuntu-touch irc channel and asked.

And I waited...three hours later on my third time asking someone suggested I try updating. There isn't wifi at work, I wondered if the ubuntu touch os is so broken it won't acknowledge user settings and will only look for updates on wifi. I tried at the hackerspace later that night and I managed to update to OTA9.1. This is the latest release version.

I don't know how this operating system is for. I have been using free unix desktops for a decade, I can't see anything in ubuntu touch that I want. The default system shows a collections of scopes, scopes are an advertisers dream.

There are default scopes like, weather, music, video, news and the scopes are populated by scope apps that create a feed for each theme. There are loads of different input services for them as well, so if you use facebook, instagram, flicker, 500px you will have a nice full feed. If you use one, or none of them you won't have anything. I saw the scopes and thought of windows crap wear.

I turned off all of the scopes as soon as I could.

Here is a short list of issues I encounter:

Out of the box

No updates were shown as available on 3G
Updates became available to download on wifi
Unable to download apps from the ubuntu app store
Unable to run any software other than browser
No way to find out the phone was an entire naming scheme behind without wifi
The UK (and English) support site doesn't mention the Ubuntu Editions at all. The Austrian one does.

General System issues

All advice assumes you are running ubuntu on the desktop
Phones always seems to be low on memory
You are stuck looking at the apps scope
No way to have a phone wallpaper, just apps
If you turn off all of the scopes there is no way to add them back in
No way to import contacts into the phonebook, yes on a phone
There isn't a calendar
There is no count down alarm timer (make noise in 20 minutes)
Bluetooth volume is massively reduced (unable to hear anything out side)
Removing wired headphones doesn't pause playback
Volume transfers when plugging/unplugging head phones. Play music full volume on speakers, plug in headphones, go deaf.
Headphone buttons don't work
It can take still running paged out apps 10 seconds to become active. This was seen on the alarm clock app
No security
No encrypted storage
No way to hide notification bodies on lock screen
Default lock screen leaks data (how many calls, how many messages)
UI Bugs everywhere
There is gravity scrolling sometimes, I can't tell you when though
The alarm time picker has gravity scroll when you don't want it and doesn't when you want it.
When you switch to paged out apps, sometimes they will be out of focus. This is your only idicator they app isn't actually running. Don't worry it will restart entirely within 10 seconds
There edge swipe gestures, but they only seem to work at the most annoying times
The left swipe menu is useless when the default screen is a page of apps
In the browser you access multiple tabs with a bottom up gesture, scroll in landscape mode is nearly impossible.

General App usage

The apps are either written by Canonical or some random person, these apps wants my login credentials
There are loads of apps available, as long as you want a news app from a minor regional newspaper
Nobody wants scopes
Built in apps (twitter, youtube, etc,) are just web views and they are terrible.
Can't tweet photos from twitter app
Can't send photos on telegram app
The Canonical provided apps have led to no good alternatives for apps (youtube, twitter, etc)

Specific Apps

Podbird looses progress in podcasts
podbird redownloads podcasts on new wifi networks
podbird doesn't have any play back indicator for podcasts other than the currently playing
podbird can't handle many rss feed urls
podbird can't bulk import podcast urls
Ureadit can't show the first item in the list in portrait
ureadit has no touch area for gifs
No way to refresh feed without a long scroll up
On out of memory feed/message thread is reset or cleared away

Other people agree , I have installed android and life is better.

Light terminal theme

Thu, 14 Apr 2016 00:00:00 +0000

For about as along as I have been using terminals I have had them set to a dark theme, for a while at the beginning I had my terminal set up to be green text on a black background. I might have been the l33test mutherfucker around.

! Zenburn theme
! black + red
urxvt*color0:      #101010
urxvt*color1:      #705050

! green + yellow
urxvt*color2:      #60b48a
urxvt*color3:      #f0dfaf

! blue + purple
urxvt*color4:      #506070
urxvt*color5:      #dc8cc3

! cyan + white
urxvt*color6:      #8cd0d3
urxvt*color7:      #dcdccc

! bright-black + bright-red
urxvt*color8:      #000000
urxvt*color9:      #dca3a3

! bright-green + bright-yellow
urxvt*color10:     #c3bf9f
urxvt*color11:     #f0dfaf

! bright-blue + bright-purple
urxvt*color12:     #94bff3
urxvt*color13:     #ec93d3

! bright-cyan + bright-white
urxvt*color14:     #93e0e3
urxvt*color15:     #ffffff

I moved to Zenburn from the unreasonably popular solarized at some point last year. When I moved I went through a few themes trying things out. I found that .Xresources supports cpp macros making it easy to swap themes.

!.Xresources 
urxvt*scrollBar:      false
urxvt*matcher.button: 1
urxvt.transparent:    false
urxvt*allow_bold:     true

Xft*dpi:              96
Xft*antialias:        true
Xft*hinting:          full

URxvt*geometry:       85x16
URxvt*fading:         0
URxvt*tintColor:      #ffffff
URxvt*shading:        0
URxvt*inheritPixmap:  False

#include ".papercolour"

URxvt.urlLauncher:    firefox
URxvt.matcher.button: 1

URxvt*font: xft:Source Code Pro:size=8

I tried to work outside quite a few times last year. Dark themes for terminals are really hard to see in bright sunlight. For some reason yeahconsole doesn't like the way I include themes and stayed with a default light theme. The light theme is really easy to see in sunlight. The contrast in readability made me really question using a dark theme at all.

At camp last year most of the daylight hours I spent on my laptop were inside our super tent, direct sunlight made my screen hard to read and the sun seems to melt my skin. My dark theme was okay to read in the tent, but it was no good outside, with the sun melting I just hid away until it was night time.

The sun being bad, I am grateful I live in Scotland, we don't have to put up with the sun very often(this is a joke, it is always fucking sunny, I want a refund).

When the sun isn't around I do quite like to sit in darkened rooms when I am hacking. If you too are a vampire you might have noticing the eyeball explosion that happens when you switch from your friendly terminal to the light explosion that is a web browser. Pretty much every web page has a light theme, I actually dislike dark themed web pages, I always think the designer is being up front with how much of a douchebag they are.

Anyway, this is a lot of words for "I have moved to a light theme"

! PaperColour Theme

URxvt.foreground: #4D4D4C
URxvt.background: #EEEEEE

! black
URxvt.color0: #EDEDED
URxvt.color8: #969694

! red
URxvt.color1: #D7005F
URxvt.color9: #D7005F

! green
URxvt.color2: #718C00
URxvt.color10: #718C00

! yellow / orange
URxvt.color3: #D75F00
URxvt.color11: #D75F00

! blue

URxvt.color4: #4271AE
URxvt.color12: #4271AE

! magenta
URxvt.color5: #8959A8
URxvt.color13: #8959A8

! cyan
URxvt.color6: #3E999F
URxvt.color14: #3E999F

! white
URxvt.color7: #4D4D4C
URxvt.color15: #F5F5F5

What do those XXX blocks mean

Thu, 12 May 2016 00:00:00 +0000

From Stevens TCP/IP Illustrated Vol 2 :

We will see the comment /* XXX */ throughout Net/3. It is a warning to the
reader that the code is obscure, contains nonobvious side effects, or is
quick solution to a more difficult problem.

The second volume of that series might be one of the best networking books ever written. Not because it is a good tome to learn networking from, it is instead a guide into the heart of a real system. It is close enough today to use as a starting point for finding out where things are and a step to finding out why they are.

It is where I go when I want to find out how my current machines get bytes from an application to packets on the wire.

Personal Area WiFI networks

Sat, 14 May 2016 00:00:00 +0000

The first step to getting my devices working for me is to set up a consistent network for them to use. To do this I am going to use a small pocket sized router that can be run from a usb battery to act as a hot spot for my devices, but also as a bridge to an internet connected wifi network.

The network I want to setup looks something like this:

                                                   +
  +---------+                                      |
  |  phone  <<-------+                             |
  +---------+        |                             |
                     |                             |
  +---------+        |  DHCP     PANWIFI           | DHCP    PUBLIC WIFI
  |  laptop <<-------+      +-----------------+    |    +-----------------+
  +---------+        |      |                 |    |    |                 |
                     +---->>>   OpenWRT       <<<----->>>                 +---->  INTERNET
  +---------+        |      |                 |    |    |                 |
  |  camera <<-------+      +-----------------+    |    +-----------------+
  +---------+        |             192.168.x.x/xx  |
                     |10.10.10.0/24                |
  +---------+        |                             |
  |  pda    <<-------+                             |
  +---------+                                      |
                                                   |
        Personal area wifi network                 |  Upstream wifi network
+--------------------------------------------------+---------------------------------------+

I struggled to find a network configuration like this in the OpenWRT wiki. I wondered for a while if it was because the network was an impossible (seemed unlikely) or if it was so obvious to not be worth documenting.

Eventually google turned up a bitbucket page with a config that worked perfectly.

I need to find a method which makes it straight forward to configure a new outgoing network. I think at the moment I am going to have to edit the wifi config files to make any changes. On the road that will be less than ideal.

Writing this takes a little too much effort

Mon, 26 Sep 2016 00:00:00 +0000

Writing blog posts and getting them out takes far too much effort. With a streamlined publishing system the author still has to manage to write something down.

I do not have a streamlined publishing system. Instead the tools I use sit in a balance between the ideal thing I want and the hacked together scripts I have. It has been 4 months since my last post, so you can join me on a refresher.

The web side of the software is written in nodejs using express (and python with flask, but that isn't finished). The node program starts up and parses in a configured directory containing the blogposts.

$ cd blogposts

$ git pull
Already up-to-date.

The blog posts live in git and are written in markdown. Images for the posts are kept in the images subdir. The blog posts themselves live in year folders (2014,2014, etc). The year folders are provided as configuration to the node web process as well, which implies there is work to do when the calendar flips around.

$ ls
2013   2014   2015    2016    drafts    images    newid.sh    old

$ ls 2016 
32c3.md                 glitchcards.md          unreasonable0x02.md
acuratefbsdaudio.md     lighttheme.md           unreasonable0x03.md
dso138kit.md            more-unreasonable.md    update.md
fbsdpixorg.md           notifications.md        vimcrypt.md
ffmpeggif.md            pan.md                  xxx.md
freebsdbuildflags.md    ubuntu-touch.md
freebsdrecaudio.md      unreasonable.md

blogposts have an id, which is used to sort and sequence them and is used for the post url. It was needed in earlier pieces of software I wrote and I would like it to go away. Until I move to something else I have a helper script to tell me what the next id is.

$ sh ./newid
last post id: 0089
next post id: 0090

blogposts use an email style header, each line is a key value pair separated by the first colon on the line. The header block is terminated with two newlines '\n\n'. I can type out the header, but normally I copy it from a blogpost. That's the sort of lazy person I am.

$ copy 2016/somepost.md to 2016/newpost.md
$ vim 2016/newpost.md

Title: Some post
Tags: meta
Date: 2016-01-01
Preview: Some post
Permalink: 0001

Hurr durr I am a blogpost

I am totally inciteful and full of useful information, like how nat punch
through works and the secret to everlasting life.

Now we have to edit all of the fields in the header, and content for the body of the blogpost. This is a great time to add the correct post id value we got way up top.

Title: Writing this takes a little too much effort
Tags: blog
Date: 2016-09-26
Preview: Writing this takes a little too much effort 
Permalink: 0090

Writing blogposts takes far too much effort...

Okay, we have now written the blogpost, maybe even spell checked, we can upload it to the web server.

$ git add 2016/newpost.md
$ git commit -m "blogpost"
$ git push

On the remote web server we need to pull from the master blogposts branch to get the new article we wrote.

$ ssh webserver
$ cd sites/blogposts
$ git pull

Now we have the updates we have to restart the node process. There is code to reload dynamically, but I could never get nodejs to behave here. I would like to use kqueue to watch posts dir, but when I last looked this wasn't supported on the platform.

$ cd ../register
$ forever restart server.js

Phew, there we go.

We are serving up the new blogpost from the site. This seems like a lot of work, but I think post of the component stages would be required with a static site generator.

I want to write some tools to help with schduling posts. At the moment I can write a post for future release, but I have to specify the date for release.

Reading: The Puzzle Palace, 802.11 Wireless Networks 2nd Edition.

Ex Machina

Tue, 27 Sep 2016 00:00:00 +0000

I think I really enjoyed Ex Machine , it has a great mixture of near scifi and technology. There is enough mystery and conspiracy in the film to keep me engaged, I am glad my world doesn't have so much intrigue. If it did I would probably be in some Billionaires dungeon for following the wrong lead.

The Ex Machina Soundtrack is even better than the film. It reminds me of the ambient music that plays in GTAV when you wander around with the radio off. A podcast with similar drones and loops would be an excellent thing to add to my work music mix.

Reading: The Puzzle Palace, 802.11 Wireless Networks 2nd Edition, MOONCOP!

Cybersofa

Wed, 28 Sep 2016 00:00:00 +0000

Yesterday I wrote about the Ex Machina soundtrack, but linked to an hour long loop of one of its tracks. Whoops. The whole soundtrack is equally great, go find it. Similar stuff on youtube lead to 9980 by CONNECT.OHM .

The Science Fiction podcast magazine I listed to, StarShipSofa has had some great CyberPunk stories recently.

Humans are going to become augmented, this is an inevitablity, we won't be able to resist making our selves better by merging computers and machinery into our body. "Must Supply Own Workboots" considers what happens when our jobs rely on expensive augmentations, but the augmentations become out of date.
“And You Shall Know Her By The Trail of Dead” is a Gibsonesque Cyber Cowboys fighting with the mob story. Really well timed with the article about the excellent CyberdeckC64

Reading: The Puzzle Palace, 802.11 Wireless Networks 2nd Edition, RFC6347!

Denial of Toasters

Thu, 29 Sep 2016 00:00:00 +0000

The news on this weeks Risky Business Podcast mentioned the record breaking DDOS against Krebs . 665 Gigabits of traffic per second is a lot of traffic, but that is probably only the start of such massive attacks.

While wondering how these attacks manifest an article about the slowloris attack popped up. This is a different sort of denial of service to the network traffic sent to Krebs and one that should be rather easy to mitigate against at the protocol layer.

The Krebs attack is the first I am aware of with a large IoT component. I think we have all been waiting for the hordes of vulnerable devices to appear in abuse logs. Maybe we can move to ipv6 and leave the Internet of Shit on a blackholed v4 Internet.

Reading: The Puzzle Palace, 802.11 Wireless Networks 2nd Edition, Packet Captures

Software Updates

Fri, 30 Sep 2016 00:00:00 +0000

Listening to this weeks ATP on the bus, they speak about the latest Mach OS release SomthingCali. It reminded me how little I really care for software updates. Of course I want things to get faster, more secure and less buggy so I have to endure updates. Most updates don't just bring clear improvements instead they bring feature updates.

I write software for fun and for a living and for a while I even wrote products that people used. I even provided training for our users on product updates. I saw first hand how annoying changes can be.

Most of the changes we delivered were customer driven (in fact, they were all paid for by individual customers). When we trained a customers users on the new software there were normally a whole bunch of changes to off path functionality that someone else had asked for.

I can't remember anyone ever being happy with changes to their workflow .

They were happy that bugs had been fixed and UI had gotten a little cleaner, they loved that the software was better on the crappy machine IT or we supplied them. But they didn't want change for changes sake.

I have been using Puzzle Alarm Clock to make me get up. It is great it can make you solve puzzles, quizzes, or it can use the NFC reader or camera to scan a QR code to turn the alarm off. Puzzle Alarm Clock updated this week. The UI was improved or something, all I can tell is that it is white instead of black now. But they also removed features, making the app much worse.

Reading: TLE Files

Satellite Pirates

Sat, 01 Oct 2016 00:00:00 +0000

Due to a Chatham House report on the latest dangers of Satellite hacking uhf_satcom was on this weeks risky business talking about Satellite pirates and exploit possibilities on the birds.

Not the Satellite Pirates of the 90s trying to access free TV and not arrgg Pirates out at sea(though maybe), but people taking advantage of the great accessible repeater in the sky.

A terrestrial repeater takes in a signal on an input frequency and rebroadcasts it on an output frequency. The repeater normally has better antennas system and is situation in a physical position to give the best area coverage.

A satellite repeater does the same thing, from its vantage point in space it can cover a much larger area. There are amateur radio satellites that provide this functionality, but from low earth orbit.

The pirates on Risky Business are probably using a satellite in geostationary orbit and taking advantage of it being a dumb pipe pointing back at earth.

Reading: TLE Files

This article will still be here when you're done, and blogs are dumb.

Sun, 02 Oct 2016 00:00:00 +0000

The excellent newsbeuter says I have 80 rss feeds that I pay occasional attention to. There are habit sites I visit like reddit and hackernews, but I fall back on the rss feeds when want to focus and read.

I put the rss feeds from peoples blogs in my reader, normally when I read an awesome article via HN or reddit. People don't normally post more than 3 times a month. This means there isn't so much I can't read it, but just enough that I can process it when I want to.

" This article will still be here when you're done, and blogs are dumb. "

It is Sunday, so that makes seven days of writing .

Reading: Butter from my Feed Reader

First SO-50 Pass

Sun, 02 Oct 2016 00:00:00 +0000

I managed my first SO-50 pass yesterday. Using a tape measure 70cm yagi I made last year and my baofeng I was able to hear chatter on the repeater for about 30 seconds.

As mediocre as it is I am really happy with success on my first try, I did attempt to listen to another pass of SO-50, but only heard a two second chirp. I used gpredict on my stream 7 for satellite tracking, and a cheap compass app on android to verify which way the building pointed.

Next I am going to try using an sdr for the downlink capture. I am hoping it will be a little easier to get the yagi pointed the correct way and give me a chance of finding the signal mid pass.

Malware

Mon, 03 Oct 2016 00:00:00 +0000

There are reports of Malware in the PS Vita Piracy scene. When you have to pursue shady enterprises to use the hardware you own this is always the risk you take. Consoles have the coolest security hardware, but it is aimed at stopping piracy rather than protecting users.

The Grey area jailbreak tools live in make it really hard for users to find the real tools. Instead the end up with malware.

Here is 50 minutes on why this was going to happen.

Reading: The Puzzle Palace

Hacking Games

Mon, 03 Oct 2016 00:00:00 +0000

I read this awesome review of hackmud , it made me think of other games about hacking or games that involve actual hacking.

I have only played Uplink and TIS-100, I have heard the others are pretty great. You should play them and tell me how they are.

WPA IS BROKEN!!!

Tue, 04 Oct 2016 00:00:00 +0000

WPA IS BROKEN!!!1

Okay it isn't, that attack is awesome, but it is a social one rather than a break of WPA. I bet it would work in a load of environments, I would be surprised if pentesters didn't already have it in their toolkits.

Really the OS should be doing much more to protect users from this class of attacks. WPA written today would not be vulnerable to this class of attack at all.

Reading: The Puzzle Palace, 802.11 Wireless Networks 2nd Edition, Packet Captures

About Electron Gnomes

Wed, 05 Oct 2016 00:00:00 +0000

As an aside form talking about the Electron Gnomes on the latest Embedded FM podcast Elecia and Christopher implored us to talk to people about their awesome podcast to everyone we know.

So, go and listen to the Embedded FM Podcast featuring excellent interviews , professional advice and something about Electron Gnomes.

Reading: Little Brother

5th post

Metadata

Thu, 06 Oct 2016 00:00:00 +0000

Recently StarShipSofa has been delivering podcast files to me that contain 3rd party ads. It is their hosting provider that is inserting the ads, but both times I have been aksed if this my client is to blame.

I am certain PocketCasts would never do this.

Maybe there is something in the file that would indicate who did the encoding?

play (from the sox package)

$ play starshipsofa-454-ads.mp3:

starshipsofa-454-ads.mp3:

File Size: 33.7M     Bit Rate: 64.0k
  Encoding: MPEG audio    
  Channels: 1 @ 16-bit   
Samplerate: 44100Hz      Album: StarShipSofa
Replaygain: off         Artist: StarShipSofa
  Duration: 01:10:10.78  Title: StarShipSofa No 454 Alex Shvartsman and Stephen S. Power

In:0.05% 00:00:02.04 [01:10:08.74] Out:90.1k [  -===|===-  ]        Clip:0

Just the file name and year, lets try ffprobe from the ffmpeg tools:

ffprobe

$ ffprobe starshipsofa-454-ads.mp3:

[mp3 @ 0x809691000] Skipping 0 bytes of junk at 159.
[mp3 @ 0x809691000] Estimating duration from bitrate, this may be inaccurate
Input #0, mp3, from 'starshipsofa-454-ads.mp3':
  Metadata:
    title           : StarShipSofa No 454 Alex Shvartsman and Stephen S. Power
    album           : StarShipSofa
    artist          : StarShipSofa
    date            : 2016
  Duration: 01:10:10.39, start: 0.000000, bitrate: 64 kb/s
    Stream #0:0: Audio: mp3, 44100 Hz, mono, s16p, 64 kb/s

Nothing more there, a google says there is something called mp3info:

mp3info

$ mp3info starshipsofa-454-ads.mp3:

starshipsofa-454-ads.mp3 does not have an ID3 1.x tag.

Well that was no good at all.

I don't have a ton of time to find the mp3 metadata might be, none of these tools show anything. I guess that means I can be happy I am not leaking info when I encode an mp3, or I can't find it with normal tools.

Reading: Little Brother

Porting a WiFi Driver

Fri, 07 Oct 2016 00:00:00 +0000

To win this bet I have with Ed I need a WiFi adapter that can do 80211n in the 5GHz band. There aren't a lot of these around and n in 2.4GHz band makes it hard to find adapters with the right support.

I got pair of AC600 generic adapters on ebay for about a tenner, a quick look showed promising Linux support. This indicated I could use one for the bet without too much hassle.

I got a second so I could work on a wireless driver for FreeBSD, what else am I to do with my time?

The adapter is a MediaTek MT7610U device, there is a whole load of information about it on Wikidevi and there are a family of forks of the vendor code on github.

Wikidevi says the MT7610U is similar to the RT28xx series , which are supported by run in FreeBSD. I started last night by taking the run driver, getting it to build as a module, then turning everything off apart from probe, attach and detach.

This is the first time I have tried to port a driver, to help I collated everything I could find written about doing it.

There is straight up FreeBSD stuff:

There are load of little posts where people have ported drivers from FreeBSD to somewhere else:

And there are a load of articles about building wifi drivers for android, these are worth read, but they are worth pointing out:

Reading: Little Brother

I had an argument with some Germans about the pronunciation of WiFi, apparently it is WeeFii using the sounds of wireless and fidelity. They also pronounced HiFi incorrectly, English is a strange language.

Cold Brew

Sat, 08 Oct 2016 00:00:00 +0000

My Cold Brew Recipe requires:

128g of Coarse ground coffee (I guess 125g is okay, if you aren't cool)
1L Vessel (I use a nalgene)
1L of potable water
Fridge
v60
Jug

Method:

Put the ground coffee in the vessel.
Fill the vessel with cold water
Place vessel in fridge

I use tap water because I live in a place with excellent drhinking water. If that isn't the case for you, you will have to figure something else out. Make sure the ground is well soaked, it will swell. I give it a good shake then add a little more water to make sure the nalgene is good and full.

After about a day take the nalgene out of the fridge.

Pour the coffee/concentrate blend into the jug.
Clean the nalgene.
using the v60 filter the concentrate back into the nalgene.

I normally end up with about 700ml of concentrated coffee. I mix it with boiling water to drink, about 120ml of concentrate to 200ml.

Reading: Little Brother

Coding Sunday

Sun, 09 Oct 2016 00:00:00 +0000

The tortoise needs an improved heating setup, now have a 'night time' buld that just puts out heat. Before I change anything I want to have numbers so I can try and quantify the change.

I knocked up a micropython script and ran it on a nodemcu board with a couple of dht11's. It looks like this:

def temperatureclient(sensors,addr="255.255.255.255"):
    print("       sending to: {} {} every {} seconds"
        .format(addr , PORT, DELAY))

    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    while True:
        pkt = takereading(sensors)
        sock.sendto(pkt, (addr, PORT))

        time.sleep(DELAY)

def takereading(sensors):
    readings = []
    for sensor in sensors:
        sensor.measure()

        reading = {}

        reading["sensor"] = str(sensor.pin)
        reading["pin"] = str(sensor.pin)
        reading["temp"] = sensor.temperature()
        reading["humditiy"] = sensor.humidity()

        readings.append(reading)
        print(reading)
    return json.dumps(readings)

It doesn't have to live for long, just a day or two.

The always on machine on my network doesn't seem to have anything useful installed and without internet at home that wasn't going to be a simple fix. Instead I used tcpdump to capture the json packets.

Tcpdump works really well in this situation, the micopython board doesn't have a RTC, but the pcap from tcpdump will have acurate timestamps for each field. I did something like:

$ tcpdump -w tempreadings.pcap udp and port 6969

Later I can process this out with a shell script or scapy or something.

It is Sunday, so that makes seven days of writing .

Reading: Butter from my Feed Reader

Network Analysis

Mon, 10 Oct 2016 00:00:00 +0000

I read this excellent post by Simone Margaritelli on hacking a network connected coffee machine. Simone reverse engineered the Android app that controls the coffee machine and wrote a command line tool for getting the machine going.

Simone took a completely different angle to solving the problem than I would. Being a network person I would have gone straight to tcpdump, grabbed some traces from the app/coffee machine and worked from that.

Instead Simone used a tool to dump a disassembly of the Android apk. I haven't done that before, I don't think it would be my first thought when I had to take something apart. From this post I think I might give it a shot on the local bus app.

The coffee machine looks awesome, you might not want an internet connected coffee machine, but I think it is an awesome idea. Coffee is a great reward for solving a problem, the machine could automate teaching people how to reverse network protocols.

Reading: Little Brother

Flashing AI-Think NodeMCU Boards

Mon, 10 Oct 2016 00:00:00 +0000

I ordered a handful of the cheapest nodemcu boards I could find from ebay. A couple of weeks later I got a nodemcu 'like' board from a company callsed AI-THINKER . The boards following instructions written on the back of them:

1. Install CH340G driver.
2. Use 9600bps baud rate.
3. Connect to WiFi.

I tried playing with two of the boards, powering them up and searching for wifi networks showed a network with a name like:

AI-THINKER_238810
AI-THINKER_23A9BF

Connecting to the wifi was fine, but I didn't really know what they expected me to do. nmap'ing the device has no results and an hour googling didn't really show up anything. Connecting over serial resulted in some noise then nothing.

I was going to flash micropython anyway, so lets do that.

Flash micropython

Connecting to the nodemcu board over serial spits out some gibberish no matter the baud rate I pick.

$ sudo cu -l /dev/ttyU1 -s 76800
Connected
Sd3²ì{£P:ýCê
ets Jan  8 2013,rst cause:2, boot mode:(3,6)

load 0x40100000, len 1856, room 16 
tail 0
chksum 0x63
load 0x3ffe8000, len 776, room 8 
tail 0
chksum 0x02
load 0x3ffe8310, len 552, room 8 
tail 0
chksum 0x79
csum 0x79

2nd boot version : 1.5
  SPI Speed      : 40MHz
  SPI Mode       : DIO
  SPI Flash Size & Map: 8Mbit(512KB+512KB)
jump to run user1 @ 1000

êñ+Pr-r+§(r

SD«¢hJëÙ-$xùÊkPx\)§k ¢ÀjtNü

Some time with a scope reveals the board is starting up at one rate then switching to another. The rate switch means the esptool is unable to do automatic baud rate detection.

With that we can flash the boards:

erase the flash
esptool.py --port /dev/tty.wchusbserial1420 erase_flash

flash the image
esptool.py --port /dev/tty.wchusbserial1420 --baud 76800 write_flash --flash_size=8m 0 esp8266-2016-05-03-v1.8.bin

reset the board

cu -l /dev/tty.wchusbserial1420 -s 115200
MicroPython v1.8.2 on 2016-08-05; ESP Module with ESP8266
Type "help()" for more information.
>>>

13cm Simplex

Mon, 10 Oct 2016 00:00:00 +0000

Fresh of great weekend at the RSGBConvention my good friend hibby was talking about doing point to point line of sight lines with 400MHz and up. He is super eager to do giant 50Km links and was suggesting hills to climb at the weekend.

I thought maybe we could try something a little easier to debug when it doesn't work. We settled to try point to point between my house and something the other side of the valley.

We did some local test and I was able to hear clear audio out to about 500m. At that distance we ran out of road to walk down. I can see the Newhills Parish Church from a rear window of my house, it is probably a little under a mile away line of sight.

While Hibby headed out there and I set up the yagi, we used 70cm as a return channel as the portapack can't transmit with the current firmware.

We ended up using the rad1o badge from cccamp last year as a 2.4GHz transmitter and a wifi yagi I had lying around. We played with settings for a while and eventually figured out the right combination of settings to do WFM voice!

Next we need to find a pair of points with los that are far enough apart to test range.

Are you awake?

Tue, 11 Oct 2016 00:00:00 +0000

It said

Are you awake? Read a blog!

And I was awake, so I opened the blog. It was about baseball.

Instead I read an actual blog post , another one about the RSGB convention. Then I looked at this bytebeat album . Fuck baseball.

Reading: Litte Brother, Transmetropolitan

Parsing data from pcaps

Tue, 11 Oct 2016 00:00:00 +0000

On Sunday I set up some quick and dirty temperature monitoring. At that point I didn't have any server code lying around to recieve the readings from the sensors. I set up tcpdump on a fileserver to capture the packets, tcpdump has the benefit of loggin a timestamp with each packet helping me get around limitations of the nodemcu hardware.

A day later I have to try and process the pcap files.

$ tcpdump -A -r temperaturevalues.pcap-1 | head -n 4
reading from file temperaturevalues.pcap-1, link-type EN10MB (Ethernet)
12:20:55.766057 IP 10.4.4.160.4097 > 10.4.4.187.acmsoda: UDP, length 134
E...........
...
......9....[{"humditiy": 47, "temp": 23, "pin": "Pin(4)", "sensor": "Pin(4)"}, {"humditiy": 45, "temp": 21, "pin": "Pin(5)", "sensor": "Pin(5)"}]

The -A flag for tcpdump will show me the packet payload as ascii, I was pushing json from the server so this is rather easy to see. I could use some shell magic to pull this out, but I wanted to play with scapy.

Scapy is a python library for dealing with packets, it does everything tcpdump will with packet injection to boot. Scapy will happily take in the pcap files.

#!/usr/bin/env python

from scapy.all import rdpcap
import json

if __name__ == "__main__":
    pcapfiles = [ "temperaturevalues.pcap-1", "temperaturevalues.pcap-2"]

    readings = []

    for files in pcapfiles:
        pkts = rdpcap(files)

        for p in pkts:
            time = p.time
            readings = json.loads(p.load)
            print("%s,%s,%s,%s,%s" % 
                (time, 
                readings[0]["sensor"],readings[0]["temp"],readings[0]["humidity"],
                readings[1]["sensor"],readings[1]["temp"],readings[1]["humidity"],
                )
            )

Running

$ python process.py  > readings.csv

Gives me a csv file with the temperature and humidity data from the sensors. Feeding this to gnuplot with something like the below results in a nice(albeit noisy) plot of the temperature from the two sensors.

set datafile sep ','
set timefmt "%s"
set format x "%m/%d/%Y %H:%M:%S"
set xdata time

set terminal png size 3000,500
set output 'data.png'

plot 'temperaturedata.csv' using 1:3 with lines, 'temperaturedata.csv' using 1:6 with lines

Triangles are my favourite shape

Wed, 12 Oct 2016 00:00:00 +0000

Damn, today has been a hard fucking start up sequence ( slow starts punk brother ). TCP jokes are the best, if you don't get them we can keep retrying until you do.

This tweet by dwf

Possibly the most unbelievable thing about Star Trek is how different alien
civilizations maintain cross-compatible video calling software.

It's a funny joke. Current humans are still competing in the name of capitalism, there is little to no incentive to build interoperable system when you can control a market sector. Of course no one actually can, but that doesn't stop facetime not being available on android.

Rants aside; We are going to solve this set of problems with automation, machine learning and AI. Here is a great talk on transport layer improvements , it talks about machine learning approaches to optimise delay/bandwidth for live streaming video connections.

It is entirely feasible that we could run similar approaches to coordinate video communication, especially if we are a civilisation that spends all of its time exploring and finding new people to speak to. Automate the boring stuff, you know?

Reading: Little Brother, Transmet

The BBC have an excellent rendition of Burning Chrome by William Gibson. I am sure a neighbour will help you out if you are geographically impaired.

Red Team Newsletter

Thu, 13 Oct 2016 00:00:00 +0000

I was pretty much dead yesterday, I didn't do anything interesting.

I signed up for an Offensive Security Newsletter from Phobos Group . I don't normally take corporate output directly, the people behind Phobos have a track record of doing awesome things. The first issue appeared today, certainly worth a read.

I have been thinking about adding more automation into my...I dunno life? This morning I was thinking about using post tags to automatically cross blog to reddit. I think that might work for well for hacking , radio definitely has a home in the ham subreddits.

I am not sure if there is somewhere that will welcome the daily morning posts. /r/Blogging has a weekly 'Check out my blog' thread, but it is limited to one post per blog per week. I wonder if there is somewhere I can feed my daily ritual, like a don't break the chain place .

I will automate everything to go out the twitter hole , I would like to do the tag thing to irc channels to. That might be a bit insane and self promotional though.

Reading: Little Brother, Transmet

I hate it here

Fri, 14 Oct 2016 00:00:00 +0000

It is raining so hard I can hear it over my music and the rumble of the bus. It is raining in the book I am reading. Completely unconnected events, but humans have this thing for making patterns where they don't exist.

In this book over centralisation leads to a complete media blackout. Decentralisation is a core ethical tenant , of course I enjoy the collapse of the media in the story.

But what can you do about centralisation?

Until the singularity you are going to be stuck as a centralised human being. I know it sucks, but one day we will be able to move past this.

The indieweb movement has great advice for getting started . The biggest single step you can take to decentralise yourself on the internet is to have another machine to represent you.

Once you have a VPS running somewhere in the internet, you have access to an constantly running, near permanent version of yourself.

Reading: Little Brother, Transmetropolitan

Saturday HF Radio

Sat, 15 Oct 2016 00:00:00 +0000

Weather is horrible againg, looks like we are getting the tail end of some dramatic weather.

Hibby and I planned ot try some more line of sight microwave , neither of us fancied climbing a hill in this storm. Instead we did a bit hf from my QTH. Radio power meter looks mental when doing hell .

Reading: Little Brother, Transmetropolitan

Sunday

Sun, 16 Oct 2016 00:00:00 +0000

Today has been a very slow start, most of yesterday was spent drinking shows and playing with radios. I wanted to post a gif from twitter, but my brain isn't work well enough to figure out how on earth to get hold of one.:

pic.twitter.com/O6m93MgvAW
— 豊井 (@1041uuu) September 13, 2016

I can embed a tweet here using the code twitter gives me, but the media preview doesn't seem to work. There aren't any errors in the console or in the network debugger in firefox.

It is Sunday, so that makes seven days of writing .

Reading: Little Brother, Transmetropolitan

Android Reverse Engineering

Mon, 17 Oct 2016 00:00:00 +0000

I seem to have a knack for finding the hardest problems to start with. Anyway I thought I would have a look at doing some android reverse engineering on a local transit app.

First you will need to get the apk application bundle for the app you want to have a look at. If you have the app installed on your phone this is really easy to do with adb .

$ adb shell
anddroid$ pm list packages
package:com.google.android.youtube
package:com.android.providers.telephony
...
package:com.android.documentsui
package:com.android.externalstorage
package:com.test.testapp

anddroid$ pm list packages | grep testapp
package:com.test.testapp
anddroid$ pm path com.test.testapp
package:/data/app/com.test.testapp/base.apk
anddroid$ exit
$ adb pull /data/app/com.test.testapp/base.apk

Now you will have the apps apk as base.apk and feed it to jadx . jadx is a dex to java decompiler with a pretty gui and the ability to deobfuscate code. When you fire up jadx with the apk you will get a complete break down of the apk bundle and decompiled classes.

At this point you should see the decomiled classes, but as I said I am great at picking hard targets. There is some decompiled java here, but there are also mono packages and a load of dlls shipped in the assemblies directory.

As I said, great at picking hard targets. To get further with this I shall have to find a c# decompiler, they seem to be quite common.

Reading: Little Brother, Transmetropolitan

bytebeat

Tue, 18 Oct 2016 00:00:00 +0000

I am struggling for something to write today. I spent last night working on the second stage of a reverse engineering project, but I haven't made much progress yet and there isn't anything to show. Windows tools seem determined to be as alien as possible to use.

I had a look through my browser tabs, I still have what I consider the canonical bytebeat reference open. bytebeat is a sort of code golf based algorithmic music generation, the tiny snippets of code can manage to create some awesome sounds.

There are quite a few people working on audio from crazy systems. Captain Credible's excellent album Dead-Cats is generated with an attiny85. I have Blooper Eel mini synth kit from him that I have toyed with a ton at my desk.

And this is just the start of the rabbit hole, if you want to go up a level you should read the excellent noisepedals blog .

Reading: Litte Brother, Transmetropolitan

HyperNormalisation

Wed, 19 Oct 2016 00:00:00 +0000

I watched the latest documentary from Adam Curtis , HyperNormalisation , instead of anything of the things I planned to do last night.

Reading: Little Brother

If you are geographically or temporally challenged I am sure a neighbour has a copy you can borrow.

Driving Cypherpunk

Thu, 20 Oct 2016 00:00:00 +0000

I spent last night working on the mt7610 driver and by that I mean I was reading the open linux source trying to work through it's general insanity. Look I found the register access isn't really meaty enough to write about.

Thanks to @adventureloop I've added the https://t.co/oog7dmXgtK archive, https://t.co/IZHLZUgrSn
— Mike Dank (@Famicoman) October 19, 2016

@Famicoman is attempting to create a full archive of the Cypherpunks mailing list . I tried to read the mailing list last year and made by own copy of an archive. My copy has been add to the github repo that is trying to capture this.

Reading: Little Brother, Autumn 2600

Mosh

Fri, 21 Oct 2016 00:00:00 +0000

I have to ssh proxy to get to my main machine, everything is filtered on the network my machine is on, apart from the ssh access box. This makes using mosh a little troublesome.

                          +-------+           
+------+                  |ssh    |          +-----------+ 
|laptop|-------ssh------->|gateway|--ssh---->|           | 
+------+<--               +-------+          |dev machine| 
           \---------mosh------------------->|           | 
                                             +-----------+

dev can only be reached via an ssh proxy, but thankfully there is an open UDP port range that works. Mosh seems to have trouble figuring out the correct ip/port pair to select in this setup, mosh is quite simple so it is easy to deal with.

Host dev
Hostname dev.domain.tld
User tj
ProxyCommand ssh -w 30 -q gateway.domain.tld nc %h 22

The mosh command is just a shell script, it sshs to the remote machine and runs mosh-server . Mosh server generates an AES session key and starts the mosh server process on the machine. mosh-client takes the session key via an environmental variable, ip address and port the server is listening on.

With that we can run mosh by hand:

[laptop] $ ssh dev
[dev] $ mosh-server
  setsockopt( IP_RECVTOS ): Invalid argument

  MOSH CONNECT 40001 pv2jeN0MJ1N4gCd1V0i21g

  mosh-server (mosh 1.2.5) [build mosh 1.2.5]
  Copyright 2012 Keith Winstein <mosh-devel@mit.edu>
  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
  This is free software: you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law.

  [mosh-server detached, pid = 19100]

  Warning: termios IUTF8 flag not defined.
  Character-erase of multibyte character sequence
  probably does not work properly on this platform.
[dev] $ exit
[laptop] $ MOSH_KEY="pv2jeN0MJ1N4gCd1V0i21g"
[laptop] $ mosh-client 143.100.67.5 40001

Once you know how to do mosh by hand there are other things we can try. I don't think it would be impossible to work around certain types of NAT using nc. It requires a third party box, but a lot of STUN can be done with just UDP packets.

Reading: Little Brother, Transmetropolitan

I am sure I have written this down before, google couldn't find it.

Decentralise as Default

Sat, 22 Oct 2016 00:00:00 +0000

Yesterday featured a massive ddos attack against DynDNS. For me, in the north of Scotland, this meant an entire shutdown of the web. ssh and mosh connections stayed, but everything from twitter to google were unreachable.

Name discovery in decentralised networks is a really hard problem, I am not aware of any really solid solutions. There is probably a large capitalist factor involved here, you really can't centralise profits from a decentralised system.

I spent some time reading about name systems for adhoc mesh networks, before I gave up on trying to build this out. It is hard and would require a load of other people to test.

A quick search of my in brain cache returns:

Multicast or mdns
namecoin

mdns is probably already running on your local network, it won't scale well and certainly not to internet sizes. namecoin is something I am just sort of aware of, I think worry of blockchain buzzword bingo has stopped me looking too hard.

I would love to know about more interesting and diverse systems, if you know of any drop me a line.

Reading: Little Brother, Transmetropolitan

Late night Cybering

Sun, 23 Oct 2016 00:00:00 +0000

Last night I finally read this giant interview in the Paris review with William Gibson . The interview is full of great quotes, insights and gems like Gibsons first published story .

The second part of Burning Chrome is available on iplayer about now, the first part was excellent.

It is Sunday, so that makes seven days of writing .

Reading: Little Brother, Transmetropolitan

If you are geographicaly impaired, I am sure a neighbour help you out.

Bookcase Pi

Mon, 24 Oct 2016 00:00:00 +0000

Last night, I drilled some holes in a book case and bricked a pi. That isn't so interesting, unless you really like holes in wood, and it leaves me at a loss what write about.

Okay, fine. I have this awesome eink screen for the pi, I got it to do something like this tide clock . I don't want single purpose things lying around, the same pi is going to be running mpd my music player of choice. It will be using the screen to show cool effects (like the thing on it now) and probably stats about things.

What things, I have no idea. Maybe: * bus times * output from the house sensors * whats playing * network uptime

See, it isn't really fleshed out yet. I do have all the code to write stuff to the screen, it took ages to get working using python, cairo and pango. Now I have holes drilled and audio cables routed through the book case, I need to get the pi up and doing music.

It is not ready yet.

Reading: Little Brother

OTG USB Hub

Tue, 25 Oct 2016 00:00:00 +0000

For my silly little tablet I got this awesome usb otg hub thing. It has 3 usb ports, a microusb hole and an otg cable, you can you it to connect 3 devices to your phone or tablet and power them all at the same time.

I got this thing so I could install something other than windows on my stream 7, to do that I need power, usb storage, usb networking and io stuff all at once.

It also comes with the most mental instructions I have seen. I am trying to figure out what it says, but man, who knows. I think there was a deal on 3 postition swtiches and they put it in instead of a 2 position one.

Reading: Little Brother, Seveneves

Boxes and lines

Wed, 26 Oct 2016 00:00:00 +0000

Osmocom can do 3G voice! Look at this awesome article about the new support, it builds on this equally awesome article that gives a status update on the 3G stack. This is excellent news, as we move through LTE into whatever the 5G tech will be called, the open source community is starting to catch up with commercial hardware.

Look at those awesome diagrams:

                                +--------+
                             ,-->| MGCPGW |<--RTP--...
                            /    |        |
                            |    |        |<--MGCP
                            |    +--------+       \
                            /                     |
        +------------+<--RTP     +--------+       `->+----------+
 UE <-->| hNodeB     |           | HNB-GW |          | OsmoCSCN |
 UE <-->|            |<--Iuh---->|        |<--IuCS-->|          |
        |            |     ...-->|        |    ...-->|          |
        |            |           |        |          +----------+
        +------------+<--GTP-U   |        |
                              \  |        |          +------+           +------+
                              |  |        |<--IuPS-->| SGSN |<--GTP-C-->| GGSN |
                              |  +--------+    ...-->|      |   GTP-U-->|      |
                              |                      +------+  /        +------+
                              \_______________________________/

I mean, look at the awesome curved line:

            +--------+
         ,-->| MGCPGW |
        /    |        |
        |    |        |
        |    +--------+
        /
-+<--RTP

I cannot draw lines like that, I can draw lines like this:

               +--------+
               >        |
             -/|        |
            /  |        |
           /   +--------+
         -/     
   -----/

[DrawIT][2], the vim plugin I use for ascii boxes and lines just can't do those amazing curved lines. I bet it is a emacs plugin or something else I can't use making those awesome lines. Man am I jealous.

Reading: Little Brother, Seveneves

Hacktoberfest

Thu, 27 Oct 2016 00:00:00 +0000

I finished Hacktoberfest Last night !!!! Hacktoberfest is a month long hackathon thing run by DigitalOcean and github, in exchange for some open source Pull Requests DO will send you stickers and a tshirt. I tried to do this last year, but found it is really hard to do small commits against projects, I ended only managed 1 commit, but DO still sent me a sticker.

This year I was determined to manage the 4 commits required to get a tshirt, silly me I thought that working on an open source github hosted project for $work would make that easy. Instead I really struggled to manage the four PR's, I only got two via the work project, small commits are hard things to find.

For the other two pull requests I looked at open source software.

gpredict is a cross platform open source satellite tracker, I have used to for following amateur satellites. gpredict has always been super buggy for me, the current packaged build for FreeBSD dumped core when I tried to open the 'sat info screen'. Firing up gpredict with debug symbols and within in gdb made it really easy to find the use after free that was the culprit.

There were a pile of issues like this, I ran the build through llvm's scanbuild tool and it showed up a bunch of potential bugs. They too went into the PR for gpredict.

Last night an email came from DO stating there was still time to get the necessary PR's in. Dern, I had only manage three of the four pull requests so far.

Kaitai Struct is an awesome project for generating code from binary formats, it is a compiler, a visualizer and a declarative language. There is a set of example formats of images, games, media, compression and network packets. I noticed that UDP was missing from the network set and shamelessly added it .

Reading: Little Brother, Autumn 2600

Modern Gonzo

Fri, 28 Oct 2016 00:00:00 +0000

Maybe because there is an election on or maybe just because I wanted a use for my new stream 7 tablet thing, I read through all of Transmetropolitan . Transmet (as I am told the cool kids call it) is a Cyberpunk comic book series written by Warren Ellis, featuring a Gonzo journalist reporting on an Election from 'The City'.

I am a huge fan of Gonzo as written by Hunter S. Thompson, but Hunter is long dead and this has limited his journalistic output severely. So here I have a problem, I would be very happy to read more high quality pieces in the Gonzo style, but I have found finding such writing to be an absolute nightmare.

Here is a list of people I know writing great stuff:

Barrett Brown has been writing from prison, his stay is nearly up, it seems they want to kick him out for some reason.

I might have to look harder.

Reading: ELEKTROGRAD

I couldn't finish Little Brother, it became too YA and it just annoyed me. I did read all of it when it came out so I am not that bothered.

Follow that robot

Sat, 29 Oct 2016 00:00:00 +0000

pic.twitter.com/DSD8e44xZJ
— Archillect (@archillect) October 29, 2016

This robot is excellent, you should follow it.

Reading: Abaddon's Gate

Coffee routine

Sun, 30 Oct 2016 00:00:00 +0000

Went to a friends and carved some pumpkins last night, that means I didn't manage to do anything interesting yesterday. Weekends are when I make coffee , Sunday is filtering day which looks something like this:

I have to run out to meet someone for lunch, tonight I am going to have a play with Scapy. I think I will try to pull an image out of a http stream, that seems like a small enough task to be doable.

It is Sunday, so that makes seven days of writing .

Reading: Abaddon's' Gate

Getting Certs Out of Wireshark

Mon, 31 Oct 2016 00:00:00 +0000

Packet capture tools are oscilloscopes to network programmers, I couldn't get anything done without near continual use of tcpdump and wireshark . In a pinch tcpdump can be used instead of writing server code .

Wireshark has support for a load of protocols and can really help with debugging. Recently I added dtls support to NEAT . DTLS is a protocol enhancement to TLS to support datagram traffic, when it is working all of the traffic is basically random noise.

I had trouble gettting server certs to work correctly with DTLS, thankfully Wireshark can reassemble the datagrams into a coherent certificate and export the data out to a file. I can use this to manually check the cert is being sent correctly.

The process is something like this:

1. Import pcap
2. Find the full reassembled server hello
3. Expand the DTLS body
4. Expand the DTLS Record, Certificate (Reassembled)
5. Right click on 'Handshake Protocol: Certificate(Reassembled)' 
6. Select Export Packet Bytes

After than I had a TLS Cert in DER format , DER is just he raw cert bytes. With this I could then verify using openssl that the cert chain was valid.

Reading: Abaddon's Gate

Getting Images Out of Wireshark

Mon, 31 Oct 2016 00:00:00 +0000

While researching extracting images with scapy I found a page describing image extraction with Wireshark, I am not sure why I didn't think to try this first. Of course Wireshark can do this super useful network task, their mission is to make the ultimate network diagnostic tool.

The information on that page seems to be a little out of date, on my Wireshark build the PDU tracing and http follow options were already selected.

Grab a dump of a http session, then feed it into Wireshark:

# tcpdump -w webimage.pcap host adventurist.me and port 80

I visited this page which I know has an image on it in FireFox's porn mode.

http.response.code==200

In Wireshark I used a http 200 response code to find all of the assets in the stream. This left only three items, the page itself, the css style sheet and the image. Expand out the TCP block in Wireshark, right click on the JPEG block and choose 'Export Packet Bytes'. I saved this as .bin, moved it to a .jpeg and was able to open the image.

Reading: Abaddon's Gate

Spooky Art-Net Pumpkins

Tue, 01 Nov 2016 00:00:00 +0000

Last night was All Hallows' Eve , I wanted to do something cool with the decorations. I repurposed an rgb neopixel board driven by a nodemcu board and gave one of our pumpkins a network controlled candle instead of the old analog kind.

I also spent some time building out a motion sensor, but I wasn't able to integrate that with the network code in time to use it. In the end the weather seems to have kept everyone at home and we didn't have any visitors.

I am going to try and get everything together tonight at the hackerspace , if I do I will write up what all the parts are.

Reading: Abaddon's Gate

Extract Images from a TCP Flow

Tue, 01 Nov 2016 00:00:00 +0000

On Sunday I thought I would try to extract images from a TCP Flow with scapy . Initial searches should how to do this with wireshark , but the idea is to do this programatically and try to learn something about scapy. At the hackerspace tonight, not wanting to work on anything I said I would, I thought I would have a play with scapy.

A little bit on TCP

A TCP Flow is how we refer to the stream of packets that make what you might call TCP Connection. The connection bit is just the start. A Flow is defined in IP by 5 numbers:

* protocol (TCP or UDP for the most part)
* source
    - address (ip address of the initiator of the connection
    - port (normally a randomly chosen emphemeral port number)
* destination
    - address (ip address of the host)
    - port (normally a well known service number, http is 80)

A lot of time we call this the 5 tuple, or 4 tuple if we know the protocol. At any one point in a time a given 5-tuple defines the connection(Flow).

To a programmer TCP presents a reliable byte orientated stream interface. This means any bytes we write into our TCP socket, will come out of the other end in order and they are guaranteed to arrive (or an error is generated).

Data written into a TCP socket is broken into chunks the network can support (normally, without fragmentation), we call these chunks of data segments. Each segment has a sequence number, which tells the remote end where it is in the stream, there can be a large number of these segments in the air at a time, the flight size.

Segments can get lost in the network (well dropped by routers), reordered or delayed.

Extracting Images

To extract images from a network capture we need to separate out the packets into flows; reassemble TCP flows into a byte stream taking into account loss and reordering; reconstruct the segments into a coherent byte stream; search the byte stream for image headers and try to extract them.

This is a none trivial amount of work for a Tuesday night.

Before writing any code I did some searching, scapy might have support for flow reconstruction(nope). I came across some references to a tool called tcpflow , tcpflow claims to be a tool for extracting TCP Flows from either a live capture interface or a pcap file.

That looked great to me, I would grab a pcap with tcpdump , process out the flows with tcpflow and then drop that into scapy and start looking for some images.

Reading the tcpflow man page I instead found a single option that would do all the work for me.

Images with `tcpflow`

It is really easy to extract images from a http TCP Flow using tcpflow , you can do this live, but I used a pcap file.

# tcpdump -w webimage.pcap host adventurist.me and port 80

I started up the dump then visited http://adventurist.me/posts/0129 in FireFox's porn mode.

tcpflow will read in a pcap file with the -r flag, the -e flag will apply magic to the flow and find you fun stuff.

$ tcpflow -r webimage.pcap -e http

tcpflow will spit out a file for each flow, to boot it will throw in extracted data for everything it understands.

$ ls
093.095.228.091.00080-172.031.005.168.58914
093.095.228.091.00080-172.031.005.168.58914-HTTPBODY-001.jpg
093.095.228.091.00080-172.031.005.168.58914-HTTPBODY-002.ico
093.095.228.091.00080-172.031.005.168.60028
093.095.228.091.00080-172.031.005.168.60028-HTTPBODY-001.html
093.095.228.091.00080-172.031.005.168.60028-HTTPBODY-002.css
172.031.005.168.58914-093.095.228.091.00080
172.031.005.168.60028-093.095.228.091.00080
report.xml
webimage.pcap

tcpflow also seems to be spitting out a report.xml , it seems to describe what it has just done. I image that is super useful when running tcpflow against a live capture. I haven't managed to get very far using scapy to pull images out of flows, I am starting to wonder if there is really any point when all these tools are available.

Making webm files

Wed, 02 Nov 2016 00:00:00 +0000

Yesterday I posted a stupidly large gif of some pumpkins flashing different colours. Then a couple of minutes later after suffering through the upload I replaced it with a teeny webm.

ffmpeg now has excellent webm support, for me it will happily pick it up with automatic detection. To make videos like the one above I use ffmpeg and let it choose its own options. I drop the audio from the video to give it a nicer gif like effect, I think from today I will tell browsers to autoplay the video snippets so they look like awesome gifs.

I do something like this:

$ ffmpeg -i VID_20161030_112003.mp4  -ss 00:00:02.0 -an coffee.webm

-i is the input video
-ss tells ffmpeg to start the encoding at this time stamp
- we could pass -t to tell ffmpeg to extract a t length section of the video
-an tells us to the use no audio
coffee.webm is the output video and format, ffmpeg will do the right thing here

Reading: Abaddon's Gate

Rotator Controller

Thu, 03 Nov 2016 00:00:00 +0000

gpredict is able to signal a rotator controller over TCP. This is awesome, I want to track satellites and I am not going to pay for a rotator controller. I am going to build something to get my antenna pointed, using servos and a wifi microcontroller board.

I have tried searching a few times, but like everything amateur radio hard facts are hard to come by, the scam artists and windows developers protecting their sacred lore abound. I was at a bit of loss until I thought to try seeing what gpredict spits out over the network.

First I created a test rotator in the gpredict settings:

Then I dug around until I could find the rotator control panel, named antenna control. In this panel there is a 'track' button and an 'engage' button, figuring engage was the test option to manually set the rotator I hit that.

After a short pause a helpful 'ERROR' pops up under the Az/El settings, Good progress. Next I started up nc pretending to be a listening rotator controller so I could see what gpredict was sending.

$ nc -l 0.0.0.0 4533  
p

P  180.00   45.00

p

P  180.00   45.00

p

P  180.00   45.00

p

P  180.00   45.00



p

P  180.00   45.00

This output is great, nc is just outputting the bytes sent down the tcp connection. It seems that gpredict sends a letter 'p', I replied with a blank line by hitting enter, this resulted in a capital 'P' and a Az and El. Some guess work interpretation suggests gpredict is asking for our position with 'p', then giving us a position to move to with 'P'.

This is a great start, next I will have a look through the gpredict source to see what it is doing. I will start with the 'engage' button from gtk-rot-ctrl.c .

Reading: Abaddon's Gate

Analysing a Network Protocol

Thu, 03 Nov 2016 00:00:00 +0000

gpredict is piece of software for tracking things in orbits, sometimes you want to automatically point things at stuff in orbit. To get things pointed at stuff in orbit we can use a rotator controller, gpredict as a piece of radio software has an antenna rotator controller built it. The gpredict rotator controller expects to speak to something over TCP.

I have not been able to find documentation for the protocol (I didn't look very hard), I thought it would be fun to reverse engineer the protocol and write a simple daemon. Earlier I took some first steps to see what gpredict was doing on the network.

If you want to play a long at home this is what I am going to do:

set up a dummy daemon using netcat (nc -l localhost 4533)
use tcpdump with -XX to watch all traffic (e.g. tcpdump -XX -ilo0 tcp and port 4533)
send data from gpredict to the daemon (hit the 'engage' button on the antenna control screen)
play with responses (type into the console running nc)
look at the gpredict code starting here: https://github.com/csete/gpredict/blob/master/src/gtk-rot-ctrl.c

The Network traffic

$ nc -l 0.0.0.0 4533  
p

P  180.00   45.00

When I press the 'engage' button, gpredict sends a single lower case 'p', if I press enter, sending a blank line, gredict responds with a capital 'P' and two numbers. To me these numbers look like an Az El pair, they correspond to the values on the antenna control screen in gpredict . No need for tcpdump this time.

We have source avaialble

With only one half of the network protocol to look at, we can't get very far. gpredict is open source and there is a github mirror where we can browse the source tree. The file names in the 'src' directory show some promising results:

gtk-rot-ctrl.c
gtk-rot-ctrl.h
gtk-rot-knob.c
gtk-rot-knob.h
rotor-conf.c
rotor-conf.h
sat-pref-rot.c
sat-pref-rot.h

The pref and conf files, are probably configuration stuff, I have no idea what is in the knob file, but the gtk-rot-ctrl set of files is what we want. I confirmed this by picking a string in the UI of the relevant screen and grepping through the code for it. This can be troublesome if the software is heavily localised, but it this case I could track down the 'Engage' button to a comment in the code .

There are two functions used for network traffic, send is used to send data into a tcp connection, recv is used to receive data from a TCP connection. If we can find these in the code, we find where the software is generating network traffic. Normally only a starting point, it is very common to wrap these two functions into other convenience functions.

A grep through the code brings up a send call in send rotctld command . More grepping and we find that send_rotctld_command is called from two places, the get pos function (which I have to guess asks for the rotators positions) and the set pos function (which must try to set the rotators position).

The get_pos function fills a format string with "p\x0a" and uses send_rotcld_command to send it. Looking up 0x0A in an ascii table shows it is Line Feed(LF) also known as a newline on a unix system. It splits buffback on newlines using g_strsplit , looking to find two floating point numbers to use as azimuth and elevation, one on each line.

get_pos :

/* send command */
buff = g_strdup_printf("p\x0a");
retcode = send_rotctld_command(ctrl, buff, buffback, 128);
...
vbuff = g_strsplit(buffback, "\n", 3);
if ((vbuff[0] != NULL) && (vbuff[1] != NULL))
{
    *az = g_strtod(vbuff[0], NULL);
    *el = g_strtod(vbuff[1], NULL);
}

This piece of code shows up something really important, gpredict is using a single function to both send a command and gather the response from the remote end. If we look at send_rotctld_command the recv call is called right after a send. Here we can see that gpredict only does a single recv to gather responses, it is expecting a reply that fits into a single read. This is a bug, but probably not one that really matters.

/* try to read answer */
size = recv(ctrl->sock, buffout, sizeout, 0);

The set_pos function fills up a format string with a capital 'P', and two floating point numbers. It doesn't do any parsing of the response, only looking at the error code from the socket call.

set_pos :

/* send command */
g_ascii_formatd(azstr, 8, "%7.2f", az);
g_ascii_formatd(elstr, 8, "%7.2f", el);
buff = g_strdup_printf("P %s %s\x0a", azstr, elstr);

retcode = send_rotctld_command(ctrl, buff, buffback, 128);

Write a Daemon

With this little bit of analysis we have enough to write an antenna control daemon that gpredict can speak to. The rotator control protocol has two simple commands, a position query which expects the currect az/el across separate lines and a position setter, which expects no response.

#!/usr/bin/env python

import socket

TCP_IP = '127.0.0.1'
TCP_PORT = 4533
BUFFER_SIZE = 100

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((TCP_IP, TCP_PORT))
s.listen(1)

conn, addr = s.accept()
print 'Connection address:', addr

az = 0.0
el = 0.0

while 1:
    data = conn.recv(BUFFER_SIZE)
    if not data:
        break

    print("received data:", data)

    if data == "p\n":
        print("pos query at az:{} el: {}", az, el);

        response = "{}\n{}\n".format(az, el)
        print("responing with: \n {}".format(response))
        conn.send(response)
    elif data.startswith("P "):
        values = data.split("  ")
        print(values)
        az = float(values[1])
        el = float(values[2])

        print("moving to az:{} el: {}".format( az, el));

        conn.send(" ")
    elif data == "q\n":
        print("close command, shutting down")
        conn.close()
        exit()
    else:
        print("unknown command, closing socket")
        conn.close()
        exit()

Using the python TCP server example as a starting point it is easy to put together a daemon that will listen to the rotator controller. The code should be pretty straight forward to read, we process the commands documented earlier. There is one addition that I didn't see in the code at first. There is a quit command that does not use the normal wrapper and instead uses send directly. This command was easy to handle.

This is how I approach network problems, whether in code I have written or code that is completely new to me. Hopefully if you have been following along at home the example above is straightforward to read.

I had to make a candle

Fri, 04 Nov 2016 00:00:00 +0000

We don't have candles lying around the lab, I wasn't going to let that stop me. I made one using an arduino mega, the single ws2812 neopixel led I could find and some diffuse that was lying around. I was really hard to capture on my phone, but the flicker effect I found on github works really well.

Reading: Abaddon's Gate, Reamde

Internet Cafes in 2016

Sat, 05 Nov 2016 00:00:00 +0000

The Starbucks I am sat in right now is the model of the modern internet cafe. There is coffee, free WiFi, chairs(!), they are happy for you to sit there all day if you order an over priced drink every so often. And other than me there are people in here using laptops, they might even be working.

In the 90's an internet cafe was a different thing, there might have been coffee and drinks, but the main feature that drew people in were the rows and rows of computers. Laptops had weedy specs and were really over priced. Many people probably visited just to use the computers, it might have been there only way to get online.

Internet cafes did not last in the west, the pc market had to make laptops affordable to live. With disposable income and infrastructure that had to appear to be world leading it quickly became expected to have a computer at home.

There is an impression in the western mindset, driven by the media, that internet cafes are still a big thing in poorer parts of the world. If you show a user in India or China using a computer from an internet cafe no one will bat an eye. Both For the Win and Reamde feature Gold Farmers playing MMO's from internet cafes.

Unfortunately Internet cafes aren't a myth, there are still many places you can find desktop computers set up for general public access. University computer rooms, public libraries, airports and hotel lobbies are some common culprits. As in the 90's and 2000's public machines are a security nightmare.

You can never be safe using someone else's computer, that is why the cloud is such a joke. General public machines are a potential goldmine to a malicious actor and maybe worse, are a breeding ground for malware that will be around even when the host isn't actively malicious.

== Can We Build An Internet Cafe in 2016? ==

People are going to no matter what, can we build something that is reasonably safe for a user? I think we first have to assume that the machines we are going to use are not actively malicious, there is very little we can do to stop someone that is actively coming after you. Active attacks are rare, most people are only targeted when they stand out from the crowd.

I think there are two ways we can do this:

1. User provides the computing and storage

In this case the user has their own computing power, but they need access to a larger screen and more capable peripherals. The venue operator just have to provide a standard interface, lets say HDMI ports on large monitors, and the keyboard and mouse.

You could carry a some sort of HDMI stick pc, a raspberry pi, or something else. This idea is the basic of Ubuntu's Convergence computing , the phone you carry around all day is already a capable enough computer. With a little hardware to connect a screen, keyboard and mouse, the convergence device goes from phone OS to full desktop OS.

The convergence idea is really interesting, but Ubuntu is starting it up very slowly. One day soon, hopefully.

2. User provides storage

The second idea is that the venue provides normal desktop computers of some sort we would expect, but they don't have a hard drive or operating system installed.

Instead the user brings a bootable USB stick with a proactively secure operating system like tails installed. The user is able to take the USB stick wherever they go and manage to maintain a session between boots.

This is possible now.

Reading: Abaddon's Gate, Reamde

The subtitle text for Neal Stevenson's website is excellent

TouchBar

Sun, 06 Nov 2016 00:00:00 +0000

The HN thread about nyancat on the new MacBook TouchBar is overwhelmingly negative, that's normal for HN, but the response I have seen to it other places has been just as bad.

The changes to the keyboard, awkwardness of touch screens and position of the board are common points of compliant. I think Keyboards have too many keys, I type on a planck which only has 40 keys. I love this keyboard. I think if you look at a layout map of the planck you will realise that you can get used to very strange layouts.

An awesome, long screen like that is a great addition the standard laptop layout. Having a way to access context while not obstructing the main display is awesome. The hardware as implemented by Apple is a problem, if it really is running WatchOS then no other OS will ever work with it, that doesn't stop over manufacturers doing it with a sensible hardware link.

I love the idea of having a secondary display built into my laptop. Look at the awesomeness @jcs managed with the RGB bar on the chromebook pixel, that bar is only RGB, a full colour display could do so much more.

My OpenBSD driver for the Chrome EC supports userland access, so now the lightbar can blink red whenever pf blocks a packet. pic.twitter.com/1wnwGOFaPq
— joshua stein (@jcs) October 8, 2016

Screens on keyboards aren't new, there have been gaming keyboards with screens for a long time. Apple might be the first to try this on a laptop, but will probably be the first to succeed with this idea. I suspect Apple will have the first implementation that sees real adoption in the secondary screen peripheral space.

It is Sunday, so that makes seven days of writing .

Reading: Abaddon's' Gate, Reamde

CC0 Images With Unsplash

Mon, 07 Nov 2016 00:00:00 +0000

This has been a hard morning, the weather is extra foul and heading into a real winter, and I sense a cold coming on. On HN today there is the curated CC0 image site unsplash , I have co-opted a giant night sky image from there for today's post.

I have been thinking for a while that every post really should have an image, ideally I would take a load of great photos and add a sort of relevant one to each post as I go. In lieu of that happening I might integrate something like unsplash and one of their great collections.

https://unsplash.it/200/300/?random

They do have a service which provides random images with a url, others have used it to make 404 pages more interesting.

Reading: Abaddon's Gate, Reamde

Collecting my own stock images

Tue, 08 Nov 2016 00:00:00 +0000

Yesterday I came across the awesome CC0 license stock image site unsplash , both today and yesterday I have used other peoples images from that site. The images aren't of anything I have been writing about, but images make blog posts looks a ton better.

I think I am going to continue to try and have an image on every blog post, even if it is just to give them some colour. There really is nothing stopping me from using my own pictures of awesome places.

This awesome picture of the Skogarfoss Waterfall (which I got from unsplash) is really strange to me. I was in Iceland in August and I visited that exact waterfall and while I have used someone else's stock photo there is nothing stopping me from using my own picture of the exact same feature.

I will attack my photo collection in the next few days and try to built up a bank of images to use. I think I want to have images sorted so I can match them against the tags on blog posts. My approach to tagging is very haphazard, I will probably make groups like:

electronics
coffee
radios and antennas
landscapes(rural)
landscapes(urban)
graffiti
tool kits, workbenchs, desks

I am going to have to start taking more photos.

Reading: Abaddon's Gate, Reamde

Pirate Searching For Images On 500px

Tue, 08 Nov 2016 00:00:00 +0000

Ormiret from the hacker space created a tool to encourage him to head out and take more photos. His random theme generator is built up from some photo theme lists. I have been wanting to have a picture on every blog post, for that to be feasible I have to take many more photos.

The theme tool is an awesome idea, I thought it would be more powerful if there were exemplar images alongside the theme. There are quite a few sites that have attracted a large number of photographers and make an excellent place to search for images matching a theme.

I looked at both flickr and 500px, but neither of these sites have an API that allows unauthenticated access. I really don't want to create account on these sites just for a throwaway image search. I did spend some time looking at their APIs but neither looked like much fun.

500px has a public search page that doesn't require auth, by using Firefox to grab the request headers it was easy in an hour or so to put together a command line search tool.

#/usr/bin/env python
import sys
import urllib2
import pprint
from StringIO import StringIO
import gzip
import json

################################################################################
#                             _               _   _    _                       #
#                _ _ ___ _ __| |__ _ __ ___  | |_| |_ (_)___                   #
#               | '_/ -_) '_ \ / _` / _/ -_) |  _| ' \| (_-<                   #
#               |_| \___| .__/_\__,_\__\___|  \__|_||_|_/__/                   #
#                       |_|                                                    #
################################################################################
header = """
User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:49.0) Gecko/20100101 Firefox/49.0
Accept: application/json, text/javascript, */*; q=0.01
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
X-CSRF-Token: J2iawi/G7GJ3cqXgKVhYVlIQjNMWCXCdSPNscW/lyZb0xk1+Y3HWlPgCW6/kYQWY6tkfSWF2IWDtx8H8Q8A4Eg==
Referer: https://500px.com/search?type=photos&utm_campaign=google_search_box&q=blue
Origin: https://500px.com
Cookie: _hpx1=BAh7CkkiD3Nlc3Npb25faWQGOgZFVEkiJTcwNWJhODM3OGE0ODIwZGRjMWMzNzBmZGY5NGU1ZTczBjsAVEkiCWhvc3QGOwBGIg41MDBweC5jb21JIhhzdXBlcl9zZWNyZXRfcGl4M2xzBjsARkZJIhBfY3NyZl90b2tlbgY7AEZJIjEwNjdYdkV5M092YVBjUDVQelRsZHpyakprNXAzZjFIOXBUU3RqU3dsOFlRPQY7AEZJIhFwcmV2aW91c191cmwGOwBGSSI%2BL3NlYXJjaD90eXBlPXBob3RvcyZ1dG1fY2FtcGFpZ249Z29vZ2xlX3NlYXJjaF9ib3gmcT1ibHVlBjsAVA%3D%3D--2a92d13e5bc840dd0de0d3d469d0cc3019e12fb3; optimizelyEndUserId=oeu1478624905590r0.6262684177302974; optimizelySegments=%7B%22569090246%22%3A%22false%22%2C%22569491641%22%3A%22campaign%22%2C%22575800731%22%3A%22ff%22%2C%22589900200%22%3A%22true%22%7D; optimizelyBuckets=%7B%227781310076%22%3A%227773970029%22%7D; _ga=GA1.2.166996737.1478624906; _gat=1; _gat_unifiedTracker=1; amplitude_id500px.com=eyJkZXZpY2VJZCI6IjJkYjljZTg3LTk5ZjktNDg4Yy1hNTFlLWNhN2M5ZGU3MGUwZFIiLCJ1c2VySWQiOm51bGwsIm9wdE91dCI6ZmFsc2UsInNlc3Npb25JZCI6MTQ3ODYyNDkzMTM1NSwibGFzdEV2ZW50VGltZSI6MTQ3ODYyNDkzMTM1NSwiZXZlbnRJZCI6MCwiaWRlbnRpZnlJZCI6MCwic2VxdWVuY2VOdW1iZXIiOjB9; optimizelyPendingLogEvents=%5B%22n%3Dhttps%253A%252F%252F500px.com%252Fsearch%253Ftype%253Dphotos%2526utm_campaign%253Dgoogle_search_box%2526q%253Dblue%26u%3Doeu1478624905590r0.6262684177302974%26wxhr%3Dtrue%26time%3D1478624986.482%26f%3D7763794202%2C7254840151%2C7769081978%2C7513516222%2C7781310076%26g%3D582890389%22%5D
Connection: keep-alive
Cache-Control: max-age=0
"""
################################################################################

host = "api.500px.com"
path = "/v1/photos/search?type=photos&term={}&image_size%5B%5D=1&image_size%5B%5D=2&image_size%5B%5D=32&image_size%5B%5D=31&image_size%5B%5D=33&image_size%5B%5D=34&image_size%5B%5D=35&image_size%5B%5D=36&image_size%5B%5D=2048&image_size%5B%5D=4&image_size%5B%5D=14&include_states=true&formats=jpeg%2Clytro&include_tags=true&exclude_nude=true&page=1&rpp=50"

themes_url="https://raw.githubusercontent.com/ormiret/photo-themes/master/themes.txt"

def search500px(searchstring):
    query = urllib2.quote(searchstring)

    url = "https://" + host + path.format(query)

    opener = urllib2.build_opener()
    for line in header.split("\n"):
        if not line: 
            continue

        s = line.split(":") 
        opener.addheaders.append((s[0], s[1].strip()))

    response = opener.open(url)

    if response.info().get('Content-Encoding') == 'gzip':
        buf = StringIO(response.read())
        f = gzip.GzipFile(fileobj=buf)
        data = f.read()

    data = json.loads(data)
    photos = data["photos"]
    print("{} photos in response".format(len(photos)))

    images = [] 
    for photo in photos:
        image = photo["image_url"][-1]
        page_url = "https://500px.com/" + photo["url"]

        datafile.write("<img src=\"{} \" href=\"{}\"></img>"
            .format(image, page_url))
        images.append({"image":image, "page_url":page_url})
    return images

if __name__ == "__main__":
    searchstring = "night time rail"

    if len(sys.argv) > 1:
        print(sys.argv)
        searchstring = " ".join(sys.argv[1:])

    print("Searching for: {}".format(searchstring))
    filename = "{}.html".format(urllib2.quote(searchstring))

    datafile = open(filename, "w")

    datafile.write("""
    <html>
        <head>
        </head>
        <body>
        <h1>
        {}
        </h1>
    """.format(searchstring))

    images = search500px(searchstring)
    for image in images:
        datafile.write("<img src=\"{} \" href=\"{}\"></img>"
            .format(image["image"], image["page_url"]))

    datafile.write("""
        </body>
    </html>""")

    print("writing out file: {}".format(filename))
    datafile.close()

This scrip paired with the request headers exported from firefox allows you to search for a string on 500px and generates a page with the first set of results. Of course I had to pull out the search and send Ormiret a pull request to add this functionality to his theme generator.

Stupid

Wed, 09 Nov 2016 00:00:00 +0000

Couldn't say it better if I tried. @warrenellis @DarickR pic.twitter.com/yswrRp9u4p
— Mireille Abrihet (@MireilleAbrihet) November 9, 2016

The ticket sales for Congress this year have been a ultra fast, it is has been entertaining to watch friends fight against server crashes and load while trying to get tickets. This year I was lucky enough to avoid that ordeal, but it has made me think about writing bots to buy tickets. I think I would be trying to do so if I was going through the public sale.

I have previously watched a defcon talk about buying cars using a set of bots, I do wonder if there is a set of literature on doing this and dealing with mitigations.

There is also this talk , it might be good.

Reading: Abaddon's Gate, Reamde

Politics

Thu, 10 Nov 2016 00:00:00 +0000

Politically the last few years have been really hard for me, 2014 , 2015 , and 2016 saw votes go completely against my expectations. This week was also surprise. It is easy to think I hold fringe views, that I am all alone surrounded by fascists, but the numbers show that only about have the electorate disagree in each of these cases.

The problem in almost all of these votes is not the right, but the inability of the left to draw people out. The fascists have it easy, they can hold a deplorable ideal, get rid of the immigrants and their supporters can galvanise around the idea. The left only seems to offer the status quo.

There are two courses of action at a time like this.

Get some supplies and a gun, go up a hill and disconnect from the world. (I can reccomend some hills)
Get involved and try to advance the causes you really care about.

Today, I really just want to climb a hill and start living in a cave. But that is the easy way out, instead I am going to start helping make the world a better place.

Reading: Abaddon's Gate, Reamde

Making MacOS be Friendly with gdb

Thu, 10 Nov 2016 00:00:00 +0000

MacOS has lots of cool security features, by default the OS will only run signed code. Great security has trade offs, tonight I was hit my MacOS restricting permissions. gdb needs to be signed before it will be allowed to debug other program. It manifests like this:

$ gdb -q neat-streamer 
Reading symbols from neat-streamer...done.
(gdb) r
Starting program: /Users/jones/code/neat-streamer/neat-streamer 
Unable to find Mach task port for process-id 13334: (os/kern) protection failure (0x2).
(please check gdb is codesigned - see taskgated(8))

Learning lldb seems like far too much work, this needs fixed. Searching brings up stackoverflow questions, with a pointer to this guide that explains the entire process. In general you need to create a code signing key, sign the gdb binary and then restart the enforcement service taskgated .

The restart commands were a little harder to track down.

Restart taskgated :

sudo launchctl unload /System/Library/LaunchDaemons/com.apple.taskgated.plistv
sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.taskgated.plistv

There are also start and stop commands, but this didn't work for me. The troubleshooting on the guide was of no help. I even went as far as trying a reboot, but no luck. Maybe I will try figuring out lldb .

If anyone has any idea hows to get this working, I would love some help.

iPhoto Mess

Fri, 11 Nov 2016 00:00:00 +0000

As I have said already, I am trying to get control of my photo collection. I want to have an image on almost every blog post, but before I can do that I need to sort out the mess that is my collection. Currently I have raws and jpegs in a directory structure, an iPhoto library and some almost structure files.

I want to have the directory layout:

year/month/day/[raw|jpeg]

For today it would be:

2016/11/11/raw
2016/11/11/jpeg

Before I can do that I need to extract images from iPhoto and collate everything together. Unfortunately iPhoto on my laptop does not want to start up at all and I suspect the App Store will want me to upgrade my OS too. I am a hacker so this isn't a problem.

Some searching turned up exportiphoto a python program that will extract images from your iPhoto library. Download, run:

python exportiphoto.py [options] "iPhoto-Library-dir" "destination-dir"

Running this script there was some crunching, some promising output and then it was done super fast, awesome! I sshfs ed out to the storage box and started looking around for my photos. Instead I found a bunch of empty directories, I must have done something wrong.

Instead of poking at the script I thought I would have a look at the iPhoto app bundle. Apps on a back are made up of a bundle, the bundle is just a directory which the finder treats in a special way. Looking into the bundle I found a Masters directory. The Masters directory was 40GB of photos in a raw format, most of the pictures that will be in the library.

The Masters directory has the photos stored in the correct directory structure, so I copied that out to use as the basis for my tidy.

Reading: Abaddon's Gate, Reamde

Pictures

Sat, 12 Nov 2016 00:00:00 +0000

I had a look through some of the pictures I have from my Iceland trip in August, but it was really painful. My network drive seems to be struggling delivering large files over sshfs, it probably doesn't help that they are 25MB raws.

I used darktable to crop the image, everythin else I had on my machine chocked on the CR2 raw files.

Reading: Abaddon's Gate, Reamde

High motion quality webm

Sat, 12 Nov 2016 00:00:00 +0000

ffmpeg by default aims for the lowest bitrate it can manage for a video when encoding webm. I have been happy with this so far, but the video I grabbed of a waterfall today does not look good in this mode. I tried changing the bitrate options as discussed on the ffmpeg wiki , I thought I would show what you can expect with a couple of differnet rates.

The original mov file generated from my camera was 21MB.

$ ffmpeg -i INPUT.mov -an output-default.webm

The original ultra low, 443kb/s that ffmpeg generates, file is 369KB.

$ ffmpeg -i DSC_3536.MOV -an -c:v libvpx -b:v 1M  output-1M.webm

Doubling the rate, file is 976KB. Still a lot of artifacts in the video.

$ ffmpeg -i DSC_3536.MOV -an -c:v libvpx -b:v 10M  output-10M.webm

This passed the smell test for me, I think it looks good enough for its size, this file is 5.6MB roughly the size of a jpeg of the same scene.

I hate software

Sun, 13 Nov 2016 00:00:00 +0000

I tried to use hugin to stitch together a panorama I took of a glacier, but the binaries they offer will only run on the next version on MacOS. Really annoying. I will give it a try tomorrow on FreeBSD, if not I will have to try some of the gimp plugins.

Facing a gimp plugin makes me think of this xkcd .

It is Sunday, so that makes seven days of writing .

Reading: Abaddon's' Gate, Reamde

Panoramas with Hugin

Mon, 14 Nov 2016 00:00:00 +0000

My Idea to use the hugin stitching software to make a panorama from some images I found on my camera seems to have hit a snag. I am convinced I didn't have a tripod with me and took the panorama in a haphazard fashion, I remember the area by the glacier being much much colder than the campsite we were staying in and I was pushed to leave.

I opened up the 8 images I had to try and stitch together and while they sort of fall out in a reasonable orde I think it is going to take some time with the software to get them together. Unless I find the more magic button .

Reading: Reamde

Nihilism

Tue, 15 Nov 2016 00:00:00 +0000

Today I've got nothing. At my desk there are a load of started and unfinished projects, parts for other things, kits from boldport club to be made. Nothing that is interesting even in its started state, components to make cools things, coolness sold separately.

At the hackerspace tonight I will try to finish my sat tracker , but even that is a fallback project. The projects I want to have completed have such a high bar to entry.

I wonder if my brain empties out in cycles.

Reading: Reamde

Sat Tracking and Killer Robots

Wed, 09 Nov 2016 00:00:00 +0000

Last night at the hacker space I finally got around to building hardware out for my emfcamp badge powered satellite tracker . Most of the time was spent hot gluing together foam board to make a stand for the servos I integrated the control code with the TCP server and the whole thing is controllable from gpredict now.

When testing servos, knifes are the recommended indicator devices.

Reading: Reamde

A mountain

Thu, 17 Nov 2016 00:00:00 +0000

Reading: Reamde

Winter Temperatures

Fri, 18 Nov 2016 00:00:00 +0000

Winter is here, stepping out this morning it was -2, hopefully the start of some nice seasonal weather with a showering of snow and not the minimum temperature for the year.

The twitters tell me that Bunnie Huang of Hacking the Xbox , Breaking SD Cards , The Essential Guide to Electronics in Shenzhen and a ton of other cool things has a new book in the works . I read Hacking the Xbox when it was released for Free after Aaron Swartz's death, the book is an excellent read and gave me a ton of insights about electronics and breaking physical things. The new book is in early access, which means you can read it if you think reading tiny bits of a book is a good idea.

While on the nostarch I looked at another early access book, Attaching Network Protocols . The cover, looking a Tardigrade at a glance(it isn't), drew me in, the awesome title didn't hurt.

Hopefully the internet will come alive and tell me when these two books are finished and available.

Reading: Reamde

Of course that snowy picture was taken up a mountain, but it was only about 4 degrees up there. Warmer than it seems it is going to get to today.

The Use of Botting

Sat, 19 Nov 2016 00:00:00 +0000

This article on the use of bots on github made me think of a different use of the github api .

The first pieces of python code I pushed to github on my own account were in my tiny-artnet mircopython artnet implementation. Soon after committing that code I started getting emails from recruiters looking to hire python developers. They would say something along the lines 'based on your github activity we think you would be perfect for a job doing django".

At first these were hilarious, micropython is nothing like python, if they had looked at my github profile they would have seen the large C projects I work on.

But after a few of these I started to get annoyed, clearly these people were finding my email from code I had written or from commit logs. Why weren't they trying a little bit harder? To me, github is the technical recruiters wet dream, but whoever was generating the leads here clearly wasn't doing a good job.

I don't think cold lead generation is a good way to sell anything, let alone a job opportunity, but this is how I would use github(bitbucket, gitlab and everything else too) to do it.

Search projects that have the correct language keywords (python, go, c)
Find any email addresses at all, sort by most recent
Attempt to resolve email addresses into real people
a) Find personal site for email address or b) (worse) find social media pages for address
Send generated lead info to recruiter

The human at the end needs to be able to do a final set of filters, but anywhere that is too high a cost isn't going to use the lead well anyway. I am sure the 100 line script that could be written on those lines that would generate substantially better leads than cold contacting any email address.

Reading: Reamde

Colder War

Sun, 20 Nov 2016 00:00:00 +0000

It is cold and I am hiding inside, today clearly things yesterday was far too warm. Morning temperature was -5, which is nothing compared to the arctic, but cold for somewhere people live. If I set up temperature sensors I could make some plots, but that seems like a lot of hassle.

It is Sunday, so that makes seven days of writing .

Reading: Cibola Burn

Glacial Progress

Mon, 21 Nov 2016 00:00:00 +0000

Again it is cold, the previous few years there really hasn't been any substantial 'winter'. This year is different.

I did some work on the wireless driver yesterday, but it was entirely refactoring. I do think I am in a point to start crashing things. I am very happy with this sort of progress, even if it isn't really interesting. The small steps are required for the big steps to work.

Reading: Cibola Burn

Kill Games

Tue, 22 Nov 2016 00:00:00 +0000

This Killscreen article, The people trying to save programming , which I found via lobste.rs really caught my attention. The article is about some people that are trying to fix the way games are made, they think that software is development is too impersonal and long for the good old days of the Apple 2. Commercial Game engines are the problem.

The article is worth a read. Digging into the community around handmade hero is interesting too, but I don't really think either of the developers mentioned are starting a movement. To me it feels like the appeal to the desire everyone has to understand everything, actioning that by inventing the universe.

That is fine and all, but far too many new people get stuck in the trap of trying to build a world before they can walk(I did). The best tools for a beginner are the ones that let them succeed as quickly as possible. The hard nitty gritty details can be learnt later on.

Reading: Cibola Burn

The Myth of Something Easy

Wed, 23 Nov 2016 00:00:00 +0000

The Myth of Something Easy was a good talk, you should watch it. I would have just embedded it and left it at that today, but I had already picked out a picture.

The panorama came from my Android phone (nexus something), if you zoom in, the cuts between frames are really jarring. It will be interesting to see how I get on with hugin , the images I have to stitch are much large (and maybe higher quality) than anything my crappy phone can do. Some of the shots are out of focus, the stitching will be really interesting to do, whenever I get around to it.

Reading: Cibola Burn, Excession

Reading Interface Speed

Thu, 24 Nov 2016 00:00:00 +0000

Q : How do I get the interface speed?

A : On Linux:

$ ethtool eth0 
    Speed: 1000Mb/s

Not what I want at all,

Q How do I get interface throughput

A iftop does what top does for network interfaces:

$ iftop
interface: em0
IP address is: 192.168.204.4
MAC address is: ffffffec:ffffffb1:ffffffd7:34:ffffffa3:ffffffa1
pcap_open_live(em0): em0: You don't have permission to capture on that device ((cannot open device) /dev/bpf: Permission denied)

Annoying

$ sudo iftop
...cool ncurses display...

A Besides iftop and iptraf, also check: bwm-ng

$ bwm-ng 
...cool ncurses display...

Not scriptable

$ bwm-ng --output csv
1479982871;em0;0.00;0.00;0.00;0;0;0.00;0.00;0.00;0;0;0.00;0.00;0;0
1479982871;lo0;0.00;0.00;0.00;0;0;0.00;0.00;0.00;0;0;0.00;0.00;0;0
1479982871;total;0.00;0.00;0.00;0;0;0.00;0.00;0.00;0;0;0.00;0.00;0;0

Q How do those commands gather their data?

A It is different everywhere

Getting a look a network rates is really easy on FreeBSD, the systat tool in ifstat ships with the base system. But if you want to do this programmatically there isn't a lot of information out there, I had to read source code to figure out how to do it.

The initial iftop error message indicates they are doing a capture of all the traffic on all interfaces and working this stuff out on their own. That requires root and I really don't want the hassle of doing it, surely the OS is capturing these stats from the network stack?

On Linux, these stats are exposed via /proc :

/sys/class/net/eth0/statistics/rx_bytes
/sys/class/net/eth0/statistics/tx_bytes

There may actually be other interfaces for Linux, but I don't think it is worth digging any further.

On FreeBSD you can do what systat does and use a sysctl call to populate a struct. The bwm-ng man page has a heap of methods for finding these numbers on different platforms, for the BSD's and MacOS it suggests the getifaddrs interface.

For portable code not written in C I will probably set up a thread running bwm-ng outputting csv data.

Reading: Cibola Burn, Excession

Lightning Talks

Fri, 25 Nov 2016 00:00:00 +0000

It seems I am submitting a lightning talk to CCC. Lightning talks a short 5 minute presentations. The format is really popular for adding a load of content to a conference, giving many more people a chance to talk.

I have watched the congress and camp lightning talk sessions before, but I can't really remember any jumping out at me. Searching today for 'best lightning talks eva' didn't have useful results. Well, wat came up, wat is an excellent talk.

I guess I will watch some lightning talks from previous congresses and see what they were like.

Reading: Cibola Burn, Excession

Sunset

Sat, 26 Nov 2016 00:00:00 +0000

Reading: Cibola Burn, Excession

Just a picture

Sun, 27 Nov 2016 00:00:00 +0000

I wrote up a script yesterday to grab the most recent file from the super awesome toshiba flashair wifi sd card . I had suggested the card to someone in the hackerspace, he planned on using it to help align a camera trap (not that model, but you get the idea).

Once you put the trap up a tree, it is a real hassle to figure out if it is really pointing the way you want it to. So use the wifi sd card to grab the latest image and confirm it is.

After writing the script I tried for a while to get my laptop connected, but it seems that the camera trap doesn't keep the card powered on for nearly long enough. I might be able to get it to work if I can get my laptop to over overzealous in connecting to the wifi.

It is Sunday, so that makes seven days of writing .

Reading: Cibola Burn, Excession Location: 57.155,-2.210

Apparently there isn't a simple API to turn a lat/lon into the weather. I have no idea why web services all seem to insist on having an API key for all requests. It is just annoying.

Live Network Tracing in Python

Mon, 28 Nov 2016 00:00:00 +0000

python-libtrace comes highly recommended over scapy . Scapy always feels a bit alien to me, I think the custom repl front end aimed at 'security people' (whatever that means). I am sure it is there to make things simple, but for me it just makes it harder to write programs with.

python-libtrace certainly isn't easy to install, all of the documentation is left to the libtrace project. Once I figured out the magic words I was able to throw together a dscp mark classifier really quickly. For live capture on your system you will probably have to change the bpf:em0 to something like pcapint:eth0 .

import plt
import time

trace = plt.trace('bpf:em0')
trace.start()

INTERVAL = 1

dscp = {}
start = time.time()

try:
    for pkt in trace:
        ip = pkt.ip
        if not ip:
            continue

        dscpvalue = ip.traffic_class >> 2

        if dscpvalue in dscp:
            dscp[dscpvalue] = dscp[dscpvalue] + 1
        else:
            dscp[dscpvalue] = 1

        done = time.time()

        if done - start > INTERVAL:
            print("marks:".format(len(dscp)), end="")
            for mark,count in dscp.items():
                print(" {}:{},".format(mark, count), end="")
            print("")
            dscp = {}
            start = done
except KeyboardInterrupt:
    trace.close()
    sys.exit()

This can be tested with netcat quite easily, though the options seem to be different everywhere.

nc -u -T ef [host] [post]

Reading: Cibola Burn, Excession

task today

Tue, 29 Nov 2016 00:00:00 +0000

I use taskwarrior to manage tasks, well sort of. Every so often I fill it with highish level tasks and leave it completely forgotten for a few weeks. On a similar frequency(though out of phase) I look through my task list and prune out the things I have done. This isn't great, I have had a lot of trouble refining down tasks, figuring out what to do, then doing it.

Last night I thought I would try to start generating a set of tasks to do TOMORROW , then when I got to work the next today I could ask task warrior what it I was to do that day. Taskwarrior makes that sort of easy with virtual tags, the virtual tags can only be generated by due dates.

$ task add due:tomorrow proj:life get milk
Created task 1

Will generate a task, due tomorrow today, but come tomorrow it will be tagged with today. Makes sense right?. We can then easily search for all tasks matching the TODAY tag:

$ task +TODAY list

ID Age   P Project  Due        Description        Urg 
1  1m    L life     2016-11-30 get milk            1

1 task

The taskwarriors output looks awesome on the command line, but it doesn't come out my thermal printer very well. Taskwarrior will output json with the export flag, json isn't very fun on the command line. Thankfully there is the jq tool. jq claims to be like sed for json, explains it's near inscrutability.

With these bits we can generate a snappy list of things we have to do today:

figlet -f small TODAY:;cat tmp.json| jq -r '''.[] | .project,.description,""'''

Something like:

 _____ ___  ___   ___   ___ 
|_   _/ _ \|   \ /_\ \ / (_)
  | || (_) | |) / _ \ V / _ 
  |_| \___/|___/_/ \_\_| (_)

schemes.crime.bank
Order drawings of bank

schemes.crime.bank
Enroll on plasma cutting course

schemes.crime.botnet
Establish control channel for bots on freenode

schemes.crime.botnet
Register spam address

life
get milk

life
put bins out

Which is really easy to spit out to my thermal printer:

I wonder if there is some way to get xscreensaver to run a script when I log in? I could use that hook to tidy away undone tasks and do the print out on my first log in of the day.

Reading: Cibola Burn, Excession

The DC3

Wed, 30 Nov 2016 00:00:00 +0000

This plane sit in a glacial outwash plain in the South of Iceland. The area around it is barren and devoid of life. We arrived in a fog bank, there was nothing to see in any direction save from the well worn path out to the wreckage.

Walking out was like being in a dream, we could see through the haze the bright clothing of other visitors to the plane.

The fog lifted for our return journey, the landscape didn't improve. The area is almost completely flat, with small undulating banks of aggregate. The entire place looked life the surface of mars renderer in black.

Reading: Cibola Burn, Excession Location: 57.168, -2.1055

Another Amazing Waterfall

Thu, 01 Dec 2016 00:00:00 +0000

Reading: Cibola Burn, Excession Location: 57.168, -2.1055

Excuses

Fri, 02 Dec 2016 00:00:00 +0000

I have felt terrible all week and haven't had energy to really do anything. Having an image to post everyday turned out to be an excellent idea. I do need to go through the archives to top up the reserve of images at some point.

Reading: Cibola Burn

Getting the Weather

Sat, 03 Dec 2016 00:00:00 +0000

pic.twitter.com/ds38znWfqF
— Warren Ellis (@warrenellis) December 1, 2016

My good friend Warren Ellis (well complete stranger, but I read his newsletter so that is pretty the same thing) tweets pictures of where he is with the weather info overlaid. I am sure he is using some sort of newfangled social media filter to provide the info. I want something similar for the footnotes on my fairly post, but social media stuff is no good for me, I need an API to use.

Now, as hard as I try I cannot find a weather service that will just spit some data at me. I really want to do curl weathersite.internet | jq... and end up with a nice summary for a location. The web is closing up and locking down, which means an API key is required.

After putting this off for a while, this morning I remembered I have previously registered for a weather service. A steaming cup of coffee later and I found the python bindings to the excellent forecast.io already installed.

import forecastio

api_key = "yer_key_here_bampot"
lat = 57.168
lng = -2.1055

forecast = forecastio.load_forecast(api_key, lat, lng)

weather = forecast.daily().data[0]

temperatureMax = int(weather.apparentTemperatureMax)
temperatureMin = int(weather.apparentTemperatureMin)
summary = weather.summary

print("{}°C {}".format(temperatureMax, summary))

Gives a nice

4°C Partly cloudy throughout the day.

There isn't anything to this, If I could find an API that didn't require a key I probably wouldn't even use python. But madness makes more madness, so here we are.

Reading: Cibola Burn, Virtual Light

Location: 57.168, -2.1055

Weather: 4°C Partly cloudy throughout the day.

Warren Ellis's morning.computer was the main driver for me to start blogging everyday. I like to think I am being influenced by someone super productive, rather than blantently copying him.

Building a dash

Sun, 04 Dec 2016 00:00:00 +0000

I think the weather stuff I played with yesterday is going to be an input to a quantified self dashboard I have been toying with building for a long time.

I have wanted to put together a dash for years, but I have always struggled to find technologies that I want to work with. For a demo at work I have had to put together a simple dash, all it does is show interface throughput for two interfaces, but it has give a chance to play with the front end UI and backend webserving components that I want to use.

I am lurking in a coffee shop now, which is a great time to have a first whack at the idea.

It is Sunday, so that makes seven days of writing .

Reading: Cibola Burn, Virtual Light

Location: 57.1446, -2.1060

Weather: 6˚C Clear.

Street Art

Mon, 05 Dec 2016 00:00:00 +0000

Union Terrace Gardens has some excellent pieces that were put up as part of a street art festival . Adding culture to the city is great, but there is something about 'santioned creativity' that really annoys me. I know the residents around here would be up in arms if someone did a giant mural overnight.

Reading: Cibola Burn, Virtual Light

Location: 57.1578,-2.2143

Weather: 2°C Partly cloudy starting in the evening.

Blogs about blogging

Tue, 06 Dec 2016 00:00:00 +0000

I am still playing with other fields to stick onto the daily post. So far I have been sticking on a reading field that can sort of track how I am progressing with books. I want to include a fuzzy location and the state of the weather around me, obviously I know where I am looking back it will be interesting to me having a record of where I was when I posted.

I have tried with outside , reality , being and a load of other vague terms, writing this out those all look ridiculous. Now I have tried just letting the info hang there instead, my current lat/long converted to a place name with a link to a map, followed by the weather.

Reading: Cibola Burn, Virtual Light

Hacktoberfest Pay off

Wed, 07 Dec 2016 00:00:00 +0000

I looked up reverse geocoding with openstreetmap and found a keyless api. Reverse geocoding is the process of turning a location as a latitude and longitude into a place name. This is handy for creating my daily post footer, I want to have a script that will take in a lat/lng pair and output the full location name and weather with a map link.

I can use the kindly provided nominatim reverse geocoding URI and a bit of python. I guess openstreetmap thinks I am in a weird parallel UK that is made up of states, that is easy to deal with thankfully.

base_url = "http://nominatim.openstreetmap.org/reverse?format=json&lat={}&lon={}&zoom=18&addressdetails=1"
uri = base_url.format(lat, lng)

fp = urllib.request.urlopen(uri)
response = fp.read()

location = json.loads(response.decode("utf8"))
fp.close()

city = location['address']['city']
country = location['address']['country']

if country == "UK" or country == "US":
    country = location['address']['state']

return {'country':country, 'city':city}

I end up with a single script for generating the location/weather block. The script will default my 'work' location or it will try and format a lat/lng out of any arguments passed in.

#!/usr/bin/env python3.5

import forecastio
import pprint
import urllib.request
import json 
import sys

api_key = "yer_key_here_bawbag"
lat = 57.168
lng = -2.1055

def forwardweather(lat, lng):
    forecast = forecastio.load_forecast(api_key, lat, lng)

    weather = forecast.daily().data[0]

    temperatureMax = int(weather.apparentTemperatureMax)
    temperatureMin = int(weather.apparentTemperatureMin)
    summary = weather.summary

    return {'temperature':temperatureMax, 'summary':summary}

def reversegeocode(lat, lng):
    base_url = "http://nominatim.openstreetmap.org/reverse?format=json&lat={}&lon={}&zoom=18&addressdetails=1"
    uri = base_url.format(lat, lng)

    fp = urllib.request.urlopen(uri)
    response = fp.read()

    location = json.loads(response.decode("utf8"))
    fp.close()

    city = location['address']['city']
    country = location['address']['country']

    if country == "UK" or country == "US":
        country = location['address']['state']

    return {'country':country, 'city':city}

if __name__ == "__main__":
    if len(sys.argv) == 2:
        loc = sys.argv[1].split(',') 
        if len(loc) != 2:
            exit()
        lat = float(loc[0])
        lng = float(loc[1])
    if len(sys.argv) == 3:
        lat = float(sys.argv[1])
        lng = float(sys.argv[2])

    print("Getting weather for: {}, {}\n\n".format(lat, lng))

    weather = forwardweather(lat, lng)
    location = reversegeocode(lat, lng)

    base_url = "http://www.openstreetmap.org/search?query={}%2C%20{}"
    uri = base_url.format(lat, lng)

    print("[{}, {}][0]: {}°C, {}".format(location['city'], location['country'], 
        weather['temperature'], weather['summary']))
    print("\n[0]: {}".format(uri))

Reading: Virtual Light

Congress Plans

Thu, 08 Dec 2016 00:00:00 +0000

The most important twitter account counts down the remaining days to congress. Today there are just 19 days left until a hole opens in the universe and excellent people appear to keep the base going.

Ignoring the large amount of realwork™ I have to do, there is a lot of important stuff to be prepared for congress. I have tried to avoid committing to doing anything on my holiday, but I plan to bring the following three ideas:

RGB Pixel Display
- I have a 8x8 neopixel display with a small case. I am planning to make it controllable via UDP packets and leave it on the open network for other people to find and play with.
Slow TV
- Multicast video feed showing relaxing slow paced video. I have a lot of driving from my trip to Iceland. I will include some other feeds I have picked up in the last few month. We might include a radio station a long side, but that bit hasn't been figured out yet.
Some sort of display showing:
- rainbowstream
- cool stuff pulled of the open wifi, images, password wall of sheep style.

Only 19 days left until #33C3
— Waiting for 33C3 (@c3daysleft) December 8, 2016

Reading: Virtual Light

UDP Panel

Fri, 09 Dec 2016 00:00:00 +0000

Congress is coming, 18 days to go!

The whole point of sharing my plans yesterday was to make sure I actually do them. Easiest on the list is to set up the UDP controlled blinkenlights panel. I am going to attach this to my bag or something, there must be cool blinkenlights everywhere I go.

The panel is a 8x8 Addressable RGB pixel array made from the super popular WS2812. Control is a NodeMCU board running micropython, the NodeMCU board is a ESP8266 broken out in a sensible way, it means I can get this cool little project on the network.

I have some pieces of clear acrylic and foam sandwiched together to diffuse the light, all held everything together with some bolts. Here is the small test script I have put together so far:

import machine
import neopixel
import time

pin = machine.Pin(14, machine.Pin.OUT)

np = neopixel.NeoPixel(pin, 64)

skull = [
0,0,1,1,1,1,1,0,
0,1,1,1,1,1,1,1,
1,0,0,1,0,0,1,1,
1,0,0,1,0,0,1,1,
0,1,1,0,1,1,1,0,
0,0,0,1,1,1,0,0,
0,0,0,1,0,1,0,0,
0,0,0,0,0,0,0,0,]

while True:
    colour = uos.urandom(3)

    for x in range(len(skull)):
        if skull[x]:
            np[x] = colour

    np.write()
    time.sleep(1)

I am going to accept any 8x8 RGB frame (any 192 byte packet) that is sent and take any other shorter packet and use it to set the colour on the skull. I will include a timeout to change the colour so it isn't just a static panel.

If I find a load of spare time between the sofa cushions I will throw together a web interface.

Reading: Virtual Light

Ubuntu 16.10 on the HP Stream 7

Sat, 10 Dec 2016 00:00:00 +0000

Yesterday I finally got sick of windows 10 and installed Ubuntu 16.10 on the HP Stream 7. There are load of instructions out there to install earlier versions, but nothing up to date.

Intel created a world of pain when it introduced 32bit UEFI for 64bit machines, the Stream 7 is one of many Baytrail devices that have this ridiculous boot firmware configuration. Linux distros have completely ignore this boot combination and 3 years after these devices started appearing haven't started to ship the bootia32.efi files on the install media.

Thankfully there are people spinning builds with the correct media and a whole load of drivers to help support silly little Intel convertible tablet things. I used Linuxium's Ubuntu 16.10 build. I flashed it to a USB stick and used my awesome little otg usb hub to connect the install media. I did have to disable secure boot in the bios to get the media to boot.

I needed a separate keyboard for the install, but the touch screen and WiFi worked out of the box. The hardware support is okay, there is no screen brightness control out of the box and suspend is missing. I spent a while setting up the on screen keyboard onboard , it looks like I am going to have to create my own layout to get a split keyboard.

The install was painless, much much easier than the Baytrail device I tried to use ubuntu on in 2014. It does still baffle me that distros don't have boot support for these devices on the installers.

Reading: Nemesis Games

You are in control

Sun, 11 Dec 2016 00:00:00 +0000

pic.twitter.com/ltRfxVCVQ1
— Archillect (@archillect) December 11, 2016

It is Sunday, so that makes seven days of writing .

Reading: Nemesis Games, Idoru

FOSDEM 2017

Mon, 12 Dec 2016 00:00:00 +0000

It seems I will speaking at FOSDEM next year in the bsd devroom. I will be presenting " Transport Evolution on top of the BSD's ", whatever that means :D. I also have another talk in a similar vein submitted to the Real Time Communications devroom, it looks like my first FOSDEM will be a busy one.

Reading: Nemesis Games, Idoru

It is getting closer

Tue, 13 Dec 2016 00:00:00 +0000

It's getting closer.... \0/ #33c3 pic.twitter.com/NcnmFf2mwM
— Edwin van Andel (@Yafsec) December 13, 2016

Okay, just 14 days to congress how is your preparation going?

I have three projects coming with me:

RGB Pixel Display
- I have the panel built and some initial code running. I still want to connect the panel to the network and do some other cool effects, this has a good start.
Slow TV
- hibby and I spent a couple of hours playing with VLC and python on Sunday. We have a script that can send out a multicast video stream which we can pick up in vlc. We need to get separate audio working and video playlists going before we can say this is ready.
Some sort of display showing:
- I am going to set up a pi today, it will boot into rainbowstream running with image on terminal. I will connect that to the cheap pico projector I have and then I will call it 'done'.

Reading: Nemesis Games, Idoru

33C3 Lightning talk

Wed, 14 Dec 2016 00:00:00 +0000

I got the confirmation email today, I will be presenting a lightning talk about internet transport at congress. There are about one hundred billion lightning talks at each congress spread over three days, the bar for entry is much lower than a real track talk. I am happy to be included with the likes of the hacker yoga guy from camp. The lightning talk reveals the secret fourth planned item for my trip to hamburg.

With my FOSDEM talks and congress I have been preparing 'external' facing presentations a lot this month. I am now sure that there isn't any fixed length of talk that really works. 55 minutes is a lot of time to speak for, writing a coherent story that will come across in that amount of time is hard.

And yet, a 5 minute lightning talk slow is a horrible thing! There isn't much time to speak, which means there is almost no time at all to get your problems out and your solutions in order.

I am quite sure the lightning talks are live streamed, they are certainly recorded. I will post a link to the timeslot once I know when it is, the live stream just before it happens and the video once it is posted.

Reading: Nemesis Games, Idoru

Wellington Suspension Bridge

Thu, 15 Dec 2016 00:00:00 +0000

Unable to remember the name of the Wellington Suspension Bridge I came across this awesome website that documents the 'Doric Columns' . The site is full of history about Aberdeen and the local area, including old photographs, paintings and etching of the local infrastructure. This Etching of the Brig o' Balgownie gives a real impression of the extent of the land reclaimed from the Sea in Aberdeen.

Reading: Nemesis Games, All Tomorrow's Parties

Puffins

Fri, 16 Dec 2016 00:00:00 +0000

Today was a very slow start, staying in bed for an extra hour really didn't help me out at all today. Normally the end of the year is quite calm, all of the deadlines seem to have concentrated themselves at the start of next year. Time to work on interesting, but not pressing problems probably won't exist next year, as much as possible has to happen in the next week.

That does make preparation for congress very interesting.

Reading: Nemesis Games, All Tomorrow's Parties

Attacking Wifi with Wireshark

Sat, 17 Dec 2016 00:00:00 +0000

For a thing , I want to dump the wlan traffic between an Android app and a wifi camera. It isn't hard to grab network traffic from Android, if you have a rooted device you can just run tcpdump . tcpdump on Android is annoying, you have to manage the pcap files and it isn't clear what you are capturing.

Thankfully, wireshark can be fed WPA and WEP keys , making snooping as a third party an absolute breeze. The key options are in the protocol preferences for IEEE 802.11 , they look something like this:

wep:a1:b2:c3:d4:e5
wpa-pwd:MyPassword:MySSID
wpa-psk:0102030405060708091011...6061626364

The protocol preferences dialog doesn't seem to do any validation of the keys, instead I had to restart wireshark to get the super unhelpful error message.

The wireshark guide mentions the wireless toolbar, but this wasn't available on my platform and I didn't need it. With just the key, WEP traffic can be decrypted. WPA traffic requires that you capture an EAPOL handshake first. The easiest way to do that is observe the device keying, for testing I just had my phone join the network.

Reading: Nemesis Games, All Tomorrows Parties

Weekend off

Sun, 18 Dec 2016 00:00:00 +0000

https://t.co/zaXVqEBQtQ pic.twitter.com/sKRhJoBvUU
— Archillect (@archillect) December 18, 2016

This weekend has to be considered a weekend off. I played with a wifi camera yesterday, but I only made a dent in the project. Today was a final(ish) run at Christmas shopping, in all I did almost nothing all weekend. It feels great.

Next week will be a hectic run to get work done, and projects ready for congress.

It is Sunday, so that makes seven days of writing .

Reading: Nemesis Games, All Tomorrows Parties

UDP Panel ✓

Mon, 19 Dec 2016 00:00:00 +0000

Okay, one CCC project done. The panel now accepts data via UDP , if you send enough it will reset the whole panel, to something. It doesn't do what I want, but what it does right now is much much cooler than what I planned to do.

If I get time during congress I will do something more I guess. Here is all of the code so you can make your own and play a long at home.

import machine, neopixel, time, socket

LEDCOUNT = 64
skull = [
0,0,1,1,1,1,1,0,
0,1,1,1,1,1,1,1,
1,0,0,1,0,0,1,1,
1,0,0,1,0,0,1,1,
0,1,1,0,1,1,1,0,
0,0,0,1,1,1,0,0,
0,0,0,1,0,1,0,0,
0,0,0,0,0,0,0,0,]

def chunks(l, n):
    n = max(1, n)
    return [l[i:i + n] for i in range(0, len(l), n)]

if __name__ == "__main__":
    addr = "0.0.0.0"
    port = 6969

    pin = machine.Pin(14, machine.Pin.OUT)
    np = neopixel.NeoPixel(pin, LEDCOUNT)

    print("receiving from {} {}".format(addr, port))

    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, True)

    sock.settimeout(1.0)
    sock.bind((addr,port))

    while True:
        try: 
            pkt,addr = sock.recvfrom(1024) #blocks
            print("addr {}".format(addr))
        except OSError: 
            pkt = b""

        colours = chunks(pkt, 3)
        if len(pkt) == 3*LEDCOUNT:
            for x in range(LEDCOUNT):
                np[x] = colour
        else:
            colour = (0,0,0)
            if not len(colours) % 3:
                colour = uos.urandom(3)
            else:
                colour = colours[0]

            for x in range(len(skull)):
                if skull[x]:
                    np[x] = colour
        np.write()
        time.sleep(0.16)

SlowTV is going slowly(lol), the project is all set up. I need to figure out how to get the pi to output at the teeny resolution it supports.

Reading: Nemesis Games, All Tomorrows Parties

Last Day!

Tue, 20 Dec 2016 00:00:00 +0000

pic.twitter.com/dkxJFVMZLR
— Archillect (@archillect) December 20, 2016

I played with the wifi camera last night, but I couldn't get my phone to connect to it when my laptop was in monitor mode. That was perplexing enough to hold up anything I was trying to do. I might try again tonight with something that isn't a mac, verifying I can intercept phone traffic is step 1 in this project.

This is my last day in the 'office' this year, apparently I am too late to wish folk a good new year as I am the only person in today.

Reading: Nemesis Games, All Tomorrows Parties

What? Internet is Evolving

Wed, 21 Dec 2016 00:00:00 +0000

Checking the FOSDEM instance of the terrible pentabarf submission system I see that my second talk proposal to a devroom has been accepted. I think, they haven't timetabled the room yet so I can't link to a timetable slot for the talk.

I have three talks coming up:

33C3 Lightning Talk
FOSDEM BSD devroom talk
FOSDEM Real Time Communications devroom Talk

All three of these talks are going to tell the same story, laid out in different ways: The Internet is Broken, Fixes are hard to deploy, Developers won't use new protocols, We have a solution.

The changes that are happening in the internet right now are really interesting, but it is really hard to get over the knowledge curve required for the solutions to make sense. It is really common to hear, "The Internet works fine, why are you trying to fix it", from people that really should know better.

If you want to find out the what and why that is internet transport evolution you should find one of these talks and watch it. Hopefully they will all be recorded and online after the events.

If you want to know more you can email or track me down in IRC.

Reading: Nemesis Games

RSS Feed Reader

Thu, 22 Dec 2016 00:00:00 +0000

For a long while I have wanted a bot to sit in irc channels spitting out updates on rss feeds. I Think I have finally found all the pieces I need to write a bot in the way I like.

I have written up a really simple IRC client on top of asyncio, but it needs much more testing than a day of dev. I want to have ssl running before I try and run the bot against anything.

The RSS code is much simplier:

import aiohttp
import asyncio
import async_timeout
import feedparser

import pprint

INTERVAL = 60

async def fetch(session, url):
    with async_timeout.timeout(10):
        async with session.get(url) as response:
            return await response.text()

async def fetchfeeds(loop, feedurls, ircsock):
    last_entry = None

    feeds = []

    for url in feedurls:
        feeds.append({'url':url, 'last':""})

    while True:
        for feed in feeds:
            async with aiohttp.ClientSession(loop=loop) as session:
                html = await fetch(session, feed['url'])
                rss = feedparser.parse(html)
                if feed['last']:
                    if feed['last']['title'] != rss['entries'][0]['title'] and feed['last']['link'] != rss['entries'][0]['link']:
                        print("new entry")
                        feed['last'] = rss['entries'][0]

                        print("MSG {}".format(feed['last']['title']))
                        print("MSG {}".format(feed['last']['link']))
                else:
                    feed['last'] = rss['entries'][0]

        await asyncio.sleep(INTERVAL)

loop = asyncio.get_event_loop()
loop.run_until_complete(fetchfeeds(loop, ['https://n-o-d-e.net/rss/rss.xml',
    "http://localhost:8000/rss.xml"], None))

This is really only a proof of concept, there needs to be much more error handling before I would expect this to run for long.

Reading: Nemesis Games

Feed Testing

Thu, 22 Dec 2016 00:00:00 +0000

For development of the RSS IRC bot I need an RSS feed that updates frequently, I could subscribe to a HN feed or something from reddit. I did a quick search for something high speed and found a SO thread][1] with exactly what I need.

The linked heroku app can provide a feed updating at any interval you want. Turns out this is great for soak testing code over night.

Reading: Nemesis Games

Bags packed, Leaving

Sat, 24 Dec 2016 00:00:00 +0000

Weather is getting cold, the storms have been beating the house.

Time to leave for somewhere, well somewhere not warmer, just different. Packing for congress was made much harder by ridiculous shipping regulations, an extra couple of bottles in my bag are worth it for the fun of a buckfast party night.

The pico projector was one of the first victims in the packing war, having spent more time with this projector I don't think it is going to be a big loss. I still have the udp panel and the slowtv project, with the lightning talk I probably have enough details to worry about at congress..

Reading: Nemesis Games

Outbound

Mon, 26 Dec 2016 00:00:00 +0000

I think the guy that brought my breakfast told me off for the table I was using. He said something, thankfully my amazing Shure SE215 isolating earphones blocked all of the sound out. I have a reason to be ignorant. I am sitting in a bar place, but I bought food so I could sit at a sensible height and type.

I figure the main purpose of airport seating is to stop you complaining and to keep you awake. The lack of somewhere to sit your laptop and type like a normal human is infuriating.

This coffee isn't even good.

ABZ->LHR->HAM

Reading: Nemesis Games, Or Nothing

And by nothing I mean I don't think I will get any more reading done this year.

Day 0

Mon, 26 Dec 2016 00:00:00 +0000

It starts!

Flight worked out well, the delay I had setting off from Aberdeen shortened my transfer, but it didn't hold anything up. I made it into the congress center around half three and was too early to get a ticket.

Rest of the day went into setting up blinkenlights and other important projects. Tomorrow I will explore the place and see what is going on.

Day 1

Tue, 27 Dec 2016 00:00:00 +0000

The first day and their are some excellent sessions lined up. All of the talks are recorded, I normally catch up with the talks that catch my interest after the event.

At congress it is best to hit the self organised sessions, they aren't recorded and are almost always excellent. Because they are excellent they are really hard to attend, loads of other people show up. The good sessions are standing room only with the corridor completely full too.

- [Are_decentralized_services_unable_to_innovate][1]
- [Mechanical_Keyboard_Meetup Tryout][2]
- [We Fix the Net][3]

In the past I lined up a busy schedule for the congress and ended up not going to any of the sessions. This year I am going to try and drop in and out of events, the sessions above are more of an intent that a plan.

The We Fix the Net session is really interesting, instead of a single event they have an afternoon of panels lined up. They are more focused on security aspects than transport, it should be a highlight of the event.

Reading: The Fahrplan, if anything

33c3 Day 2

Wed, 28 Dec 2016 00:00:00 +0000

Day 1 became Day 2 with the industry standard partying all night transition. This morning was a very slow start, with my lightning talk somewhere in there, talk came out okay I think.

Keeping track of time in here is really difficult, the leds sort of merge everything together, windows would ruin the atmosphere so that isn't available as a measure of time. We know that day lights hurts hackers brains.

So far things have been a flop on the project front. The congress network doesn't support multicast on the wireless, the wired segment is fine, but it has caused us to run out of steam. Multicast traffic on the wifi has to be sent at the lowest rate connected clients support, this burns a lot of airtime leaving multicast blocked on wifi access points.

The UDP panel hasn't been set up yet, the projector didn't make the trip across.

Reading: Fahrplan

33c3 Day 3

Thu, 29 Dec 2016 00:00:00 +0000

Hackaday are covering 33c3 , mostly talks so far, but there may also be articles about the all the amazing projects that fill up the CCH. There are so many awesome internet controlled projects around here that it is probably impossible to see all of them. The contents of the rooms in the hack center is changing all the time as well.

I think today I am going to see how many network blinkenlights projects I can find and make a little catalog. A metablinkenlights controller would be awesome to build out.

Reading: Fahrplan

Rumble Mesh networking

Thu, 29 Dec 2016 00:00:00 +0000

Congress is a great place to try out apps or networking things that require a lot of people involved. All over the building there are posters up with apps, network services, political action, calls for poets, puzzles, manifestos. Following up on these ideas could fill your time at congress.

One poster than caught my eye was a call to use a decentralised mesh networked micro blogging service. The post links to an app called Rumble that is available with Android and iOS. There are enough people willing to try things out at congress that a meshed messaging app could be great fun to use.

Unfortunately it seems the app can't handle the network conditions at congress. The meshing can work over wifi or via bluetooth. I suspect the mesh over the wifi uses something like mdns for neighbour discovery. We found when we tried the SlowTV that multicast is blocked on the wifi for performance reasons.

The bluetooth option for the app seems unable to find any neighbours in the hackcenter either. It might be that the rf conditions are making this nearly impossible.

I will keep trying to play with the app after the event, but it would have been awesome if it had been usable at congress.

Network Blinkenlights

Thu, 29 Dec 2016 00:00:00 +0000

Blinkenlights are really big in hacker culture, the hackcenter where our table is located is completely full of led strips, installations, projectors and a ton of other things that glow, flash or blink.

ATTENTION
This room is fullfilled mit special electronische equippment.
Fingergrabbing and pressing the cnoeppkes from the computers is allowed for die experts only!
So all the “lefthanders” stay away and do not disturben the brainstorming von here working intelligencies.
Otherwise you will be out thrown and kicked anderswhere!
Also: please keep still and only watchen astaunished the blinkenlights.

The obvious think to do with blinkenlights is to get them onto the network, my udp panel continues this glorious tradition. There are loads of awesome blinkenligths on the network:

The Milliways sign is made from addressable led strips mounted on a frame. There is an awesome ethernet connected controller board that drives all of the leds.

There is a group of hackers sat in front of the flipdot sign day and night playing with it. The flipdots are small electromechanical modules that can be either white or black, the modules take a fraction of a second to swith and make an awesome sound as they do.

The flipdot sign is controllable from a (probably tempory) website .

33c3 Day 4

Fri, 30 Dec 2016 00:00:00 +0000

Day 4 is the sad day, at a certain time this evening a switch will flip, everyone will stop what they are doing and start packing up. Right away the hackcenter will go from being another world back to a boring hall.

Reading: Fahrplan

Books I read in 2016

Fri, 30 Dec 2016 00:00:00 +0000

Presented with out comment are the books I read in 2016, ordered with the most recently read first:

* All Tomorrow's Parties
* Idoru 
* Virtual Light
* Excession
* Cibola Burn
* Reamde
* Abandon's Gate
* Seveneves
* ELEKTROGRAD
* Ashes of Victory
* Little Brother
* Transmetropolitan Book Vol 3
* Transmetropolitan Book Vol 2
* Transmetropolitan Book Vol 1
* Transmetropolitan Book Vol 10
* Transmetropolitan Book Vol 9
* Transmetropolitan Book Vol 8
* Transmetropolitan Book Vol 7
* Transmetropolitan Book Vol 6
* Transmetropolitan Book Vol 5
* Transmetropolitan Book Vol 4
* The Cuckoos Egg
* Networks of New York
* Hydrogen Sonata
* Surface Detail
* Matter
* Look to Windward
* Inversions
* Use of Weapons
* The Player of Games   
* Consider Phlebaa
* The Man in the High Castle
* Overtime
* Eqoid
* Down on the Farm
* Neuromancer 
* The Soul of a New Machine
* The Wise Man's Fear
* Name of the Wind 
* Zero History 
* Spook Country 
* The Atrocity Archives
* The Fuller Memorandum
* The Jennifer Morgue 
* The Apocalypse Codex
* The Annihilation Score
* The Rhesus Chart
* The Long Dark Tea Time of the Soul
* Dirk Gently's Holistic Detective Agency
* Titus Groan
* Next Stop Execution
* Mona Lisa Overdrive
* Pattern Recognition
* Gormenghast
* Cunning Plans
* Tubes

Stranded in Hamburg

Sat, 31 Dec 2016 00:00:00 +0000

An abundance of particles in the air forced BA to cancel my flights home. They knocked my flight back 24 hours without any choice in the timing of my new flight. This means I get to spend an extra night in Hamburg and I get to move all of my new year plans to somewhere in the future.

Congress is done for another year, it has been any amazing event. There really is nothing like it on the planet, attempts at describing the event and conveying that different world always seem to fail.

The Chaos Communication Congress really is a place that must be seen, with the CCH being knocked down next year the event you go to will certainly be materially different that the one I have attended for the past three years.

I cannot wait to see how the deal with loosing the CCH and where CCC ends up in the future.

Reading: PO-14 User Manual

Inbound

Sun, 01 Jan 2017 00:00:00 +0000

I was hoping to write the first blog post of this year from the airport, but time conspired against me. With BA moving my flight 24 hours I had to decided between having a quiet New Year and a fun one.

I certainly did not go to 'There is No Party' which was not held somewhere in HH. The music was excellent, the crew that did the lighting and audio did amazing work. It makes me wonder what could happen here if there was space where people could play.

From the party I headed back to the apartment, packed and set off for the airport. I turned a long day with a weird sense of time into an adventure across Hamburg at New Year and through the Airport.

By the time I made Heathrow I was on about 4 hours sleep in a 36 hour window, I opted to nap instead of writing..

It is Sunday, so that makes seven days of writing .

Reading: Nemesis Games

HAM->LHR->ABZ

MacOS Malware

Mon, 02 Jan 2017 00:00:00 +0000

14 hours of sleep, I feel like I have woken up in another dimension.

There is a ton of congress stuff floating around twitter, Here is a list of talks ranked from the number of tweets and retweets mentioning them. If you are only going to watch a couple of sessions from 33c3 that list is probably great.

I caught an awesome article classifying the MacOS malware found in 2016. I am glad there is so little malware aimed at this platform.

Reading: Nemesis Games

A little more Chaos

Tue, 03 Jan 2017 00:00:00 +0000

Chaos is an important part of CCC, most of the best things that happen are pranks that only a small number of people experience. The Fnord News Show has a large audience German speaking audience, I am pretty sure this awesome 'event' is unknown outside of the German crowd.

Reading: Nemesis Games

33c3 Talks

Wed, 04 Jan 2017 00:00:00 +0000

God damn it! I won't be downloading all the 33c3 talks this year to watch offline, instead I will stream them from the excellent media.ccc.de . No good reason, I am only doing this because when making a list to feed to wget I did:

$ cat 33c3list.txt| grep -v deu | wc -l > tmp.txt
$ mv tmp.txt 33c3list.txt

I didn't really have the disc spare to store 100GB or so of talks anyway. I will stream the videos in my browser instead. I don't really have set approach to watching the CCC talks. I normally work through the list watching things that other have said were good, or talks whose title catches my eye.

Reading: Nemesis Games

33c3 Hardware Hacking

Thu, 05 Jan 2017 00:00:00 +0000

bunnie has a long history of doing really cool things in hardware hacking, his book Hacking the Xbox is a great read (he has another book in the works too ). bunnie and xobs presented a complete tear down and reverse engineering of sd cards at 30c3, at 33c3 they were back talking about their education project chibitronics .

bunnie's talk is about the project it self, technical design and motivations, if the front matter of the talk turn you off believe me when I say it is worth powering through and watching the whole thing.

xob's presents an excellent session of bit banging out usb from a low power Cortex-M0+ microcontroller. This talk is a great introduction into the low level details of the usb protocol.

Reading: Babylon's Ashes

33c3 Spaaaaaaaaaaace!

Fri, 06 Jan 2017 00:00:00 +0000

The CCC put together an excellent track of talks about Science and Space technology. I chewed through a lot of them yesterday, they set a really great tone and are aimed really well at their audience.

I have been thinking recently about organising events locally that have much more technical content than the current things that happen. Up here there isn't the density of expertise required to run a monthly or even quarterly event without running out of fresh speakers very quickly.

Techmeetup Aberdeen really struggles to bring speakers in, very have many times falling back to a set of 'known good' speakers from the local hackerspace .

Sessions by experts in a field with technical content, aimed at Non Cyber Muggles from other fields (similar pitching as the space track talks) could work very well. I will have to play with this idea and see if people from other fields are interesting in taking part.

Reading: Babylon's Ashes

Hot Adventure

Sat, 07 Jan 2017 00:00:00 +0000

What do you do when you find a USB stick on the ground?

Clearly you take it to work, plug it into a computer with network admin privileges to make sure there is nothing funny about it.

I guess something could go wrong, I saw a documentary once where criminals dropped a load of USB sticks on the ground which an unsuspecting prison guard used in a computer. They probably put some malware on that USB stick and all, not cool.

Anyway, at congress I saw this sign, sans stick. I hope there was both something horrible on it and something that made it worth the hassle.

Reading: Babylon's Ashes

Loch Brandy

Sun, 08 Jan 2017 00:00:00 +0000

Went up a hill today.

It was a change from sitting inside, the view was really nice. On the way up I was thinking about photography and finding the right equipment. It is pretty clear my J1 with a 10mm pancake lens isn't ideal for landscape photography, but I am not really sure how to get a set of gear to make the photos I want to take possible.

Sitting down with books and reviews are the obvious way to figure this out, but maybe there is a more 'fun' solution. Here's an idea for free:

We take in the camera equipment you already have.
You go through flickr, 500px or something else and tag photos you wish you had taken.
We parse out the lens/camera used
We recommend the gear to help take the photos you want

Skill will have to be provided by the user.

It is Sunday, so that makes seven days of writing .

Reading: Babylon's Ashes

State of Interner Censorship

Mon, 09 Jan 2017 00:00:00 +0000

One of the speakers asks the audience early on 'Do you think Internet Censorship should be allowed?' and gets about half the crowd showing hands. I really cannot understand that sort of response, clearly there are things we don't want people to see, but I can't support a blanket censorship system to block that content.

If there was a way to block really dangerous material, without risking blocking completely reasonable material I am sure that is what we would be implementing.

The slides for the presentation are here , there are some other slides here .

Reading: Babylon's Ashes

This podcast it is quite nice.

JTAG on USB3

Tue, 10 Jan 2017 00:00:00 +0000

Physical access is pretty much always game over, apart from the iPhone there are not many devices that can stand up to attack. Intel seem to want to make physical access even easier and are now offering JTAG access on USB.

JTAG is a hardware debugging protocol normally seen on embedded systems or accessed through a special adapter on the motherboard. You can use JTAG to pause a processor, step through the instructions being executed and read into memory. With JTAG access you have full access to the machine.

Reading: Babylon's Ashes

c720 Trackpad set up

Tue, 10 Jan 2017 00:00:00 +0000

I reinstalled or upgraded my c720 or something and things are a bit all over the place. Tonight I started firefox in the hackerspace and noticed my trackpad wasn't working, it needs to be explicitly setup. This is mentioned on the comprehensive FreeBSD c720 guide , but there have been some updates to the driver that aren't reflected on the page. You now need to load the chromebook_platform driver manually.

# kldload chromebook_platform
# kldload ig4
# kldload cyapa

The cyapa driver offers all the features you would want from a trackpad, two finger dragging, thresholds for taps and an three button mouse emulation mode.

# sysctl debug.cyapa_enable_tapclick=3

Which gives me the following awesome mouse button layout on the trackpad.

        Trackpad layout

         2/3               1/3
+--------------------+------------+
|                    |   Middle   |
|                    |   Button   |
|       Left         |            |
|      Button        +------------+
|                    |   Right    |
|                    |   Button   |
+--------------------+............|
|     Thumb/Button Area           | 15%
+---------------------------------+

Also disable super danger mode:

# echo "hw.acpi.power_button_state=NONE" >> /etc/sysctl.conf

William Binney

Wed, 11 Jan 2017 00:00:00 +0000

My head is pretty full writing slides for FOSDEM. Here is an interview with William Binney , if you don't know of Binney this interview is a great introduction. Binney is credited by Snowden as one of the motivators behind his set of leaks.

Binney also gave the keynote at Hope 9, which is a great watch.

Reading: Babylon's Ashes

33c3 Wireshark Workshop

Thu, 12 Jan 2017 00:00:00 +0000

I use Wireshark quite all the time. I was lucky to get a copy of Hacking: The Art of Exploitation when I was a teenager, the book gave me an excellent introduction to using tcpdump to perform network analysis. tcpdump is the first tool I reach for when I wonder where the packets are going, but for anything higher level (breaking down http, checking wlan flags) I use wireshark , I am always impressed.

At 33c3 there was a wireshark introductory self organised session run by kirils . I did not go to this session, but the slides I found look to be an excellent introduction to using wireshark .

Reading: Babylon's Ashes

Spooky Friday

Fri, 13 Jan 2017 00:00:00 +0000

It is Friday the 13th, wooooo spooky!!!!

Rudy_Giuliani was nominated Cyber Tzar or something yesterday, the hacker community suddenly became very interested in this credentials. This morning twitter was filled with the results of int gathering exercises.

Found on /r/sysadmin, presented without comment. pic.twitter.com/UmWe7tHURv
— Ryan Castellucci (@ryancdotorg) January 12, 2017

Giuliani uses bad SSH key.

h/t @n0x00 pic.twitter.com/GI2l90wuty
— Rob Graham (@ErrataRob) January 13, 2017

The domain now points to localhost, someone clearly got a late night phone call. It is strange that only now is noise being made about this, Ruddy isn't exactly a popular figure in America. He made a lot of mistakes in high profile positions. The big scary guys in the Int agencies will have pursued all these leads a long time ago.

Of course, that is assuming the site wasn't a honeypot.

Reading: Babylon's Ashes

Walking

Sat, 14 Jan 2017 00:00:00 +0000

Thought we had hit all of the peaks on Bennachie , but looking at stuff later it seems there are about 7 'summits' to hit. That's annoying, living in Aberdeen I have done the Mither Tap walk loads of times. Today was my first time taking the trek over to Oxencraig.

That was most of today, I poked some wireless driver stuff, but it is all initial steps.

Reading: Babylon's Ashes

Fairy Dust

Sun, 15 Jan 2017 00:00:00 +0000

It is Sunday, so that makes seven days of writing .

Reading: Babylon's Ashes

PO-14

Mon, 16 Jan 2017 00:00:00 +0000

I was given a Teenage Engineering PO-14 for Christmas and took it with me to congress for entertainment on the way. The pocket operator has a load of functions hidden behind very few buttons, I had a lot of fun playing with it on the flight. I am still to really figure out everything this board can do.

Watching some OP-1 videos (their much bigger synth) TE manage to pack a ton of functionality into hardly any keys.

Reading: Babylon's Ashes

BIOS Engines

Tue, 17 Jan 2017 00:00:00 +0000

I read this cool article on trying to get the Purism laptop booting with coreboot instead of the proprietary bios. Quite a lot of people having been trying to open up the Intel hardware ecosystem in the past few years, all those closed bits make it very hard to say that hardware is secure.

It would be nice if we could leave the Intel world and use ARM or MIPS processors, but I think the graphics situation holds us back.

Reading: Babylon's Ashes, Cryptonomicon

Loch Brandy

Wed, 18 Jan 2017 00:00:00 +0000

Reading: Babylon's Ashes, Cryptonomicon

Are you a real hacker?

Thu, 19 Jan 2017 00:00:00 +0000

Are you a fed? You have to tell me if you are. pic.twitter.com/mMtmNA8oOW
— GonzoHacker (@GonzoHacker) January 19, 2017

Pretty sure I am not, I lost the git branch with a new feature on it and it took an hour to find. I didn't delete it, I just couldn't remember where it was.

Reading: Babylon's Ashes, Cryptonomicon

5GHz Problems

Fri, 20 Jan 2017 00:00:00 +0000

Somebody's illegally configured 5GHz wireless is making a mess of the @MetService rain radar. pic.twitter.com/10noLJuGyG
— Steve Biddle (@stevebiddle) January 19, 2017

Interference is a bitch, you should heed the warnings about cheap Chinese video systems they can make a lot of noise. This weather sat didn't stand a chance, Here is an article explaining what is going on.

Reading: Babylon's Ashes, Cryptonomicon

Setting SYSDIR

Sat, 21 Jan 2017 00:00:00 +0000

Poking at the mt7620 wifi driver today, but I don't have a FreeBSD source tree in /usr/src. Trying to build spits out this message:

$ make
make: "/usr/share/mk/bsd.kmod.mk" line 12: Unable to locate the kernel source tree. Set SYSDIR to override.

Searching around, I could find others with this problem, mostly they had had forgotten to checkout a source tree into /usr/src . With a source tree in /home/user/code/freebsd I needed to set SYSDIR.

SYSDIR must point to the sys subdir in the FreeBSD source rather than the location of the whole tree(i.e. /usr/src). I modified my module Makefile list so:

SRCS=bus_if.h device_if.h opt_usb.h usbdevs.h if_run.c
KMOD=run_mt
SYSDIR=/home/user/code/freebsd/sys

.include <bsd.kmod.mk>

Reading: Babylon's Ashes, Cryptonomicon

Tracing Call Graphs

Sun, 22 Jan 2017 00:00:00 +0000

Modern IDEs have a load of functionality to help trace function call and data accesses through large code bases. cscope is an interactive command line tool that helps with searching codebases based on C symbols. With cscope you can find all the callers of a function, every function a function calls, or by C type.

$ cd code/repo
$ cscope -R

Working on this wireless I have spent a lot of time digging down the callgraph with cscope manually figuring out how things tie together. Today I looked to see if there was a tool I could use to generate a callgraph from the cscope database files.

I found a stackoverflow thread with the recommendation of a shell script that could generate a dot file with the callgraph. The script is unfortunately very basic, rather that something to run against a code base it is a set of bash functions.

$ . ~/tmp/calltree.sh               # load functions in
$ _relate rt28xx_open RTUSBWriteMACRegister| dot2png out.png

The graph below was ( generated from this repo) took about 1 minute to generate on my reasonably fast laptop. You will need to install the graphviz tools to generate the png.

I had to make some modifications to get the script to run, here is my version:

#!/bin/bash

echo "loading calltree.sh functions"

#use cscope to build reference files (./cscope.out by default, use set_graphdb to override name or location)
set_graphdb() { export GRAPHDB=$1; }
unset_graphdb() { unset GRAPHDB; }
build_graphdb() { cscope -bkRu ${GRAPHDB:+-f $GRAPHDB} && echo Created ${GRAPHDB:-cscope.out}...; }

# cscope queries
lsyms() { cscope -R ${GRAPHDB:+-f $GRAPHDB} -L0 $1 | grep -v "<global>" | grep "="; }
fdefine() { cscope -R ${GRAPHDB:+-f $GRAPHDB} -L1 $1; }
callees() { cscope -R ${GRAPHDB:+-f $GRAPHDB} -L2 $1; }
callers() { cscope -R ${GRAPHDB:+-f $GRAPHDB} -L3 $1; }

# show which functions refer to a set of symbols
filter_syms() { local sym cscope_line
    while read -a sym; do
        lsyms $sym | while read -a cscope_line; do
            printf "${cscope_line[1]}\n"
        done
    done
}

# given a set of function names, find out how they're related
filter_edges() { local sym cscope_line
    while read -a sym; do
        fdefine $sym | while read -a cscope_line; do
            grep -wq ${cscope_line[1]} ${1:-<(echo)} &&
            printf "${cscope_line[1]}\t[href=\"${cscope_line[0]}:${cscope_line[2]}\"]\t/*fdefine*/\n"
        done
        callees $sym | while read -a cscope_line; do
            grep -wq ${cscope_line[1]} ${1:-<(echo)} &&
            printf "$sym->${cscope_line[1]}\t[label=\"${cscope_line[0]}:${cscope_line[2]}\"]\t/*callee*/\n"
        done
        callers $sym | while read -a cscope_line; do
            grep -wq ${cscope_line[1]} ${1:-<(echo)} &&
            printf "${cscope_line[1]}->$sym\t[label=\"${cscope_line[0]}:${cscope_line[2]}\"]\t/*caller*/\n"
        done
    done
}

# dump args one-per-line
largs() { for a; do echo $a; done; }
toargs() { local symbol
    while read -a symbol; do
        printf "%s " $symbol
    done
    echo
}

# present list of symbols to filter_syms properly
refs() { local tfile=/tmp/refs.$RANDOM
    cat ${1:+<(largs $@)} > $tfile
    filter_syms $tfile <$tfile | sort -u
    rm $tfile
}

# present list of function names to filter_edges properly
edges() { local tfile=/tmp/edges.$RANDOM
    cat ${1:+<(largs $@)} > $tfile
    filter_edges $tfile <$tfile
    rm $tfile
}

# append unknown symbol names out of lines of cscope output
filter_cscope_lines() { local cscope_line
    while read -a cscope_line; do
        grep -wq ${cscope_line[1]} ${1:-/dev/null} || echo ${cscope_line[1]}
    done 
}

# given a set of function names piped in, help spit out all their callers or callees that aren't already in the set
descend() { local symbol
    while read -a symbol; do
        $1 $symbol | filter_cscope_lines $2
    done
}

# discover functions upstream of initial set
all_callers() { local tfile=/tmp/all_callers.$RANDOM
    cat ${1:+<(largs $@)} > $tfile
    descend callers $tfile <$tfile >>$tfile
    cat $tfile; rm $tfile
}

# discover functions downstream of initial set
all_callees() { local tfile=/tmp/all_callees.$RANDOM
    cat ${1:+<(largs $@)} > $tfile
    descend callees $tfile <$tfile >>$tfile
    cat $tfile; rm $tfile
}

# all the ways to get from (a,b,...z) to (a,b,...z), i.e. intersect all_callers and all_callees of initial set
call_graph() { local tfile=/tmp/subgraph.$RANDOM; local args=/tmp/subgraph_args.$RANDOM
    cat ${1:+<(largs $@)} > $args
    cat $args | all_callers | sort -u > $tfile
    comm -12 $tfile <(cat $args | all_callees | sort -u)
    rm $tfile $args
}

# all functions downstream of callers of argument
all_callerees() { callers $1 | filter_cscope_lines | all_callees; }

# odd experimental set of calls that might help spot potential memory leaks
call_leaks() { local tfile=/tmp/graph_filter.$RANDOM
    all_callerees $1 | sort -u > $tfile
    comm -2 $tfile <(all_callers $2 | sort -u)
    rm $tfile
}

# wrap dot-format node and edge info with dot-format whole-graph description
graph() { printf "digraph iftree {\ngraph [rankdir=LR, ratio=compress, concentrate=true];\nnode [shape=record, style=filled]\nedge [color="navy"];\n"; cat | sort -u; printf "}\n"; }

# filter out unwanted (as specified in “~/calltree.deny”) and/or unnecessary edges
graph_filter() { local tfile=/tmp/graph_filter.$RANDOM
    cat > $tfile
    grep fdefine $tfile
    grep $1 $tfile | grep -v ~/calltree.deny | cut -f1,3
    rm $tfile
}

# how to invoke zgrviewer as a viewer
zgrviewer() { ~/bin/zgrviewer -Pdot $@; }
# how to invoke xfig as a viewer
figviewer() { xfig <(dot -Tfig $@); }
# how to create and view a png image
pngviewer() { dot -Tpng $@ -o /tmp/ct.png && gqview -t /tmp/ct.png; }

# specify a viewer
ctviewer() { pngviewer $@; }

# add color to specified nodes
colornodes() { (cat; for x in $@; do echo "$x [color=red]"; done;) }

# generate dot files
_upstream() { all_callers $1 | edges | graph_filter ${2:-caller} | colornodes $1 | graph; }
_downstream() { all_callees $1 | edges | graph_filter ${2:-callee} | colornodes $1 | graph; }
_upndown() { (all_callers $1; all_callees $1) | edges | graph_filter ${2:-callee} | colornodes $1 | graph; }
_relate() { call_graph $@ | edges | graph_filter callee | colornodes $@ | graph; }
_leaks() { call_leaks $1 $2 | edges | graph_filter ${3:-callee} | colornodes $1 $2 | graph; }

# generate dot files and invoke ctviewer
upstream() { _upstream $@ > /tmp/tfile; ctviewer /tmp/tfile; rm -f /tmp/tfile; }
downstream() { _downstream $@ > /tmp/tfile; ctviewer /tmp/tfile; rm -f /tmp/tfile; }
upndown() { _upndown $@ > /tmp/tfile; ctviewer /tmp/tfile; rm -f /tmp/tfile; }
relate() { _relate $@ > /tmp/tfile; ctviewer /tmp/tfile; rm -f /tmp/tfile; }
leaks() { _leaks $@ > /tmp/tfile; ctviewer /tmp/tfile; rm -f /tmp/tfile; }

# dot file conversions
dot2png() { dot -s36 -Tpng -o $1; }
dot2jpg() { dot -Tjpg -o $1; }
dot2html() { dot -Tpng -o $1.png -Tcmapx -o $1.map; (echo "<IMG SRC="$1.png" USEMAP="#iftree" />"; cat $1.map)  > $1.html; }

It is Sunday, so that makes seven days of writing .

Doing this I also found out about vim s cscope integration.

Reading: Babylon's Ashes, Cryptonomicon

Some Posters

Mon, 23 Jan 2017 00:00:00 +0000

Reading: Babylon's Ashes, Cryptonomicon

Binary Image Analysis

Tue, 24 Jan 2017 00:00:00 +0000

I have gotten to the point with the MT76x0U driver where I need to load the firmware image onto the MCU. Unlike the older hardware on which the driver is based, the firmware image is much more complicated. The old images are 4KB and can be directly DMA'd across to the MCU, the newer image is around 80KB, contains 2 sections and the reference driver does a complicated dance to copy them across.

There is quite a lot of pointer magic in setting up the DMA buffers in the reference image, I need to understand the firmware layout to know what this is trying to accomplish.

First thing, binwalk is of no help:

$ binwalk mcu/bin/MT7610_formal_2.6.bin

DECIMAL       HEXADECIMAL     DESCRIPTION
-------------------------------------------------------------------------------

The image size is:

$ ls -l 
-rw-r--r--  1 hacker hacker  80288 Dec 21 19:40 mcu/bin/MT7610_formal_2.6.bin

The reference code does some work to extract out build info from the 32 byte header, I put together this python script to do the same:

import sys
import struct

if __name__ == "__main__":
    if len(sys.argv) != 2:
        print("usage: {} firmware.bin".format(sys.argv[0]))
        exit()

    filename = sys.argv[1]
    data = None
    with open(filename, "rb") as f:
        data = f.read()

    print("Image size: {}".fomrat(len(data)))

#    ilm_len   4 bytes
#    dlm_len   4 bytes
#    fw_ver    2 bytes
#    build_ver 2 bytes
#
#    4 bytes of something?
#
#    build_time 16 byte str starting from byte 16 (base+16) """
    hdr = data[:32]

    ilm_len, dlm_len, fw_ver, build_ver, something, build_time = struct.unpack("<IIHH4s16s", hdr)

    print("ilm_len:    {}".format(ilm_len))
    print("dlm_len:    {}".format(dlm_len))
    print("fw_ver:     {}".format(fw_ver))
    print("build_ver:  {}".format(build_ver))
    print("something:  {}".format(something))
    print("build_time: {}".format(build_time))

    print("fw version: {}.{}.{}"
        .format(
            (fw_ver & 0xf000) >> 8,
            (fw_ver & 0x0f00) >> 8,
            fw_ver & 0x00ff))

I know from the reference driver that there are two images shipped in the firmware, called ILM and DLM ( accoring to this Instruction and Data Local Memory).

$ python3.5 parsefirmware.py mcu/bin/MT7610_formal_2.6.bin 
Image size: 80288
ilm_len:    68780
dlm_len:    11476
fw_ver:     30272
build_ver:  256
something:  b'Bv\x11\x02'
build_time: b'201308221655____'
fw version: 112.6.64

The build time is happily just an ascii string, the first output in strings

$ strings ../../mcu/bin/MT7610_formal_2.6.bin| head -n 5 
201308221655____H
s@!
ELq@'P
ELq@
<@!H

and easy to spot in a hexdump

$ head -c 64 ../../mcu/bin/MT7610_formal_2.6.bin| hexdump -C
00000000  ac 0c 01 00 d4 2c 00 00  40 76 00 01 42 76 11 02  |.....,..@v..Bv..|
00000010  32 30 31 33 30 38 32 32  31 36 35 35 5f 5f 5f 5f  |201308221655____|
00000020  48 00 00 78 48 00 00 1e  48 00 00 1c 48 00 00 1a  |H..xH...H...H...|
00000030  48 00 00 18 48 00 00 16  48 00 00 14 48 00 00 12  |H...H...H...H...|

The ILM and DLM sizes are also very useful, with the header they add up to the firmware size!

Image size = hdr_size + ilm_len + dlm_len
80288 = 32 + 68780 + 11476

Currently I think the reference driver skips some further data in the ILM, I need to see if there is documentation for the format so I can make an informed guess.

Reading: Babylon's Ashes, Cryptonomicon

VR Shit

Wed, 25 Jan 2017 00:00:00 +0000

You'll be sick!

Reading: Babylon's Ashes, Cryptonomicon

The Network is susceptible to security violations

Thu, 26 Jan 2017 00:00:00 +0000

RFC602 published in December 1973:

           "The Stockings Were Hung by the Chimney with Care"


The ARPA Computer Network is susceptible to security violations for at least
the three following reasons:

(1)  Individual sites, used to physical limitations on machine access, have
     not yet taken sufficient precautions toward securing their systems
     against unauthorized remote use.  For example, many people still use
     passwords which are easy to guess:  their fist names, their initials,
     their host name spelled backwards, a string of characters which are
     easy to type in sequence (e.g. ZXCVBNM).

(2)  The TIP allows access to the ARPANET to a much wider audience than
     is thought or intended.  TIP phone numbers are posted, like those
     scribbled hastily on the walls of phone booths and men's rooms.  The
     TIP required no user identification before giving service.  Thus,
     many people, including those who used to spend their time ripping off
     Ma Bell, get access to our stockings in a most anonymous way.

(3)  There is lingering affection for the challenge of breaking
     someone's system.  This affection lingers despite the fact that
     everyone knows that it's easy to break systems, even easier to
     crash them.

All of this would be quite humorous and cause for raucous eye
winking and elbow nudging, if it weren't for the fact that in
recent weeks at least two major serving hosts were crashed
under suspicious circumstances by people who knew what they
were risking; on yet a third system, the system wheel password
was compromised -- by two high school students in Los Angeles
no less.

We suspect that the number of dangerous security violations is
larger than any of us know is growing.  You are advised
not to sit "in hope that Saint Nicholas would soon be there".

via

First, read this: https://t.co/vNjckyS1NB
Second, realize that this is from 1973 (yes, 43 years ago) pic.twitter.com/O4u7NVKuyX
— Sebastian Schinzel (@seecurity) January 26, 2017

Reading: Babylon's Ashes, Cryptonomicon

There is no lounge

Fri, 27 Jan 2017 00:00:00 +0000

I wonder if a CCC style lounge would work as a real club, I guess the status of DNA Lounge indicates it isn't a great proposition. Hacker bars for console jockeys are really appealing, the properties that would make them a nice place to be; not too crowded, room to hack, exclusive, don't really translate to a successful business.

Still, one can dream.

Reading: Babylon's Ashes, Cryptonomicon

Edge Measurements

Sat, 28 Jan 2017 00:00:00 +0000

Something for work meant I had to throw together a DSCP probing tool at the last minute. It still needs lots of work, but I needed to get out today and test on some real networks (just work and my house don't count). If I have to spend a lot of time in cafes, pubs and chains doing measurements from edges I might as well enjoy it.

If I can manage more than a couple of shots like this I can put together an excellent slide when presenting this work.

Reading: Babylon's Ashes, Cryptonomicon

On the road

Sun, 29 Jan 2017 00:00:00 +0000

Time for a quick international measurement tour!

Well no, nothing as cool. I have meeting and a hackathon in Paris this week, then on Friday I am jetting (on a train) up to Brussles for FOSDEM. Travelling gives me a chance to run my network tracing tool from lots of strange places, hopefully strange places reveal strange networks.

Now I am going to have a wander around this city for a bit, before a great day of coding tomorrow.

It is Sunday, so that makes seven days of writing .

Reading: Babylon's Ashes, Cryptonomicon

I hate phones

Mon, 30 Jan 2017 00:00:00 +0000

I wandered down to the Louvre last night and took some pictures, but my phone is refusing to speak to my laptop, so here is the Seidenstrasse instead. I am hacking from the amazing Mozilla Paris offices this week, for some reason this is the projects favourite place to meet.

Reading: Babylon's Ashes, Cryptonomicon

Travel

Tue, 31 Jan 2017 00:00:00 +0000

Reading: Babylon's Ashes, Cryptonomicon

They can't take spring

Wed, 01 Feb 2017 00:00:00 +0000

George Orwell, Some Thoughts On The Common Toad (1946). https://t.co/pD5a8GKvsj pic.twitter.com/BWSMf8Em8c
— Graham Sleight (@grahamsleight) February 1, 2017

Reading: Babylon's Ashes, Cryptonomicon

Don't Violate DiffServ

Thu, 02 Feb 2017 00:00:00 +0000

Today's #network wisdom: You have a problem. You implement #QoS . Now you have 9 problems.
— Allan Eising (@dubcroster) December 1, 2016

Reading: Babylon's Ashes, Cryptonomicon

FOSDEM Prep

Fri, 03 Feb 2017 00:00:00 +0000

Tomorrow, I am presenting at FOSDEM . Somehow I have gotten roped into doing two talks at almost the same time.

At 1555, I will present QoS Challenges for Real Time Traffic in the Real Time Communications devroom. I am pretty certain I am going to explain what the hell QoS in the internet is, some of the benefit you might see and how to do any of this with NEAT. The recent measurements we have been doing suggest that QoS shouldn't be a deal breaker for any connection, great news, but it doesn't make much of a discussion.

At 1630, I will present Transport Evolution on top of the BSD's in the BSD devroom. The two talks are in the same building, but 15 minutes is a bit tighter than I would have liked. This talk will be about the problems faced in making the internet better and how NEAT will help.

I know the BSD devroom will be recorded, but not streamed. I haven't heard anything about the RTC rooms video setup. If you are around for FOSDEM you should drop by and ask me difficult questions. If not I will take them over email.

Reading: Babylon's Ashes, Cryptonomicon

FOSDEM 2017

Sat, 04 Feb 2017 00:00:00 +0000

I did my talks today, there is video coming, but I am told the audio is out of sync. My hotel doesn't seem to be up to doing streaming video so I haven't even tried.

Here are the slides:

Which should be clickable, but I can't figure out how to trick my site to do it

Reading: Babylon's Ashes, Cryptonomicon

FOSDEM is Done

Sun, 05 Feb 2017 00:00:00 +0000

We're baaaack to talk about live tweeting. TRANSMET: Human Reaction and Criminal Enterprise #3 https://t.co/yP1N8bE5Zf @warrenellis @DarickR pic.twitter.com/qDTBELFwtO
— Comicosity (@comicosity) February 5, 2017

FOSDEM is done! I want to start writing about the things I saw, but my phone still won't speak to my laptop. Tomorrow should be more differenter, once the travel is done I will have access to sane computing.

It is Sunday, so that makes seven days of writing .

Reading: Babylon's Ashes, Cryptonomicon

Trains Home

Mon, 06 Feb 2017 00:00:00 +0000

Long journey back from FOSDEM, a train to Paris, train to the Airport and a hop through Amsterdam before making it home.

Reading: Babylon's Ashes, Cryptonomicon

Xen DNA TShirt

Tue, 07 Feb 2017 00:00:00 +0000

I got this TShirt at FOSDEM, I was told that it means something .

First here is all of the text from the tshirt, which should make the page more discoverable:

Xen 
  Project
  @FOSDEM 2017

Forging the 
DNA of the Cloud

AGTCTACTCAGCCGTACTTCATCGACTTAGCGTTGTCTCAGC
GCTTGTCACGTCCTGCTCGTCAGCTAGCGTCTAAGCTACCTC
ACTAGCTGTGCTAGCTAGCGTCTAAGCTCAATCTGTGATTAC

These three strings are a bit unweildly to work with, lets break them down into groups of three.

AGT CTA CTC AGC CGT ACT TCA TCG ACT TAG CGT TGT CTC AGC
GCT TGT CAC GTC CTG CTC GTC AGC TAG CGT CTA AGC TAC CTC
ACT AGC TGT GCT AGC TAG CGT CTA AGC TCA ATC TGT GAT TAC

Now to wikipedia to get a DNA Codon Table to refer to:

Amino acid  Codons                          Compressed                      
Ala/A       GCT GCC GCA GCG                 GCN                     
Arg/R       CGT CGC CGA CGG AGA AGG         CGN MGR     
Asn/N       AAT AAC                         AAY     
Asp/D       GAT GAC                         GAY     
Cys/C       TGT TGC                         TGY     
Gln/Q       CAA CAG                         CAR     
Glu/E       GAA GAG                         GAR     
Gly/G       GGT GGC GGA GGG                 GGN     
His/H       CAT CAC                         CAY     
Ile/I       ATT ATC ATA                     ATH                         
START       ATG                                 

Amino acid  Codons                          Compressed                      
Leu/L       TTA TTG CTT CTC CTA CTG         YTR CTN
Lys/K       AAA AAG                         AAR
Met/M       ATG
Phe/F       TTT TTC                         TTY
Pro/P       CCT CCC CCA CCG                 CCN
Ser/S       TCT TCC TCA TCG AGT AGC         TCN AGY
Thr/T       ACT ACC ACA ACG                 ACN
Trp/W       TGG
Tyr/Y       TAT TAC                         TAY
Val/V       GTT GTC GTA GTG                 GTN
STOP        TAA TGA TAG                     TAR TRA

I poked at this for ages, I wondered if I had need to the take the codons in as they are written. A column at a time. But even looking at that, there was nothing obvious about the codon sequence.

After a while I started searching for subsequences from the string. This didn't help at all, but one page on google had a related search for ['dna message translator'][3]. Annoyingly I fed in this string:

AGTCTACTCAGCCGTACTTCATCGACTTAGCGTTGTCTCAGCGCTTGTCACGTCCTGCTCGTCAGCTAGCGTCTAAGCTACCTCACTAGCTGTGCTAGCTAGCGTCTAAGCTCAATCTGTGATTAC

And got the result:

XEN HACKATHON FORGING THE DNA OF THE CLOUD

Reading: Babylon's Ashes, Cryptonomicon

FOSDEM Booths

Wed, 01 Feb 2017 00:00:00 +0000

FOSDEM is split between devrooms and booths, there are booths for a load of different projects. The Olimex DIY Laptop was announced the week before, they were showing it off at a stand at FOSDEM.

The hardware is quite nice, the case feels a little cheap, but what can you expect for that price? The keyboard they have on it is horrible. It would be much better served with much fewer keys on the layout, something more like a chromebook would be good (I would settle for 40%, but the market is probably small).

I unplugged the assembled model they had on display and they got upset and it was quickly plugged back in. The hardware is still quite early, it looks like a really solid start.

I don't think I would pick up the first generation of hardware, if they continue with the project it will be really promising.

Reading: Babylon's Ashes, Cryptonomicon

OONI Probe Mobile

Thu, 09 Feb 2017 00:00:00 +0000

Awesome news, OONI probe now has an android tool. OONI probe uses tor to map out censorship on the internet. If you want to make the internet a much better place I really suggest running this app.

Boom! Now almost anyone can help fight Internet #censorship with the OONIprobe phone app! https://t.co/2xmq79rjls … #networking #MobileApp pic.twitter.com/rdUgVV8NTV
— torproject (@torproject) February 9, 2017

Unrelated , I read an awesome write up on reverse engineering a book cover . It is a shame I don't read Polish, the author including something interesting in the cover is a real draw to the book for me.

Reading: Babylon's Ashes, Cryptonomicon

Evil Bit

Fri, 10 Feb 2017 00:00:00 +0000

An email I just received... I should probably add it to https://t.co/tvQqQJ4ogR pic.twitter.com/am46Lu3wkc
— Steven Bellovin (@SteveBellovin) February 9, 2017

It appears that people mistake this April 1st RFC for a genuine one, the author has a page of replies here .

Reading: Babylon's Ashes, Diamond Age

Headless CHIP WiFi

Sat, 11 Feb 2017 00:00:00 +0000

I am working on a modification to hostapd and I really have to run this on linux. The pi I was planning to use has seems to have finally given up the ghost after 4 years of use. Never fear, I failed over to using the chip , that came with my pocketchip flashed with the headless firmware.

The chip will use the microusb connection as a serial port by default, similar to the way the BBB uses the usb port to do ethernet. With a serial connection, I needed to figure out how to get the chip onto wifi.

There is a command line tool for interacting with network manager listed in the chip documentation . nmcli is a pretty great tool for network access.

$ nmcli device wifi list
*  SSID             MODE   CHAN  RATE       SIGNAL  BARS  SECURITY    
*  HameNetwork      Infra  11    54 Mbit/s  100     ▂▄▆█  WPA2        
   NetGear-AWQ4D8   Infra  1     54 Mbit/s  69      ▂▄▆_  WPA1 WPA2   
   BTWifi-X         Infra  6     54 Mbit/s  30      ▂___  WPA2 802.1X 
   BTWifi-with-FON  Infra  6     54 Mbit/s  30      ▂___  --

Better yet, there is a nmtui interface that presents a nice ncurses way to configure the network.

Reading: Babylon's Ashes, Diamond Age

Ethics in Internet Archiving

Sun, 12 Feb 2017 00:00:00 +0000

The Internet Archive is my favourite thing on the internet, it is much much more than just the wayback machine . It is a massive archive of human culture, it might be one of the most important things being created right now.

There is much more information being added to the IA now than any one person could process themselves. But it isn't that hard for individuals to pick an upload or a topic and process through the files and provide some sort of best of list.

The video below is a chat about archiving, I think the most important take away is that we really need people to review the material and make sections accessible.

Paraphrased:

I am waiting for someone to go through 200 floppy discs and write a blogpost: 
    "I looked at all this junk, these 7 are great."

It is Sunday, so that makes seven days of writing .

Reading: Babylon's Ashes, Diamond Age

Archive this page now

Sun, 12 Feb 2017 00:00:00 +0000

The Internet Archive's Wayback Machine has an 'archive this page now' button, which I know I was aware of, but I hadn't ever used it. Watching this chat about archiving I thought I would have a look at it.

My preferred option would be to add a hook to my post script that triggers the IA to scrap any new pages. Poking around the FAQ and the scant API page didn't reveal any recommendations for how to use the tools. With no advice I tried curl with the URL they used:

curl https://web.archive.org/save/http://adventurist.me/posts/0244

That URL worked great, the history page now has a new scrap entry from today. Having done that I started looking to see how well covered my site was, it turns out very few pages have been captured into the global history.

Seems like an easy fix:

curl -s -S "https://web.archive.org/save/http://adventurist.me/posts/0[040-243]" > /dev/null

Assorted CHIP Stuff

Mon, 13 Feb 2017 00:00:00 +0000

Some stuff from playing with the stupidly named CHIP .

Battery

There is a script shipped with the CHIP images that will dump some information from the battery controller. Which is sort of useful I guess.

[chip@chip] $ sudo battery.sh
BAT_STATUS=0
CHARG_IND=1
BAT_EXIST=1
CHARGE_CTL=0xc9
CHARGE_CTL2=0x45
Battery voltage = 3930.3mV
Battery discharge current = 0mA
Battery charge current = 882.5mA
Internal temperature = 51.9c

LEDs

There are two leds on board, a pink one that is directly wired into power and a status led connected over i2c. The led can be control directly over i2c with the i2cset command.

[chip@chip] $ sudo i2cset -f -y 0 0x34 0x93 0x0 #turn off

[chip@chip] $ sudo i2cset -f -y 0 0x34 0x93 0x1 #turn on

On my CHIP image the led is showing some sort of heartbeat that isn't stopped when I manually intervene. On their forums the i2cset method is reccomended to control the led, but the heartbeat made this impossible.

After a ton of poking and searching, trying to see if you can get strace to log processes that access a path (doesn't look like you can) I came across ledtrig-cpu in the dmesg .

[    2.315000] ledtrig-cpu: registered to indicate activity on CPUs

ledtrig-cpu is a kernel module for showing event status on built in leds, there is some inscruitable BBB documentation that somewhat shows how to control it.

In /sys/class there is an entry for each on the leds on board, listed with their colour. We can have a play with the led by looking at the following:

[root@chip] # cd /sys/class/leds/chip:white:status
[root@chip] # ls
brightness  device  max_brightness  power  subsystem  trigger  uevent
[root@chip] # cat brightness 
0
[root@chip] # cat max_brightness 
255
[root@chip] # echo 24 > brightness 
[root@chip] # echo 10 > brightness
[root@chip] # echo 255 > brightness
[root@chip] # echo 0 > brightness

Changing the value in brightness didn't dim the STAT led at all, I can only set it on or off.

[root@chip] # cat trigger
[none] kbd-scrollock kbd-numlock kbd-capslock kbd-kanalock kbd-shiftlock kbd-altgrlock kbd-ctrllock kbd-altlock kbd-shiftllock kbd-shiftrlock kbd-ctrlllock kbd-ctrlrlock nand-disk usb-gadget usb-host axp20x-usb-online timer oneshot heartbeat backlight gpio cpu0 default-on transient flash torch mmc0 rfkill0 rfkill1 rfkill2 rfkill3

The trigger functionality was much more fun. Trigger modes can be changed by writing their name to the file

[root@chip] # echo backlight > trigger
[root@chip] # echo transient > trigger
[root@chip] # echo torch > trigger    
[root@chip] # echo mmc0 > trigger 
[root@chip] # echo timer > trigger

Setting the trigger mode to timer added two more files the /sys entry:

[root@chip] # ls
brightness  delay_off  delay_on  device  max_brightness  power  subsystem  trigger  uevent
[root@chip] # cat delay_on 
500
[root@chip] # cat delay_off
500
[root@chip] # echo 2000 > delay_off

We can restore the trigger to the heartbeat with:

[root@chip] # echo heartbeat > trigger

Reading: Babylon's Ashes, Diamond Age

This Skull thing

Tue, 14 Feb 2017 00:00:00 +0000

. @BSidesSF pic.twitter.com/QqPNngF6Cc
— Dan Tentler (@Viss) February 14, 2017

Reading: Babylon's Ashes, Diamond Age

FOSDEM Talk Videos

Thu, 16 Feb 2017 00:00:00 +0000

It looks like all the FOSDEM 2017 videos are up and available. Direct links for my two talks:

I posted the slides earlier here

Reading: Babylon's Ashes, Diamond Age

Changing font size in urxvt

Fri, 17 Feb 2017 00:00:00 +0000

I finally had the need to dynamically change my font size in urxvt. If you search you will find keybindings to do this, such as the ones recommended in this thread . With my teeny planck keyboard, i3 , and tmux I don't really have room in my head for learning other keybindings.

In that thread there is also a printf command that changes the font size via a terminal escape code.

alias biggest="printf '\33]50;%s\007' \"xft:Source Code Pro:pixelsize=30\""
alias big="printf '\33]50;%s\007' \"xft:Source Code Pro:pixelsize=20\""
alias small="printf '\33]50;%s\007' \"xft:Source Code Pro:pixelsize=10\""
alias teeny="printf '\33]50;%s\007' \"xft:Source Code Pro:pixelsize=8\""
alias normal="printf '\33]50;%s\007' \"xft:Source Code Pro:pixelsize=12\""

I added the above aliases to my zshrc .

Reading: Babylon's Ashes, Diamond Age

It's Just Emulation

Sat, 18 Feb 2017 00:00:00 +0000

I have done a lot today, but accomplished very little. I think more often than not these days happen. Here is a cool talk about archivist activities around games:

Reading: Diamond Age

Kaitai Struct Andes firmware image

Sat, 18 Feb 2017 00:00:00 +0000

I have added some parsing data for the andes core firmware format I wrote about before . kaitai is quite nice to write binary formats out with the result is reasonably understandable:

meta:
  id: andes_firmware
  endian: le
seq:
  - id: image_header
    type: image_header
    size: 32
  - id: ilm
    size: image_header.ilm_len
  - id: dlm
    size: image_header.dlm_len
types:
  image_header:
    seq:
      - id: ilm_len
        type: u4
      - id: dlm_len
        type: u4
      - id: fw_ver
        type: u2
      - id: build_ver
        type: u2
      - id: extra
        type: u4
      - id: build_time
        type: str
        size: 16
        encoding: UTF-8

The power of kaitai comes from its integration into languages, there is a compiler output to dot that you can play with online . Using that compiler I could generate a png from the dot file like so:

dot -Tpng andes.dot > andes.png

70cm Radio

Sun, 19 Feb 2017 00:00:00 +0000

We finally managed to get the programming software, radios and cables in the same room for the Motorola GM4340s we have. The radios are for doing 70cm packet data in Aberdeen, with them finally programmed and tested we need to make a computer radio interface.

The GM340 has an accessory connector on the rear, rather than hacking together something using the microphone jack we can use this. The connector is 0.1" pitch so we can just connect female pin headers to it.

Our connector needs to hook into audio in, audio out, PTT and ground. I soldered these connections on a strip of female pin header and connected them to two TRRS jacks that were lying around. I pulled off the PTT and a ground line to another block of female header so we could connect that to a GPIO on a pi.

With this hacked together connector we were able to do some packet between a pi and another radio.

It is Sunday, so that makes seven days of writing .

Reading: Diamond Age, Crooked Little Vein

mosh NAT Script

Mon, 20 Feb 2017 00:00:00 +0000

In the past I required a custom script to mosh into my desktop, I was updating the script I use for this last night when I discovered it was no longer needed. I figured I might as well post the script here in case it helps someone else.

#!/bin/sh

# ########################################################## #
# wrapper for mosh to work with ssh's proxycommand directive #
# this only makes sense if the machine is directly reachable #
# from the internet using udp.                               #
# ########################################################## #

THISSCRIPT="`basename \"$0\"`"
REMOTE="$1"
REMOTEIP="$2"
NUM=`od -An -N1 -i /dev/random`
PORT=$((60000+NUM))

debug() {
    echo "[$THISSCRIPT] $@" >&2
}

usage() {
    debug "use me like this: $THISSCRIPT host [ip]"
}


# some default values
if [ -z "$REMOTEIP" ]; then
    if [ -z "$REMOTE" ]; then
        usage
        exit 1
    fi
    # does the remote have a hostname listed in .ssh/config
    REMOTEHOST="`grep -E -C1 \"^Host ([a-zA-Z0-9 ]+ )?$REMOTE( [a-zA-Z0-9 ]+)?$\" ~/.ssh/config | tail -n1 | gsed -r 's/\W*Hostname\W+//'`"
    if [ -z "$REMOTEHOST" ]; then REMOTEHOST="$REMOTE"; fi
    # resolve hostname
    REMOTEIP=`host -4 -t a $REMOTEHOST | head -n 1 | awk '{print $4}'`
    if [ -z "$REMOTEIP" ]; then
        debug "could not resolve hostname $REMOTE"
        exit 1
    fi
fi

debug "starting mosh-server on remote server $REMOTE $REMOTEIP:$PORT"
MOSHDATA="`ssh -t \"$REMOTE\" mosh-server new -i $REMOTEIP -p $PORT | grep '^MOSH' | tr -d \"\r\n\"`"
if [ -z "$MOSHDATA" ]; then
    debug "mosh-server could not be started"
    exit 1
fi

PORT="`echo -n \"$MOSHDATA\" | awk '{print \$4}'`"
MKEY="`echo -n \"$MOSHDATA\" | awk '{print \$5}'`"

if [ -z "$PORT" -o -z "$MKEY" ]; then
    debug "got no parseable answer"
    exit 1
fi

debug "starting local mosh-client to $REMOTEIP $PORT" 
MOSH_KEY="$MKEY" exec mosh-client "$REMOTEIP" "$PORT"

Reading: Crooked Little Vein

Know your Rockets

Wed, 22 Feb 2017 00:00:00 +0000

Detailed Tech Overview of the venerable #Soyuz U, flying its last mission today: https://t.co/03JBYQBtJr pic.twitter.com/xmHOf51voA
— ISS Updates (@ISS101) February 22, 2017

Reading: Crooked Little Vein

SHA1 Hash Collision

Thu, 23 Feb 2017 00:00:00 +0000

Its really windy, so windy that this plane couldn't land first try:

It’s windy today: here’s 500 tons of super-jumbo going wobble on the wing! https://t.co/VGjfOctkqg
— Charlie Stross (@cstross) February 23, 2017

Google found a SHA1 Collision their blog post is much smarter than anything I could write about it.

Reading: Babylon's Ashes, Diamond Age

RF Path Probing

Fri, 24 Feb 2017 00:00:00 +0000

We are trying to put up a 70cm amateur radio packet network, I started this by hacking together a connector last week. I really need to figure out where I can reach on 70cm.

Hibby and I tried to voice between my home and his, but were unable to hear each other. That isn't so bad, we really want to be able to get into my office where a friend can host a packet repeater.

I have remote access to work, so if I can get audio out of a radio onto the internet I will be able to do a remote check. The plan was to setup a pi, with an [rtl-sdr][6] and [rtl_fm][7] doing demodulation, ssh in and stream the audio back to where ever I am.

rtl_fm is an excellent tool, it can connect to an rtl-sdr stick and provide samples from the sdr. It can give you raw iq output or demodulate audio. This rtl_fm command can be used with the play (from the [sox][8] package to play broadcast fm.

$ rtl_fm -M wbfm -f 90.9M | play -r 32k -t raw -e s -b 16 -c 1 -V1 -

I referred to a blog post with an example of streaming audio over ssh using sox. To test audio streaming over ssh I pushed a copy of this amazing album and got lost for a while listening to it.

$ ssh -C sdr@192.168.1.181 "cat test.flac"| play -q -

rtl_fm will happily dump samples to stdout, a test with broadcast fm over ssh is:

$ ssh -C localhost "rtl_fm -M wbfm -f 90.9M " | play -r 32k -t raw -e s -b 16 -c 1 -V1 -

Broadcast FM is an excellent way to figure out if your demodulate pipeline is working, it is always running and it shits out a fuckton of power.

The command to demod narrow fm looks like:

$ rtl_fm -M fm -s 1000000 -f 145.800M -r 48k | play -r 32k -t raw -e s -b 16 -c 1 -V1 -

And it can be run over ssh:

$ ssh -C localhost "rtl_fm -M fm -s 1000000 -f 433.550M -r 48k "| play -r 48k -t raw -e s -b 16 -c 1 -V1 -

Hibby and I tried some calls to this station from his, but I wasn't able to hear anything. Probably something to do with both of us being in buildings facing away from each other. I will give this a try from mine and see what happens.

Reading: Normal

Of course before trying to do all this from my desktop I faffed about for two hours getting a pi up and running with FreeBSD. The pi wasn't able to handle the sdr without really choppy audio. Eventually while writing this up I noticed that the cable from the antenna was long enough to read my desktop. Oh well.

Look it is post 0x100

IETF Document to Bibtex

Sat, 25 Feb 2017 00:00:00 +0000

I am writing stuff up right now so I am not really doing anything that interesting. Users of latex for academic work will know the nightmare that is generating bibtex entries. Thankfully for ietf rfc and drafts entries are automatically generate. I found this blog post that includes a tool for looking up an entry and getting its bibtex.

Reading: Normal

curling down bibtex

Sat, 25 Feb 2017 00:00:00 +0000

When I wrote the last post I really wanted to use curl to turn rfc numbers and drafts into bibtex entries. I did have a look, but I had other things to do that seem urgent and I didn't follow it through.

That was lazy of me, the page will generate an error message with a url when given an rfc or draft that doesn't exist. I looked at this url with a valid rfc, but it wasn't clear how to turn the returned info into a bibtex entry.

Stripping that div off the page makes the url visible:

Failed to read RFC or Internt-Draft resource at http://xml2rfc.tools.ietf.org/public/rfc/bibxml/reference.RFC.9999.xml

Using that url format with a valid rfc number (Our beloved RFC768 ) spits out this xml document:

$ curl  http://xml2rfc.tools.ietf.org/public/rfc/bibxml/reference.RFC.0768.xml
<?xml version='1.0' encoding='UTF-8'?>

<reference  anchor='RFC0768' target='http://www.rfc-editor.org/info/rfc768'>
<front>
<title>User Datagram Protocol</title>
<author initials='J.' surname='Postel' fullname='J. Postel'><organization /></author>
<date year='1980' month='August' />
</front>
<seriesInfo name='STD' value='6'/>
<seriesInfo name='RFC' value='768'/>
<seriesInfo name='DOI' value='10.17487/RFC0768'/>
</reference>

That is how far I got when I gave up earlier. Looking at the page again I thought I might try looking at the network traffic it generates.

That is much more interesting, the page itself is doing a request to https://sysnetgrp.no-ip.org . Lets try a curl there and see what we get:

$ curl "https://sysnetgrp.no-ip.org/rfc/rfcbibtex.php?type=RFC&number=768"    
@techreport{RFC0768,
  author = {J. Postel},
  title = {User Datagram Protocol},
  howpublished = {Internet Requests for Comments},
  type = {STD},
  number = {6},
  year = {1980},
  month = {August},
  issn = {2070-1721},
  publisher = {RFC Editor},
  institution = {RFC Editor},
  url = {http://www.rfc-editor.org/rfc/rfc768.txt},
  note = {\url{http://www.rfc-editor.org/rfc/rfc768.txt}},
}

Much better!

Don't use SHA1

Sun, 26 Feb 2017 00:00:00 +0000

2005 John Gilmore vs Linus Torvalds on SHA1 "debate" in a nutshell (h/t @zmanian ) https://t.co/mzUNaQPPF9 pic.twitter.com/VbwphiiNcj
— Tony Arcieri (@bascule) February 26, 2017

It is Sunday, so that makes seven days of writing .

Have you subscribed to orbital operations yet? You should.

Reading: Normal

Why can't I send UDP packets from a browser?

Mon, 27 Feb 2017 00:00:00 +0000

You can't because it is a terrible idea, and yet this post explains a really reasonable way forward. My first thought was hopefully the same as yours, "That is absolute insanity, won't someone think of the DDOS?". Glenn Fiedler is responsible for the most pervasive game networking tutorial , it is the beej net guide for games.

This isn't a general interface for UDP, that is of course insane. Instead it is networking library specifically for real time games. It uses http authentication to generate a security token then offers a frame locked secure datagram API for moving real time data. The proposed API has hard timeouts, latency and bandwidth expectations, really not useful for anything other than games.

Right now there is a c library available on github. It will be interesting to see a prototype javascript interface.

Reading: Normal

Stupid Puzzle box

Tue, 28 Feb 2017 00:00:00 +0000

The space was sent a cool puzzle box as part of a secret santa. One of the puzzles involves getting the output of a flashing light into an LDR on the other side of the box. I played with this with another member last week, we decided to try and decode the light output.

I am sure we don't need to do this, but I wanted to try out my idea. He tried to use FinalCut to process a vide of the output, but this didn't work. I suggested we try breaking the video down to frames, running a brightness threshold over them, then generating a plot of the output.

Turn the video into frames:

$ ffmpeg -i VID.mp4  -f image2 frames/image-%07d.png

We can use convert from imagemagick to threshhold an image, here we convert its colour space into hue saturation and brightness and resize the image down to 1x1. Convert will give a text description of what it did so we don't have loads of temporary images.

I hand picked a 'dark' image:

$ convert dark.png -colorspace hsb -resize 1x1 txt:- 
# ImageMagick pixel enumeration: 1,1,65535,hsb                        
0,0: (10086,47272,7376)  #27B81D  hsb(55,72%,11%)

And a 'light' image:

$ convert light.png -colorspace hsb -resize 1x1 txt:-    
# ImageMagick pixel enumeration: 1,1,65535,hsb
0,0: (32525,17838,41551)  #7F45A2  hsb(179,27%,63%)

I ran convert over each file with some awk magic:

for filename in frames/*png; do
    echo $filename
    convert $filename -colorspace hsb -resize 1x1 txt:- | tail -n 1 | awk '{ print $4}' | awk -F "," '{ print $3}' | sed -e "s/%)//" >> outfile
done

I ran the outputfile through matplotlib to generate a nice plot of the light values.

import matplotlib.pyplot as plt

values = []

with open("outfile", "r") as f:
    for line in f.readlines():
        value = int(line)

        if value > 40:
            value = 100
        else:
            value = 10
        values.append(value)

plt.figure(figsize=(30,2))

plt.plot(values)
plt.ylabel('brightness')

plt.savefig("plot.png")

The hardest part of this was getting the video off my phone, the most time consuming was installing matplotlib.

Reading: Normal

Bread

Wed, 01 Mar 2017 00:00:00 +0000

Making bread seems important.

Loaf 1 pic.twitter.com/viNe0MpgvF
— [tj] (@adventureloop) March 1, 2017

I asked my father how I should start, how I should approach making bread for the first time. He told me just to follow the recipe on the packet of flour. Doing so seems to have worked out okay.

Due to a misunderstanding about yeast I ended up making two loafs.

Load 2 #bread pic.twitter.com/uKOKJaRt8X
— [tj] (@adventureloop) March 1, 2017

Reading: Normal

The Sky is not falling

Wed, 08 Mar 2017 00:00:00 +0000

Here is an article that explains in clear simple terms why the CIA Vault7 leaks are not the end of the world. If you consider yourself technical (which you do, you are reading a ｂｌｏｇ after all) you really have to help constrain the insanity in the face of leaks.

Just because you can read 'breaking signal by attacking the device' it does not mean that signal is broken. You have a responsibility to your friends and family, if they panic when they read the news and fall back to SMS because whatsapp is ｂｒｏｋｅｎ , the world is not becoming a better place.

Read what trusted security people say, validate their comments, help your family.

The Forgotten Abbey. #dailyart pic.twitter.com/tKjU8gBO8Z
— Stephan Hövelbrinks (@talecrafter) March 7, 2017

Reading: Gun Machine, The Difference Engine

Death

Thu, 02 Mar 2017 00:00:00 +0000

My chemex is dead.

The chemex is dead :'( pic.twitter.com/TzsrGfzeC4
— [tj] (@adventureloop) March 2, 2017

The poor thing was murdered.

Reading: Gun Machine

Please hold on

Fri, 03 Mar 2017 00:00:00 +0000

pic.twitter.com/XwLAENeM7C
— Archillect (@archillect) March 3, 2017

Today was insanity with a lab going haywire and then revisions on a paper. I am heading south tomorrow which means pictures of trains and complaining about trains.

Reading: Gun Machine

Train Wifi

Sat, 04 Mar 2017 00:00:00 +0000

On a 'high speed train' which means it is a directish train that barely stops. The wifi is running through the GSM network which in Scotland means it is really spotty. Of course it is unsecured with a captive portal. It is surprising that there aren't more terrible people around. It would be pretty easy to run your own bridging access point.

I saw an article this week that claimed something along the lines of "50% of mobile traffic is facebook". That makes sense, if most of the traffic is just a few sites (it is) and those sites have strong protections like certificate pinning, then it doesn't really matter if on hop 1 all of your traffic is exposed.

It doesn't stop people fucking with you, but the apps should offer some protection.

Reading: Gun Machine

#bread number 3

Sun, 05 Mar 2017 00:00:00 +0000

Loaf #3 pic.twitter.com/7a2ho647WU
— [tj] (@adventureloop) March 5, 2017

Attempt number 3 at making bread, I was much closer to the recipe this time. I will be able to tell in the morning. I wanted to make buns this time, but I chickened out. I looked at the size of the dough ball and it looked like it would surely burn.

The rest of the weekend I spent drinking, which is only interesting if you were both there and drunk.

It is Sunday, so that makes seven days of writing .

Reading: Gun Machine

802.Eleventy

Mon, 06 Mar 2017 00:00:00 +0000

This is an article on 802.11 and its problems , the picture is from there and it is awesome.

Reading: Gun Machine

If your terminal can't render this ◕ ◡ ◕ you have a problem

Tue, 07 Mar 2017 00:00:00 +0000

Imagine a world where the actual CIA spends its time figuring out how to spy on you through your TV. That's today. https://t.co/dQHBrsyIoI
— Edward Snowden (@Snowden) March 7, 2017

It is the end of the world!

The collection of Japanese faces is the best part of the leak ◕ ◡ ◕ .

Reading: Gun Machine, The Difference Engine

Defrag the Fake News

Thu, 09 Mar 2017 00:00:00 +0000

pic.twitter.com/Di70dMOrsB
— Archillect (@archillect) March 9, 2017

There is nothing good to report, maybe running a good old fashioned tool can help?

Reading: Gun Machine, The Difference Engine

SSL vhost stuff

Sat, 11 Mar 2017 00:00:00 +0000

Ode To My Family #comic #sysadmin #linux #development #CloudComputing https://t.co/hu0A3odXn2 pic.twitter.com/RiawlwbyzA
— turnoff.us (@turnoff_us) March 9, 2017

Not that I can fix any of those either.

I set up ssl with Let's Encrypt for an experiment yesterday following a handy guide on the FreeBSD wiki . The guide suggested this mozilla tool for generating server configs with good parameters.

With the tool I was only able to hit an A rating on the ssllabs testing site, the A+ rating was annoyingly elusive. I am using nginx as vhost for a go web service, for HSTS a header has to be appended to the response. The config from Mozilla does this for nginx like this:

# HSTS (ngx_http_headers_module is required) (15768000 seconds = 6 months)
add_header Strict-Transport-Security max-age=15768000;

But, the hosted application has control over the response headers. nginx can be configured to always set the header with the always flag:

# HSTS (ngx_http_headers_module is required) (15768000 seconds = 6 months)
add_header Strict-Transport-Security "max-age=15768000" always;

Reading: Gun Machine, The Difference Engine

PXE Boot FreeBSD Install

Sun, 12 Mar 2017 00:00:00 +0000

I am finally starting to make a dent in the pile of things I could be using, but aren't. A friend gave me a motherboard, case, graphics card and power supply over about 18 months, in the past fortnight I finally put it all together and had a working computer.

The machine came up no problem, one of the drives I recycled from another machine and it already had FreeBSD on it. It turns out the motherboard I was given doesn't want to boot from USB at all.

We tried all the different configurations and eventually fell back to using PXE. There is an excellent graphic PXE boot environment available from netboot.xyz , there was a FreeBSD entry in the OS boot menu, but it this is not a supported boot method for FreeBSD.

netboot.xyz uses a mfsboot FreeBSD image to launch a live system over PXE. The image is created with a set of scripts available on github . FreeBSD supports booting from a bundled memory image configured with the kernel config, it looks like that is the feature that makes all of this possible.

It is Sunday, so that makes seven days of writing .

Reading: Gun Machine, The Difference Engine

GSM CellID

Mon, 13 Mar 2017 00:00:00 +0000

With installing my new desktop I am also going to move my 4G modem. I wanted to get some signal strength numbers so I could be sure I wasn't completely ruining things for myself. My router has a hand status page that among sensitive private information has signal strength, SNR and noise numbers.

On that page there is also a CELL_ID field. The field is the unique network id of the base station you are connected to. This is apparently useful for location lookups, the wikipedia page has a list of databases that use this field.

I tried to feed my CELL_ID into some of these databases, but they all wanted more information. MCC and MCN are pretty easy to find, there is a big table on the wikipedia page . I was not able to resolve down a LAC from anywhere. There are apps I could try, but I don't really want to install any of them on my phone.

Reading: Gun Machine, The Difference Engine

Dewatering a pdf

Tue, 14 Mar 2017 00:00:00 +0000

There is this pdf ebook that I want to read, but it has a really annoying 'DRAFT' water mark on every page. I looked for an automatic way to remove the watermark and found a really handly superuser answer that completely covers it.

$ pdftk original.pdf output uncompressed.pdf uncompress 
$ sed -e "s/watermarktextstring/ /" uncompressed.pdf > unwatermarked.pdf
$ pdftk unwatermarked.pdf output fixed.pdf compress

Before I ran that I tried grepping through the pdf for the string 'DRAFT', now the pdf was compressed so I didn't find anything. I wanted to make sure the watermark was just a string so I extract just the first page with pdfseperate .

$ pdfseparate -f 1 -l 1 ipv6_for_ipv4_experts_en_a4.pdf out.pdf

I opened out.pdf with inkscape and played with editing the watermark, which was indeed just text. I then round tripped the pdf through pdftk and generated a watermark free pdf.

Reading: Gun Machine, The Difference Engine

UDP Options with Scapy

Wed, 15 Mar 2017 00:00:00 +0000

I am working on an implementation of the UDP Options draft at work, this morning I got the udp_input side of processing building. This needs to be test and gotten working before moving on, before setting up some VMs to test this I need a way to generate packets with UDP Option data appended.

This seemed like a great occasion to use go a little more. There is the gopacket library from google that provides raw packet stuff.

images/

I tried for ages to put together a send example that didn't depend on linux. Eventually I got to the point where I could form crazy malformed arp packets. I got to the point of generating the above traces in wireshark , for some reason go was sticking 16 bytes into the address fields and creating madness. You will note in the above arp packet that the length is much longer, that is because go is appending some extra data for shits and giggles.

Giving up on go I had a look at the python libraries for generating packets, they are all about the same level of insanity. The pathspider project has some test probes for UDP Options using scapy .

pathspider is a lot of stuff to pull in to generate UDP datagrams, I extracted out the relevant stuff to use with scapy directly:

from scapy.all import IP
from scapy.all import UDP 
from scapy.all import *

if __name__ == "__main__":

    ip = IP(src="139.133.204.4", dst="139.133.204.64")
    udp = UDP(sport=2600, dport=2600)

    pkt = ip/udp/"Hello world"

    pkt.getlayer(1).len = len(pkt.getlayer(1)) #force UDP len
    send((pkt/"\x01\x01\x01\x00"))

You can add the numbers together to find that the extra option space is include, you can also see the 01 01 01 00 bytes at the end of the packet which are the options I add.

Reading: Gun Machine, The Difference Engine

3 Commands to bhyve

Thu, 16 Mar 2017 00:00:00 +0000

Get a vm image and decompress it:

$ fetch http://ftp.freebsd.org/pub/FreeBSD/releases/VM-IMAGES/11.0-RELEASE/amd64/Latest/FreeBSD-11.0-RELEASE-amd64.raw.xz
$ xz -d FreeBSD-11.0-RELEASE-amd64.raw.xz

3 commands to FreeBSD runningin bhyve on FreeBSD:

# kldload vmm
# ifconfig tap0 create
# sh /usr/share/examples/bhyve/vmrun.sh -c 4 -m 1024M -t tap0 -d FreeBSD-11.0-RELEASE-amd64.raw test

Of course that misses out loads of stuff, the network won't work for one. Real instructions are in the handbook . Following in Hiren Panchasara's foot steps I am going to use bhyve to test and develop some network modification in FreeBSD.

I might try and automate the deployment a bit, so I can run a single command and have fresh vms on a configured network up and running. I suspect I will have to make some changes that involve rebuilding the whole world tree, if that is the case I will be trying to figure out how to get builds much much faster.

Reading: Gun Machine, The Difference Engine

100km

Fri, 17 Mar 2017 00:00:00 +0000

At the end of this month they will stop running buses to where I live, it seems basic services aren't available to those that aren't quite rural enough. Preempting the hard switch over I started cycling to work this week.

@mgdm this is what the tire whole looked like pic.twitter.com/7ZplhtlvCQ
— [tj] (@adventureloop) March 17, 2017

Work is not close (hence the whole bus thing), at 20km a day commuting I have done the first 100km week of what will probably be many. Week one has seen two puncture from a hole in my tyre, hopefully I will have better luck next week.

Reading: The Difference Engine

The Mess We're In

Sat, 18 Mar 2017 00:00:00 +0000

Reading: The Moon is a Harsh Mistress, The Difference Engine

haskell and git annex

Sun, 19 Mar 2017 00:00:00 +0000

I have finally after nearly a year started setting up data stores with git annex , I am going to try it out with my stash of datasheets, documents and books for a while. If it holds up to what I expect I will use it for the rest of my static binary media, video, audio and images.

I have also been revisiting the infuriating torture of learning haskell, with the real world haskell book . I did a haskell course and uni and it was horrible, so far the real world haskell book has been equally unenjoyable and slow.

git annex is written in haskell so the two things sort of tie together. Not that I plan to hack on git annex .

It is Sunday, so that makes seven days of writing .

Reading: The Moon is a Hard Mistress, The Difference Engine

The entire planet

Mon, 20 Mar 2017 00:00:00 +0000

11:38 on Saturday March 18th, over the South Atlantic Ocean pic.twitter.com/PrY4GTXV3c
— DSCOVR:EPIC (@dscovr_epic) March 20, 2017

Look at this amazing gem floating in space.

Paper deadline was today, I have to set up a large survey this week, but I am starting to surface again from this insane series of deadlines. There is a lot of FreeBSD Kernel work coming up, hopefully both at work and at home.

I have already poking at an implementation of UDP Options, there is also the possibility of me being given a TCP ABE implementation to port. For this work, unlike the stuff I did before for NewCWV I am going to provide a solid set of tests in the form of VM images. To do that I will need to figure out generation of images from just a git commit id.

Reading: The Moon is a Harsh Mistress, The Difference Engine

UUCP

Tue, 21 Mar 2017 00:00:00 +0000

Oh heck yes! pic.twitter.com/FR8ErNif0x
— Seth Morabito (@Twylo) March 21, 2017

The Simulator Image and the linked picture

POC||GTFO 14 dropped today.

Reading: The Moon is a Harsh Mistress, The Difference Engine

The unix mail command

Wed, 22 Mar 2017 00:00:00 +0000

This xkcd has been relevant today

If you wanted to know how to use the mail command you could look here .

> d *

Might just make it all go away.

Reading: The Moon is a Harsh Mistress, The Difference Engine

Help make the internet better

Thu, 23 Mar 2017 00:00:00 +0000

Back in January I wrote about a small tool I had thrown together to do some internet measurements. Back then we decided not to take the next step and attempt to roll the tool out to a large audience.

We have decided we need the network edge data after all and I need your help.

Want to help measure the internet? You could do it from the pub https://t.co/drfs5wBvZl pic.twitter.com/thD8zMbk2H
— [tj] (@adventureloop) March 23, 2017

First, you can get edgetrace from https://trace.erg.abdn.ac.uk

In short: We need measurements from as many network edges as possible. Places where people connect are almost always near the edges of the internet. Your home, office, the pub or a park with WiFi is probably near the edge. We need your help by running our tool from these sorts of places. The more the better.

In full: Packets on the internet are given a Best Effort service by default, everything is treated the same. The packets for your video call are treated the same way as a large download, but that means there is more latency when queues grow and packets in your file transfer are dropped when there is network pressure. With Quality of Service and Active Queue Management we can build networks that allow latency sensitive packets through the queue quicker while also stopping packets that shouldn't be dropped from being dropped.

The DSCP Bits in the IP header are used give different IP packets different Quality of Service classes. Right now, no one is really sure how these marks are treated; Are they removed? Changed in someone way? Or much worse, does the presence of these marks lead to packets being dropped?

To find this out we need to perform a survey, we can (and have) bought time on virtual machines in data centers, but that only measures things that are close to the network core. We also need to measure how these marks are treated at the edge, on connections that real people use.

There isn't anyway to easily perform these measurements without asking a whole lot of people for help. This is where you come in.

We need you to download and run our tool. If you can do it from home, the bus or the train that is excellent. Every run of the tool helps us build up more data about what is happening in the internet.

Thank you for helping make the internet better.

Reading: The Moon is a Harsh Mistress, The Difference Engine

Build a FreeBSD VM Image Release

Fri, 24 Mar 2017 00:00:00 +0000

release(7) documents a set of shell scripts for creating FreeBSD release files in same manner as the release engineering team. The script creates a new chroot environment, checks out a fresh tree, doing the release builds in a clean environment.

That might be what you want.

I want to write some scripts that take in a specified network, some git commit ids and generates a set of virtual machine images running in bhyve to reproduce a test environment. Building in a clean environment isn't what I need.

The Makefiles in release expect to be run from a tree that already has a built kernel and world. They make building the VM images really easy, but apart from comments in the files aren't documented.

I am going to use a directory for all of the stuff:

freebsd/
    -> src      # freebsd src tree
    -> obj      # object directory
    -> destdir  # freebsd destination direcory

$ cd freebsd
$ git clone https://github.com/freebsd/freebsd.git src
$ cd src

Build the kernel and world, setting the object directory to the one in our tree.

$ env MAKEOBJDIRPREFIX=/home/user/freebsd/obj time make -j4 -DKERNFAST buildkernel
$ env MAKEOBJDIRPREFIX=/home/user/freebsd/obj make -j4 buildworld -DWITH_META_MODE=yes -DWITH_CCACHE_BUILD -DNO_CLEAN

Move to the release directory to build our VM images:

$ cd release

# env MAKEOBJDIRPREFIX=/home/user/freebsd/obj make vm-release -j4 DESTDIR=/home/user/freebsd/destdir WITH_VMIMAGES=yes VMFORMATS=raw NOPKG=yes NOPORTS=yes NOSRC=yes  
# env MAKEOBJDIRPREFIX=/home/user/freebsd/obj make vm-install -j4 DESTDIR=/home/user/freebsd/destdir WITH_VMIMAGES=yes VMFORMATS=raw NOPKG=yes NOPORTS=yes NOSRC=yes

I exclude, packages, ports and the src distribution in the images.

As a test launch a bhyve VM with our created disk image:

# sh /usr/share/examples/bhyve/vmrun.sh -c 4 -m 1024M -t tap0 -d ../../destdir/vmimages/FreeBSD-12.0-CURRENT-amd64.raw test

Reading: The Moon is a Harsh Mistress, The Difference Engine

Notification Band Thing

Sat, 25 Mar 2017 00:00:00 +0000

Last night I converted by pebble from being a single contained unit, to a 3 part kit.

Found my pebble pic.twitter.com/uLVyENji2Q
— [tj] (@adventureloop) March 25, 2017

I am probably going to have to replace it.

Pebble the company is dead, I can still get replacement hardware from amazon or ebay and I suspect it will be generally available at reasonable prices for a year or two.

I used my pebble for 3 things

It's a smart watch, so I used it as a watch for time and date
I used it for weather, with the awesome relaxing watch face
The vibrate function is amazing for notifications. My phone hasn't been off silent for since I got the pebble, notifications for calls and messages are awesome. Better I can forward notifications from a service bus app like pushover and generate them based on things I want.

I can just wear a watch to deal with 1, for 2 I am probably going to use the awesome forecast.io app and not rely on being able to casually check the temperature.

For 3 I am really at a loss what to do. I could just replace the pebble, but really I think I want a smart band with a vibration motor for notifications.

If what I want doesn't already exist, it is probably too niche to ever become a thing.

Reading: The Moon is a Harsh Mistress, The Difference Engine

More bread

Sun, 26 Mar 2017 00:00:00 +0000

Bread 8, 1/3 wholemeal four this time. #bread pic.twitter.com/dGswrqVGCt
— [tj] (@adventureloop) March 26, 2017

I did more bread, but at batch 8 this is no longer really interesting to anyone other than me.

People have been complaining that my tweets are marked as offensive material, which is really funny I only really tweet about bread and technology. I looked at my settings and the 'mark as offensive' option was enabled on my output.

I'm sure I accidentally enabled it, but the twitter documentation does say they will add it to accounts that have flagged posts.

I have no love for twitter, if literally anything else had the communities I want to pay attention to posting I would move away. Ideally something federated, but that is only a pipe dream.

Yes my phone autocompleted flour to four, you can't edit twitter posts and phones are the worst thing ever.

It is Sunday, so that makes seven days of writing .

Reading: The Moon is a Hard Mistress, The Difference Engine

Findlater Castle

Mon, 27 Mar 2017 00:00:00 +0000

Reading: The Moon is a Harsh Mistress, The Difference Engine

Gherkin 30% keyboard

Mon, 21 Aug 2017 00:00:00 +0000

I like keyboards, I have been using an OLKB Planck as my daily driver for 18 months now. I saw a really nice ortholinear 30% keyboard go by on mastodon and I had to have one.

The keyboard I saw was actually the excellent gherkin by di0ib . di0ib has worked in the true spirit of open source and provided all of the design files and firmware for the gherkin. Beyond that they have included child proof instructions to order pcbs .

I tricked some friends into agreeing to build boards if I got a run of PCBS and set off. Amazingly easyeda.com was offering 5 more boards (10 vs 5) for just $2 extra. I managed to get 10 sets (board, key plate and base) of the PCBs for about £80.

Build

The build was really easy to do, there is some advice for the socket on 40 percent club, but if you test fit everything as you go it should be straight forward. A build is probably around 2 hours depending on proficiency.

Parts Per Keyboard:

1  Keyplate PCB
1  Bottom PCB
1  Main PCB

16 M2 Spacers (14mm length)
32 M2 screws

30 key switches
30 key caps

1  Arduino Pro micro
1  machine pin socket (wide 24 pin (2x12))

30 3mm leds (your choice of colour)
30 1N4148 diodes
1  100 ohm resistors
1  100k ohm resitors
30 470 ohm reistors
1  mosfet (probs A04406A 4406A)

Key caps are a harder thing to buy (so many awesome choices) so I ended up using some spares I found in a desk drawer.

Flash

Flashing the firmware to the keyboard was a little harder to figure out. Eventually I found some instructions that included the correct avrdude flags on hackaday.io , you also need to use a switch pulling RST down to GND to put the micro controller in programming mode.

Most of the work is done by the TMK make file, but you must manually specify a target for the program command. The command I used looks like:

# programming directive
MCU = atmega32u4
OPT_DEFS += -DBOOTLOADER_SIZE=512
PROGRAM_CMD = avrdude -p $(MCU) -P /dev/tty.usbmodem1411 -c avr109 -U flash:w:$(TARGET).hex

Use

With the board built and programmed (first try) it is time to figure out how to use it. It took a couple of months of daily use to get used to using the planck, it will be the same with the gherkin. To help learn I have printed out the keyboard layout and the combination of layers.

I modified the default layout a little to make it more similar to how I normally type. I moved space bar to my left hand, made 'X' a repeatable key(gotta be able to delete chars in vim) and added a 'CMD' key. I have a fork of the repo with my layout and Makefile changes.

The layer system is easy to use, if you hold any of the keys on the base layer it will enable the alternate function for a meta key or it will switch to another layer for a layer key.

FreeBSD on the GPD Pocket

Tue, 22 Aug 2017 00:00:00 +0000

In the distant past before smart phones became identical black rectangles there was a category of devices called palmtops . Palmtops were a class of PDA PC thing that fit in the palm of your hand. Today the Psion 5 series of devices most often capture peoples attention. Not only are they small and awesome, but they have something like a real keyboard.

This form factor is so popular that there are projects trying to update Psion 5 devices with new internals. The Psion 5 is the sort of device I have complained isn't made for a long time, at some point I picked one up on ebay with the intention of running the NetBSD port on it.

Earlier this year the world caught up and two big crowd funding projects appeared for modern Psion like palmtop devices. Neither the Gemini or the GPD Pocket campaigns convinced me that real hardware would ever appear. In May reviews of the GPD Pocket started to appear and I became aware of people that had backed and received their earlier campaign for the GPD WIN .

With a quirk in indiegogo allowing me to still back the campaign I jumped on board and ordered a tiny little laptop computer.

FreeBSD

FreeBSD is the only choice of OS for a pc computer . Support is good enough that I could boot and install without any real issues, but there was enough hardware support missing that I wanted to fix things before writing a blog post about it.

Somethings don't work out of the box others will need drivers before they will work:

~~Display rotation~~
WiFi (broadcom 4356)
Bluetooth (broadcom BCM2045A0)
Audio (cherry trail audio chrt54...)
Graphics
~~Nipple~~
USB C
~~Keyboard vanishes sometimes~~
Battery
Suspend
Touch Screen (goodix)
fan (there is some pwm hardware)
backlight
~~I2C~~
gpio

Display

The most obvious issue is the display panel, the panel it self reports as being a high resolution portrait device. This problem exists in the bios menus and the windows boot splash is rotated for most of the time.

Of course the FreeBSD bootsplash and framebuffer are also rotated, but a little neck turning makes the installer usable. Once installed we can address the rotated panel in X, accelerated graphics are probably in the future for this device, but the X framebuffer drive is good enough for FreeBSD hacking.

With X we can sort of the rotation problem. xf86-video-scfb is required to use the framebuffer .

# pkg install xf86-video-scfb

And the following lines have to be added to /usr/local/etc/X11/xorg.conf.d/driver-scfb.conf

Section "Device"
    Identifier "Generic FB"
    Driver "scfb"
    Option "Rotate" "CW"
EndSection

 Section "Device"
     Identifier    "Card0"
     Driver        "scfb"
 EndSection

The screen resolution is still super high, there doesn't seem to be anyway to do DPI hinting with the framebuffer driver (or in i3 at all), but I can make terminals usable by cranking up the font size.

Keyboard and touchpoint

A Keyboard is vital for a usable computer, out of the box the keyboard works, but the touch point does not. Worse, touching the touch point caused the built in USB keyboard to die.

Some faffing trying to debug the problem with gavin@ at BSDCam and we got both keyboard and mouse working. For some reason my planck keyboard presents as a mouse among other things, pluggin in a mouse and power cycling the USB device caused ums(4) to correctly probe and attach.

Manually loading ums(4) at boot got the touch point working correctly. In fact, ig4(4) also attaches when manually loaded.

Add these lines to /boot/loader.conf

ums_load="YES"
ig4_load="YES"

The dmesg shows some problems with ACPI probing, this is probably the source of some of the device problems.

Other devices

Wifi, bluetooth and graphics are bigger problems that will hopefully be caught up in others work and made to work soon. The touchscreen controller is adding a driver and support for Cherry View GPIO, there are datasheets for these and I am working on them.

No battery level indicator makes it annoying to use the GPD Pocket out and about. Without a driver the charge controller is using a really low current to recharge the battery. Datasheets are quite readily available for these devices and I am writing drivers now.

GPD Pocket

The Pocket is a great little device, I think its 'cuteness' makes everyone fall in love with it on first sight. I am really looking forward to getting the final things working and using this as a daily device.

FreeBSD on an Intel x5-z8350 tv box

Sat, 20 Jan 2018 00:00:00 +0000

There are a load of sort of generic tv boxes on ebay with an Intel x5-z8350 processor.

The x5 SOC (formerly Cherry Tree, formerly Cherry View) is the same family as the SOC in my beloved GPD Pocket . I was really having trouble with the i2c hardware in the GPD Pocket and wanted something I could take apart and poke with an oscilloscope. I looked first at the UP Board , an x5-z8350 in a raspberry pi form factor, but not only was it much more expensive than this tv box, but it has a CPLD between the SOC io and the pin header.

Installing FreeBSD

First I needed to get the board to boot from USB, the listing I bought came with both android and windows 10 (I guess that is what dual os means). In both android and windows 10 there was a handy reboot to other os application.

From installing on the GPD Pocket I suspected that the bios boot menu key would be F7 so I used that. Windows 10 also includes a handy reboot to uefi config option which makes it easy to get into a bios menu. I used it to disable quiet boot and set the boot delay to a more sensible number.

With those changes I rebooted and got a familiar AMI bios boot screen hit F7 and choose your usb stick from the menu. The FreeBSD loader menu came up and continued into a boot from the usb stick, but it hung probing ppc0 .

I found a solution on the freebsd forum post about the upboard which suggested running:

OK unset hint.uart.1.at

at the loader prompt. With that you I could boot and do an install.

Before you reboot make sure to make that change permanent, by removing this line in /boot/device.hints

... 
hint.sc.0.flags="0x100"
#hint.uart.0.at="isa"       # comment this line out
hint.uart.0.port="0x3F8
hint.uart.0.flags="0x10
...

now reboot.

Setup

I setup the the drm-next-kmod driver, but the machine froze during boot. Next I tried using a frame buffer driver, which required the collowing config in /usr/local/etc/X11/xorg.conf.d/driver-scfb.conf :

Section "Device"
    Identifier "Generic FB"
    Driver "scfb"
EndSection

 Section "Device"
     Identifier    "Card0"
     Driver        "scfb"
 EndSection

Hardware

The box has:

Blue/Red LED
External Power button
External(ish) reset button
- Pressing the reset button caused an instant power cycle.
4 usb ports
- 1 USB 3
- 2 external USB 2
- 1 internal USB 2
sd card reader
- but it doesn't seem to be hotpluggable
ethernet
hdmi

The x5 box also has bluetooth and wifi, but neither currently have FreeBSD drivers.

Internally there are a whole bunch of unpopulated things that might be interesting.

On the top left there is an unpopulated 2.54mm pin header slot next to the led, silkscreen on the board has 1 and a 7 on either end. Probing around with a multimeter suggested that P7 was ground.

I spent quite a while poking the board with a multimeter and osclloscope to see if any gpio or buses were exposed on the headers or the board. I did find that if you connect pin 1 to gnd (or pin 7) the red led comes on and the board goes off.

I did not find any useful or even really interesting signals.

On the bottom right there is an unpopulate 15 pin header, all but two of these were connect to ground.

MEAT

Some more gory insides:

Copyright (c) 1992-2018 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.0-CURRENT #0 r328126: Thu Jan 18 15:25:44 UTC 2018
    root@releng3.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64
FreeBSD clang version 6.0.0 (branches/release_60 321788) (based on LLVM 6.0.0)
WARNING: WITNESS option enabled, expect reduced performance.
VT(efifb): resolution 1920x1080
CPU: Intel(R) Atom(TM) x5-Z8350  CPU @ 1.44GHz (1440.00-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x406c4  Family=0x6  Model=0x4c  Stepping=4
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x43d8e3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,TSCDLT,AESNI,RDRAND>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x101<LAHF,Prefetch>
  Structured Extended Features=0x2282<TSCADJ,SMEP,ERMS,NFPUSG>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 2147483648 (2048 MB)
avail memory = 1946144768 (1855 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <ALASKA A M I >
WARNING: L1 data cache covers fewer APIC IDs than a core (0 < 1)
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
random: unblocking device.
ioapic0 <Version 2.0> irqs 0-114 on motherboard
SMP: AP CPU #2 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
Timecounter "TSC" frequency 1440001458 Hz quality 1000
random: entropy device external interface
netmap: loaded module
[ath_hal] loaded
module_register_init: MOD_LOAD (vesa, 0xffffffff80ff8620, 0) error 19
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
kbd1 at kbdmux0
nexus0
cryptosoft0: <software crypto> on motherboard
acpi0: <ALASKA A M I > on motherboard
Firmware Error (ACPI): Failure creating [BDLI], AE_ALREADY_EXISTS (20180105/dswload-498)
ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20180105/psobject-371)
ACPI Error: AE_ALREADY_EXISTS, (SSDT: DptfTab) while loading table (20180105/tbxfload-355)
ACPI Error: 1 table load failures, 8 successful (20180105/tbxfload-378)
acpi0: Power Button (fixed)
unknown: I/O range not supported
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> port 0x70-0x77 on acpi0
atrtc0: Warning: Couldn't map I/O.
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff irq 8 on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 450
Event timer "HPET1" frequency 14318180 Hz quality 440
Event timer "HPET2" frequency 14318180 Hz quality 440
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0xf000-0xf03f mem 0x90000000-0x90ffffff,0x80000000-0x8fffffff at device 2.0 on pci0
vgapci0: Boot video device
xhci0: <Intel Braswell USB 3.0 controller> mem 0x91700000-0x9170ffff at device 20.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
pci0: <serial bus, USB> at device 22.0 (no driver attached)
pci0: <encrypt/decrypt> at device 26.0 (no driver attached)
pcib1: <ACPI PCI-PCI bridge> at device 28.0 on pci0
pci1: <ACPI PCI bus> on pcib1
re0: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0xe000-0xe0ff mem 0x91604000-0x91604fff,0x91600000-0x91603fff at device 0.0 on pci1
re0: Using 1 MSI-X message
re0: turning off MSI enable bit.
re0: Chip rev. 0x4c000000
re0: MAC rev. 0x00000000
miibus0: <MII bus> on re0
rgephy0: <RTL8251/8153 1000BASE-T media interface> PHY 1 on miibus0
rgephy0:  none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
re0: Using defaults for TSO: 65518/35/2048
re0: Ethernet address: 84:39:be:65:0d:60
re0: netmap queues/slots: TX 1/256, RX 1/256
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
acpi_button0: <Power Button> on acpi0
acpi_tz0: <Thermal Zone> on acpi0
sdhci_acpi0: <Intel Bay Trail/Braswell eMMC 4.5/4.5.1 Controller> iomem 0x9173c000-0x9173cfff irq 45 on acpi0
mmc0: <MMC/SD bus> on sdhci_acpi0
sdhci_acpi1: <Intel Bay Trail/Braswell SDXC Controller> iomem 0x91738000-0x91738fff irq 47 on acpi0
mmc1: <MMC/SD bus> on sdhci_acpi1
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbdc0: non-PNP ISA device will be removed from GENERIC in FreeBSD 12.
est0: <Enhanced SpeedStep Frequency Control> on cpu0
est1: <Enhanced SpeedStep Frequency Control> on cpu1
est2: <Enhanced SpeedStep Frequency Control> on cpu2
est3: <Enhanced SpeedStep Frequency Control> on cpu3
Timecounters tick every 1.000 msec
ugen0.1: <0x8086 XHCI root HUB> at usbus0
uhub0: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
mmcsd0: 31GB <MMCHC NCard  4.5 SN 6E7E9160 MFG 06/2017 by 136 0x0003> at mmc0 200.0MHz/8bit/8192-block
mmcsd0boot0: 4MB partion 1 at mmcsd0
mmcsd0boot1: 4MB partion 2 at mmcsd0
mmcsd0rpmb: 4MB partion 3 at mmcsd0
mmc1: No compatible cards found on bus
WARNING: WITNESS option enabled, expect reduced performance.
Trying to mount root from ufs:/dev/mmcsd0p2 [rw]...
uhub0: 13 ports with 13 removable, self powered
lock order reversal:
 1st 0xfffff8000417e240 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2607
 2nd 0xfffffe0000e46500 bufwait (bufwait) @ /usr/src/sys/ufs/ffs/ffs_vnops.c:282
 3rd 0xfffff800042a09a0 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2607
stack backtrace:
#0 0xffffffff80b2bba3 at witness_debugger+0x73
#1 0xffffffff80b2ba24 at witness_checkorder+0xe34
#2 0xffffffff80a9cbeb at __lockmgr_args+0x88b
#3 0xffffffff80dc2565 at ffs_lock+0xa5
#4 0xffffffff810f7af9 at VOP_LOCK1_APV+0xd9
#5 0xffffffff80ba7006 at _vn_lock+0x66
#6 0xffffffff80b9599f at vget+0x7f
#7 0xffffffff80b87891 at vfs_hash_get+0xd1
#8 0xffffffff80dbe25f at ffs_vgetf+0x3f
#9 0xffffffff80db4886 at softdep_sync_buf+0xd16
#10 0xffffffff80dc3354 at ffs_syncvnode+0x294
#11 0xffffffff80d999ff at ffs_truncate+0x6df
#12 0xffffffff80dca7f1 at ufs_direnter+0x641
#13 0xffffffff80dd393c at ufs_makeinode+0x61c
#14 0xffffffff80dcf5b4 at ufs_create+0x34
#15 0xffffffff810f51d3 at VOP_CREATE_APV+0xd3
#16 0xffffffff80ba6908 at vn_open_cred+0x2a8
#17 0xffffffff80b9f14c at kern_openat+0x20c
ugen0.2: <Dell Dell USB Entry Keyboard> at usbus0
ukbd0 on uhub0
ukbd0: <Dell Dell USB Entry Keyboard, class 0/0, rev 1.10/1.15, addr 1> on usbus0
kbd2 at ukbd0
ugen0.3: <SanDisk Cruzer Fit> at usbus0
umass0 on uhub0
umass0: <SanDisk Cruzer Fit, class 0/0, rev 2.00/2.01, addr 2> on usbus0
umass0:  SCSI over Bulk-Only; quirks = 0x8100
umass0:0:0: Attached to scbus0
da0 at umass-sim0 bus 0 scbus0 target 0 lun 0
da0: <SanDisk Cruzer Fit 2.01> Fixed Direct Access SPC-4 SCSI device
da0: Serial Number 4C530302741216116074
da0: 40.000MB/s transfers
da0: 3819MB (7821312 512 byte sectors)
da0: quirks=0x2<NO_6_BYTE>
re0: link state changed to DOWN
GEOM_PART: integrity check failed (da0s4, BSD)
GEOM_PART: integrity check failed (ufsid/5a1180062a826673, BSD)
GEOM_PART: integrity check failed (diskid/DISK-4C530302741216116074s4, BSD)

dmesg on dmesgd.nycbug.org

Far Too Much Summer

Sat, 29 Sep 2018 00:00:00 +0000

There was frost on the car this morning we can declare summer concluded. These last 3 months have been very intense, an absolute ton of fun, but the intensity meant very little down time.

I plan to write a series of blog posts to capture some of excellent adventures I had. As always the best things required participation to spare you from inside jokes I will stay close to easily shared realities.

campgnd - 1st to 3rd June
ToorCamp - 18th to 26th June
Montreal - 10th - 21st July
- Linux NetDevConf - 11th to 13th July
- IETF Hackathon - 14 and 15th July
- IETF 101 - 16th to 20th July
BSDCam - 15th to 17th August
EMF Camp 2018 - 31st Augst to 2nd September
Bucharest - 20th to 23rd September
- FreeBSD DevSummit - 20th and 21st September
- EuroBSDCon 2018 - 22nd and 23rd September

Somehow there was space in this calendar to start running a monthly pancake breakfast at the hackerspace . The next Hacker Breakfast be Sunday the 14th of October.

It is hard to admit. This summer was too much.

It is hard to admin because throughout, despite the travel exhaustion, hangovers and mild illness, it was a ton of fun. The fun came at a cost, post IETF my brain was a puddle and I still had to build a streaming system and write a slide deck based on it.

EuroBSDCon was a major stress inducer for me, I submitted to the CFP with a Proof of Concept, which did work. Getting from the PoC to a presentable system was a lot of work. I allocated time to do this and then filled that time with travel and conferences and my job.

Six large blocks of travel in a row were too many. I need to figure out how to control the commitments I make so I don't become overwhelmed by saying yes.

campgnd 2018

Sun, 30 Sep 2018 00:00:00 +0000

                                                              __,--'\
                                                        __,--'    :. \.
                                                   _,--'              \`.
                                                  /|\       `          \ `.
                            ____ _   _ ____      / | \        `:        \  `/
  ___ __ _ _ __ ___  _ __  / ___| \ | |  _ \    / '|  \        `:.       \
 / __/ _` | '_ ` _ \| '_ \| |  _|  \| | | | |  / , |   \                  \
| (_| (_| | | | | | | |_) | |_| | |\  | |_| | /    |:   \              `:. \
 \___\__,_|_| |_| |_| .__/ \____|_| \_|____/ /| '  |     \ :.           _,-'`.
                    |_|                    \' |,  / \   ` \ `:.     _,-'_|    `/
                                              '._;   \ .   \   `_,-'_,-'
                                            \'    `- .\_   |\,-'_,-'
Scotland's first Hacker camp.                            `--|_,`'

One of the campgnds I used the tag line 'it happened again!'. It keeps happening and people are still upset about that year we missed. At this point it is easier to keep doing it.

campgnd is the annual camping trip for the hackerspace . We take 10-20 people off into a remote field build up an unreasonable shanty town of tents, feed it with power and data and let our minds go.

I love campgnd, it is a chance to escape and an opportunity to test out Village for visiting larger camps around Europe. Getting away and going somewhere is a great way to increase focus, if camping isn't your thing then taking your hackerspace to a makerfaire is a great way to focus on getting projects ready to show.

It seems we are already planning campgnd 2019, if you want to join the madness drop into #scottishconsulate on freenode and ask.

FreeBSD on the Intel Compute Stick STK1AW32SC

Sun, 30 Sep 2018 00:00:00 +0000

A FreeBSD developer has been tricked somehow into working on EFI boot. A large missing piece has been support for 32 bit EFI. Many devices with Intel mobile SOCs have shipped with bios which only support 32 bit EFI for boot even on 64 bit processors.

Rumour had it the Intel Compute Stick STK1AW32SC was one of the platforms with only 32bit EFI. This compute stick has a SOC from the Cherryview family, the same as the GPD Pocket , I want FreeBSD to support this SOC well and 32 bit EFI to boot is a part of that.

This compute stick is end of life and looking around I saw a few on ebay. I managed to win an auction for a new in box compute stick, getting it for about £50. For that I got:

x5-Z8330 4 Cores 1.44 GHz
2GB Ram
32GB Internal Flash
1 USB 2 Port
1 USB 3 Port
MicroSD Card slot
Intel Wireless-AC 7265 + Bluetooth 4.2
Intel Integrated Graphics

I asked Allan Jude to take his compute stick to the DevSummit at EuroBSDCon , while he was grabbing it someone else piped up and claimed to have run FreeBSD on the compute stick before. Turns out there is a bios option to switch between 32bit boot and 64bit boot.

Yes, our deliberate FreeBSD brick actually works. Here is how to install FreeBSD on a Compute Stick:

Break into the bios by hitting F2 at boot.

In 'Configuration' change Operating System from 'Windows 32-bit' to 'Windows 64-bit'

Reboot and break into the boot menu and choose your FreeBSD USB stick.

As with the x5 box there is an issue where the uart causes the compute stick to hang.

Break into the loader menu and set:

OK unset hint.uart.1.at
OK boot

Install as normal

Before rebooting at the end of the installer you need to edit device.hints to disable the uart again.

# chmod +w /boot/device.hints
# vi /boot/device.hints
....
hint.sc.0.flags="0x100"
hint.uart.0.at="isa"    # comment this line out
hint.uart.0.port="0x3F8"
....

WiFi

Bluetooth is present in the dmesg, but we need to load the iwm kernel module then we can configure WiFi as normal .

# kldload if_iwm

Graphics

Since setting up the x5 box in January our FreeBSD has has gained support for integrated graphics on CherryView SoCs. Now graphics support is available by installing and loading the drm-next-kmod .

# pkg install drm-next-kmod
# kldload /boot/modules/i915kms

Meat

I was unable to find any tear down pictures of the compute stick so I had to make some. The cast is easy to take a part, there is a single screw under a rubber foot once that is removed the rest of the top case is held on with snap fits. Inside the fan is connected with a tiny cable, the 2.4GHz and 5GHz antennas are glued to the side of the case, everything else is held down with 2 screws. 3 screws hold the heat sink assembly to the pcb.

Inside there is very little to see

On the top is the SOC is in a puddble of goop, an AXP288 PMIC, 64Mb of Winbond flash and two Kingston 4Gb DDR3 Ram modules.

On the bottom there are two more DDR3 modules (taking us up to 2GB), a SanDisk SDINADF4A 32GB eMMC and an Intel 7265D2W WiFi + Bluetooth module..

---<<BOOT>>---
Copyright (c) 1992-2018 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.0-ALPHA7  r338849 amd64
FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on LLVM 6.0.1)
WARNING: WITNESS option enabled, expect reduced performance.
VT(efifb): resolution 1920x1080
CPU: Intel(R) Atom(TM) x5-Z8330  CPU @ 1.44GHz (1440.00-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x406c4  Family=0x6  Model=0x4c  Stepping=4
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x43d8e3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,TSCDLT,AESNI,RDRAND>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x101<LAHF,Prefetch>
  Structured Extended Features=0x2282<TSCADJ,SMEP,ERMS,NFPUSG>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 2147483648 (2048 MB)
avail memory = 1955004416 (1864 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <Intel  COMSTKFC>
WARNING: L1 data cache covers fewer APIC IDs than a core (0 < 1)
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
random: unblocking device.
ioapic0 <Version 2.0> irqs 0-114 on motherboard
Launching APs: 2 1 3
Timecounter "TSC" frequency 1439997858 Hz quality 1000
random: entropy device external interface
netmap: loaded module
[ath_hal] loaded
module_register_init: MOD_LOAD (vesa, 0xffffffff810e1920, 0) error 19
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
kbd1 at kbdmux0
nexus0
efirtc0: <EFI Realtime Clock> on motherboard
efirtc0: registered as a time-of-day clock, resolution 1.000000s
cryptosoft0: <software crypto> on motherboard
acpi0: <Intel COMSTKFC> on motherboard
acpi0: Power Button (fixed)
unknown: I/O range not supported
cpu0: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> port 0x70-0x77 on acpi0
atrtc0: Warning: Couldn't map I/O.
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff irq 8 on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 450
Event timer "HPET1" frequency 14318180 Hz quality 440
Event timer "HPET2" frequency 14318180 Hz quality 440
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0xf000-0xf03f mem 0x90000000-0x90ffffff,0x80000000-0x8fffffff at device 2.0 on pci0
vgapci0: Boot video device
xhci0: <Intel Braswell USB 3.0 controller> mem 0x91500000-0x9150ffff at device 20.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
pci0: <encrypt/decrypt> at device 26.0 (no driver attached)
pcib1: <ACPI PCI-PCI bridge> at device 28.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pci1: <network> at device 0.0 (no driver attached)
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
acpi_button0: <Power Button> on acpi0
acpi_tz0: <Thermal Zone> on acpi0
sdhci_acpi0: <Intel Bay Trail/Braswell eMMC 4.5/4.5.1 Controller> iomem 0x9152c000-0x9152cfff irq 45 on acpi0
mmc0: <MMC/SD bus> on sdhci_acpi0
sdhci_acpi1: <Intel Bay Trail/Braswell SDXC Controller> iomem 0x9152a000-0x9152afff irq 47 on acpi0
mmc1: <MMC/SD bus> on sdhci_acpi1
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbdc0: non-PNP ISA device will be removed from GENERIC in FreeBSD 12.
est0: <Enhanced SpeedStep Frequency Control> on cpu0
Timecounters tick every 1.000 msec
ugen0.1: <0x8086 XHCI root HUB> at usbus0
uhub0: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
mmcsd0: 31GB <MMCHC DF4032 0.1 SN 9557679A MFG 05/2016 by 69 0x0000> at mmc0 200.0MHz/8bit/8192-block
mmcsd0boot0: 4MB partion 1 at mmcsd0
mmcsd0boot1: 4MB partion 2 at mmcsd0
mmcsd0rpmb: 4MB partion 3 at mmcsd0
mmc1: No compatible cards found on bus
WARNING: WITNESS option enabled, expect reduced performance.
Trying to mount root from ufs:/dev/mmcsd0p2 [rw]...
uhub0: 13 ports with 13 removable, self powered
lo0: link state changed to UP

America's Hackercamp

Mon, 01 Oct 2018 00:00:00 +0000

Toorcamp is America's Hackercamp, it happens on the stunning Orcas Island an hour or so North West of Seattle. Hacker events always manage to create their own neon lit world, Toorcamp took this to another level and sequestered 500 hackers away in a idillic resort in the Pacific North west and even then it poured on the neon lighting effects to keep us in a dream world.

Doe Bay resort spreads over three regions, a bay area (were I camped with Milliways), an island outcrop and a field at the top of a hill. This division (especially the hill) make it less enticing to move around the site. It also meant that the nosiy area in the bay, by being far away from most of the camping, was able to go all night long without disturbing too many people.

Toorcamp is serviced by a group of telephone enthusiasts called Shady Tel. They operate a highly reptuable phone company in the American fashion, offering service anywhere on the camp site, whether near an exchange or on a boat out in the bay.

I hate talking to people on the phone, but I found this limited network to be a ton of fun. I must have spent hours wardialling around trying to find people to call. Once I discovered the maintainence line that echo'd back your phone number I started going around and collecting interesting phones.

Because we are hackers on top of this phone network highly ammusing things pop up. Milliways ran a pager network and from their payphone I spent many hours paging people to call numbers. Knowing how to find numbers for a phone I started paging people to call me at random places.

The Doe Bay resort that hosted Toorcamp would be a wonderful place to go even without an amazing hacker camp in toe. Rather than attempt to describe the event it is easier to link to the 10 intervies the amp hour podcast did on site.

The final night nature decided to turn on a smoke machine and join the party.

EMF Camp 2018

Tue, 02 Oct 2018 00:00:00 +0000

EMF Camp is a giant hacker camp that occurs in the deep South of England. It managed to attract nearly 2500 people into a field for four days at the end August.

EMF Camp 2018 was the first time I have volunteered to help with the organisation. I volunteered to help out the content team earlier in the year, it wasn't until the week before that we realised lightning talks needed more organisation. Foolishly I stepped up and got a weird split experience between attending the camp and running a tiny slice of it.

It wasn't sooooo awful, I'll probably do it again.

I attended EMF Camp 2014, since then they have really managed to integrate well with the village system used at other camps. The map shows all the spontaneous events that people put together during the camp, the adage 'it is what you make it' really comes out at these events with many participants helping to make it hole.

In our own way the Scottish Consulate contributed too, with our bureaucratic role playing game going beyond the pale and expanding into operation of a phone network (cups and string) and a Hard border from the rest of the camp.

EuroBSDCon Bucharest Romania

Wed, 03 Oct 2018 00:00:00 +0000

The Wikitravel page for Bucharest has some scary warnings about taxis. I didn't heaer any horror stories from conference goers, but there was a large variation in prices for the same journey.

He held a two day DevSummit before the conference proper. A DevSummit is a chance to talk through technical issues and hash things out face to face. We did some planning for FreeBSD 13 with the idea of setting GGoals for the release.

We tried to match a bit of a hackathon with the DevSummit, but the tutorial schedules meant we couldn't focus the time very well and it was broken up.

EuroBSDCon

Day One :

Keynote1: Lightweight virtualization with LightVM and Unikraft
Hacking together a FreeBSD presentation streaming box – For as little as possible
- That was me, I thought it was quite good :D
The Evolution of FreeBSD Governance
Using Boot Environments at Scale
The End of DNS as we know it
Keynote2: Some computing and networking historical perspectives
- Ron's keynote was unreal and it is a massive shame that this sessions wasn't recorded. Ron has a ton of experience with working with network systems since 1976, he shared some stories and anecdotes. The one closest to my heart was pirating away an IMP front pannel and saving it from the scrappers. If you get a chance to see Ron speak you should jump up and down at it.

Day Two :

Taking NetBSD kernel bug roast to the next level : Kernel Sanitizers
Livepatching FreeBSD kernel
- This was an interesting study into how many different platforms do live patching. The FreeBSD way to do live patching could be simplified to 'use dtrace fbt probes'. Which is super reductive of all of the work invovled, but it shows the power of the system we have with dtrace.
Profiling Packet Processing: Performance & Peculiarities
Debugging lessons learned as a newbie fixing NetBSD
- Maya is a terrifying person. Somehow she manages to hack productivly across the entire range of the stack and across many different architectures. There were many debuggin gems in here, I hope she continues to present on this the information was great.
FreeBSD/VPC: a new kernel subsystem for cloud workloads
- This was a reprisal of a bsdcan talk .
FreeBSD on IBM PowerNV
- An recount of the porting work Semihalf did to POWER8. Interesting, I hope it is also sumbitted to AsiaBSDCon. There need to be more written account of bringing up on different architectures.

Day Two concluded with announcing the location of the next EuroBSDCon, Lillehammer Norway.

My FreeBSD Development Setup

Wed, 14 Aug 2019 00:00:00 +0000

I do my FreeBSD development using git , tmux , vim and cscope .

I keep a FreeBSD fork on my github, I have forked https://github.com/freebsd/freebsd to https://github.com/adventureloop/freebsd

On my fork I have the freebsd/freebsd repo set as an upstream

$ git remote -v
origin  git@github.com:adventureloop/freebsd.git (fetch)
origin  git@github.com:adventureloop/freebsd.git (push)
upstream        https://github.com/freebsd/freebsd.git (fetch)
upstream        https://github.com/freebsd/freebsd.git (push)

See this article for information on setting this up https://help.github.com/en/articles/configuring-a-remote-for-a-fork

I do all work on branches using worktrees, keeping the master branch clean.

Periodically I sync the master branch with the FreeBSD upstream:

$ cd ~/code/freebsd-dev/freebsd-git
$ git checkout master
$ git fetch upstream
$ git merge upstream/master
$ git push

I have a development setup based on Ian Lapore's arm set up documented on the FreeBSD wiki https://wiki.freebsd.org/FreeBSD/arm/crossbuild

I have a freebsd-dev directory in my code directory. It their I keep a copy of FreeBSD in freebsd-git , and obj directory for build output and a projects directory for in progress code.

$ tree -L2
.
├── diffs
│   ├── D15222.diff
│   └── old
├── dstdir
│   ├── boot
│   ├── METALOG
│   └── usr
├── freebsd-git
│   ├── bin
│   ├── sbin
...
│   └── usr.sbin
├── obj
│   └── usr
├── projects
│   ├── atomicstats
│   ├── axp288
│   ├── bugsquash
│   ├── byebyejumbograms
...

I use git worktrees for ongoing projects. git worktrees allow you to have a shallow file system copy on a git branch in a directory.

When starting a new project I do something like:

$ cd ~/code/freebsd-dev/freebsd-git
$ git worktree add thj/newdevelopment ../projects/newdevelopment master
$ cd ../projects/newdevelopment

Once the worktree is set up I launch a tmux session in the projects directory. Each random idea or itch I have, if there is enough there, ends up with a project worktree and a tmux session.

tmux allows me to have many windows in a session, I have a serious tmux problem. Right now I have 11 sessions with 42 windows across them. This is a good indicator of my focus level.

I do FreeBSD development with cscope and vim . With tmux splits I normally have an open file and I use other cscope instances in tmux windows to search for things I need in the tree.

I do testing in a bhyve vm and leave the serial port in a tmux window somewhere. I follow the setup in the FreeBSD handbook and back each vm with a zfs dataset.

I do FreeBSD kernel builds using a command like:

env MAKEOBJDIRPREFIX=/home/tom/code/freebsd-dev/obj make -j 44 buildkernel \
        -DKERNFAST installkernel \
    -DNO_ROOT DESTDIR=/home/tom/code/freebsd-dev/dstdir

I then ship kernels to the test vm with scp. jhb@ has a nicer method using the bhyve-loader , but I am yet to try it.

When changes are maturing I create reviews for them using arcanist, manu@ has a good article on doing this

FreeBSD on the NanoPi NEOLTS

Sat, 31 Aug 2019 00:00:00 +0000

The NanoPi NEOLTS is a SBC from FriendlyElec that uses the Allwinner H3 SOC. The NanoPi NEOLTS has a nice selection of hardware including 100Mbit Ethernet, 3 USB Ports and a bunch of exposed GPIO.

FreeBSD on the NanoPi uses GENERICSD image. This image requires a bootloader to be added before it will work. We can prepare a single image to be copied to many SD cards by using a memory disk as an intermediate step.

We need to:

Get the latest GENERICSD card image snapshot
Install the correct boot loader pkg
Create a memory disk
Copy the GENERICSD image to memory disk
Copy the bootloader to the memory disk
Mount the root partition of the sd card image
Copy the programs and files we need for the tutorial to the sd card

The latest image is as I write is 13 CURRENT from 20190829:

$ fetch ftp://ftp.freebsd.org/pub/FreeBSD/snapshots/arm/armv7/ISO-IMAGES/13.0/FreeBSD-13.0-CURRENT-arm-armv7-GENERICSD-20190829-r351591.img.xz

We have to decompress the image before we can use it

$ xz -d FreeBSD-13.0-CURRENT-arm-armv7-GENERICSD-20190829-r351591.img.xz

Each u-boot bootloader platform has its own package, currently there are 46 different bootloaders in the FreeBSD ports system. We want the u-boot for the nanopi_neo (our target).

$ pkg search nanopi     
u-boot-nanopi-neo2-2019.07     Cross-build das u-boot for model nanopi-neo2
u-boot-nanopi_a64-2019.07      Cross-build das u-boot for model nanopi_a64
u-boot-nanopi_m1plus-2019.07   Cross-build das u-boot for model nanopi_m1plus
u-boot-nanopi_neo-2019.07      Cross-build das u-boot for model nanopi_neo
u-boot-nanopi_neo_air-2019.07  Cross-build das u-boot for model nanopi_neo_air

# pkg install u-boot-nanopi_neo-2019.07

The u-boot-nanopi_neo package contains the binary bootloader we need in u-boot-sunxi-with-spl.bin

$ pkg info -l u-boot-nanopi_neo-2019.07

u-boot-nanopi_neo-2019.07:
    /usr/local/share/licenses/u-boot-nanopi_neo-2019.07/GPLv2
    /usr/local/share/licenses/u-boot-nanopi_neo-2019.07/LICENSE
    /usr/local/share/licenses/u-boot-nanopi_neo-2019.07/catalog.mk
    /usr/local/share/u-boot/u-boot-nanopi_neo/README
    /usr/local/share/u-boot/u-boot-nanopi_neo/boot.scr
    /usr/local/share/u-boot/u-boot-nanopi_neo/metadata
    /usr/local/share/u-boot/u-boot-nanopi_neo/u-boot-sunxi-with-spl.bin

With the GENERICSD image and the bootloader we need to create the memory disk image we will use for staging. First we need to create a large enough backing file.

$ truncate -s 8G nanopi.img
# mdconfig -f nanopi.img
md0

Now we can dd the GENERICSD image to the memory disk

# dd if=FreeBSD-13.0-CURRENT-arm-armv7-GENERICSD-20190829-r351591.img of=/dev/md0 bs=1m

We need to dd the bootloader to the start of the SD card, i.e. the entire device and not a partition.

# dd if=/usr/local/share/u-boot/u-boot-nanopi_neo/u-boot-sunxi-with-spl.bin of=/dev/da0 bs=1k seek=8 conv=sync

With the memory disk attached we can interact with the image file as if it were a real USB drive or SD card.

$ gpart show md0
=>      63  16777153  md0  MBR  (8.0G)
        63      2016       - free -  (1.0M)
      2079    102312    1  fat32lba  [active]  (50M)
    104391   6187041    2  freebsd  (3.0G)
   6291432  10485784       - free -  (5.0G)

We can mount the root partition of the SD card and modify or add any files we wish:

# mount /dev/md0sa mnt

When we are done changing things we have to disconnect the memory disk:

# sudo mdconfig -d -u md0

Finally we can copy the memory disk to a real sd card using dd :

# sudo dd if=nanopi.img of=/dev/da0 bs=1m

Help me blog more in June

Tue, 02 Jun 2020 00:00:00 +0000

This is a post on my blog.

I both have a blog and enjoy blogging. I think a blog is the perfect way to keep notes for myself with the nice possibility that they might help someone else. I frequently look up my own post on how to take screenshots with imagemagick on this blog.

I wrote a blog post everyday for 6 months, from the 26th September 2016 to the 27th of March 2017 , in the end I added 196 new entries to this blog in 182 days. I can't write an epic detailed post everyday, though I might manage a week or so like that if I have a backlog. Most posts were little more than an image and some text, little more than tweets.

But I still wrote, on a few days I even wrote more than one blog post. Under each blog post I included where in the world I was (borrowed from a characters quirk in Cryptonomicom), the weather and the books I was reading at the time.

These posts cover a range, pictures I was proud of , planning for events , travel and cool projects I worked on . Reviewing these has been a fun experience.

All that said, I haven't published a blog post for 276 days and I think I would like some help getting back into the public writing groove.

I tend to be better at delivering projects when I am held accountable by other people. Think of it as peer help rather than peer pressure.

I think it would be really cool if you too wrote some blog posts. Not for me, not for anyone else, but for you. You today and in the future. Write something now to clear your head, write something today that will help you tomorrow. Writing things down helps you remember them, but when you forget you will get to try searching your own log rather than having to scour the depths of the Internet.

I asked this weekend at campgndd for some peer help so that I would actually write blog posts in June. I suggested we each try to write four this month.

These kind people said they would try and write too to keep my going:

It would be cool if you were to join us. I will look for your response on your blog.

Capturing a screen sub section with ffmpeg

Sat, 13 Jun 2020 00:00:00 +0000

I tooted me scrolling through the browser tabs I opened from the Black Producers bandcamp list . I used ffmpeg to to grab a subsection of the screen with this command:

ffmpeg -f x11grab -show_region 1 -framerate 10 -video_size 720x480+100,200 -i $DISPLAY scroll.webm

video_size sets the size and offset in the window (needed if grabbing a subsection)
show_region paints a box around the active part, this helps dial in what is captured.

It took me a while to figure out what the input (-i) should be, in the end I figured out that it needs to be the X display, the easiest way to grab this is from the environment variable $DISPLAY .

I first made mp4's from the capture, but they were coming out all garbled, firefox refused to play and vlc was pretty sad. I move to webm and the capture is smaller and works in firefox.

FreeBSD on the Intel 10th Gen i3 NUC

Fri, 26 Jun 2020 00:00:00 +0000

I have ended up with some 10th Gen i3 NUC's ( NUC10i3FNH to be specific) to put to work in my testbed. These are quite new devices, the build date on the boxes is 13APR2020 . Before I figure out what their true role is (one of them might have to run linux) I need to install FreeBSD -CURRENT and see how performance and hardware support is.

They have an Intel i3-10110U with 2 cores and 4 threads at 2ish GHz (with 4GHz boost), I got a single 32GB DIMM of RAM for each and a 480GB Western Digital M.2 SSD. This configuration came in just under £500 for each NUC.

The NUCs are pretty small, they have pretty beefy fans taking up about a cm of the top of the enclosure. They certainly aren't silent, without any load I could hear the NUC sat on the desk next to me. When building at full steam the fan in the NUC is about as loud as my x270 Thinkpad is when it is building.

What works?

Out of the box I had to break into the bios and disable secure boot to boot the FreeBSD installer. I did this by hitting every FN key as the NUC booted, I think FN2 was the correct choice. At the time my keyboard was being fought over by USB and Bluetooth on my MacBook Pro.

FreeBSD install was problem free. I set up the M.2 as a single drive with ZFS, datasets and snapshots are a magic power.

Before I tried anything else I had to get an idea at how well this NUC would build FreeBSD. I don't expect this to be a build machine, but having spent a while shopping for build machines recently (I settled on a VM in hetzner) it is the only benchmark I really care about. make -j4 buildworld buildkernel took 2:45, that isn't the fastest in the world, but it is about an 1:15 faster than my x270 with its 2015 i5-6300U . The difference 5ish years makes.

Graphics in this 10th Gen Intel processor wasn't supported by drm-current-kmod and I had to install drm-devel-kmod . With the devel kmod the NUC is happy to push all the pixels of the 4k TV I have here and even drive additional second monitor connected with USB-C->HDMI adapter at the same time.

Audio works through the front 3.5mm jack and after changing hw.snd.default_unit from HDMI too. There is an issue where when the display is blanked (turned off), audio will stop, but won't come back when the display does. manu@ suggested setting hw.i915kms.disable_power_well=0 in loader.conf , this resolved the problem for me.

The Intel 9462 WiFi in the NUC is not supported (yet), I stuck a cheap rtwn based USB WiFi dongle in the front port. External USB WiFi is cumbersome on a laptop, but on a machine that will rarely move it is fine. The onboard Ethernet is supported by em and is happy to do 940ishMbit/s with iperf3 with TCP and UDP.

The NUC has 5 USB ports, 3 USB-A and two USB-C. The USB-C port on the back has a little lightning bolt next to it and I imagine that means it is the one with thunderbolt support. I did a disk speed test to a Sandisk Extreme Portable SSD. The front port managed 200MB/s or so while the rear thunderbolt one only got to about 160MB/s for read and write. My MacBook Pro manages the advertised 500MB/s read speeds, so I need to dig into whether this is a hardware problem or a FreeBSD problem. Luckily I have two so installing Linux to test performance won't be a bother.

I quite like these little boxes, you can get more hardware for your money with a tower pc, but not in this form factor. I wouldn't be surprised if the i7 versions of these were quite good at building FreeBSD. The fans are a bit loud, but that can be fixed for me by playing blaring techno.

I think the NUC in this configuration is going to make a great test node in my satellite testbed or a nice little desktop box for doing video tasks.

Hardware Support

Storage         Yes     M.2 is fine, I didn't have a drive to test SATA
Graphics        Yes     drm-devel-kmod (on r362612)
Audio           Yes     hda(4)
Ethernet        Yes     Supported by em(4)
Wireless        No      Intel AC 9462 not supported :(
USB             Yes     All ports work, 
Suspend/resume  Yes     suspend is fine, resume required drm-devel-kmod
SD slot         Yes     Comes up as a mmcsdX device
Thunderbolt     N/A     I have no idea what the implications of this are or how to test

Command Line Bug Hunting in FreeBSD

Sat, 27 Jun 2020 00:00:00 +0000

FreeBSD uses bugzilla for tracking bugs, taking feature requests, regressions and issues in the Operating System. The web interface for bugzilla is okay, but if you want to do a lot of batch operations it is slow to deal with. We are planning to run a bugsquash in July and that really needs some tooling to help any hackers that show up process the giant bug list we have.

Thankfully there is a python3 command line tool for interacting with bugzilla, called pybugz . bugz allows you to search through, up date and modify bugs without having to use a web browser. Getting bugz going was not very intuitive and it took me a bit of faffing.

bugz ships with a configuration for connecting to the FreeBSD bugzilla you use it by selecting it as a connection . The supported connections are dumped out if you try and do an operation with the -d 3 debug flag. This flag is really helpful for figuring out how to use bugz because while it documents itself, it holds on to the documentation like a powerful secret.

bugz really wants you to authenticate before you do anything, it won't show you help for commands without auth. There is however a --skip-auth flag. With this you can search for bugs, lets look for ipv6 issues with a patch in the base system:

$ bugz --connection FreeBSD --skip-auth search --product "Base System" "patch ipv6"
 * Info: Using [FreeBSD] (https://bugs.freebsd.org/bugzilla/xmlrpc.cgi)
 * Info: Searching for bugs meeting the following criteria:
 * Info:    product              = ['Base System']
 * Info:    status               = ['New', 'Open', 'In Progress', 'UNCONFIRMED']
 * Info:    summary              = ['patch ipv6']
88821 bugs                 [patch] IPv6 support for ggated(8)
186133 bugs                 [patch] tcpdump(1): zero checksums are invalid for UDP over IPv6
174225 bugs                 [network.subr] [patch] add support for ipv6_addrs_IF style aliases to rc.conf(5)
178881 bdrewery             [patch] getifaddrs(3) does not report IPv6 addresses properly in 32-bit compatibility mode
180572 rc                   [network.subr] [patch] SLAAC is enabled for ipv6_cpe_wanif
133227 bugs                 [patch] whois(1): add support for SLD whois server lookups and IPv6 address lookups
104851 bugs                 [inet6] [patch] On link routes not configured when using both IPv6 autoconfiguration and manual configuration
147681 bugs                 [network.subr][patch] Add inet6 keyword if it wasn't specified in ifconfig_IF_ipv6
130657 bugs                 [ip6] [patch] ipv6 class option
165190 bugs                 [ipfw] [lo] [patch] loopback interface is not marking ipv6 packets
245103 bz                   [patch] [ipv6] IPv6: update v6 temporary address lifetime according to rfc4941bis
 * Info: 11 bug(s) found.

We can also filter our search by component:

$ bugz --connection FreeBSD --skip-auth search --product "Base System" --component bhyve

bugz supports modifying and updating bugz from the command line, this is the main focus of the README on github . To authenticate bugz takes a username and password on the command line, I am not suggesting you fill your history with your bugzilla password, I did something like:

$ bugz --connection FreeBSD -u thj@freebsd.org --password `pass show FreeBSD/bugz | head -n 1` search udp

There is a -k flag that takes a key file, but I didn't want to dig into the source of bugz to figure out what this actually is.

Our bug squash is probably going to focus on clearing bugs with patches and the readme has a workflow for finding bugs and grabbing any diffs. More tools can and should be written around bugz this is just a start. Just playing with this I have spotted bugs than can easily be closed from the tracker.

Finally, you are going to need the help while using bugz , it took me longer than I liked to figure out that each sub command documents its own help and they all take the -h after the command. You need auth (or to skip auth) before you can use this flag.

$ bugz --connection FreeBSD --skip-auth search -h

My Streaming Setup

Tue, 30 Jun 2020 00:00:00 +0000

In April and May I did some streaming on twitch of hardware hacking projects . I started this as a way to work through the material for my cancelled BSDCan hardware hacking tutorial. With the tutorial cancelled I have been left with quite a few NanoPi Neo LTS and I was thinking about doing the tutorial as a series of videos with the idea of selling intro kits with the boards.

So far I have done four streams aiming for about an hour for each. I will say now that I haven't streamed again in June, I am not saying it is the end, but I (and I bet you too) need to control my commitments or I just never get anything done.

I have also been asked to write up my streaming set up so other people can use it. This vanity pic I tooted seems to include most of it.

Equipment

I have a bunch of equipment because I want to stream stuff in the real world. If you just wanted to share some windows and your webcam as Kristof Provost does then you can get by just with your laptop.

Canon 600D
Nikon J1
MacBook Pro 13,1 for streaming
Zoom H3 microphone
Aputure AL-M9 Mini LED Light
Tripods for the cameras
An arm for the microphone
LKV373 HDMI as a capture device for a camera

This is quite a lot of stuff, but other than the Aputure light I had all of it already from other projects. I would like to be able to capture the display of my work oscilloscope, thankfully the scope has VGA out so it is just a matter of figuring out a way to capture.

Software

I am using Open Broadcast Studio (OBS) on macOS to mix video and feed it to twitch.

OBS supports a rich variety of input sources that can mixed to make a scene. I have been using three scenes to make it easy to setup what is streamed in advance:

Everything
Just the camera
Just the presentation

Everything is the mixed view of my terminal, the camera output and the slides in a web browser. My tutorial slides are derived from my EuroBSDCan Tutorial and render in a web browser natively. Having a browser in the video mix quickly turned out to be really helpful.

OBS has been rock solid and I have had no problems with stability while streaming. The interface is a bit of a maze, but I suspect that is a natural result of the power it offers.

Canon Camera

For most of the top down shots I used a canon 600D with a 18-55mm kit lens. The battery in this thing is quite old and only manages about an hour and a half of video output. I have since gotten a USB powered battery insert that should allow me to run forever, but am yet to try streaming with it.

My canon camera does not support acting as a webcam, but gphoto2 does support grabbing live video from it. This should have been easy to feed into OBS, but I couldn't get it working and ended up instead using ffplay to render the video and grabbed that with a video capture. To get video from gphoto2 into ffplay I ran:

gphoto2 --stdout --capture-movie | ffplay -

This turned out to be very stable and easy to set up. There is quite high latency between the capture output and the gphoto2 capture, but it worked fine if I didn't have to move the camera much.

Nikon Camera

I also have a Nikon J1 and while the camera is awful to use I do have a 10mm lens for it with a macro extender. This allows me to take very high detail pictures of PCBs. I wanted to add this to the streaming set up too.

The Nikon firmware is garbage and while gphoto2 does support the camera I found I could only get a single image from the camera before having to reset it and was completely unable to get video from it. Never mind, my 2018 EuroBSDCon Talk used a HDMI network extender as a capture device. In the stream Crimes Against Computers3 I did something awful with a BeagleBone Black and got video from the Nikon through the HDMI extender into OBS.

This was a lot of faff to setup (though I since did get a USB-C Ethernet for the MacBook) and I am still kind of dubious of the stability of the LKV373 so I have only used this in the stream where I figured it out. I might try this again later.

Since doing that stream cheap USB2 and USB3 HDMI capture devices have appeared. These are supposed to appear as UVC webcams to the system. I suspect these will be a better method than the LKV373, but I want to try before recommending them.

Audio

For audio I am using a Zoom H3 dictaphone thing. I really like the audio from the Zoom, but I seem to be in the minority. In the end it is the microphone I have.

It is stereo which should be mixed down or you will drift across the channels when you move your head. OBS supports this so it wasn't a problem, but I did have to be told about it.

The firmware for this thing isn't so great either, it cannot record to the Micro SD card while acting as a USB interface and Annoyingly there is a bug where it defaults to 44KHz for the audio. The USB driver doesn't seem to advertise this to the properly and if you continue with the default you get weird audio. I dealt with this before in FreeeBSD , but was surprised to see it appear in macOS. You should always double check your audio before starting a stream.

Why not stream from FreeBSD?

I wanted a pain free approach where I could set up and go for my first streams so while OBS is ported to FreeBSD I expected it to be a lot more work than the much more common macOS/OBS combination. I expect FreeBSD/OBS support to get better if people continue streaming FreeBSD stuff.

The MacBook is similar specs to my Thinkpad, but the wireless card in the Thinkpad doesn't have great support and only manages 80211g rates. That might just be enough to stream out the video, but it seems very risky to me.

I will look at streaming with OBS from FreeBSD in the future.

The streams have done okay so far, there is a big social media boost network of FreeBSD users and developers and that has helped people find the content. I am sure the next stream will have fewer viewers as there has been a gap in my streaming.

You can support this

Finally, I set up a ko-fi page to support me writing and streaming. I plan to write more this year and if there is interest I will do more streams too. The streams so far have mostly been using equipment and parts I already had. Doing more is going to require me spending money.

If you have enjoyed my streams, or if you just want me to do more a very solid signal of your support would be tipping me a cup of coffee or something through ko-fi. You can support my hardware habbit here .

Quick and Dirty Network Scanning

Tue, 30 Jun 2020 00:00:00 +0000

Ever want to scan a subnet in the nosiest, least reliable way and generate too many processes while doing so? Yes? Well do I have a script for you:

#!/bin/sh

default=172.20.10

if [ -z $1 ]
then
        prefix=$default
else
        prefix=$1
fi

pinghost ()
{
        ping -t 1 -c 1 $1 > /dev/null
        if [ $? -eq 0 ]
        then
                echo hit $1
        fi
}

for x in `jot 254`
do
        pinghost $prefix.$x &
done

I wrote this while I was doing hack the box challenges and it was a fun and quick way to look to actually find things on my test network. I do not recommend using this. Some operating systems won't let you run it twice in succession as it generates a lot of processes.

Simple ipfw NAT for bhyve virtual machines and vnet jails

Wed, 01 Jul 2020 00:00:00 +0000

Most of the time, I want to do some throw away networking temporally to play with something or to try something out. I really don't like changing all the config on a machine just to try something. The FreeBSD documentation leans the other way first showing you what to edit in rc.conf before maybe mentioning that actual commands to run.

The ipfw documentation has a different problem. The example in the handbook and online are both very verbose and very complicated. Because ipfw is normally configured with a shell script the authors go absolutely wild with all the features they can.

I had a hard time figuring out ipfw in-kernel NAT from these guides. Instead here I present the simplest set of commands I could find to set up a NAT and a little explanation to help you debug when it doesn't work.

This is based on a great email from Allan Jude on the freebsd-virtualization list from 2014 that laid out the basics of this setup.

Set up Overview

For testing I want to run virtual machines and vnet jails on my laptop and give them have access to the internet. I want a throw away NAT setup that is ready to go quickly.

My laptop connects to my home network (and eventually the internet) over wifi. The wifi network offers me an address in the 192.168.1.0/24 subnet. On my laptop I want to have multiple guests. To do this we are going to use ipfw NAT and a bridge interface. It will look something like this:

         TO INTERNET
          ^
          |
          |
          v
          +-------+  192.168.1.x
+-------------| wlan0 |---------------+
|             +-------+               |
|                 ^                   |
|                 |                   |
|              ipfw nat               |
|                 |                   |
|                 V                   |
|            +---------+              |
| 10.0.4.1   | bridge0 |              |
|            +----+----+              |
|                 ^                   |
|                 | 10.0.4.0/24       |
|      ___________+_______________    |
|      |       |        |        |    |
|      v       v        v        v    |
|  +---+--+ +--+---+ +--+---+ +--+---+|
|  | jail | |  vm  | | jail | | ...  ||
|  +------+ +------+ +------+ +------+|
+-------------- laptop ---------------+

The interfaces in the jails (the b half of the epair) and the virtual machines (the vtnet in the V) won't be visible to ipfw, but will exist in their own world. To work around this we will use a bridge with the epairs and tap interfaces.

Setting up ipfw NAT

We need to load the kernel modules for ipfw and the ipfw in kernel NAT. ipfw has the frustrating default (and annoyingly different to ipf and pf) of to dening all traffic. This default has the great property of locking you out of a machine you are setting up remotely.

This is control by a sysctl that cannot be changed at run time, but we can change the default behaviour with kenv before we load the module:

# kenv net.inet.ip.fw.default_to_accept=1

Now we can safely load ipfw and the in-kernel NAT.

# kldload ipfw ipfw_nat

ipfw should load enabled, if you are having trouble later on double check that the firewall is actually enabled.

# sysctl net.inet.ip.fw.enable
net.inet.ip.fw.enable: 1

When we do NAT we are acting as a gateway between the traffic on the NATd interface and the real interface. For any packets to be passed we need to enable forwarding.

# sysctl net.inet.ip.forwarding=1
# sysctl net.inet6.ip6.forwarding=1

ipfw rule set

We need to create an IPFW NAT instance configured with the interface we want to NAT (wlan0 in this case) and configure rules to pass all traffic from the bridge through the NAT.

# ipfw nat 1 config if wlan0
# ipfw add 101 nat 1 ip from 10.0.4.0/24 to any out via wlan0
# ipfw add 103 nat 1 ip from any to any in via wlan0

I like to leave a gap between rules like this so I can insert an ipfw log command for the eventual case that nothing makes sense and everything is broken.

set up interfaces

A bridge is the center of our guest network, we will give it the default root address that all of our guests will speak to.

# ifconfig bridge create
bridge0
# ifconfig bridge0 inet 10.0.4.1/24 up

Our jail will use an epair interface to speak to the outside world. They come as an a and a b part, ifconfig only tells us about the a part when it clones the interface. When we give a vnet jail an interface it is no longer visible to the host system. An epair gives us two interfaces that act like a virtual ethernet cable, we stick one end into the jail and the other is connected to the bridge.

# ifconfig epair create
epair0a

Our virtual machine will use a tap interface to access the world. The tap interface needs to be brought up. There is a helpful sysctl that is off by default which will trigger the interface to be brought up when it is first opened. I like to set this to one, otherwise I find myself debugging networking inside the VM alot with little success.

# ifconfig tap create
tap0
# sysctl net.link.tap.up_on_open=1

With all the interfaces set up we need to add them to our bridge.

# ifconfig bridge0 addm epair0a addm tap0

Create jail

Never spoken about is the bsdinstall jail command. It takes a directory and installs a jail into it. This command will ask you some questions, it would be cool if it didn't, that would make automating jail creation in scripts much easier for me.

# mkdir testjail
# bsdinstall jail testjail

We make our jail persist so it will stick around as we experiment. The following command creates the jail on the host:

# jail -c name=testjail persist vnet path=testjail vnet.interface=epair0b

Now we can jexec into the jail and configure the epair. When you bring one end of an epair up, the other end comes up, when it goes down the other end goes down. We just need to configure an address and a default route in our jail.

# jexec testjail sh
[testjail] # ifconfig epair0b inet 10.0.4.4/24 up
[testjail] # route add default 10.0.4.1
[testjail] # ping -c 1 10.0.4.1
[testjail] # ping -c 1 192.168.1.1
[testjail] # ping -c 1 8.8.8.8

With this setup the jail can speak to our bridge, the local network and the wider Internet.

Create and config a VM

The FreeBSD offers prebuilt virtual machine images, The latest current one is available from a url like this:

# fetch ftp://ftp.freebsd.org/pub/FreeBSD/snapshots/VM-IMAGES/13.0-CURRENT/amd64/Latest/FreeBSD-13.0-CURRENT-amd64.raw.xz

It would be cool if there was a latest symlink that gave you a new head VM from one static place. The image comes xz compressed, we need to unpack it and I like to move it to a consistent place:

# xz -d FreeBSD-13.0-CURRENT-amd64.raw.xz
# mv FreeBSD-13.0-CURRENT-amd64.raw /vms/freebsd-current

bhyve requires we load the vmm kernel module, with that we can use the excellent vmrun.sh script to launch our vm.

# kldload vmm
# sh /usr/share/examples/bhyve/vmrun.sh -c 4 -m 1024 -t tap0 -d /vms/freebsd-current freebsd-current

Once that comes up you can log in and do some manual config.

[vm] # ifconfig vtnet0 inet 10.0.4.5/24 up
[vm] # route add default 10.0.4.1
[vm] # ping 8.8.8.8

For DNS in both the jail and the virtual machines I have to manually set up the name server local from my network.

/etc/resolv.conf

search lan
nameserver 192.168.1.1

This won't be valid as I move to other networks, but I am sure I will remember after only a little confusion and debugging.

Conclusion

That is all it takes. The NAT configuration is 3 firewall rules and enabling forwarding. None of this is persistent and that isn't great practice for a production environment, but it you just want to experiment with ipfw and NAT, or spin up a VM for today knowing how to do this in a non-persistent way is really helpful.

Blog more in 2020

Thu, 02 Jul 2020 00:00:00 +0000

In June I tried to write 4 blog posts and I elicited help from some of my friends to do this. I managed to write 5 posts beyond the announcement I would blog:

Of course it wasn't just me, I asked other people to blog to help me stay on track. The idea here was that seeing other peoples blog posts would inspire and force me to keep going. This worked reasonably well. The pressure to write the blog posts was there, but publishing was harder. This ended up with me pushing several posts in the final few days of June.

The pressure didn't really show up either, I know that the others wrote blog posts, but they didn't tell me!

They were great sports to get involved and help me with this, you should look up their blogs and drop them into your rss reader.

Because this wort of worked I think we should aim to keep doing this. Now 4 posts a month is a lot (maybe even too much) and so I thought that 8 more this year would be good. That is about 1.3333333... a month and seems entirely achievable.

I am going to try and blow this number out the water, but even if I fail completely and only manage one or two more post that will still be great.

Advanced Documentation Retrieval on FreeBSD

Thu, 28 Oct 2021 00:00:00 +0000

On Fri, Jul 16, 2021 at 04:23:20PM -0400, ░▒▓░░ ░▐░▒ wrote:
> Hi Tom.
> I just not realized you've not have your inbox assaulted by our listener
> feedback. Is there a specific address you'd like that routed to?
> 
> Anyway, here is a choice one.
> 

*sigh*

░▒▓░░ I don't know what we should do about Michael . I spoke to a priest
about an exorcism and he said "I'm not going near that monster, not on your
life", which I thought was pretty alarmist for a priest.

Follows is a rough markdown draft of the article "Advanced Documentation
Retrieval on FreeBSD" as we discussed

- Tom

-----------

Advanced Documentation Retrieval on FreeBSD

FreeBSD is renowned for its very high quality documentation. For many queries the man pages have a wealth of accurate and up to date documentation that is frequently a surprise to uses of other operating systems. It is not uncommon to hear from new FreeBSD users that they have to relearn to try the man pages before searching on the web.

Beyond man pages FreeBSD also has very high quality documentation in the form of the FreeBSD handbook. Only talking about the FreeBSD Handbook actually cuts short the range of really high quality documentation that the FreeBSD project offers.

The FreeBSD Handbook covers installation and day to day usage of a FreeBSD system and it is kept reasonably up to date by the FreeBSD documentation team. The handbook is an amazing document that comes from a time before blog posts and wikis and it contains a mixture of official project direction and tutorial style walkthroughs on how to use different FreeBSD subsystems and third party software. Not everything is covered by the handbook and many times only one path of a piece of software use is covered, but it is an excellent resource to get started using and configuring a FreeBSD system.

Deeper into FreeBSD there are several other 'books' that the FreeBSD project maintains. The full list of books is available from https://docs.freebsd.org/en/books/ , it includes technical information about how the FreeBSD kernel works in the forms of the Design and Implementation of the 4.4BSD Operating System and the Architecture handbook. Documentation on how to contribute to different parts of the operating system as the porters-handbook, the fdp-primer and the developers-handbook.

The FreeBSD project also hosts a wiki which contains less formal and in progress documentation, written by users and developers. The wiki can sometimes be much more like a temporary source of information, but it does contain valuable guides. It is the right place for pages such as the laptop compatibility matrix . The wiki is a unique FreeBSD project resource in that users are also able to have edit access.

What else is there?

Beyond the documentation the project provides there are outside sources of information on how to use and configure a FreeBSD system. Searching the web will bring up a lot of Technical information in the form of blog posts and articles.

Searching the web is not the only way to get more information on FreeBSD systems. We can use external 'daemonised' resources by using the FreeBSD base system tool invoke . This tool is a little esoteric to use and sadly it is one of the excellent FreeBSD tools written by developers that just quite haven't seen the light of day.

invoke ships in the FreeBSD source tree and is in src/tools/tools/invoke . invoke isn't built by default, but it is easy to build on a system using its Makefile:

# cd /usr/src/tools/tools/invoke
# make
# make install

invoke requires quite a lot of information to be useful, annoying the author ��@ was in the process of writing a book on invoke when they went missing travelling in rural Romania. However from reading the source we can see the list of information or 'principals' required to correctly invoke documentation.

Principals are information locators which are tied closely to the source of the information. Personal web pages of authors work well and are easy to obtain, more potent sources such as hand written notes or the authors blood work the best, but we can substitute social media accounts to get a similar level familiar information about the author.

XXX more on principals XXX

For our example I have collected the personal web page, blog and twitter account of the source we want to use, we can pass them to invoke as arguments or in a configuration file.

In addition to 'principals' the invoke tool needs to be run from a special environment. A comment in the source code describes this, but it took the author some trial an error to figure it out in practice.

/*
 * invoke must be run from either a larger or lesser circle. These
 * can be ancient such as the very high quality circle at Midmar,
 * however, if you are unable to travel a Ars Theurgia Goetia will
 * suffice. You must interface the machine running invoke via a 
 * galvanic isolator. The transformer in an Ethernet connector with
 * magnetics works great, if you only have an SBC you'll have to 
 * figure out something with transformers
 */

From this comment we can see that we need to create a substitute circle to use with invoke and connect it to our computer, but isolate it from the machine. If we fail to isolate properly we can damage the machine and likely destroy it.

We can create the substitute circle by using our preferred ethereal bonding fluid. Blood works very well, but collection can be legally tricky. Luckily we can use a vegan alternative in the form of beeted oat milk. If your local shops don't carry beeted oat milk you can make your own by mixing oat milk with beetroot juice during a full moon.

With the substitute bonding fluid we need to draw our circle for summoning and holding triangle (you can see a good example here ). In this example we are going to use an Ethernet interface with magnetics, all you really need to do is to plug it into the circle.

With the interface connected to the circle we can check for state using ifconfig, we need to look for the ETHER flag in the list of options.

igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500                            
options=e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,ETHER>
ether ac:1f:6b:46:9e:da                                                                               
inet 11.14.17.13 netmask 0xffffff00 broadcast 137.50.17.255 
inet6 fe80::ae1f:6bff:fe46:9eda%igb0 prefixlen 64 scopeid 0x1                                         
media: Ethernet autoselect (1000baseT <full-duplex>)                                                  
status: active                                                                                        
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

Running Invoke

With our principals collected, and our link established to the circle we can run the invoke command as follows:

# invoke -p [principals] [-i interface] [question]

We pass in the principals we have gather and tell invoke the interface name. If you hacked together a device you'll need to pass in the gpio controller and pin using -d, for the gpio controller and -P for the pin number.

# invoke -p [principals] -d /dev/gpioc0 -P 1 [question]

The final argument to invoke is a quoted string that contains the question.

If you are all set up then you can use invoke to summon your documentation author of choice, for this example I am using Michael W. Lucas :

# invoke -p https://mwl.io https://twitter.com/mwlauthor -i igb0 "How do I configure dummynet with weighted fair queues?"

If all goes well you should hear (an normally quite grumpy) disembodied voice bark the answer to your question back to you.

As great as this is for single questions sometimes you want the author to hang around and help with protracted debugging sessions. We can achieve this with the manifest option to invoke:

# invoke -p https://mwl.io https://twitter.com/mwlauthor -i igb0 manifest

With this option you will get the author directly summoned into the triangle attached to your summoning circle. Care should be taken with manifestations, the circle and triangle must remain intact for the duration of the session. If you are using a laptop (which I recommend and you need to use with an existing circle). Then make sure to watch the battery life careful.

If you don't properly shutdown the session then you risk getting the spirit of the author trapped in your machine and no one wants that.

Conclusion

FreeBSD has a great range of available documentation, but sometimes you hit the limits of information that is readily available online. Here we discussed the invoke command and the ways it can be used to get direct help from the authors of high quality documentation that are a little too public.

FreeBSD/Ubuntu Dual Boot Homelab in The Bedroom by the bed testbed

Sun, 07 Nov 2021 00:00:00 +0000

Current events have meant that my work place is now my home office, frustratingly this is also where I sleep. On one hand this has resulted in a very short commute, but on the other hand it does mean that I am living in close quarters with the computers I use for experiments, on the third hand (where did that come from?) it means that I get to have a testbed in the room where I keep my bed.

Of course this raises the serious question, if I write tests from my bed, which bed is the testbed? Unfortunately I only did one year of philosophy and so others will have to offer answers to this grand question.

I am in the unusual (for me) situation of needing to (well getting do, I love perf) do network performance tests on real hardware, thankfully my friend Tony was able to lend me two machines from his Bioinformatics cluster to play with for a couple of months.

This testbed exists to answer questions of the form:

"How does stock Ubuntu compare to FreeBSD?"

For these tests to be as fair as possible I need to have as identical hardware for the tests as possible. Tony enabled this by giving machines built out to the same spec, annoyingly his target wasn't "push packets as fast as possible", but was instead "give a reasonable mix of a ton of storage and compute to look at DNA sequences. Anything weird in these computers is clearly his fault and we are very greatful to get to experience them.

A dmesg from one of the boxes is here , but roughly they are:

CPU: AMD Opteron(tm) Processor 6380 (2500.05-MHz K8-class CPU)
128GB RAM
SuperMicro MNL-H8DGI6 motherboard
A pair of SSDs on a PCI-e SATA controller

On top of that I added a pair of dual port 10GbE interfaces I found 'lying' around the labs. One is an Intel X520 82599ES and the other is a Mellanox ConnectX-3 Pro. These interfaces don't match and have different performance characteristics, this is fine for setting up the experiments, but I am going to replace them with a matched pair of single Interface 10Gb adapters for 'production' experiments..

My normal method for evaluating if a machine is fast is to build FreeBSD, they managed a buildworld buildkernel in a respectable 58 minutes.

Setup

                        -left                                 -right                        
                 +------------------+    10.0.x.x      +------------------+                
                 |                  |.10.2        .10.1|                  |                
     ipmi        |            mlxen0+<---------------->+ix0               |     ipmi          
192.168.100.173  |                  |                  |                  | 192.168.100.167
                 |            mlxen1+<---------------->+ix1               |                
                 |                  |                  |                  |                
                 |          igb0    |                  |     igb0         |                
                 +-----------+------+                  +------+-----------+                
                             |___                          ___|                            
      freebsd 192.168.100.10     \_______          _______/   freebsd 192.168.100.20       
       linux  192.168.100.11             V        V           linux   192.168.100.21
                                      +--------------+                                     
                                      |    switch    |                                     
                                      +--------------+                                     
                                             ^                                             
                                             |
                                             | freebsd 192.168.100.2                     
                                        +---------+                                        
                                        | control |                                        
                                        |  host   |                                        
                                        +---------+

The two boxes are named with the suffixes '-left' and '-right' with the running OS setting the prefix, so we have freebsd-left, freebsd-right, ubuntu-left and ubuntu-right.

The machines have 3 network interfaces on the mother board, Dual Gigabit Intel Ethernet and an interface for IPMI. On each, one interface and the IPMI are connected to a switch which is in turn connected to the switch in the wireless router that bridges to the WiFi in my house. This setup is a little complicated, but because there isn't ethernet run up to my bedroom WiFi is the only sensible way to connect to the Internet. I'd much preferred a bit of NAT weirdness compared to having to set up WiFi in testbed machines.

I connected the serial ports on '-left' and '-right' to my control host, which is in the same switch domain as the hosts. I configured the SuperMicro motherboard to send the bios to the serial port.

I am really not using IPMI for all its abilities and instead it is a fancy remote power button I can press from the control host:

    [control] $ ipmitool -I lanplus -H 192.168.100.173 -U ADMIN -P ADMIN chassis power on   # power on -left
    [control] $ ipmitool -I lanplus -H 192.168.100.167 -U ADMIN -P ADMIN chassis power on   # power on -right

Serial Console

The serial ports are then connected to the control computer with an awesome two headed usb serial cable (I don't know what it is and would buy more if I did). The operating systems on -left and -right are configured to offer consoles over serial so I don't have to worry about breaking the network and locking myself out when I am far away.

On FreeBSD getting serial for loader and the system requires adding config to loader.conf and is documented in the FreeBSD handbook . Look at the bottom where it says "Setting a Faster Serial Port Speed" (I think the rest of the stuff on the page is out of date and rebuilding with a custom config is no longer required):

/boot/loader.conf:

    boot_multicons="YES"
    boot_serial="YES"
    comconsole_speed="115200"
    console="comconsole,vidconsole"

This configures loader to use both the video console and the serial console, tells it to use serial and sets the serial to the baud rate '115200' from the slow default of 9600. This baud rate matches between FreeBSD, Ubuntu and the BIOS so I don't have to reconfigure my serial terminal.

Getty (the thing that gives you login prompts) on FreeBSD is configured as 'onifconsole', so no further config is required. You can check this in /etc/ttys :

/etc/ttys:

    ...
    # The 'dialup' keyword identifies dialin lines to login, fingerd etc.
    ttyu0   "/usr/libexec/getty 3wire"      vt100   onifconsole secure
    ttyu1   "/usr/libexec/getty 3wire"      vt100   onifconsole secure
    ttyu2   "/usr/libexec/getty 3wire"      vt100   onifconsole secure
    ttyu3   "/usr/libexec/getty 3wire"      vt100   onifconsole secure

Getting Serial for GRUB on Ubuntu (in 2021) requires adding the to /etc/default/grub.d and rebuilding the config file with update-grub , this isn't really documented, but information can be found in the grub manual and in a selection of blogposts . For grub serial we need to add:

/etc/default/grub.d

    GRUB_TERMINAL="console serial"             
    GRUB_SERIAL_COMMAND="serial --speed=115200"

I am pretty sure the grub.d that ships with ubuntu is out of date with the actual file, when I rebuilt the menu timeout broke, it went from the default 10 seconds to the 0 seconds in the grub.d that I edited. I didn't care enough to file a bug report, this was a lot of faff.

To get console message from the Linux kernel you need to change the flags passed to the kernel when it is booted, you can do this too from grub.d :

/etc/default/grub.d:

    GRUB_CMDLINE_LINUX_DEFAULT="console=ttyS0,115200 console=tty0"

This tells the Linux kernel to use ttyS0 (com0) as the console, configures the baud rate to '115200' and tells the kernel to use tty0 as the console. After a few messages the kernel will hand over to something else that will ignore the console config if you haven't also configured systemd to offer a console.

To configure systemd to use a console you need to create a services file and it will handle the magic for you. This is documented on the Ubuntu wiki, but everything there is wrong. Instead the systemd versions are available in blog posts you can find online. I found the best results following documentation for a different distro targeting the Raspberry Pi 3 .

    # systemctl enable serial-getty@ttyS0.service
    # systemctl start serial-getty@ttyS0.service

You should now have a working getty on serial, but I think you then need to kick something else, I could only get this to work by rebooting.

Booting

To do comparisons I need to be able to boot both Operating Systems and manage them remotely. Dual boot of some sort means that I can dig into differences on the two the platforms quickly and get answers from the running systems.

Dual booting the machines turned out to be a lot harder than I expected. When I got the bios output working on serial I thought I was on to a winner, but that pesky SATA controller doesn't play well with the BIOS boot menu. Only the first drive in the SATA controller pair appears in the boot selector leaving me plumb out of luck.

Instead I dove into the Linux world. Knowing that grub knows how to boot FreeBSD I went with using grub to get a boot menu that I can control from the serial port. ( side note: I know that grub is a multiboot compatible boot loader, meaning that it will boot anything that matches that spec. I think it is also multiboot compatible and can be chained, i.e. grub can boot grub. If that is the case then FreeBSD's loader is also multiboot compatible and the FreeBSD kernel is probably too, can loader then boot grub? It will take a truly brave person to figure this particular puzzle out. )

After installing FreeBSD to the second SSD in the SATA controller. I got messages about a FreeBSD install being detected when I ran update-grub . I think these were just for fun though, I didn't get any new menu entries when I test rebooted. I installed FreeBSD by pulling the drive I installed Ubuntu to, booting a FreeBSD USB installer and installing to the only drive (thanks hot swap bay!).

Configuring grub to boot FreeBSD requires adding an entry to one of the extra config files. Internet searching suggested /etc/grub.d/40_custom which I filled out with:

/etc/grub.d/40_custom:

    #!/bin/sh
    exec tail -n +3 $0
    # This file provides an easy way to add custom menu entries.  Simply type the
    # menu entries you want to add after this comment.  Be careful not to change 
    # the 'exec tail' line above.

    menuentry "FreeBSD 13.0" {
    set root=(hd1)
    chainloader +1
    }

A Unix StackExchange Answer helped me figure out the rough grub commands to use and I tried them out on the grub command line (press 'c' from the menu).

The final grub config looks like this ( notice that default grub is friendly and doesn't beep by default ), with above /etc/grub.d/40_custom :

/etc/default/grub:

    # If you change this file, run 'update-grub' afterwards to update            
    # /boot/grub/grub.cfg.                                                       
    # For full documentation of the options in this file, see:                   
    #   info -f grub -n 'Simple configuration'                                   

    GRUB_DEFAULT=0                                                               
    GRUB_TIMEOUT_STYLE=menu                                                      
    GRUB_TIMEOUT=-1                 # pause at bootloader menu                   
    GRUB_DISTRIBUTOR=lsb_release -i -s 2> /dev/null || echo Debian               
    GRUB_CMDLINE_LINUX_DEFAULT="console=ttyS0,115200 console=tty0"               
    GRUB_CMDLINE_LINUX=""                                                        

    # Uncomment to enable BadRAM filtering, modify to suit your needs            
    # This works with Linux (no patch required) and with any kernel that obtains 
    # the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)     
    #GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"                   

    # Uncomment to disable graphical terminal (grub-pc only)                     
    #GRUB_TERMINAL=console                                                       
    GRUB_TERMINAL="console serial"                                               
    GRUB_SERIAL_COMMAND="serial --speed=115200"                                  

    # The resolution used on graphical terminal                                  
    # note that you can use only modes which your graphic card supports via VBE  
    # you can see them in real GRUB with the command `vbeinfo'                   
    #GRUB_GFXMODE=640x480                                                        

    # Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
    #GRUB_DISABLE_LINUX_UUID=true                                                

    # Uncomment to disable generation of recovery mode menu entries              
    #GRUB_DISABLE_RECOVERY="true"                                                

    # Uncomment to get a beep at grub start                                      
    #GRUB_INIT_TUNE="480 440 1"

Baseline Measurements

Before running more enjoyable experiments it is a requirement to get baseline measurements for what the systems can do. I think I need a bit more of a test framework for network performance tests, I want to sample memory usage, CPU usage and get flame graphs for tests, but for starters it is good to get raw iperf3 numbers.

For each configuration, I ran forward and backward iperf3 tests with UDP and TCP. I let iperf3 run in its default 10 seconds measurement mode, for UDP I requested it try infinite bandwidth (-b 0).

For each case I ran iperf3 as a server on *-right and the client on *-left .

Remember for these that by default the client iperf3 process sends and the server receives, this is swapped with the -R flag.

freebsd-left -> freebsd-right (server)

    tcp             iperf3 -c 10.0.10.1
            [  5]   0.00-10.00  sec  6.58 GBytes  5.66 Gbits/sec    0             sender
            [  5]   0.00-10.00  sec  6.58 GBytes  5.65 Gbits/sec                  receiver

    tcp             iperf3 -c 10.0.10.1 -R
            [  5]   0.00-10.00  sec  8.34 GBytes  7.16 Gbits/sec  2485            sender
            [  5]   0.00-10.00  sec  8.34 GBytes  7.16 Gbits/sec                  receiver

    udp             iperf3 -c 10.0.10.1 -u -b 0

            [  5]   0.00-10.00  sec  3.63 GBytes  3.11 Gbits/sec  0.000 ms  0/2666610 (0%)  sender
            [  5]   0.00-10.00  sec  2.03 GBytes  1.74 Gbits/sec  0.006 ms  1173881/2666555 (44%)  receiver

    udp             iperf3 -c 10.0.10.1 -u -b 0 -R

            [  5]   0.00-10.00  sec  3.24 GBytes  2.79 Gbits/sec  0.000 ms  0/2384960 (0%)  sender
            [  5]   0.00-10.00  sec  1.91 GBytes  1.64 Gbits/sec  0.003 ms  977341/2384881 (41%)  receiver

We run baselines so we can understand what future measurements show. Care has to be take that things are actually fair.

FreeBSD -> FreeBSD on the same hardware is a fair test, but it isn't what we have here. When we compare the forward and reverse modes for the iperf3 measurement we see that when the freebsd-left is the sender for TCP we get a much lower through put than when freebsd-right is the sender. My guess is that this is the difference between the offload engines in the Intel and Mellanox cards.

FreeBSD -> FreeBSD for UDP has interesting results. freebsd-left with the Mellanox card is able to sink more packets into the network than freebsd-right with Intel. Annoyingly these are opposite to the TCP results, where freebsd-right can send more.

This might already be highlighting an interesting place to dig, and it is where I would look next, IF I were comparing network interfaces.

ubuntu-left -> ubuntu-right (server)

    tcp             iperf3 -c 10.0.10.1

    [  5]   0.00-10.00  sec  9.59 GBytes  8.24 Gbits/sec  823             sender
    [  5]   0.00-10.00  sec  9.59 GBytes  8.23 Gbits/sec                  receiver

    tcp             iperf3 -c 10.0.10.1 -R

    [  5]   0.00-10.00  sec  11.0 GBytes  9.41 Gbits/sec    0             sender
    [  5]   0.00-10.00  sec  11.0 GBytes  9.41 Gbits/sec                  receiver

    udp             iperf3 -c 10.0.10.1 -u -b 0

    [  5]   0.00-10.00  sec  2.09 GBytes  1.79 Gbits/sec  0.000 ms  0/1546210 (0%)  sender
    [  5]   0.00-10.00  sec  1.50 GBytes  1.29 Gbits/sec  0.006 ms  436284/1546104 (28%)  receiver

    udp             iperf3 -c 10.0.10.1 -u -b 0 -R

    [  5]   0.00-10.00  sec  2.08 GBytes  1.79 Gbits/sec  0.000 ms  0/1544000 (0%)  sender
    [  5]   0.00-10.00  sec  1.12 GBytes   965 Mbits/sec  0.015 ms  710878/1543876 (46%)  receiver

Next up Ubuntu -> Ubuntu. When looking at later measurements we need an idea of where changes come from isolating out variables is a good thing to do.

Again with the TCP tests we see a difference in performance between the two systems, for Ubuntu -> Ubuntu it is approximately 1.2 Gbit/s, whereas for FreeBSD it is around 1.5Gbit/s, but FreeBSD has a lower baseline for comparison.

Next up for UDP with Ubuntu -> Ubuntu we see something really weird, for the ubuntu-left sender and the ubuntu-right sender the packets we try to send are much lower than the FreeBSD hosts. This correlates with lower overall throughput in both tests, but almost half the performance for ubuntu-right looks really weird.

I have a hunch that the lower sending rate is related to better pacing interactions between iperf3 and the Linux kernel. I have no idea why the received rate is so low for ubuntu-right .

freebsd-left -> ubuntu-right (server)

    tcp             iperf3 -c 10.0.10.1

            [  5]   0.00-10.00  sec  10.7 GBytes  9.19 Gbits/sec  1720             sender
            [  5]   0.00-10.22  sec  10.7 GBytes  8.98 Gbits/sec                  receiver


    tcp             iperf3 -c 10.0.10.1 -R

            [  5]   0.00-10.22  sec  8.75 GBytes  7.35 Gbits/sec  1464             sender
            [  5]   0.00-10.00  sec  8.75 GBytes  7.52 Gbits/sec                  receiver


    udp             iperf3 -c 10.0.10.1 -u -b 0

            [  5]   0.00-10.00  sec  3.63 GBytes  3.12 Gbits/sec  0.000 ms  0/2667830 (0%)  sender
            [  5]   0.00-10.04  sec  1.29 GBytes  1.11 Gbits/sec  0.011 ms  1717151/2667664 (64%)  receiver


    udp             iperf3 -c 10.0.10.1 -u -b 0 -R

            [  5]   0.00-10.04  sec  2.07 GBytes  1.77 Gbits/sec  0.000 ms  0/1524220 (0%)  sender
            [  5]   0.00-10.00  sec  1.90 GBytes  1.63 Gbits/sec  0.003 ms  125112/1524149 (8.2%)  receiver

Finally we get to run it all again with both operating systems in play. If everything was optimal (and we therefore had no work to do) there would be no difference in their performance, but we already know that this isn't true.

Running the tests with differing operating systems gives us an opportunity to see if the receiver side of the test has an impact (rather than just the sender). We can do this by pair the faster side with the slower side.

For now though, I think the different network cards are introducing too much variation between the systems. The numbers differ here and that on its own is quite interesting, but there seem to be too many choices for why. I find the Ubuntu -> Ubuntu reverse test halving the rate very suspicious and want to run the tests again.

The variation in the network cards is actually too much for me, I think it is a red flag in the measurements and would only encourage stupid review comments. This annoyed me enough that I bought a pair of Single Port Mellanox ConnectX-3 EN PCIe 10GbE to evaluate before running any meaningful experiments.

These systems are up and running, even with the questions that the baselines raised they are functional enough to start developing the interesting parts of the experiments and writing enough automation glue to rule out me making mistakes. I can then return, rerun the automated experiments and get better numbers.

FreeBSD/Ubuntu Dual-boot testbed using Desktop Hardware

Tue, 09 Nov 2021 00:00:00 +0000

Tony needed his computers back, he is a great friend and so he offered me some replacements. This means that we get to make version 2 of the "The Bedroom by the bed testbed" .

The first version of the testbed was built to answer questions of the form:

"How does stock Ubuntu compare to FreeBSD?"

Version 2 of the testbed continues this approach with an addition:

How does stock Ubuntu compare to FreeBSD, and what if we try emulating a satellite network?"

Tony and I spoke for a while about my needs from a testbed, the machines he lent me before were part of a set, they were two boxes in a 6 node Bioinfromatics cluster. He needed them back to start doing shake out tests of the cluster so he can start selling time on it ( check out his excellent company ).

While I have been using his machines all of the work so far has been developing experiment tooling. For me the time to replace the machines was actually quite fortunate, I have managed to get automation working, but I don't yet have any finished results. This means that I can move to new machines and apart from a short down time reconfiguring things there shouldn't be any disruption.

To replace the Opteron 6380 systems he offered me 2 Threadripper 1950X systems.

-left , -right

Threadripper 1950X
Asrock X399M Taichi motherboard
32GB RAM @2666MHz ( -right has 64GB, -left will get more once it arrives in the mail)
SSD storage

The Opteron machines were server motherboards, in massive whale sized cases and they made whale sized sounds when they were running. The Threadrippers are in lovely little mini-ATX cases and when they are running the make a lovely little hum that easily vanished into the background when I type on my cherry blue keyboard.

The smaller quieter form factor comes from the use of desktop hardware, sadly this also means the loss of lights out management.

When Tony listened to my needs he also offered me a third system (so I can answer the questions the addition raises), but I turned him down. For longer term plans I need to own machines and I am grudgingly happy to buy hardware to get experiments running.

I wanted to lean hard into using desktop hardware for non desktop tasks (my computers don't have an SLA to fulfil). With this goal in mind I speced out a Ryzen 5950X system that was a bit of a monster. I got cold feet at the price and decided to build a compatible system using a lower end Ryzen 7 processor. Gazing into the future I think this might be a safe bet, if I need more compute I should be able to pick up a 5950X on ebay for ~50% of the list price.

For 10GbE network I put a single port Mellanox Connect-X 3 Pro interface in each.

pokeitwithastick

Ryzen 3700x
Asrock X570 PRO4
32GB RAM @2666MHz
NVME storage

From the 'big machine' spec I culled this down to using less and slower RAM and lower end processor. I think I should be able to push up this rig with more faster RAM and a bigger processor, but I get the flexibility to try doing this on the cheap first.

The Ryzen system has a Mellanox Connect-X 3 Pro interface with Dual ports. It is going to be routing packets.

This machine is able to be a more moving target so I installed FreeBSD-14-CURRENT on it from a recent snapshot. The use of stock CURRENT is worth noting when you think about the performance of the Ryzen system compared to the others.

I only know 1 functional test that I really care about and that is building FreeBSD. I pulled the pairs of drives from the testbed v1 machines and transferred them over (the lack of 2.5" drive hot swap on the Threadrippers was annoying). Once I got the drives on the correct SATA cables the boxes came up and I was able to see how all three machines did:

host            processor       RAM time buildworl buildkernel

freebsd-left        Opteron 6380        128GB   58:00
freebsd-left            Threadripper 19050X  32GB   30:45
freebsd-right           Threadripper 1950X   64GB   30:06
pokeitwithastick        Ryzen 3700X      32GB   33:09

Network Setup

With a third testbed machine the network diagram from before changes slightly. Rather than the interfaces of the two machines being connected back to back, they are now connected to pokeitwithastick which is acting as a router.

The network now looks like this:

              -left                pokeitwithastick        -right

                          10.0.10.x               10.0.20.x
           +-------------+         +-------------+         +-------------+
           |           .2|         |.1         .1|         |.2           |
           |             |         |             |         |             |
           |       mlxen0+<------->+mlxen0 mlxen1+<------->+mlxen0       |
           |             |         |             |         |             |
           |     igb0    |         |    igb0     |         |     igb0    |
           +------+------+         +-----+-------+         +------+------+
                  |                      |                        |
                  |               freebsd|192.168.100.50          |
                  ._______               |                    ____.
                          \________.     |    .______________/
freebsd 192.168.100.10             V     V    V              freebsd 192.168.100.20
linux   192.168.100.11           +--------------+             linux   192.168.100.21
                                 |    switch    |
                                 |   (openwrt)  |
                                 +--------------+
                                        ^
                                        |
                                        | freebsd 192.168.100.2
                                   +---------+
                                   | control |
                                   +---------+

This is a pretty standard dumbell network and is good set up for performance work when you need a bottleneck.

Remote Power On

There were two features of the previous hardware that I really liked. Both machines had serial ports, which gave me a last ditch management interface option if I completley hosed the network while I wasn't at home. And the machines support lights out management with IPMI. In a sheer irony, even if the boxes hadn't had serial broken out on the motherboard, IPMI would have given me access.

I was using serial and IPMI to allow me to power on the machines remotely and control which Operating System they booted into. IPMI allowed power on, grub was configured to output to serial and video and that gave me boot control.

Tony doesn't have the same requirements as me for machines so while my Asrock X570 motherboard has a COM Port header, the Taichi motherboards don't.

Wake on LAN (WOL) is a poor replacement for IPMI power control. It is sort of famously badly implemented and it is sort of clear why. It is a packet with the MAC addresses repeated a bunch of times that turns on the system by magic. No matter your opinion of WOL the Intel network interfaces on all three machines seem to be very good at booting with WOL when they get the packets.

WOL had to be configured in the BIOS before it could be used:

In the X570 PRO4 bios configure:        
    Advanced->ACPI Configuration->PCIE Devices Power On
        "Allow the system to be wakeed up by a
        PCIE device and enable wake on LAN"

In the Taichi bios configure:           
    Advanced->ACPI Configuration->PCIE Devices Power On
        "Allow the system to be wakeed up by a
        PCIE device and enable wake on LAN"

With the BIOS set up remote power on requires using a WOL tool to send a magic packet, the control host is well placed on the network to do this with the wol command:

control $ wol a8:a1:59:95:87:60

After running the command I got nothing.

This is fine, I am a network engineer and a hacker(!), I can debug this sort of issue. Some time with tcpdump showed that I wasn't getting broadcast traffic through at all.

I tested this assertion by using a host directed WOL packet:

control $ wol a8:a1:59:95:77:ab -i 192.168.100.50

These packets appear on the host in tcpdump and after a power off are able to wake the machines up. Well at first, if I waited a while then the machine was still not responding to the WOL packet.

The diagram in the v1 network and the diagram I would have drawn for the v2 network at first was a lie. control is connected to the switch on the back of an OpenWRT router that is acting as a WiFi client to the network in the house. That switch is in turn connected to an unmanaged Netgear switch that all the rest of the network is connected to (IPMI and useful interfaces).

One of these switches was not forwarding broadcast traffic and it was only forwarding unicast traffic when the host was 'alive' enough. Not having IPMI anymore I was able to remove the Netgear switch, my needs now fit onto the four port switch on the OpenWRT router. Removing the Netgear switch didn't solve the problem and I can't remove the OpenWRT router so a different solution is required.

OpenWRT has a tool called etherwake to support Wake On LAN, I installed this on the router and I immediately got consistently working WOL:

root@OpenWrt:~# etherwake a8:a1:59:95:77:ab

I have to ssh to the router to run power on commands, but that is enough for the testbed to be useful now.

Controlling the booted OS

Having a serial interface to access the grub menu and select the booted OS was great. But I have to wipe away my tears and accept that this isn't possible with this hardware.

Grub supports something called grub-reboot , normally grub tries very hard to not write anything to disk in normal operation. You can however configure grub to use a scratch space and remember which operating system was booted before (and maybe other things).

grub-reboot uses this mechanism from the Operating System to control which menu item grub uses as default when it boots. This is part of the grub environment and is documented here .

/etc/default/grub

# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=saved
GRUB_TIMEOUT_STYLE=menu
GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="console=ttyS0,115200 console=tty0"
GRUB_CMDLINE_LINUX=""

# Uncomment to disable graphical terminal (grub-pc only)
GRUB_TERMINAL="console serial"
GRUB_SERIAL_COMMAND="serial --speed=115200"

To use grub-reboot you need to set the default menu entry to 'saved'. If you changed grub to enable grub-reboot make sure to update grub:

# sudo update-grub

With grub configured with a default menu entry of saved , we can configure grub to boot the next time from a different menu entry other than the first one.

# sudo grub-reboot 4
# sudo reboot

On these systems this allows me to boot from the FreeBSD menu entry (it is fifth in the list). Without serial to boot into FreeBSD I have to do a round trip into Linux, but this is a lot better than having to sit near the hot loud computers.

Performance

The generation change in AMD hardware had a huge improvement in processing speed, going from 58 minutes for a FreeBSD build to 30 minutes is amazing. These machines do networking stuff so it is good to look at network benchmarks for a baseline.

On Ubuntu and FreeBSD, the Threadripper machines are able to saturate 10GbE with TCP and get a about 6Gbit/s of UDP traffic. They can generate enough UDP to saturate the link so this is a huge step forward.

iperf3 benchmarks where the Ryzen system is the receiver manage half the traffic that the Threadripper systems do, capping out at about 4.5Gbit/s, the Ryzen can however send enough to saturate the link.

Base forwarding tests show no change in the throughput of the Threadripper systems. I had never considered that receive could be harder than transmit, but these baselines seem (and chatting to FreeBSD developers) seem to suggest that this isn't uncommon. For now this isn't a problem, but later I might need the Ryzen system to have more head room when running tests. If that happens I'll have a good reason to get a faster processor :D

Updating mlx4en Firmware on FreeBSD

Thu, 11 Nov 2021 00:00:00 +0000

My testbed has Nvidia/Mellanox/Chelsio 10GbE network cards which are quite old, but sit well in the price (super cheap) usability (they work great on FreeBSD ond Linux) spectrum.

There is an issue on FreeBSD 14 -CURRENT where when you load the kernel module for the card (mlx4en) kldload hangs. If you hit control C the process will continue and the module will load properly. This is also an issue when you load the module using kld_list in rc.conf and as my router machine can't be managed with serial yet I have no way to press control C when it is booting.

On the bug report I was asked if the firmware is up to date. It wasn't and doing so was not fun.

You should follow the Mellanox instructions rather than my blog post to do a firmware update, but The Mellanox FreeBSD documentation for ( Linux ) the cards is from 2015 and this is the process that worked for me in 2021.

Mellanox have a tools package you can download, there is also a port called mstflint you can install:

# pkg install mstflint

The Mellanox tools need to know which card they are speaking to, you can find the card with pciconf once you have loaded the kernel module:

# pciconf -lv | grep mlx4
mlx4_core0@pci0:9:0:0:  class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1007 subvendor=0x15b3 subdevice=0x000c

Downloading the firmware for card required a OPN and a PSID from the card. You can use mstflint to get information about the card with the pci address and the 'q' query command:

# mstflint -d pci0:9:0:0 q          
Image type:            FS2
FW Version:            2.40.5030
FW Release Date:       4.1.2017
Product Version:       02.40.50.30
Rom Info:              type=PXE version=3.4.746
Device ID:             4103
Description:           Node             Port1            Port2            Sys image
GUIDs:                 ffffffffffffffff ffffffffffffffff ffffffffffffffff ffffffffffffffff 
MACs:                                       ec0d9ae13420     ec0d9ae13421
VSD:                   
PSID:                  MT_1200111023

This didn't give me an OPN so instead I looked at every entry on the download page until I found the correct PSID.

With the firmware downloaded and unzipped you can flash it using mstflint :

# mstflint -d pci0:9:0:0 -i fw-ConnectX3Pro-rel-2_42_5000-MCX312B-XCC_Ax-FlexBoot-3.4.752.bin b

Current FW version on flash:  2.40.5030
New FW version:               2.42.5000

Burning FS2 FW image without signatures - OK  
Restoring signature                     - OK

Some Links

Fri, 19 Nov 2021 00:00:00 +0000

Once a month for the podcast I read through about 20 articles so I know what to talk about on the show . This week when I was reading the article I thought 'this is really easy, maybe I can do this for all my browser tabs'

This isn't really a fair comparison because the articles for the show are mostly short and the articles that fill out my tabs are the ones that were too long to read immediately or I wanted to keep them around.

Instead of keeping them in firefox I am going to try doing link posts again :

I think they should have just left it alone and kept calling it eBPF.

Some Classic Computing Links

Sun, 21 Nov 2021 00:00:00 +0000

Some Sunday evening reading about keeping old computers alive.

Assembling a Dual Processor VAX

Mon, 22 Nov 2021 00:00:00 +0000

Here are some links on lower level stuff. I have only read the linker script and the paper on the dual processor VAX. The other two are books that I want to get to one day and this blog is as good a place as any to keep them stashed away.

MIPS is going away in FreeBSD so this is a great time to read about MIPS assembly.

The 1982 paper on the Dual Processor VAX is excellent and you should read it. Papers aren't that interesting anymore.

Some Links About Thinking and Thinking About Work

Mon, 22 Nov 2021 00:00:00 +0000

The irony of writing a links post containing a bunch of links about productivity as a form of procrastination isn't lost on me.

part of the process means doing sub-par work to get to the good work

There are a couple of themes in here, I have been writing a lot recently and especially this year. I feel like I am starting to come across advice that makes sense to me, if you want to do things you need to well, do things. There is no substitute for practice.

While I am not going to write daily posts again, I miss the format I used before. I will look at resurrecting the tools I used to generate the daily posts before, maybe a weekly update post would be nice to do.

I plan to return to 'Work with the Garage Door Up', but I have also adopted a policy of not talking about work in progress, so you will have to wait and see.

Some ideas I picked from the electromagnetic fields

Thu, 09 Jun 2022 00:00:00 +0000

I made it to and from the Electromagnetic Field . It was quite an experience to return to a hacker large event after so many years.

A real review of the event will appear in the FreeBSD Journal this summer and I rambled a little at the end of BSDNow 459.

I don't think I could have articulated before hand exactly how much I missed the massive influx of energy being around people doing things gives you. To be honest, it probably explains why I spent so many years running things myself. The sheer magic of other people doing things seems to relax my brain and lets it fill up to the brim with new ideas.

In the vein of "“Make 50 of Something" ( which I learned about from Vi Hart making 50 fizzbuzzers ), I thought it was finally time to write down the ideas in the aftermath of the event and have a record of them.

50 is a lot of things, but I don't think this list actually captures everything that popped into my head. Much of this list is ideas I have been having over the last decade of going to hacker events, but unexecuted ideas can still be good.

ID Badge
Buckfast FM
Buckfast FM badge
BBS serial network
Government style news distribution network with backdoor leading to a bbs
TIC-80 simulator for the badge
shit camera ID printer
badge tamogotchi thing
"Cyberman" led outfit
kit dome of trees
web bbs
retro bbs
web bbs serial links to esp consoles
demo display on a dome
longest serial transmission competition
internet of buckets
network of floaty led things in lake
posters!
stickers!
sticker: The Internet is Bullshit
sticker: Computers are a fuck
sticker: "I'll drive if you feed me cheese"
sticker: a big pile of tofu
talk: 10 years of Scotland as performance art
Old Unix BBS
something targetting the badge
"set" mainframe backdrop
weather reports
Scotcon TV
An actual good computer book shop
BSD meet up
hacker breakfast
'drugwars' with real location integration
CTF coffee puzzle
folding bikes!
portraits with a camera
portraits with a paint brush
computer church
beacon hunt
demo party
paiting party
talk: Learning a place by drawing its buildings
SBC colo
deerocracy
tshirts
sticker: No one hsa found a good use for a networked computer
sketching workshop
thicknet, network
industrial quantities of mauve
$5 electronics macro photo workshop
silent journalling with [tj]

Yes, It is super vague. There are thousands of words of explanation behind some of these. That you will have to invent for yourself.

Steal what you want, I won't remember publishing this and will love seeing your implementation.

Booting FreeBSD on the RISC-V VisionFive2

Sat, 27 May 2023 00:00:00 +0000

The VisionFive 2 is a RISC-V based single board computer from Starfive integrating their JH7110 SoC. The board is available from Waveshare on its own or with some handy components to get you started.

The SoC is pitched to be a competitor to the RaspberryPi 4 in specs and along with oodles of interfaces it has a raspberrypi compatible 40 pin header exposing pins for io.

The SoC is also in the Star64 and PinetabV from Pine64 (I didn't know that the StarFive was for sale until checking when writing this).

StarFive is a Chinese integrator using IP blocks from SiFive and so it integrates cores and peripherals from the HiFive Unleashed board.

I picked one of these up to play with FreeBSD/RISC-V on a computer with a real MMU. Before doing anything I wanted to be sure that I could get Linux to boot on the board. I sort of followed the instructions from the [Waveshare wiki page][wavesharewiki], ignoring the Windows GUI tools and using dd and cu to write the image and connect to serial.

With an SD card image with their published I powered up the board and tried to connect. The wiki instructions highlight that you need to break into the boot loader during boot and manually load segments from the SD card. I tried this for about an hour or two with 3 different images (the 69 image took 40 minutes to write to the SD card), but I didn't have any luck.

Giving up I just let the u-boot run and got a booting Debian system immediately:

U-Boot 2021.10 (Feb 12 2023 - 18:15:33 +0800), Build: jenkins-VF2_515_Branch_SDK_Release-24

CPU:   rv64imacu
Model: StarFive VisionFive V2
DRAM:  8 GiB
MMC:   sdio0@16010000: 0, sdio1@16020000: 1
Loading Environment from SPIFlash... SF: Detected gd25lq128 with page size 256 Bytes, erase size 4 KiB, total 16 MiB
*** Warning - bad CRC, using default environment

StarFive EEPROM format v2

--------EEPROM INFO--------
Vendor : StarFive Technology Co., Ltd.
Product full SN: VF7110B1-2253-D008E000-00004015
data version: 0x2
PCB revision: 0xb2
BOM revision: A
Ethernet MAC0 address: 6c:cf:39:00:42:ba
Ethernet MAC1 address: 6c:cf:39:00:42:bb
--------EEPROM INFO--------

In:    serial@10000000
Out:   serial@10000000
Err:   serial@10000000
Model: StarFive VisionFive V2
Net:   eth0: ethernet@16030000, eth1: ethernet@16040000
switch to partitions #0, OK
mmc1 is current device
found device 1
bootmode flash device 1
385 bytes read in 6 ms (62.5 KiB/s)
Importing environment from mmc1 ...
Can't set block device
Hit any key to stop autoboot:  0
Can't set block device
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
Retrieving file: /boot/extlinux/extlinux.conf
Can't set block device
Error reading config file
StarFive #
StarFive #
U-Boot SPL 2021.10 (Feb 12 2023 - 18:15:33 +0800)
DDR version: dc2e84f0.
Trying to boot from SPI

OpenSBI v1.2
   ____                    _____ ____ _____
  / __ \                  / ____|  _ \_   _|
 | |  | |_ __   ___ _ __ | (___ | |_) || |
 | |  | | '_ \ / _ \ '_ \ \___ \|  _ < | |
 | |__| | |_) |  __/ | | |____) | |_) || |_
  \____/| .__/ \___|_| |_|_____/|____/_____|
        | |
        |_|

Platform Name             : StarFive VisionFive V2
Platform Features         : medeleg
Platform HART Count       : 5
Platform IPI Device       : aclint-mswi
Platform Timer Device     : aclint-mtimer @ 4000000Hz
Platform Console Device   : uart8250
Platform HSM Device       : jh7110-hsm
Platform PMU Device       : ---
Platform Reboot Device    : pm-reset
Platform Shutdown Device  : pm-reset
Firmware Base             : 0x40000000
Firmware Size             : 292 KB
Runtime SBI Version       : 1.0

Domain0 Name              : root
Domain0 Boot HART         : 1
Domain0 HARTs             : 0*,1*,2*,3*,4*
Domain0 Region00          : 0x0000000002000000-0x000000000200ffff (I)
Domain0 Region01          : 0x0000000040000000-0x000000004007ffff ()
Domain0 Region02          : 0x0000000000000000-0xffffffffffffffff (R,W,X)
Domain0 Next Address      : 0x0000000040200000
Domain0 Next Arg1         : 0x0000000042200000
Domain0 Next Mode         : S-mode
Domain0 SysReset          : yes

Boot HART ID              : 1
Boot HART Domain          : root
Boot HART Priv Version    : v1.11
Boot HART Base ISA        : rv64imafdcbx
Boot HART ISA Extensions  : none
Boot HART PMP Count       : 8
Boot HART PMP Granularity : 4096
Boot HART PMP Address Bits: 34
Boot HART MHPM Count      : 2
Boot HART MIDELEG         : 0x0000000000000222
Boot HART MEDELEG         : 0x000000000000b109


U-Boot 2021.10 (Feb 12 2023 - 18:15:33 +0800), Build: jenkins-VF2_515_Branch_SDK_Release-24

CPU:   rv64imacu
Model: StarFive VisionFive V2
DRAM:  8 GiB
MMC:   sdio0@16010000: 0, sdio1@16020000: 1
Loading Environment from SPIFlash... SF: Detected gd25lq128 with page size 256 Bytes, erase size 4 KiB, total 16 MiB
*** Warning - bad CRC, using default environment

StarFive EEPROM format v2

--------EEPROM INFO--------
Vendor : StarFive Technology Co., Ltd.
Product full SN: VF7110B1-2253-D008E000-00004015
data version: 0x2
PCB revision: 0xb2
BOM revision: A
Ethernet MAC0 address: 6c:cf:39:00:42:ba
Ethernet MAC1 address: 6c:cf:39:00:42:bb
--------EEPROM INFO--------

In:    serial@10000000
Out:   serial@10000000
Err:   serial@10000000
Model: StarFive VisionFive V2
Net:   eth0: ethernet@16030000, eth1: ethernet@16040000
switch to partitions #0, OK
mmc1 is current device
found device 1
bootmode flash device 1
Can't set block device
Failed to load '/boot/uEnv.txt'
Hit any key to stop autoboot:  0
Can't set block device
Importing environment from mmc1 ...
## Warning: Input data exceeds 1048576 bytes - truncated
## Info: input data size = 1048578 = 0x100002
385 bytes read in 6 ms (62.5 KiB/s)
## Warning: defaulting to text format
## Error: "boot2" not defined
47546 bytes read in 13 ms (3.5 MiB/s)
47546 bytes written in 563 ms (82 KiB/s)
Retrieving file: /boot/extlinux/extlinux.conf
823 bytes read in 10 ms (80.1 KiB/s)
U-Boot menu
1:  Debian GNU/Linux bookworm/sid 5.15.0-starfive
2:  Debian GNU/Linux bookworm/sid 5.15.0-starfive (rescue target)
Enter choice: 1:    Debian GNU/Linux bookworm/sid 5.15.0-starfive
Retrieving file: /boot/initrd.img-5.15.0-starfive
9684953 bytes read in 421 ms (21.9 MiB/s)
Retrieving file: /boot/vmlinuz-5.15.0-starfive
8015200 bytes read in 348 ms (22 MiB/s)
append: root=/dev/mmcblk1p3 rw console=tty0 console=ttyS0,115200 earlycon rootwait stmmaceth=chain_mode:1 selinux=0
Retrieving file: /boot/dtbs/starfive/jh7110-visionfive-v2.dtb
47546 bytes read in 13 ms (3.5 MiB/s)
   Uncompressing Kernel Image
Moving Image from 0x44000000 to 0x40200000, end=41767000
## Flattened Device Tree blob at 48000000
   Booting using the fdt blob at 0x48000000
   Using Device Tree in place at 0000000048000000, end 000000004800e9b9

Starting kernel ...

clk u5_dw_i2c_clk_core already disabled
clk u5_dw_i2c_clk_apb already disabled
[    0.000000] Linux version 5.15.0-starfive (sw_buildbot@mdcsw02) (riscv64-unknown-linux-gnu-gcc (GCC) 10.2.0, GNU ld (GNU Binutils) 2.35) #1 SMP Mon Dec 19 07:56:37 EST 2022
[    0.000000] OF: fdt: Ignoring memory range 0x40000000 - 0x40200000
[    0.000000] Machine model: StarFive VisionFive V2
[    0.000000] earlycon: uart0 at MMIO32 0x0000000010000000 (options '115200')
[    0.000000] printk: bootconsole [uart0] enabled
[    0.000000] efi: UEFI not found.
[    0.000000] Reserved memory: created CMA memory pool at 0x0000000080000000, size 512 MiB
[    0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
[    0.000000] Zone ranges:
[    0.000000]   DMA32    [mem 0x0000000040200000-0x00000000ffffffff]
[    0.000000]   Normal   [mem 0x0000000100000000-0x000000023fffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000040200000-0x00000000c010ffff]
[    0.000000]   node   0: [mem 0x00000000c0110000-0x00000000c01fffff]
[    0.000000]   node   0: [mem 0x00000000c0200000-0x000000023fffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000040200000-0x000000023fffffff]
[    0.000000] SBI specification v1.0 detected
[    0.000000] SBI implementation ID=0x1 Version=0x10002
[    0.000000] SBI TIME extension detected
[    0.000000] SBI IPI extension detected
[    0.000000] SBI RFENCE extension detected
[    0.000000] SBI SRST extension detected
[    0.000000] SBI v0.2 HSM extension detected
[    0.000000] CPU with hartid=0 is not available
[    0.000000] CPU with hartid=0 is not available
[    0.000000] riscv: ISA extensions acdfim
[    0.000000] riscv: ELF capabilities acdfim
[    0.000000] percpu: Embedded 17 pages/cpu s31528 r8192 d29912 u69632
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 2067975
[    0.000000] Kernel command line: root=/dev/mmcblk1p3 rw console=tty0 console=ttyS0,115200 earlycon rootwait stmmaceth=chain_mode:1 selinux=0
[    0.000000] Unknown command line parameters: stmmaceth=chain_mode:1 selinux=0
[    0.000000] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, linear)
[    0.000000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] software IO TLB: mapped [mem 0x00000000fbfff000-0x00000000fffff000] (64MB)
[    0.000000] Memory: 7582168K/8386560K available (9884K kernel code, 4982K rwdata, 4096K rodata, 2191K init, 401K bss, 280104K reserved, 524288K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu:     RCU restricting CPUs from NR_CPUS=8 to nr_cpu_ids=4.
[    0.000000] rcu:     RCU debug extended QS entry/exit.
[    0.000000]  Tracing variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] CPU with hartid=0 is not available
[    0.000000] riscv-intc: unable to find hart id for /cpus/cpu@0/interrupt-controller
[    0.000000] riscv-intc: 64 local interrupts mapped
[    0.000000] plic: plic@c000000: mapped 136 interrupts with 4 handlers for 9 contexts.
[    0.000000] random: get_random_bytes called from start_kernel+0x4d0/0x6e2 with crng_init=0
[    0.000000] riscv_timer_init_dt: Registering clocksource cpuid [0] hartid [1]
[    0.000000] clocksource: riscv_clocksource: mask: 0xffffffffffffffff max_cycles: 0x1d854df40, max_idle_ns: 881590404240 ns
[    0.000001] sched_clock: 64 bits at 4MHz, resolution 250ns, wraps every 2199023255500ns
[    0.009005] clocksource: timer@13050000.ch0: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 79635851949 ns
[    0.020330] clocksource: timer@13050000.ch1: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 79635851949 ns
[    0.031662] clocksource: timer@13050000.ch2: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 79635851949 ns
[    0.042988] clocksource: timer@13050000.ch3: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 79635851949 ns
[    0.054477] Console: colour dummy device 80x25
[    0.060704] printk: console [tty0] enabled
[    0.065259] Calibrating delay loop (skipped), value calculated using timer frequency.. 8.00 BogoMIPS (lpj=40000)
[    0.076482] pid_max: default: 32768 minimum: 301
[    0.081782] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, linear)
[    0.090298] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, linear)
[    0.100574] ASID allocator disabled
[    0.104515] rcu: Hierarchical SRCU implementation.
[    0.109965] EFI services will not be available.
[    0.115349] smp: Bringing up secondary CPUs ...
[    0.121954] smp: Brought up 1 node, 4 CPUs
[    0.128390] devtmpfs: initialized
[    0.139598] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.150460] futex hash table entries: 1024 (order: 4, 65536 bytes, linear)
[    0.171957] pinctrl core: initialized pinctrl subsystem
[    0.178604] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    0.185824] cpuidle: using governor menu
[    0.212080] platform soc:dsi-output: Fixing up cyclic dependency with 29400000.dc8200
[    0.221266] platform 295d0000.mipi: Fixing up cyclic dependency with soc:dsi-output
[    0.230103] platform 29590000.hdmi: Fixing up cyclic dependency with 29400000.dc8200
[    0.250583] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[    0.257968] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[    0.268496] vgaarb: loaded
[    0.271719] SCSI subsystem initialized
[    0.276289] usbcore: registered new interface driver usbfs
[    0.282365] usbcore: registered new interface driver hub
[    0.288257] usbcore: registered new device driver usb
[    0.294100] mc: Linux media interface: v0.10
[    0.298849] videodev: Linux video capture interface: v2.00
[    0.305076] Advanced Linux Sound Architecture Driver Initialized.
[    0.312174] Bluetooth: Core ver 2.22
[    0.316137] NET: Registered PF_BLUETOOTH protocol family
[    0.321995] Bluetooth: HCI device and connection manager initialized
[    0.328993] Bluetooth: HCI socket layer initialized
[    0.334376] Bluetooth: L2CAP socket layer initialized
[    0.339947] Bluetooth: SCO socket layer initialized
[    0.345655] clocksource: Switched to clocksource riscv_clocksource
[    0.359945] NET: Registered PF_INET protocol family
[    0.366328] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, linear)
[    0.382150] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, linear)
[    0.391902] TCP established hash table entries: 65536 (order: 7, 524288 bytes, linear)
[    0.401467] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, linear)
[    0.411645] TCP: Hash tables configured (established 65536 bind 65536)
[    0.419151] UDP hash table entries: 4096 (order: 6, 393216 bytes, linear)
[    0.427156] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, linear)
[    0.435832] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    0.442719] RPC: Registered named UNIX socket transport module.
[    0.449269] RPC: Registered udp transport module.
[    0.454444] RPC: Registered tcp transport module.
[    0.459629] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.467439] PCI: CLS 0 bytes, default 64
[    0.472432] Initialise system trusted keyrings
[    0.477573] workingset: timestamp_bits=62 max_order=21 bucket_order=0
[    0.477763] Unpacking initramfs...
[    0.490491] NFS: Registering the id_resolver key type
[    0.496101] Key type id_resolver registered
[    0.500717] Key type id_legacy registered
[    0.505234] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    0.512649] nfs4flexfilelayout_init: NFSv4 Flexfile Layout Driver Registering...
[    0.520870] ntfs: driver 2.1.32 [Flags: R/W].
[    0.525964] jffs2: version 2.2. (NAND) © 2001-2006 Red Hat, Inc.
[    0.533157] fuse: init (API version 7.34)
[    0.577419] NET: Registered PF_ALG protocol family
[    0.582730] Key type asymmetric registered
[    0.587294] Asymmetric key parser 'x509' registered
[    0.592771] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 249)
[    0.600969] io scheduler mq-deadline registered
[    0.605983] io scheduler kyber registered
[    0.712788] clk-starfive-jh7110 13020000.clock-controller: starfive JH7110 clkgen init successfully.
[    0.724053] L2CACHE: DataError @ 0x00000000.08040140
[    0.729629] L2CACHE: DataFail @ 0x00000000.0804006C
[    0.735093] L2CACHE: No. of Banks in the cache: 8
[    0.740308] L2CACHE: No. of ways per bank: 16
[    0.745115] L2CACHE: Sets per bank: 256
[    0.749369] L2CACHE: Bytes per cache block: 64
[    0.754279] L2CACHE: Index of the largest way enabled: 15
[    0.760564] jh7110-pmu 17030000.power-controller: registered 8 power domains
[    0.828245] Serial: 8250/16550 driver, 6 ports, IRQ sharing disabled
[    0.838268] @@#########################@@
[    0.880540] @@ dev ptr:ffffffe0bff00000/1500/1
[    0.885723] PVR_K:  1: Read BVNC 36.50.54.182 from HW device registers
[    0.893003] PVR_K:  1: RGX Device registered BVNC 36.50.54.182 with 1 core in the system
[    0.903216] [drm] Initialized pvr 1.17.6210866 20170530 for 18000000.gpu on minor 0
[    0.924159] loop: module loaded
[    0.930154] spi-nor spi0.0: gd25lq128d (16384 Kbytes)
[    1.191214] Freeing initrd memory: 9456K
[    1.212281] 3 fixed-partitions partitions found on MTD device 13010000.spi.0
[    1.220111] Creating 3 MTD partitions on "13010000.spi.0":
[    1.226190] 0x000000000000-0x000000020000 : "spl"
[    1.233161] 0x000000100000-0x000000400000 : "uboot"
[    1.240125] 0x000000f00000-0x000001000000 : "data"
[    1.248093] libphy: Fixed MDIO Bus: probed
[    1.254081] CAN device driver interface
[    1.258941] starfive-eth-plat 16030000.ethernet: force_sf_dma_mode is ignored if force_thresh_dma_mode is set.
[    1.270362] starfive-eth-plat 16030000.ethernet: User ID: 0x41, Synopsys ID: 0x52
[    1.278643] starfive-eth-plat 16030000.ethernet:     DWMAC4/5
[    1.284695] starfive-eth-plat 16030000.ethernet: DMA HW capability register supported
[    1.293336] starfive-eth-plat 16030000.ethernet: RX Checksum Offload Engine supported
[    1.301978] starfive-eth-plat 16030000.ethernet: Wake-Up On Lan supported
[    1.309464] starfive-eth-plat 16030000.ethernet: TSO supported
[    1.315906] starfive-eth-plat 16030000.ethernet: Enable RX Mitigation via HW Watchdog Timer
[    1.325125] starfive-eth-plat 16030000.ethernet: Enabled Flow TC (entries=1)
[    1.332900] starfive-eth-plat 16030000.ethernet: TSO feature enabled
[    1.339913] starfive-eth-plat 16030000.ethernet: Using 40 bits DMA width
[    1.598015] libphy: stmmac: probed
[    1.601791] YT8531 Gigabit Ethernet stmmac-0:00: attached PHY driver (mii_bus:phy_addr=stmmac-0:00, irq=POLL)
[    1.612755] YT8531 Gigabit Ethernet stmmac-0:01: attached PHY driver (mii_bus:phy_addr=stmmac-0:01, irq=POLL)
[    1.624914] starfive-eth-plat 16040000.ethernet: force_sf_dma_mode is ignored if force_thresh_dma_mode is set.
[    1.636328] starfive-eth-plat 16040000.ethernet: User ID: 0x41, Synopsys ID: 0x52
[    1.644586] starfive-eth-plat 16040000.ethernet:     DWMAC4/5
[    1.650657] starfive-eth-plat 16040000.ethernet: DMA HW capability register supported
[    1.659300] starfive-eth-plat 16040000.ethernet: RX Checksum Offload Engine supported
[    1.667947] starfive-eth-plat 16040000.ethernet: Wake-Up On Lan supported
[    1.675420] starfive-eth-plat 16040000.ethernet: TSO supported
[    1.681863] starfive-eth-plat 16040000.ethernet: Enable RX Mitigation via HW Watchdog Timer
[    1.691113] starfive-eth-plat 16040000.ethernet: Enabled Flow TC (entries=1)
[    1.698891] starfive-eth-plat 16040000.ethernet: TSO feature enabled
[    1.705903] starfive-eth-plat 16040000.ethernet: Using 40 bits DMA width
[    1.962922] libphy: stmmac: probed
[    1.966737] YT8531 Gigabit Ethernet stmmac-1:00: attached PHY driver (mii_bus:phy_addr=stmmac-1:00, irq=POLL)
[    1.977684] YT8531 Gigabit Ethernet stmmac-1:01: attached PHY driver (mii_bus:phy_addr=stmmac-1:01, irq=POLL)
[    1.990177] Intel(R) Wireless WiFi driver for Linux
[    1.997530] cdns3-starfive 10210000.usbdrd: usb mode 2 2.0 probe success
[    2.005728] usbcore: registered new interface driver uas
[    2.011632] usbcore: registered new interface driver usb-storage
[    2.035993] starfive-rtc 17040000.rtc: registered as rtc0
[    2.041959] starfive-rtc 17040000.rtc: setting system clock to 2001-01-01T00:00:00 UTC (978307200)
[    2.052010] i2c_dev: i2c /dev entries driver
[    2.056986] usbcore: registered new interface driver uvcvideo
[    2.064494] starfive-wdt 13070000.wdog: Heartbeat: timeout=15, count/2=180000000 (0aba9500)
[    2.074249] Bluetooth: HCI UART driver ver 2.3
[    2.079178] Bluetooth: HCI UART protocol H4 registered
[    2.085056] starfive-cpufreq soc:starfive,jh7110-cpufreq: Failed to get regulator for cpu!
[    2.094201] starfive-cpufreq soc:starfive,jh7110-cpufreq: Failed to init starfive cpu dvfs info
[    2.104367] sdhci: Secure Digital Host Controller Interface driver
[    2.111207] sdhci: Copyright(c) Pierre Ossman
[    2.116050] Synopsys Designware Multimedia Card Interface Driver
[    2.122972] sdhci-pltfm: SDHCI platform and OF driver helper
[    2.130039] jh7110-sec 16000000.crypto: Unable to request sec_m dma channel in DMA channel
[    2.139189] jh7110-sec 16000000.crypto: Cannot initial dma chan
[    2.146014] usbcore: registered new interface driver usbhid
[    2.152166] usbhid: USB HID core driver
[    2.156657] usbcore: registered new interface driver snd-usb-audio
[    2.172080] NET: Registered PF_PACKET protocol family
[    2.177724] can: controller area network core
[    2.182629] NET: Registered PF_CAN protocol family
[    2.187939] can: raw protocol
[    2.191215] can: broadcast manager protocol
[    2.195856] can: netlink gateway - max_hops=1
[    2.200998] Bluetooth: RFCOMM TTY layer initialized
[    2.206424] Bluetooth: RFCOMM socket layer initialized
[    2.212118] Bluetooth: RFCOMM ver 1.11
[    2.216311] Bluetooth: BNEP (Ethernet Emulation) ver 1.3
[    2.222169] Bluetooth: BNEP filters: protocol multicast
[    2.227956] Bluetooth: BNEP socket layer initialized
[    2.233611] 9pnet: Installing 9P2000 support
[    2.238405] Key type dns_resolver registered
[    2.243868] Loading compiled-in X.509 certificates
[    2.285477] starfive_jh7110-pinctrl 13040000.gpio: SiFive GPIO chip registered 64 GPIOs
[    2.295094] starfive_jh7110-pinctrl 17020000.gpio: SiFive GPIO chip registered 4 GPIOs
[    2.304078] pl08xdmac 16008000.sec_dma: initialized 8 virtual memcpy channels
[    2.311992] pl08xdmac 16008000.sec_dma: initialized 16 virtual slave channels
[    2.321510] debugfs: Directory '16008000.sec_dma' with parent 'dmaengine' already present!
[    2.330687] pl08xdmac 16008000.sec_dma: DMA: PL080 rev0 at 0x16008000 irq 23
[    2.338771] ssp-pl022 10060000.spi: ARM PL022 driver for StarFive SoC platform, device ID: 0x00041022
[    2.348977] ssp-pl022 10060000.spi: mapped registers from 0x0000000010060000 to (____ptrval____)
[    2.359120] ssp-pl022 10060000.spi: Requested frequency: 10000000 Hz is unsupported,select by default 8250000 Hz
[    2.370741] ssp-pl022 10060000.spi: will use autosuspend for runtime pm, delay 100ms
[    2.380629] i2c 2-0045: Fixing up cyclic dependency with 295d0000.mipi
[    2.388122] seeed_panel 2-0045: Unknown Atmel firmware revision: 0x00
[    2.395431] i2c 2-0019: Fixing up cyclic dependency with 295d0000.mipi
[    2.404073] at24 5-0050: supply vcc not found, using dummy regulator
[    2.411894] at24 5-0050: 512 byte 24c04 EEPROM, writable, 16 bytes/write
[    2.421270] axp15060-regulator 5-0036: Register mipi_0p9 done! vol range:900 ~ 900 mV
[    2.431418] axp15060-regulator 5-0036: Register hdmi_1p8 done! vol range:1800 ~ 1800 mV
[    2.441765] axp15060-regulator 5-0036: Register hdmi_0p9 done! vol range:900 ~ 900 mV
[    2.451931] axp15060-regulator 5-0036: Register cpu_vdd done! vol range:500 ~ 1540 mV
[    2.461375] i2c 6-0010: Fixing up cyclic dependency with 19800000.vin_sysctl
[    2.469570] imx219 6-0010: supply VANA not found, using dummy regulator
[    2.477029] imx219 6-0010: supply VDIG not found, using dummy regulator
[    2.484377] imx219 6-0010: supply VDDL not found, using dummy regulator
[    2.499168] imx219 6-0010: failed to read chip id 219
[    2.504953] imx219: probe of 6-0010 failed with error -5
[    2.513417] pcie_plda 2b000000.pcie: host bridge /soc/pcie@2B000000 ranges:
[    2.521191] pcie_plda 2b000000.pcie:      MEM 0x0030000000..0x0037ffffff -> 0x0030000000
[    2.530144] pcie_plda 2b000000.pcie:      MEM 0x0900000000..0x093fffffff -> 0x0900000000
[    2.539130] ATR entry: 0x0940000000 -> 0x0000000000 [0x0010000000] (param: 0x000001)
[    2.547698] ATR entry: 0x0030000000 -> 0x0030000000 [0x0008000000] (param: 0x000000)
[    2.556265] ATR entry: 0x0900000000 -> 0x0900000000 [0x0040000000] (param: 0x000000)
[    2.905756] pcie_plda 2b000000.pcie: Port link up.
[    2.911202] pcie_plda 2b000000.pcie: PCI host bridge to bus 0000:00
[    2.918138] pci_bus 0000:00: root bus resource [bus 00-ff]
[    2.924191] pci_bus 0000:00: root bus resource [mem 0x30000000-0x37ffffff]
[    2.931793] pci_bus 0000:00: root bus resource [mem 0x900000000-0x93fffffff pref]
[    2.940084] pci 0000:00:00.0: [1556:1111] type 01 class 0x060400
[    2.946728] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0xffffffff 64bit pref]
[    2.954755] pci 0000:00:00.0: supports D1 D2
[    2.959479] pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    2.970356] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    2.979343] pci 0000:01:00.0: [1106:3483] type 00 class 0x0c0330
[    2.986000] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00000fff 64bit]
[    2.993564] pci 0000:01:00.0: PME# supported from D0 D3cold
[    3.003108] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
[    3.010448] pci 0000:00:00.0: BAR 0: no space for [mem size 0x100000000 64bit pref]
[    3.018918] pci 0000:00:00.0: BAR 0: failed to assign [mem size 0x100000000 64bit pref]
[    3.027773] pci 0000:00:00.0: BAR 8: assigned [mem 0x30000000-0x300fffff]
[    3.035257] pci 0000:01:00.0: BAR 0: assigned [mem 0x30000000-0x30000fff 64bit]
[    3.043349] pci 0000:00:00.0: PCI bridge to [bus 01]
[    3.048836] pci 0000:00:00.0:   bridge window [mem 0x30000000-0x300fffff]
[    3.056432] pci 0000:00:00.0: enabling device (0000 -> 0002)
[    3.062673] pci 0000:01:00.0: enabling device (0000 -> 0002)
[    3.068956] pci 0000:01:00.0: quirk_usb_early_handoff+0x0/0x9d4 took 12239 usecs
[    3.077348] xhci_hcd 0000:01:00.0: xHCI Host Controller
[    3.083132] xhci_hcd 0000:01:00.0: new USB bus registered, assigned bus number 1
[    3.091725] xhci_hcd 0000:01:00.0: hcc params 0x002841eb hci version 0x100 quirks 0x0000040000000890
[    3.101883] pcie_plda 2b000000.pcie: msi#0 address_hi 0x0 address_lo 0x190
[    3.109864] xhci_hcd 0000:01:00.0: xHCI Host Controller
[    3.115640] xhci_hcd 0000:01:00.0: new USB bus registered, assigned bus number 2
[    3.123831] xhci_hcd 0000:01:00.0: Host supports USB 3.0 SuperSpeed
[    3.131508] hub 1-0:1.0: USB hub found
[    3.135704] hub 1-0:1.0: 1 port detected
[    3.141068] hub 2-0:1.0: USB hub found
[    3.145241] hub 2-0:1.0: 4 ports detected
[    3.152489] pcie_plda 2c000000.pcie: host bridge /soc/pcie@2C000000 ranges:
[    3.160247] pcie_plda 2c000000.pcie:      MEM 0x0038000000..0x003fffffff -> 0x0038000000
[    3.169208] pcie_plda 2c000000.pcie:      MEM 0x0980000000..0x09bfffffff -> 0x0980000000
[    3.178197] ATR entry: 0x09c0000000 -> 0x0000000000 [0x0010000000] (param: 0x000001)
[    3.186760] ATR entry: 0x0038000000 -> 0x0038000000 [0x0008000000] (param: 0x000000)
[    3.195309] ATR entry: 0x0980000000 -> 0x0980000000 [0x0040000000] (param: 0x000000)
[    3.355708] usb usb2-port2: over-current condition
[    3.455731] usb 1-1: new high-speed USB device number 2 using xhci_hcd
[    3.515736] usb usb2-port4: over-current condition
[    3.625932] pcie_plda 2c000000.pcie: Port link down, exit.
[    3.645690] clk-starfive-jh7110-vout 295c0000.clock-controller: starfive JH7110 clk_vout init successfully.
[    3.666898] hub 1-1:1.0: USB hub found
[    3.668071] clk-starfive-jh7110-isp 19810000.clock-controller: starfive JH7110 clk_isp init successfully.
done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done.
Begin: Running /scripts/local-premount ... done.
Warning: fsck not present, so skipping root file system
[    7.114158] EXT4-fs (mmcblk1p3): mounted filesystem with ordered data mode. Opts: (null). Quota mode: disabled.
done.
Begin: Running /scripts/local-bottom ... done.
Begin: Running /scripts/init-bottom ... done.
[    7.883228] systemd[1]: System time before build time, advancing clock.
[    7.980379] systemd[1]: systemd 251.2-5 running in system mode (+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[    8.012496] systemd[1]: Detected architecture riscv64.

Welcome to D[    8.020576] systemd[1]: Hostname set to <starfive>.
ebian GNU/Linux bookworm/sid!

With that working it was time to try FreeBSD. A little searching unearthed a forums post showing a success opensbi->u-boot->loader process, but almost immediate failure once the kernel started. The forums post used a stock 13 image to boot which made it look like getting to a partially booting kernel wouldn't require too much extra work even if there was an immediate boot issue to fix.

The Debian boot process initially recommended manually picking up the kernel and device tree images to boot. A little googling pulled up a git repo with a script and instructions on how to boot . With that magic I grabbed a current snapshot and tried to reproduce the failing boot.

U-Boot SPL 2021.10 (Feb 12 2023 - 18:15:33 +0800)
DDR version: dc2e84f0.
Trying to boot from SPI

OpenSBI v1.2
   ____                    _____ ____ _____
  / __ \                  / ____|  _ \_   _|
 | |  | |_ __   ___ _ __ | (___ | |_) || |
 | |  | | '_ \ / _ \ '_ \ \___ \|  _ < | |
 | |__| | |_) |  __/ | | |____) | |_) || |_
  \____/| .__/ \___|_| |_|_____/|____/_____|
        | |
        |_|

Platform Name             : StarFive VisionFive V2
Platform Features         : medeleg
Platform HART Count       : 5
Platform IPI Device       : aclint-mswi
Platform Timer Device     : aclint-mtimer @ 4000000Hz
Platform Console Device   : uart8250
Platform HSM Device       : jh7110-hsm
Platform PMU Device       : ---
Platform Reboot Device    : pm-reset
Platform Shutdown Device  : pm-reset
Firmware Base             : 0x40000000
Firmware Size             : 292 KB
Runtime SBI Version       : 1.0

Domain0 Name              : root
Domain0 Boot HART         : 1
Domain0 HARTs             : 0*,1*,2*,3*,4*
Domain0 Region00          : 0x0000000002000000-0x000000000200ffff (I)
Domain0 Region01          : 0x0000000040000000-0x000000004007ffff ()
Domain0 Region02          : 0x0000000000000000-0xffffffffffffffff (R,W,X)
Loading kernel...
/boot/kernel/kernel text=0x5c324c text=0x182fa4 data=0xf5508 data=0x2dc4+0x264a4c 0x8+0x1ffc5c0+0x8+0xfb1fc/
Loading configured modules...
/boot/kernel/umodem.ko text=0x2080 text=0x1280 data=0x6f0+0x4 0x8+0x71b8+0x8+0xdcc
loading required module 'ucom'
/boot/kernel/ucom.ko text=0x2618 text=0x2d28 data=0x960+0x858 0x8+0xfae0+0x8+0x1492
can't find '/etc/hostid'
can't find '/boot/entropy'

Hit [Enter] to boot immediately, or any other key for command prompt.
Booting [/boot/kernel/kernel]...               
Using DTB provided by EFI at 0x47ef2000.
Kernel entry at 0xf680002e...
Kernel args: (null)
clk u5_dw_i2c_clk_core already disabled
clk u5_dw_i2c_clk_apb already disabled
---<<BOOT>>---
GDB: no debug ports present
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2023 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 14.0-CURRENT #0 main-n263139-baef3a5b585f: Thu May 25 06:56:04 UTC 2023
    root@releng1.nyi.freebsd.org:/usr/obj/usr/src/riscv.riscv64/sys/GENERIC riscv
FreeBSD clang version 15.0.7 (https://github.com/llvm/llvm-project.git llvmorg-15.0.7-0-g8dfdcc7b7bf6)
WARNING: WITNESS option enabled, expect reduced performance.
VT: init without driver.
SBI: OpenSBI v1.2
SBI Specification Version: 1.0
CPU 0  : Vendor=SiFive Core=6/7/P200/X200-Series Processor (Hart 1)
  marchid=0x8000000000000007, mimpid=0x4210427
  MMU: 0x1<Sv39>
  ISA: 0x112d<Atomic,Compressed,Double,Float,Mult/Div>
real memory  = 4294967296 (4096 MB)
avail memory = 4122566656 (3931 MB)
sbi_trap_error: hart0: trap handler failed (error -2)
sbi_trap_error: hart0: mcause=0x0000000000000005 mtval=0x0000000040048060
sbi_trap_error: hart0: mepc=0x0000000040004cac mstatus=0x0000000200001800
sbi_trap_error: hart0: ra=0x0000000040009ee2 sp=0x0000000040047f10
sbi_trap_error: hart0: gp=0x0000000000000000 tp=0x0000000040048000
sbi_trap_error: hart0: s0=0x0000000040047f20 s1=0x0000000040048000
sbi_trap_error: hart0: a0=0x0000000040048060 a1=0x0000000000000002
sbi_trap_error: hart0: a2=0x0000000000000000 a3=0x0000000000000019
sbi_trap_error: hart0: a4=0x0000000000000001 a5=0x0000000040048060
sbi_trap_error: hart0: a6=0x00000000400480a8 a7=0x0000000000000004
sbi_trap_error: hart0: s2=0x00000000400241a8 s3=0x0000000000000000
sbi_trap_error: hart0: s4=0x0000000000000000 s5=0x0000000040029000
sbi_trap_error: hart0: s6=0x0000000040029020 s7=0x0000000000000000
sbi_trap_error: hart0: s8=0x000000000000001c s9=0x0000000040035ab0
sbi_trap_error: hart0: s10=0x0000000000000000 s11=0x0000000000000000
sbi_trap_error: hart0: t0=0x0000000000000000 t1=0x0000000000000000
sbi_trap_error: hart0: t2=0x0000000000000000 t3=0x0000000000002000
sbi_trap_error: hart0: t4=0x0000000000000000 t5=0x0000000000000000
sbi_trap_error: hart0: t6=0x0000000000000000

Cool, I can reproduce the failure. I love it when that happens.

The git repo I got the u-boot commands from was adding a memory disk image to the boot process and showed a system log of the host getting all the way up to single user (without any device specific devices discovered). Digging into their script they pull down a different device tree blob and use that with their manual u-boot boot.

There was a follow up post by Jess (jrtc27@) linking to a issue with the device tree blob . I guess her suggestions were merged into the visionfive2 tree and they have yet to make it through Linux and into FreeBSD.

We need an updated DTB.

Loading kernel...
/boot/kernel/kernel text=0x5c324c text=0x182fa4 data=0xf5508 data=0x2dc4+0x264a4c 0x8+0x1ffc5c0+0x8+0xfb1fc/
Loading configured modules...
can't find '/boot/entropy'
/boot/kernel/umodem.ko text=0x2080 text=0x1280 data=0x6f0+0x4 0x8+0x71b8+0x8+0xdcc
loading required module 'ucom'
/boot/kernel/ucom.ko text=0x2618 text=0x2d28 data=0x960+0x858 0x8+0xfae0+0x8+0x1492
can't find '/etc/hostid'

Hit [Enter] to boot immediately, or any other key for command prompt.
Booting [/boot/kernel/kernel]...               
Using DTB provided by EFI at 0x47f00000.
Kernel entry at 0xf680002e...
Kernel args: (null)
clk u5_dw_i2c_clk_core already disabled
clk u5_dw_i2c_clk_apb already disabled
---<<BOOT>>---
GDB: no debug ports present
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2023 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 14.0-CURRENT #0 main-n263139-baef3a5b585f: Thu May 25 06:56:04 UTC 2023
    root@releng1.nyi.freebsd.org:/usr/obj/usr/src/riscv.riscv64/sys/GENERIC riscv
FreeBSD clang version 15.0.7 (https://github.com/llvm/llvm-project.git llvmorg-15.0.7-0-g8dfdcc7b7bf6)
WARNING: WITNESS option enabled, expect reduced performance.
VT: init without driver.
SBI: OpenSBI v1.2
SBI Specification Version: 1.0
CPU 0  : Vendor=SiFive Core=6/7/P200/X200-Series Processor (Hart 1)
  marchid=0x8000000000000007, mimpid=0x4210427
  MMU: 0x1<Sv39>
  ISA: 0x112d<Atomic,Compressed,Double,Float,Mult/Div>
real memory  = 4294967296 (4096 MB)
avail memory = 4121706496 (3930 MB)
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
CPU 1  : Vendor=SiFive Core=6/7/P200/X200-Series Processor (Hart 2)
CPU 2  : Vendor=SiFive Core=6/7/P200/X200-Series Processor (Hart 3)
CPU 3  : Vendor=SiFive Core=6/7/P200/X200-Series Processor (Hart 4)
arc4random: WARNING: initial seeding bypassed the cryptographic random device because it was not yet seeded and the knob 'bypass_before_seeding' was enabled.
random: entropy device external interface
kbd0 at kbdmux0
ofwbus0: <Open Firmware Device Tree>
clk_fixed0: <Fixed clock> on ofwbus0
clk_fixed1: <Fixed clock> on ofwbus0
clk_fixed2: <Fixed clock> on ofwbus0
clk_fixed3: <Fixed clock> on ofwbus0
clk_fixed4: <Fixed clock> on ofwbus0
clk_fixed5: <Fixed clock> on ofwbus0
clk_fixed6: <Fixed clock> on ofwbus0
clk_fixed7: <Fixed clock> on ofwbus0
clk_fixed8: <Fixed clock> on ofwbus0
clk_fixed9: <Fixed clock> on ofwbus0
clk_fixed10: <Fixed clock> on ofwbus0
clk_fixed11: <Fixed clock> on ofwbus0
clk_fixed12: <Fixed clock> on ofwbus0
clk_fixed13: <Fixed clock> on ofwbus0
clk_fixed14: <Fixed clock> on ofwbus0
clk_fixed15: <Fixed clock> on ofwbus0
clk_fixed16: <Fixed clock> on ofwbus0
clk_fixed17: <Fixed clock> on ofwbus0
clk_fixed18: <Fixed clock> on ofwbus0
clk_fixed19: <Fixed clock> on ofwbus0
simplebus0: <Flattened device tree simple bus> on ofwbus0
plic0: <RISC-V PLIC> mem 0xc000000-0xfffffff irq 14,15,16,17,18,19,20,21,22 on simplebus0
timer0: <RISC-V Timer>
Timecounter "RISC-V Timecounter" frequency 4000000 Hz quality 1000
Event timer "RISC-V Eventtimer" frequency 4000000 Hz quality 1000
rcons0: <RISC-V console>
cpulist0: <Open Firmware CPU Group> on ofwbus0
cpu0: <Open Firmware CPU> on cpulist0
Timecounters tick every 1.000 msec
usb_needs_explore_all: no devclass
Release APs
WARNING: WITNESS option enabled, expect reduced performance.
Trying to mount root from ufs:/dev/ufs/rootfs [rw]...
mountroot: waiting for device /dev/ufs/rootfs...
random: unblocking device.
Mounting from ufs:/dev/ufs/rootfs failed with error 19.

Loader variables:
  vfs.root.mountfrom=ufs:/dev/ufs/rootfs
  vfs.root.mountfrom.options=rw

Manual root filesystem specification:
  <fstype>:<device> [options]
      Mount <device> using filesystem <fstype>
      and with the specified (optional) option list.

    eg. ufs:/dev/da0s1a
    zfs:zroot/ROOT/default
    cd9660:/dev/cd0 ro
      (which is equivalent to: mount -t cd9660 -o ro /dev/cd0 /)

  ?               List valid disk boot devices
  .               Yield 1 second (for background tasks)
  <empty line>    Abort manual input

mountroot>

Cool that works, but it is a very annoying way to boot. Can we fix it?

StarFive # printenv
baudrate=115200
boot_a_script=load ${devtype} ${devnum}:${distro_bootpart} ${scriptaddr} ${prefix}${script}; source ${scriptaddr}
boot_efi_binary=load ${devtype} ${devnum}:${distro_bootpart} ${kernel_addr_r} efi/boot/bootriscv64.efi; if fdt addr ${fdt_addr_r}; then bootefi ${kernel_addr_r} ${fdt_addr_r};else bootefi ${kernel_addr_r} ${fdtcontroladdr};fi
boot_efi_bootmgr=if fdt addr ${fdt_addr_r}; then bootefi bootmgr ${fdt_addr_r};else bootefi bootmgr;fi
boot_extlinux=sysboot ${devtype} ${devnum}:${distro_bootpart} any ${scriptaddr} ${prefix}${boot_syslinux_conf}
boot_prefixes=/ /boot/
boot_script_dhcp=boot.scr.uimg
boot_scripts=boot.scr.uimg boot.scr
boot_syslinux_conf=extlinux/extlinux.conf
boot_targets=mmc0 dhcp
bootargs=console=ttyS0,115200  debug rootwait  earlycon=sbi
bootcmd=run load_vf2_env;run importbootenv;run load_distro_uenv;run boot2;run distro_bootcmd
bootcmd_dhcp=devtype=dhcp; if dhcp ${scriptaddr} ${boot_script_dhcp}; then source ${scriptaddr}; fi;setenv efi_fdtfile ${fdtfile}; setenv efi_old_vci ${bootp_vci};setenv efi_old_arch ${bootp_arch};setenv bootp_vci PXEClient:Arch:00027:UNDI:003000;setenv bootp_arch 0x1b;if dhcp ${kernel_addr_r}; then tftpboot ${fdt_addr_r} dtb/${efi_fdtfile};if fdt addr ${fdt_addr_r}; then bootefi ${kernel_addr_r} ${fdt_addr_r}; else bootefi ${kernel_addr_r} ${fdtcontroladdr};fi;fi;setenv bootp_vci ${efi_old_vci};setenv bootp_arch ${efi_old_arch};setenv efi_fdtfile;setenv efi_old_arch;setenv efi_old_vci;
bootcmd_distro=run fdt_loaddtb; run fdt_sizecheck; run set_fdt_distro; sysboot mmc ${fatbootpart} fat c0000000 ${bootdir}/${boot_syslinux_conf};
bootcmd_mmc0=devnum=0; run mmc_boot
bootdelay=2
bootdir=/boot
bootenv=uEnv.txt
bootmode=flash
bootpart=1:3
chip_vision=B
chipa_gmac_set=fdt set /soc/ethernet@16030000/ethernet-phy@0 tx_inverted_10 <0x0>;fdt set /soc/ethernet@16030000/ethernet-phy@0 tx_inverted_100 <0x0>;fdt set /soc/ethernet@16030000/ethernet-phy@0 tx_inverted_1000 <0x0>;fdt set /soc/ethernet@16030000/ethernet-phy@0 tx_delay_sel <0x9>;fdt set /soc/ethernet@16040000/ethernet-phy@1 tx_inverted_10 <0x0>;fdt set /soc/ethernet@16040000/ethernet-phy@1 tx_inverted_100 <0x0>;fdt set /soc/ethernet@16040000/ethernet-phy@1 tx_inverted_1000 <0x0>;fdt set /soc/ethernet@16040000/ethernet-phy@1 tx_delay_sel <0x9>
chipa_set=if test ${chip_vision} = A; then run chipa_gmac_set;fi;
chipa_set_linux=fdt addr ${fdt_addr_r};run visionfive2_mem_set;run chipa_set;
chipa_set_linux_force=fdt addr ${fdt_addr_r};run visionfive2_mem_set;run chipa_gmac_set;
chipa_set_uboot=fdt addr ${uboot_fdt_addr};run chipa_set;
chipa_set_uboot_force=fdt addr ${uboot_fdt_addr};run chipa_gmac_set;
devnum=1
distro_bootcmd=for target in ${boot_targets}; do run bootcmd_${target}; done
distroloadaddr=0xb0000000
efi_dtb_prefixes=/ /dtb/ /dtb/current/
eth0addr=6c:cf:39:00:42:ba
eth1addr=6c:cf:39:00:42:bb
ethaddr=6c:cf:39:00:42:ba
ext4bootenv=ext4load mmc ${bootpart} ${loadaddr} ${bootdir}/${bootenv}
fatbootpart=1:2
fdt_addr_r=0x46000000
fdt_high=0xffffffffffffffff
fdt_loaddtb=fatload mmc ${fatbootpart} ${fdt_addr_r} ${bootdir}/dtbs/${fdtfile}; fdt addr ${fdt_addr_r};
fdt_sizecheck=fatsize mmc ${fatbootpart} ${bootdir}/dtbs/${fdtfile};
fdtaddr=fffc5dd0
fdtcontroladdr=fffc5dd0
fdtfile=starfive/starfive_visionfive2.dtb
importbootenv=echo Importing environment from mmc${devnum} ...; env import -t ${loadaddr} ${filesize}
initrd_high=0xffffffffffffffff
ipaddr=192.168.120.230
kernel_addr_r=0x40200000
load_distro_uenv=fatload mmc ${fatbootpart} ${distroloadaddr} ${bootdir}/${bootenv}; env import ${distroloadaddr} 17c;
load_efi_dtb=load ${devtype} ${devnum}:${distro_bootpart} ${fdt_addr_r} ${prefix}${efi_fdtfile}
load_vf2_env=fatload mmc ${bootpart} ${loadaddr} ${testenv}
loadaddr=0xa0000000
loadbootenv=fatload mmc ${bootpart} ${loadaddr} ${bootenv}
memory_addr=40000000
memory_size=200000000
mmc_boot=if mmc dev ${devnum}; then devtype=mmc; run scan_dev_for_boot_part; fi
mmcbootenv=run scan_mmc_dev; setenv bootpart ${devnum}:${mmcpart}; if mmc rescan; then run loadbootenv && run importbootenv; run ext4bootenv && run importbootenv; if test -n $uenvcmd; then echo Running uenvcmd ...; run uenvcmd; fi; fi
mmcpart=3
netmask=255.255.255.0
partitions=name=loader1,start=17K,size=1M,type=${type_guid_gpt_loader1};name=loader2,size=4MB,type=${type_guid_gpt_loader2};name=system,size=-,bootable,type=${type_guid_gpt_system};
preboot=run chipa_set_uboot;run mmcbootenv
pxefile_addr_r=0x45900000
ramdisk_addr_r=0x46100000
scan_dev_for_boot=echo Scanning ${devtype} ${devnum}:${distro_bootpart}...; for prefix in ${boot_prefixes}; do run scan_dev_for_extlinux; run scan_dev_for_scripts; done;run scan_dev_for_efi;
scan_dev_for_boot_part=part list ${devtype} ${devnum} -bootable devplist; env exists devplist || setenv devplist 1; for distro_bootpart in ${devplist}; do if fstype ${devtype} ${devnum}:${distro_bootpart} bootfstype; then run scan_dev_for_boot; fi; done; setenv devplist
scan_dev_for_efi=setenv efi_fdtfile ${fdtfile}; for prefix in ${efi_dtb_prefixes}; do if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${efi_fdtfile}; then run load_efi_dtb; fi;done;run boot_efi_bootmgr;if test -e ${devtype} ${devnum}:${distro_bootpart} efi/boot/bootriscv64.efi; then echo Found EFI removable media binary efi/boot/bootriscv64.efi; run boot_efi_binary; echo EFI LOAD FAILED: continuing...; fi; setenv efi_fdtfile
scan_dev_for_extlinux=if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${boot_syslinux_conf}; then echo Found ${prefix}${boot_syslinux_conf}; run boot_extlinux; echo SCRIPT FAILED: continuing...; fi
scan_dev_for_scripts=for script in ${boot_scripts}; do if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${script}; then echo Found U-Boot script ${prefix}${script}; run boot_a_script; echo SCRIPT FAILED: continuing...; fi; done
scan_mmc_dev=if test ${bootmode} = flash; then if mmc dev ${devnum}; then echo found device ${devnum};else setenv devnum 0;mmc dev 0;fi; fi; echo bootmode ${bootmode} device ${devnum};
scan_sf_for_scripts=${devtype} read ${scriptaddr} ${script_offset_f} ${script_size_f}; source ${scriptaddr}; echo SCRIPT FAILED: continuing...
script_offset_f=0x1fff000
script_size_f=0x1000
scriptaddr=0x43900000
serial#=VF7110B1-2253-D008E000-00004015
set_fdt_distro=if test ${chip_vision} = A; then if test ${memory_size} = 200000000; then run chipa_gmac_set;run visionfive2_mem_set;fatwrite mmc ${fatbootpart} ${fdt_addr_r} ${bootdir}/dtbs/${fdtfile} ${filesize};else run chipa_gmac_set;run visionfive2_mem_set;fatwrite mmc ${fatbootpart} ${fdt_addr_r} ${bootdir}/dtbs/${fdtfile} ${filesize};fi;else run visionfive2_mem_set;fatwrite mmc ${fatbootpart} ${fdt_addr_r} ${bootdir}/dtbs/${fdtfile} ${filesize};fi;
sf_boot=if sf probe ${busnum}; then devtype=sf; run scan_sf_for_scripts; fi
stderr=serial@10000000
stdin=serial@10000000
stdout=serial@10000000
testenv=vf2_uEnv.txt
type_guid_gpt_loader1=5B193300-FC78-40CD-8002-E86C45580B47
type_guid_gpt_loader2=2E54B353-1271-4842-806F-E436D6AF6985
type_guid_gpt_system=0FC63DAF-8483-4772-8E79-3D69D8477DE4
uboot_fdt_addr=0xfffc5dd0
ver=U-Boot 2021.10 (Feb 12 2023 - 18:15:33 +0800)
visionfive2_mem_set=fdt memory ${memory_addr} ${memory_size};

Environment size: 7216/65532 bytes

From the u-boot environment we can find the DTB file u-boot is passing on to the EFI loader. If we move our DTB to be in that location on disk we should be able to boot automatically.

fdtfile=starfive/starfive_visionfive2.dtb

With the working DTB in the correct place we can boot the FreeBSD image from power on without any manual intervention. We fail at mount root because we are missing drivers for any sort of storage medium, eMMC or SD (both are on a SDIO controller), USB or NVMe. We don't have any drivers for this board yet, the stuff detected in the dmesg above is devices in the DTB which exist elsewhere and so have FreeBSD drivers.

To get past mount root (without using a memory disk) we need to have a storage driver, so lets pull up the docs and write one!

Much angered reading and intense searching later

It looks like there isn't useful documentation available for a driver writer. There are open source drivers on their way into the Linux Kernel, but the Software TRM is a high level overview, the software developer guides explain how to use interfaces from Linux and everything else is a useless summary.

From searching I found that Haiku is booting and running on the VisionFive2 . They have drivers for the PCIe controller giving them USB and NVMe and graphics on the frame buffer because u-boot is kind enough to set it up. As far as I can tell (I hope I'm wrong), they PCIe controller driver has been cribbed from the Linux Kernel.

While writing drivers from others source is possible, it is a bit hairy going from the GPL to BSD licenses and it is terribly annoying when you hit issues to not have documentation.

I can't see FreeBSD supporting the VisionFive2 without there being proper open documentation to write drivers from.

BSDCan 2023

Sun, 28 May 2023 00:00:00 +0000

In the Days before my fiancée and I explored Ottawa a bit, got stranded in a forest in Quebec and did our usual tour of coffee shops and book stores (got to make sure your bags are heavier on the way home somehow).

The evening before BSDCan and the Devsummits is traditionally a meeting of the Editorial Board of the FreeBSD Journal and accomplices to discuss what we want to cover in the next year. We had a brain storming session that picked up nearly 40 ideas for articles. From this we will figure out topics for the issues in the next year.

Devsummit Day 1

Typically in the days before a BSD conference the FreeBSD project holds a developer summit with some time aside for hacking . BSDCan is a big event for this and the Devsummit before was well attended.

Foundation update

The Devsummit normally starts with an update from the FreeBSD foundation on what is going on. This summits update included a lot of information about on going projects, new hires and direction the foundation is working in. After a pandemic gap in major updates there was a lot of new information from the foundation.

This year is the 30th anniversary of the start of the FreeBSD project and the foundation got us cake to celebrate!

Core update

Next up is normally an update from core letting us know what they are doing about it.

Guided code reading with Mark J.

For a new Devsummit thing Mark Johnson did a guided code reading session in the scheduler. This was an introduction to the files and functions of the ULE scheduler ifreebsd which has been a topic of discussion in recent years and more in the last two months.

I think as a new format this worked quite well, taking 40 minutes or so to walk around a sun system might be a good way to give devs kick start introductions when help is needed.

Story time with Mike Karels

Wrapping up the first Devsummit day we had a story time session with Mike Karels. Mike is one of the OG BSD people, it was great to hear his experiences working with and on BSD for the last 4 decades. This one was recorded and will be worth revisiting in the future.

Devsummit Day 2

I took the morning of the second day of the Devsummit to stare into my slides and wonder why I had signed up to talk again. For me this is normal fair in the lead up to a conference, but boy am I good at forgetting about all the lead in anxiety before presenting a talk.

15 planning

I made it to the Devsummit to join the have/need/want/axe session for FreeBSD 15. These sessions are where we try to figure out what we might have for an upcoming release and what we need to get devs to sign up to do.

In the last few years we've had more desire to kill off old subsystems and tools, making the code base both smaller and easier to maintain. 15 will continue this and there is a nice list of things to get rid of.

Group photo and off to the pub to help set up for registration

Post lunch we had a project group photo of everyone that made the Devsummit. I decided to duck out for the rest of the day and help Dan prepare stuff for registration for start of the conference. It helped that this prep was in a pub.

BSDCan Day 1

I took the morning of the first day to review my slides, drink coffee and try to avoid becoming agitated before my talk skipping the two sessions which happened in the slots before.

My talk - Making FreeBSD QUIC

I reprised my talk from EuroBSDCon, I had hoped to expand on it, but procrastination and time limitations got in the way. I’m working on a hacky GSO for FreeBSD, while it would have been great to present early results, they just weren't ready in time.

I think the talk went well, but as it was a second presentation that isn’t too surprising. If I present at a BSD conference again it’ll be something completely new and weirdly out of the box.

Mateusz - Dtrace vs eBPF

An update from EuroBSDCon 2022, Mateusz looked at the tracing overheads for eBPF and DTrace . There are a lot of claims out there about how things work and the impact performance tools have ranging from no overhead so the tool is safe to use all the way up to too much overhead so tools are never safe to use.

Mateusz normalised out differences between FreeBSD and Linux and explored the overheads of both sets of tools on each OS.

Kirk - gunion

Kirk came to give an overview of a new geom module call gunion:

“Turns out explains gunion takes about 10 minutes so I’ll give some background on geom too”

Kirk covered why we have geom and how it developed over time. He talked about the nop geom which has grown enough features to not be a no operation anymore. He wrapped up talking about gunion - a new geom writable layer that grabs writes to a paired geom allowing you to dry run potentially destructive operations. The use came for this is a quick fsck -y without the pitfall fall of potentially killing your disk.

Me again, with Benedict and Allan - BSDNow Live

Final session for day 1 a live performance of BSDNow. We went into this one with as much prep as we normally do and that just wasn't enough.

Our big announcement at the end is that Allan is going to retire from hosting the show with the live show being his final episode.

I got positive feedback after and I may be tempted to try this format again. The recording is the record of if this was a good use of everyone’s time.

In the future I'll ask to have the hour before the live show blocked out for set up and preparation and bring enough of a mic set up that we can be heard in the room, on the stream and have a good mix for the show. This 'speak into the goat' experience while very funny was not a great recording environment.

BSDCan Day 2

I started the second day with a detour to get good coffee from Little Victories and got caught in the start of the day of rain. The little victories chain (there are 2 in 2023) is my recommendation for the best coffee near the venue, I made the 10 minute detour twice on my way in to have “good” coffee.

Kristof - if_ovpn

The first session was Kristof talking about the design and implementation of the if_ovpn kernel module. This modules allows crypto and packet processing to be handled in the kernel avoiding a round trip to user space for every packet.

This works in a similar way to TLS offload where the meat is handled in the kernel, but the slower control traffic has to dealt with my a user space process.

It seemed to offer pretty good performance improvements over the user space side; almost 5x

Taylor Campbell - NetBSD disk encryption

It is always good to know what is going on in other parts of the BSD world and with two competing talks which I’d watch on YouTube I decided to see a NetBSD talk.

Taylor spoke about the design and usage of CGD disk device which offers encryption in NetBSD. He talked about some of the details and limitations of the current system. I’m glad that there are modern descriptions of NetBSD kernel internals happening at conferences. I do wish I’d managed to see one of the OpenBSD sessions but it didn't work out.

Corey Stephan - Using BSD in the liberal arts

Corey’s talk was a change of pace from the normal content at a BSD conference. Corey is a theologian teaching at a Texan university.

“Who has sufficient time to build a better computer workflow”

He spoke about the role open software and our communities can have in the study and teaching of liberal arts. Corey has had good experiences working with us BSD folk and is able to work well with packaged software. I’m hoping he’ll give an update in a few years and tell us if he managed to convert anyone else.

Lunch interview with the freebsd foundation

During lunch I did a short “how you got into bsd” interview with Drew from the foundation. I think it’s going into TikTok or something similar.

Wuyang Chung - single address space capability system

Another change of pace and a very academic style presentation.

Warner Losh - kboot: Booting FreeBSD with LinuxBoot

LinuxBoot offers a mechanism for a Linux kernel to boot another kernel, this can have a huge advantage if you need to reboot something that takes a long time to move through firmware initialisation. With LinuxBoot you kexec the new image and can do a warm restart.

Warner did work to enable FreeBSD to be booted with kexec/Linuxboot. This is similar to the way FreeBSD could be booted on the PlayStation 3, but with a decade plus of new stuff added in.

Closing and Auction

BSDCan ends each year with a closing ceremony and auction. In the closing ceremony Dan announced that he was stepping down after 20 years of running the conference. New team is coming together to keep BSDCan going into its third decade.

The final event is an auction to raise money for the Ottawa mission. I joined in and picked up a 5th copy of the 30th anniversary journal edition with signatures from the people at the conference.

Booting Linux on the Pine64 Ox64 SBC

Sun, 28 May 2023 00:00:00 +0000

The Ox64 (notice that is a letter 'O' not a 0) is a tiny microcontroller sized single board computer from Pine64. The Ox64 is built around the bl808 SOC from Bouffalo Lab. It has one application level 64 bit RISC-V core, a 32 bit RISC-V core and an extra little 32 bit RISC-V core for handling io.

It is a computer in the form factor of a microcontroller and it is built around what seems to be an open eco system. The Ox64 is early in its lifetime and Pine64 are quite good at making hardware for the community to build software around. That means that software is early and there is much to do (as I learned) to get code running.

Lets get Bouffalo's example Linux build going on this tiny thing.

First signs of life

There is quite a bit of documentation and write ups in the wiki page which helped me get going.

The links in the wiki aren't super clear about how to get any sort of working Linux image on the Ox64. Before trying any of those I wanted to get some signs of life out of the board. A two hour round trip to the hackerspace later I had headers soldered on to my two boards and connected up the Waveshare USB uart I got with the VisionFive2 kit.

The instructions aren't clear about where to expect the initial uart output to appear. The pinout on the wiki has RXD and TXD marked in several places, but also GPIO14/GPIO15 for UART0 TX adn UART0 RX. Two baud rates are listed 115200 and 2000000 (this might be faster than your usb uart can go), I got the following out of GPIO14/15 with a baud rate of 2000000.

$ sudo cu -l /dev/ttyU0 -s 2000000

 ____               __  __      _       _       _     
|  _ \             / _|/ _|    | |     | |     | |    
| |_) | ___  _   _| |_| |_ __ _| | ___ | | __ _| |__  
|  _ < / _ \| | | |  _|  _/ _` | |/ _ \| |/ _` | '_ \ 
| |_) | (_) | |_| | | | || (_| | | (_) | | (_| | |_) |
|____/ \___/ \__,_|_| |_| \__,_|_|\___/|_|\__,_|_.__/ 

Build:19:50:39,Nov 20 2022
Copyright (c) 2022 Bouffalolab team
dynamic memory init success,heap size = 93 Kbyte 
sig1:ffff32ff
sig2:0000ffff
Pong!
Ping!
Pong!
Ping!
Pong!

Flashing with DevCube

The buildroot instructions on the wiki have an example usage of the SDK to build firmware and how to flash boot firmware for the two cores.

Bouffalo's flashing tool is called Devcube, they provide binaries for Linux, Mac OS and Windows. The Linux Binary (version 1.8.4) refused to run on FreeBSD with Linux compat giving me:

$ ./BLDevCube-ubuntu 
PySide2/__init__.py: Unable to import shiboken2 from /tmp/_MEIN3lvWA/base_library.zip, /tmp/_MEIN3lvWA/lib-dynload, /tmp/_MEIN3lvWA
/lib64/libc.so.6: version `GLIBC_2.18' not found (required by /tmp/_MEIN3lvWA/libstdc++.so.6)

I don't have a lot of experience using Linux compat and I want to get this board going in less than a month of work so I flashed from Mac OS.

In Mac OS I was able to run Devcube, but kept hitting failures trying to flash the firmware images onto the board. Eventually I started trying more extreme things, the wiki page suggested that certain USB uart chips just don't work, so I tried another adapter. Another suggested solution was to write a USB uart firmware on to a pi pico and use that. I pulled the pi pico from my BluesSCSI v2, copied the firmware on to the virtual msdos file system and that finally worked.

With the firmware written the output on uart0 changed to:

[I][] 
[I][]   ____                   ____               __  __      _       
[I][]  / __ \                 |  _ \             / _|/ _|    | |      
[I][] | |  | |_ __   ___ _ __ | |_) | ___  _   _| |_| |_ __ _| | ___  
[I][] | |  | | '_ \ / _ \ '_ \|  _ < / _ \| | | |  _|  _/ _` | |/ _ \ 
[I][] | |__| | |_) |  __/ | | | |_) | (_) | |_| | | | || (_| | | (_) |
[I][]  \____/| .__/ \___|_| |_|____/ \___/ \__,_|_| |_| \__,_|_|\___/ 
[I][]        | |                                                      
[I][]        |_|                                                      
[I][] 
[I][] Powered by BouffaloLab
[I][] Build:11:52:22,Mar  6 2023
[I][] Copyright (c) 2023 OpenBouffalo team
[I][] Copyright (c) 2022 Bouffalolab team
[I][] =========== flash cfg ==============
[I][] jedec id   0xEF6018
[I][] mid            0xEF
[I][] iomode         0x04
[I][] clk delay      0x01
[I][] clk invert     0x01
[I][] read reg cmd0  0x05
[I][] read reg cmd1  0x35
[I][] write reg cmd0 0x01
[I][] write reg cmd1 0x31
[I][] qe write len   0x01
[I][] cread support  0x00
[I][] cread code     0xFF
[I][] burst wrap cmd 0x77
[I][] sector size:   0x04
[I][] =====================================
[I][] dynamic memory init success,heap size = 156 Kbyte 
[I][MAIN] Starting Mailbox Handlers
[I][MBOX] Forwarding Interupt SDH (33) to D0 (0x58008bbc)
[I][MBOX] Forwarding Interupt GPIO (60) to D0 (0x58008d0e)
[I][MAIN] Running...
[I][MBOX] Mailbox IRQ Stats:
[I][MBOX]   Peripheral SDH (33): 0
[I][MBOX]   Peripheral GPIO (60): 0
[I][MBOX] Unhandled Interupts: 0 Unhandled Signals 0
[I][MBOX] ====================================
[I][MBOX] Mailbox IRQ Stats:
[I][MBOX]   Peripheral SDH (33): 0
[I][MBOX]   Peripheral GPIO (60): 0
[I][MBOX] Unhandled Interupts: 0 Unhandled Signals 0
[I][MBOX] ====================================
[I][MBOX] Mailbox IRQ Stats:
[I][MBOX]   Peripheral SDH (33): 0
[I][MBOX]   Peripheral GPIO (60): 0
[I][MBOX] Unhandled Interupts: 0 Unhandled Signals 0
[I][MBOX] ====================================

This is promising and nice to see firmware giving us a channel for further debugging in the future.

I wrote the example sd card image onto a card and tried to boot and got nothing.

In the instructions I spotted that the uart for u-boot/linux was on gpio 16/17 on the other side of the board to where I had soldered headers (I only had enough right angled headers to do one side). Luckily, I soldered straight headers onto all pins of the second Ox64 I got for working on this.

With those connected I still got nothing, I tried all the uarts on the pinout in the wiki. Dropping to 115200 for the baud because some disconnected messages in the discord suggested that might be the issue. I verified that the baud should be 2000000 in the DTB files.

I gave in, joined the Pine64 discord and asked what might be up. Arne said that Devcube 1.8.4 has some konwn issues and I should try Devcube 1.8.3 . This version didn't like uploading the bl808-firmware.bin file, it was unable to establish a connection to the firmware and gave me the same errors I saw before when I was using the USB uart adapter.

I tried fruitlessly many combinations of boot switches and power cycles, but I couldn't get the firmware to write.

Not sure what to try next I connected up to the system uart to see if I could get any further hints and instead I got a booting Linux:

[I][] 
[I][]   ____                   ____               __  __      _       
[I][]  / __ \                 |  _ \             / _|/ _|    | |      
[I][] | |  | |_ __   ___ _ __ | |_) | ___  _   _| |_| |_ __ _| | ___  
[I][] | |  | | '_ \ / _ \ '_ \|  _ < / _ \| | | |  _|  _/ _` | |/ _ \ 
[I][] | |__| | |_) |  __/ | | | |_) | (_) | |_| | | | || (_| | | (_) |
[I][]  \____/| .__/ \___|_| |_|____/ \___/ \__,_|_| |_| \__,_|_|\___/ 
[I][]        | |                                                      
[I][]        |_|                                                      
[I][] 
[I][] Powered by BouffaloLab
[I][] Build:11:52:04,Mar  6 2023
[I][] Copyright (c) 2023 OpenBouffalo team
[I][] Copyright (c) 2022 Bouffalolab team
[I][] dynamic memory init success,heap s[I][LowLoad] D0 start...
[I][LowLoad] low_load start... 
[I][LowLoad] Header at 0x5d5ff000
[I][LowLoad] Section dtb(1) - Start 0x5d5ff100, Size 14314
[I][LowLoad] Copying DTB to 0x51ff8000...0x51ffb7ea
[I][LowLoad] Done!
[I][LowLoad] Section OpenSBI(2) - Start 0x5d60f100, Size 109864
[I][LowLoad] Copying OpenSBI to 0x3ef80000...0x3ef9ad28
[I][LowLoad] Done!
[I][LowLoad] Section Kernel(3) - Start 0x5d62f100, Size 315597
[I][LowLoad] Uncompressing Kernel to 0x50000000...
[I][LowLoad] Done!
[I][LowLoad] CRC: 00000000
[I][LowLoad] load time: 61311 us 
[I][LowLoad] Setting PMP
[I][LowLoad] Booting OpenSBI at 0x000000003ef80000 with DTB at 0x51ff8000

OpenSBI v1.2
  ____                    _____ ____ _____
 / __ \                  / ____|  _ \_   _|
| |  | |_ __   ___ _ __ | (___ | |_) || |
| |  | | '_ \ / _ \ '_ \ \___ \|  _ < | |
| |__| | |_) |  __/ | | |____) | |_) || |_
 \____/| .__/ \___|_| |_|_____/|____/_____|
       | |
       |_|

  Platform Name             : Pine64 Ox64 (D0)
  Platform Features         : medeleg
  Platform HART Count       : 1
  Platform IPI Device       : aclint-mswi
  Platform Timer Device     : aclint-mtimer @ 1000000Hz
  Platform Console Device   : bflb_uart
  Platform HSM Device       : ---
  Platform PMU Device       : ---
  Platform Reboot Device    : ---
  Platform Shutdown Device  : ---
  Firmware Base             : 0x3ef80000
  Firmware Size             : 200 KB
  Runtime SBI Version       : 1.0

  Domain0 Name              : root
  Domain0 Boot HART         : 0
  Domain0 HARTs             : 0*
  Domain0 Region00          : 0x00000000e4008000-0x00000000e400bfff (I)
  Domain0 Region01          : 0x00000000e4000000-0x00000000e4007fff (I)
  Domain0 Region02          : 0x000000003ef80000-0x000000003efbffff ()
  Domain0 Region03          : 0x0000000000000000-0xffffffffffffffff (R,W,X)
  Domain0 Next Address      : 0x0000000050000000
  Domain0 Next Arg1         : 0x0000000051ff8000
  Domain0 Next Mode         : S-mode
  Domain0 SysReset          : yes

  Boot HART ID              : 0
  Boot HART Domain          : root
  Boot HART Priv Version    : v1.11
  Boot HART Base ISA        : rv64imafdcvx
  Boot HART ISA Extensions  : time
  Boot HART PMP Count       : 8
  Boot HART PMP Granularity : 4096
  Boot HART PMP Address Bits: 38
  Boot HART MHPM Count      : 8
  Boot HART MIDELEG         : 0x0000000000000222
  Boot HART MEDELEG         : 0x000000000000b109


  U-Boot 2023.04-rc2 (Mar 06 2023 - 11:48:40 +0000)

  DRAM:  64 MiB
  Core:  36 devices, 17 uclasses, devicetree: board
  MMC:   mmc@20060000: 0
  Loading Environment from FAT... Unable to read "uboot.env" from mmc0:2... 
  Loading Environment from nowhere... OK
  In:    serial@30002000
  Out:   serial@30002000
  Err:   serial@30002000
  Net:   
  Warning: emac@20070000 (eth0) using random MAC address - fa:6f:53:9f:5c:36
  eth0: emac@20070000
  Hit any key to stop autoboot:  0 
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:2...
Found /extlinux/extlinux.conf
Retrieving file: /extlinux/extlinux.conf
Select the boot mode
1:  Pine64 0X64 Kernel
2:  Sipeed M1SDock Kernel
Enter choice: 1
1:  Pine64 0X64 Kernel
Retrieving file: /extlinux/../Image
append: root=PARTLABEL=rootfs rootwait rw rootfstype=ext4 console=ttyS0,2000000 loglevel=8 earlycon=sbi
Retrieving file: /extlinux/../bl808-pine64-ox64.dtb
## Flattened Device Tree blob at 51ff8000
Booting using the fdt blob at 0x51ff8000
Working FDT set to 51ff8000
Loading Device Tree to 0000000053f22000, end 0000000053f25fab ... OK
Working FDT set to 53f22000

Starting kernel ...

[    0.000000] Linux version 6.2.0 (runner@fv-az587-938) (riscv64-unknown-linux-gnu-gcc (Xuantie-900 linux-5.10.4 glibc gcc Toolchain V2.6.1 B-20220906) 10.2.0, GNU ld (GNU Binutils) 2.35) #1 Mon Mar  6 11:17:27 UTC 2023
[    0.000000] OF: fdt: Ignoring memory range 0x50000000 - 0x50200000
[    0.000000] Machine model: Pine64 Ox64
[    0.000000] earlycon: sbi0 at I/O port 0x0 (options '')
[    0.000000] printk: bootconsole [sbi0] enabled
[    0.000000] efi: UEFI not found.
[    0.000000] Zone ranges:
[    0.000000]   DMA32    [mem 0x0000000050200000-0x0000000053ffffff]
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000050200000-0x0000000053ffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000050200000-0x0000000053ffffff]
[    0.000000] SBI specification v1.0 detected
[    0.000000] SBI implementation ID=0x1 Version=0x10002
[    0.000000] SBI TIME extension detected
[    0.000000] SBI IPI extension detected
[    0.000000] SBI RFENCE extension detected
[    0.000000] riscv: base ISA extensions acdfim
[    0.000000] riscv: ELF capabilities acdfim
[    0.000000] pcpu-alloc: s0 r0 d32768 u32768 alloc=1*32768
[    0.000000] pcpu-alloc: [0] 0 
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 15655
[    0.000000] Kernel command line: root=PARTLABEL=rootfs rootwait rw rootfstype=ext4 console=ttyS0,2000000 loglevel=8 earlycon=sbi
[    0.000000] Dentry cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    0.000000] Inode-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 48084K/63488K available (3941K kernel code, 4584K rwdata, 2048K rodata, 2118K init, 301K bss, 15404K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] riscv-intc: 64 local interrupts mapped
[    0.000000] plic: interrupt-controller@e0000000: mapped 64 interrupts with 1 handlers for 2 contexts.
[    0.000000] riscv-timer: riscv_timer_init_dt: Registering clocksource cpuid [0] hartid [0]
[    0.000000] clocksource: riscv_clocksource: mask: 0xffffffffffffffff max_cycles: 0x1d854df40, max_idle_ns: 3526361616960 ns
[    0.000003] sched_clock: 64 bits at 1000kHz, resolution 1000ns, wraps every 2199023255500ns
[    0.000920] Console: colour dummy device 80x25
[    0.001166] Calibrating delay loop (skipped), value calculated using timer frequency.. 2.00 BogoMIPS (lpj=4000)
[    0.001687] pid_max: default: 32768 minimum: 301
[    0.002408] Mount-cache hash table entries: 512 (order: 0, 4096 bytes, linear)
[    0.002749] Mountpoint-cache hash table entries: 512 (order: 0, 4096 bytes, linear)
[    0.007710] cblist_init_generic: Setting adjustable number of callback queues.
[    0.008028] cblist_init_generic: Setting shift to 0 and lim to 1.
[    0.008940] ASID allocator using 16 bits (65536 entries)
[    0.010160] EFI services will not be available.
[    0.011191] devtmpfs: initialized
[    0.014879] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.015375] futex hash table entries: 256 (order: 0, 6144 bytes, linear)
[    0.016003] pinctrl core: initialized pinctrl subsystem
[    0.018536] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    0.019435] DMA: preallocated 128 KiB GFP_KERNEL pool for atomic allocations
[    0.019823] DMA: preallocated 128 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[    0.025441] bflb-ipc 30005000.mailbox: Bouffalo Lab IPC mailbox interrupt controller
[    0.030626] SCSI subsystem initialized
[    0.033265] clocksource: Switched to clocksource riscv_clocksource
[    0.061833] NET: Registered PF_INET protocol family
[    0.062497] IP idents hash table entries: 2048 (order: 2, 16384 bytes, linear)
[    0.064593] tcp_listen_portaddr_hash hash table entries: 512 (order: 0, 4096 bytes, linear)
[    0.065029] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.065544] TCP established hash table entries: 512 (order: 0, 4096 bytes, linear)
[    0.065953] TCP bind hash table entries: 512 (order: 1, 8192 bytes, linear)
[    0.066387] TCP: Hash tables configured (established 512 bind 512)
[    0.067071] UDP hash table entries: 256 (order: 1, 8192 bytes, linear)
[    0.067407] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes, linear)
[    0.068175] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    0.070709] workingset: timestamp_bits=62 max_order=14 bucket_order=0
[    0.073019] NET: Registered PF_ALG protocol family
[    0.073455] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
[    0.073801] io scheduler mq-deadline registered
[    0.074056] io scheduler kyber registered
[    0.074387] io scheduler bfq registered
[    0.075570] bflb-gpio-pinctrl 200008c4.pinctrl: No cache defaults, reading back from HW
[    0.077954] bflb-gpio-pinctrl 200008c4.pinctrl: Bouffalo Lab pinctrl+GPIO(+interrupt) controller - Registered 32 function(s) for 46 pin(s)
[    0.083806] 30002000.serial: ttyS0 at MMIO 0x30002000 (irq = 2, base_baud = 2500000) is a BFLB UART
[    0.084288] printk: console [ttyS0] enabled
[    0.084288] printk: console [ttyS0] enabled
[    0.084735] printk: bootconsole [sbi0] disabled
[    0.084735] printk: bootconsole [sbi0] disabled
[    0.086406] 2000aa00.serial: ttyS1 at MMIO 0x2000aa00 (irq = 3, base_baud = 2500000) is a BFLB UART
[    0.115172] brd: module loaded
[    0.131893] loop: module loaded
[    0.133183] physmap-flash 58500000.xip_flash: physmap platform flash device: [mem 0x58500000-0x588fffff]
[    0.139081] wireguard: WireGuard 1.0.0 loaded. See www.wireguard.com for information.
[    0.139577] wireguard: Copyright (C) 2015-2019 Jason A. Donenfeld <Jason@zx2c4.com>. All Rights Reserved.
[    0.140210] PPP generic driver version 2.4.2
[    0.140910] PPP BSD Compression module registered
[    0.141223] PPP Deflate Compression module registered
[    0.142332] sdhci: Secure Digital Host Controller Interface driver
[    0.142723] sdhci: Copyright(c) Pierre Ossman
[    0.142993] sdhci-pltfm: SDHCI platform and OF driver helper
[    0.144330] mmc0 bounce up to 128 segments into one, max segment size 65536 bytes
[    0.145494] ledtrig-cpu: registered to indicate activity on CPUs
[    0.146299] bflb-seceng 20004000.seceng: No cache defaults, reading back from HW
[    0.149836] random: crng init done
[    0.150084] bflb-seceng 20004000.seceng: Bouffalo Lab Secure Engine
[    0.151045] riscv-pmu-sbi: SBI PMU extension is available
[    0.151453] riscv-pmu-sbi: 16 firmware and 10 hardware counters
[    0.153359] NET: Registered PF_INET6 protocol family
[    0.156148] Segment Routing with IPv6
[    0.156517] In-situ OAM (IOAM) with IPv6
[    0.156953] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[    0.158856] NET: Registered PF_PACKET protocol family
[    0.159229] Key type dns_resolver registered
[    0.183293] debug_vm_pgtable: [debug_vm_pgtable         ]: Validating architecture page table helpers
[    0.197399] mmc0: SDHCI controller on 20060000.sdhci [20060000.sdhci] using DMA
[    0.198161] Waiting for root device PARTLABEL=rootfs...
[    0.489067] mmc0: new high speed SDHC card at address b368
[    0.491548] mmcblk0: mmc0:b368 NCard 29.1 GiB 
[    0.510192]  mmcblk0: p1 p2 p3
[    0.551381] EXT4-fs (mmcblk0p3): mounted filesystem c0eebeb6-58dd-4946-bc48-e8c2c43098cc with ordered data mode. Quota mode: disabled.
[    0.552227] VFS: Mounted root (ext4 filesystem) on device 179:3.
[    0.559988] devtmpfs: mounted
[    0.568251] Freeing unused kernel image (initmem) memory: 2116K
[    0.568740] Run /sbin/init as init process
[    0.569018]   with arguments:
[    0.569212]     /sbin/init
[    0.569447]   with environment:
[    0.569654]     HOME=/
[    0.569812]     TERM=linux
[    0.841528] EXT4-fs (mmcblk0p3): re-mounted c0eebeb6-58dd-4946-bc48-e8c2c43098cc. Quota mode: disabled.
[    1.647489] Adding 1048572k swap on /dev/mmcblk0p1.  Priority:-2 extents:1 across:1048572k SS
Root Partition Already resized
Starting syslogd: OK
Starting klogd: OK
Running sysctl: OK
Starting mdev... OK
Initializing random number generator: OK
Saving random seed: OK
Starting rpcbind: OK
Starting iptables: OK
Starting network: OK
Starting dhcpcd...
dhcpcd-9.4.1 starting
dhcp_vendor: Invalid argument
forked to background, child pid 103
Starting sntp: sntp 4.2.8p15@1.3728-o Mon Mar  6 10:45:15 UTC 2023 (1)
pool.ntp.org lookup error Temporary failure in name resolution
FAIL
Starting ntpd: OK
Starting dropbear sshd: OK

Welcome to Buildroot
ox64 login:

Cool, that is Linux going, with only a lot of hassle. I wonder what I could try next?

netmap on cxgbe interfaces

Fri, 23 Feb 2024 00:00:00 +0000

Last year my testbed machine got the luxury of 100Gbit Chelsio T6 network interfaces with a pair of T62100-LP-CR cards. I have been using these recently for VPP development on FreeBSD and while they work well I have been having a lot of trouble doing any packet generation with them.

I got pretty miserable performance using netmaps pkt-gen on FreeBSD and trying Linux the comparative oddness of the T6 drivers seems to trip on the TRex packet generator suite on Linux.

From reading Chelsio's own documentation it seemed I was missing something, they have benchmarks using pkt-gen showing link saturation for small packets at 40Gbit/s on their T5 cards. What was up?

When I run a pkt-gen (lives in tools/tools/netmap ) benchmark I get a little under 1 Gbit/s of traffic:

$ sudo ./pkt-gen -f tx -i cc0 -S 00:07:43:74:3f:e9 -D 00:07:43:74:3f:e1
...
250.554583 main_thread [2713] 9.012 Mpps (9.587 Mpkts 4.326 Gbps in 1063799 usec) 512.00 avg_batch 0 min_space

~9 Mpps isn't that great, I paid for 100 Gbit and 4 just isn't enough.

FreeBSD has a zero copy userspace networking framework called netmap , it cleverly hooks into drivers early in their packet processing pipeline and if there is a netmap application running steals them so the host doesn't see them. Aiding with support, if the app doesn't want the given packet it can be given back to the driver and normal processing can occur.

Drivers can be natively supported with netmap or if support isn't available an emulated mode driver can be used.

Eventually after looking really really hard at the Chelsio examples I saw that the network interface names were different from mine. They were prefixed with a 'v', 'vcxl' rather than 'cxl'.

A bunch of googling later I read the top cxgbe(4) correctly:

The cxgbe driver uses different names for devices based on the associated
ASIC:

      ASIC    Port Name    Parent Device    Virtual Interface
      T4      cxgbe        t4nex            vcxgbe
      T5      cxl          t5nex            vcxl
      T6      cc           t6nex            vcc

To get a virtual interface we need to set the hw.cxgbe.num_vis sysctl to a value greater than 1. I added the following to my /boot/loader.conf :

# Add a virtual port for cxgbe                                     
hw.cxgbe.num_vis="2"            # Number of VIs per port           
hw.cxgbe.nnmrxq_vi="8"          # Number of netmap RX queues per VI
hw.cxgbe.nnmtxq_vi="8"          # Number of netmap TX queues per VI
hw.cxgbe.nrxq_vi="8"            # Number of RX queues per VI       
hw.cxgbe.ntxq_vi="8"            # Number of TX queues per VI

On a reboot I have a shiny new vcc0 device, letting pkt-gen run there I get much nicer results:

$ sudo ./pkt-gen -f tx -i vcc0 -S 00:07:43:74:3f:e9 -D 00:07:43:74:3f:e1 
...
329.706767 main_thread [2713] 68.770 Mpps (73.060 Mpkts 33.010 Gbps in 1062380 usec) 480.61 avg_batch 99999 min_space

~70 Mpps and 33 Gbps of 64 byte packets is pretty great , with an imix I can saturate the 100Gbps with pkt-gen .

I didn't find anything about this differentiator by searching, pkt-gen doesn't mention it. There are some dmesg lines, but if you don't know, you don't know.

238.184192 [1135] generic_netmap_attach     Emulated adapter for cc0 created (prev was NULL)
238.192821 [ 320] generic_netmap_register   Emulated adapter for cc0 activated
238.200244 [1889] netmap_interp_ringid      invalid ring id 1
238.206046 [ 295] generic_netmap_unregister Emulated adapter for cc0 deactivated

Adventurist.me: Letters into the void

Presentations with mdp

Enter mdp

Exporting to html

Presentations with remarkjs

Integrating diagrams

I really like remarkjs

Joining two sets with LINQ

FreeBSD USB Installer

Wot Happened; The break in

Atari used to 'make' printers

These guys are lucky they can walk.

Stripe now takes Bitcoins

Printing code with Enscript

Computer are very complex

This is the worst day of the year

Imprecision

Silly time changes

Raspberry Pi Compute board

Resize VDI

Use tcpdump to save wireless bridge

Videos I have watched from BSDCan 2014

Announcing Builds

Minimal R plots

It is actually

campGND network

pv

GPGME with Mutt on OS X

urtwn on FreeBSD ARM

Strange Mast

31c3

User SPI Adventures

libnfc i2c usage

msp430 hello world

Tech seen debugging bus sign

Attaching a debugger to mspdebug

Reading analog values on the msp430

Using portmaster to bulk install ports

Controlling keyboard leds

wait_on

Adding a TrueType font

Screenshotting in i3

rtl_sdr and OpenWRT

WR703N

rtl_tcp

Viewing the data

Setting serial baud rate on FreeBSD

Talks from BSDCan2015

The laddie and the Tramp

The fox in the city

Paris loves the theatre

Radio day to Balmedie Beach

More BSDCan Videos

Open Screenshotting

Touch Screen and Tablet on x220 Tablet and FreeBSD

gimme pcbs

rfcat on FreeBSD

IM-ME Specan

57N Stupid Shit No One Needs Hackathon

57N Stupid Shit No One Needs Hackathon - Results

FreeBSD on a pi with a small screen

Super simple presentations

Quick gifs

Tamogotchi Hive

Happy Hackmas from 57North

DSO138 Kit

The Thirty Second Chaos Communication Congress

Making GIFS with FFMPEG

Setting up xorg on the pi

Glitch Cards

Unreasonable Podcast

Recording Audio on FreeBSD

How to do(bad) encryption in vim

Command line notifications

Unreasonable Podcast Episode 0x02

Fixing rate issues with audio on FreeBSD

Unreasonable Podcast Episode 0x03

More Unreasonable Podcast Stuff

Finding package build options

Ubuntu Touch, Nowhere near ready yet

Images with `tcpflow`