Analysing a Network Protocol

gpredict is piece of software for tracking things in orbits, sometimes you want to automatically point things at stuff in orbit. To get things pointed at stuff in orbit we can use a rotator controller, gpredict as a piece of radio software has an antenna rotator controller built it. The gpredict rotator controller expects to speak to something over TCP.

I have not been able to find documentation for the protocol (I didn't look very hard), I thought it would be fun to reverse engineer the protocol and write a simple daemon. Earlier I took some first steps to see what gpredict was doing on the network.

If you want to play a long at home this is what I am going to do:

  • set up a dummy daemon using netcat (nc -l localhost 4533)
  • use tcpdump with -XX to watch all traffic (e.g. tcpdump -XX -ilo0 tcp and port 4533)
  • send data from gpredict to the daemon (hit the 'engage' button on the antenna control screen)
  • play with responses (type into the console running nc)
  • look at the gpredict code starting here: https://github.com/csete/gpredict/blob/master/src/gtk-rot-ctrl.c

The Network traffic

$ nc -l 0.0.0.0 4533  
p

P  180.00   45.00

When I press the 'engage' button, gpredict sends a single lower case 'p', if I press enter, sending a blank line, gredict responds with a capital 'P' and two numbers. To me these numbers look like an Az El pair, they correspond to the values on the antenna control screen in gpredict . No need for tcpdump this time.

We have source avaialble

With only one half of the network protocol to look at, we can't get very far. gpredict is open source and there is a github mirror where we can browse the source tree. The file names in the 'src' directory show some promising results:

gtk-rot-ctrl.c
gtk-rot-ctrl.h
gtk-rot-knob.c
gtk-rot-knob.h
rotor-conf.c
rotor-conf.h
sat-pref-rot.c
sat-pref-rot.h

The pref and conf files, are probably configuration stuff, I have no idea what is in the knob file, but the gtk-rot-ctrl set of files is what we want. I confirmed this by picking a string in the UI of the relevant screen and grepping through the code for it. This can be troublesome if the software is heavily localised, but it this case I could track down the 'Engage' button to a comment in the code .

There are two functions used for network traffic, send is used to send data into a tcp connection, recv is used to receive data from a TCP connection. If we can find these in the code, we find where the software is generating network traffic. Normally only a starting point, it is very common to wrap these two functions into other convenience functions.

A grep through the code brings up a send call in send rotctld command . More grepping and we find that send_rotctld_command is called from two places, the get pos function (which I have to guess asks for the rotators positions) and the set pos function (which must try to set the rotators position).

The get_pos function fills a format string with "p\x0a" and uses send_rotcld_command to send it. Looking up 0x0A in an ascii table shows it is Line Feed(LF) also known as a newline on a unix system. It splits buffback on newlines using g_strsplit , looking to find two floating point numbers to use as azimuth and elevation, one on each line.

get_pos :

/* send command */
buff = g_strdup_printf("p\x0a");
retcode = send_rotctld_command(ctrl, buff, buffback, 128);
...
vbuff = g_strsplit(buffback, "\n", 3);
if ((vbuff[0] != NULL) && (vbuff[1] != NULL))
{
    *az = g_strtod(vbuff[0], NULL);
    *el = g_strtod(vbuff[1], NULL);
}

This piece of code shows up something really important, gpredict is using a single function to both send a command and gather the response from the remote end. If we look at send_rotctld_command the recv call is called right after a send. Here we can see that gpredict only does a single recv to gather responses, it is expecting a reply that fits into a single read. This is a bug, but probably not one that really matters.

/* try to read answer */
size = recv(ctrl->sock, buffout, sizeout, 0);

The set_pos function fills up a format string with a capital 'P', and two floating point numbers. It doesn't do any parsing of the response, only looking at the error code from the socket call.

set_pos :

/* send command */
g_ascii_formatd(azstr, 8, "%7.2f", az);
g_ascii_formatd(elstr, 8, "%7.2f", el);
buff = g_strdup_printf("P %s %s\x0a", azstr, elstr);

retcode = send_rotctld_command(ctrl, buff, buffback, 128);

Write a Daemon

With this little bit of analysis we have enough to write an antenna control daemon that gpredict can speak to. The rotator control protocol has two simple commands, a position query which expects the currect az/el across separate lines and a position setter, which expects no response.

#!/usr/bin/env python

import socket

TCP_IP = '127.0.0.1'
TCP_PORT = 4533
BUFFER_SIZE = 100

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((TCP_IP, TCP_PORT))
s.listen(1)

conn, addr = s.accept()
print 'Connection address:', addr

az = 0.0
el = 0.0

while 1:
    data = conn.recv(BUFFER_SIZE)
    if not data:
        break

    print("received data:", data)

    if data == "p\n":
        print("pos query at az:{} el: {}", az, el);

        response = "{}\n{}\n".format(az, el)
        print("responing with: \n {}".format(response))
        conn.send(response)
    elif data.startswith("P "):
        values = data.split("  ")
        print(values)
        az = float(values[1])
        el = float(values[2])

        print("moving to az:{} el: {}".format( az, el));

        conn.send(" ")
    elif data == "q\n":
        print("close command, shutting down")
        conn.close()
        exit()
    else:
        print("unknown command, closing socket")
        conn.close()
        exit()

Using the python TCP server example as a starting point it is easy to put together a daemon that will listen to the rotator controller. The code should be pretty straight forward to read, we process the commands documented earlier. There is one addition that I didn't see in the code at first. There is a quit command that does not use the normal wrapper and instead uses send directly. This command was easy to handle.

This is how I approach network problems, whether in code I have written or code that is completely new to me. Hopefully if you have been following along at home the example above is straightforward to read.

Rotator Controller

gpredict is able to signal a rotator controller over TCP. This is awesome, I want to track satellites and I am not going to pay for a rotator controller. I am going to build something to get my antenna pointed, using servos and a wifi microcontroller board.

I have tried searching a few times, but like everything amateur radio hard facts are hard to come by, the scam artists and windows developers protecting their sacred lore abound. I was at a bit of loss until I thought to try seeing what gpredict spits out over the network.

First I created a test rotator in the gpredict settings:

Then I dug around until I could find the rotator control panel, named antenna control. In this panel there is a 'track' button and an 'engage' button, figuring engage was the test option to manually set the rotator I hit that.

After a short pause a helpful 'ERROR' pops up under the Az/El settings, Good progress. Next I started up nc pretending to be a listening rotator controller so I could see what gpredict was sending.

$ nc -l 0.0.0.0 4533  
p

P  180.00   45.00

p

P  180.00   45.00

p

P  180.00   45.00

p

P  180.00   45.00



p

P  180.00   45.00

This output is great, nc is just outputting the bytes sent down the tcp connection. It seems that gpredict sends a letter 'p', I replied with a blank line by hitting enter, this resulted in a capital 'P' and a Az and El. Some guess work interpretation suggests gpredict is asking for our position with 'p', then giving us a position to move to with 'P'.

This is a great start, next I will have a look through the gpredict source to see what it is doing. I will start with the 'engage' button from gtk-rot-ctrl.c .


Reading: Abaddon's Gate

Making webm files

Yesterday I posted a stupidly large gif of some pumpkins flashing different colours. Then a couple of minutes later after suffering through the upload I replaced it with a teeny webm.

ffmpeg now has excellent webm support, for me it will happily pick it up with automatic detection. To make videos like the one above I use ffmpeg and let it choose its own options. I drop the audio from the video to give it a nicer gif like effect, I think from today I will tell browsers to autoplay the video snippets so they look like awesome gifs.

I do something like this:

$ ffmpeg -i VID_20161030_112003.mp4  -ss 00:00:02.0 -an coffee.webm
  • -i is the input video
  • -ss tells ffmpeg to start the encoding at this time stamp
    • we could pass -t to tell ffmpeg to extract a t length section of the video
  • -an tells us to the use no audio
  • coffee.webm is the output video and format, ffmpeg will do the right thing here

Reading: Abaddon's Gate

Extract Images from a TCP Flow

On Sunday I thought I would try to extract images from a TCP Flow with scapy . Initial searches should how to do this with wireshark , but the idea is to do this programatically and try to learn something about scapy. At the hackerspace tonight, not wanting to work on anything I said I would, I thought I would have a play with scapy.

A little bit on TCP

A TCP Flow is how we refer to the stream of packets that make what you might call TCP Connection. The connection bit is just the start. A Flow is defined in IP by 5 numbers:

* protocol (TCP or UDP for the most part)
* source
    - address (ip address of the initiator of the connection
    - port (normally a randomly chosen emphemeral port number)
* destination
    - address (ip address of the host)
    - port (normally a well known service number, http is 80)

A lot of time we call this the 5 tuple, or 4 tuple if we know the protocol. At any one point in a time a given 5-tuple defines the connection(Flow).

To a programmer TCP presents a reliable byte orientated stream interface. This means any bytes we write into our TCP socket, will come out of the other end in order and they are guaranteed to arrive (or an error is generated).

Data written into a TCP socket is broken into chunks the network can support (normally, without fragmentation), we call these chunks of data segments. Each segment has a sequence number, which tells the remote end where it is in the stream, there can be a large number of these segments in the air at a time, the flight size.

Segments can get lost in the network (well dropped by routers), reordered or delayed.

Extracting Images

To extract images from a network capture we need to separate out the packets into flows; reassemble TCP flows into a byte stream taking into account loss and reordering; reconstruct the segments into a coherent byte stream; search the byte stream for image headers and try to extract them.

This is a none trivial amount of work for a Tuesday night.

Before writing any code I did some searching, scapy might have support for flow reconstruction(nope). I came across some references to a tool called tcpflow , tcpflow claims to be a tool for extracting TCP Flows from either a live capture interface or a pcap file.

That looked great to me, I would grab a pcap with tcpdump , process out the flows with tcpflow and then drop that into scapy and start looking for some images.

Reading the tcpflow man page I instead found a single option that would do all the work for me.

Images with tcpflow

It is really easy to extract images from a http TCP Flow using tcpflow , you can do this live, but I used a pcap file.

# tcpdump -w webimage.pcap host adventurist.me and port 80

I started up the dump then visited http://adventurist.me/posts/0129 in FireFox's porn mode.

tcpflow will read in a pcap file with the -r flag, the -e flag will apply magic to the flow and find you fun stuff.

$ tcpflow -r webimage.pcap -e http

tcpflow will spit out a file for each flow, to boot it will throw in extracted data for everything it understands.

$ ls
093.095.228.091.00080-172.031.005.168.58914
093.095.228.091.00080-172.031.005.168.58914-HTTPBODY-001.jpg
093.095.228.091.00080-172.031.005.168.58914-HTTPBODY-002.ico
093.095.228.091.00080-172.031.005.168.60028
093.095.228.091.00080-172.031.005.168.60028-HTTPBODY-001.html
093.095.228.091.00080-172.031.005.168.60028-HTTPBODY-002.css
172.031.005.168.58914-093.095.228.091.00080
172.031.005.168.60028-093.095.228.091.00080
report.xml
webimage.pcap

tcpflow also seems to be spitting out a report.xml , it seems to describe what it has just done. I image that is super useful when running tcpflow against a live capture. I haven't managed to get very far using scapy to pull images out of flows, I am starting to wonder if there is really any point when all these tools are available.

Spooky Art-Net Pumpkins

Last night was All Hallows' Eve , I wanted to do something cool with the decorations. I repurposed an rgb neopixel board driven by a nodemcu board and gave one of our pumpkins a network controlled candle instead of the old analog kind.

I also spent some time building out a motion sensor, but I wasn't able to integrate that with the network code in time to use it. In the end the weather seems to have kept everyone at home and we didn't have any visitors.

I am going to try and get everything together tonight at the hackerspace , if I do I will write up what all the parts are.


Reading: Abaddon's Gate