Booting Linux on the Pine64 Ox64 SBC
The Ox64 (notice that is a letter 'O' not a 0) is a tiny microcontroller sized single board computer from Pine64. The Ox64 is built around the bl808 SOC from Bouffalo Lab. It has one application level 64 bit RISC-V core, a 32 bit RISC-V core and an extra little 32 bit RISC-V core for handling io.
It is a computer in the form factor of a microcontroller and it is built around what seems to be an open eco system. The Ox64 is early in its lifetime and Pine64 are quite good at making hardware for the community to build software around. That means that software is early and there is much to do (as I learned) to get code running.
Lets get Bouffalo's example Linux build going on this tiny thing.
First signs of life
There is quite a bit of documentation and write ups in the wiki page which helped me get going.
The links in the wiki aren't super clear about how to get any sort of working Linux image on the Ox64. Before trying any of those I wanted to get some signs of life out of the board. A two hour round trip to the hackerspace later I had headers soldered on to my two boards and connected up the Waveshare USB uart I got with the VisionFive2 kit.
The instructions aren't clear about where to expect the initial uart output to appear. The pinout on the wiki has RXD and TXD marked in several places, but also GPIO14/GPIO15 for UART0 TX adn UART0 RX. Two baud rates are listed 115200 and 2000000 (this might be faster than your usb uart can go), I got the following out of GPIO14/15 with a baud rate of 2000000.
$ sudo cu -l /dev/ttyU0 -s 2000000
____ __ __ _ _ _
| _ \ / _|/ _| | | | | | |
| |_) | ___ _ _| |_| |_ __ _| | ___ | | __ _| |__
| _ < / _ \| | | | _| _/ _` | |/ _ \| |/ _` | '_ \
| |_) | (_) | |_| | | | || (_| | | (_) | | (_| | |_) |
|____/ \___/ \__,_|_| |_| \__,_|_|\___/|_|\__,_|_.__/
Build:19:50:39,Nov 20 2022
Copyright (c) 2022 Bouffalolab team
dynamic memory init success,heap size = 93 Kbyte
sig1:ffff32ff
sig2:0000ffff
Pong!
Ping!
Pong!
Ping!
Pong!
Flashing with DevCube
The buildroot instructions on the wiki have an example usage of the SDK to build firmware and how to flash boot firmware for the two cores.
Bouffalo's flashing tool is called Devcube, they provide binaries for Linux, Mac OS and Windows. The Linux Binary (version 1.8.4) refused to run on FreeBSD with Linux compat giving me:
$ ./BLDevCube-ubuntu
PySide2/__init__.py: Unable to import shiboken2 from /tmp/_MEIN3lvWA/base_library.zip, /tmp/_MEIN3lvWA/lib-dynload, /tmp/_MEIN3lvWA
/lib64/libc.so.6: version `GLIBC_2.18' not found (required by /tmp/_MEIN3lvWA/libstdc++.so.6)
I don't have a lot of experience using Linux compat and I want to get this board going in less than a month of work so I flashed from Mac OS.
In Mac OS I was able to run Devcube, but kept hitting failures trying to flash the firmware images onto the board. Eventually I started trying more extreme things, the wiki page suggested that certain USB uart chips just don't work, so I tried another adapter. Another suggested solution was to write a USB uart firmware on to a pi pico and use that. I pulled the pi pico from my BluesSCSI v2, copied the firmware on to the virtual msdos file system and that finally worked.
With the firmware written the output on uart0 changed to:
[I][]
[I][] ____ ____ __ __ _
[I][] / __ \ | _ \ / _|/ _| | |
[I][] | | | |_ __ ___ _ __ | |_) | ___ _ _| |_| |_ __ _| | ___
[I][] | | | | '_ \ / _ \ '_ \| _ < / _ \| | | | _| _/ _` | |/ _ \
[I][] | |__| | |_) | __/ | | | |_) | (_) | |_| | | | || (_| | | (_) |
[I][] \____/| .__/ \___|_| |_|____/ \___/ \__,_|_| |_| \__,_|_|\___/
[I][] | |
[I][] |_|
[I][]
[I][] Powered by BouffaloLab
[I][] Build:11:52:22,Mar 6 2023
[I][] Copyright (c) 2023 OpenBouffalo team
[I][] Copyright (c) 2022 Bouffalolab team
[I][] =========== flash cfg ==============
[I][] jedec id 0xEF6018
[I][] mid 0xEF
[I][] iomode 0x04
[I][] clk delay 0x01
[I][] clk invert 0x01
[I][] read reg cmd0 0x05
[I][] read reg cmd1 0x35
[I][] write reg cmd0 0x01
[I][] write reg cmd1 0x31
[I][] qe write len 0x01
[I][] cread support 0x00
[I][] cread code 0xFF
[I][] burst wrap cmd 0x77
[I][] sector size: 0x04
[I][] =====================================
[I][] dynamic memory init success,heap size = 156 Kbyte
[I][MAIN] Starting Mailbox Handlers
[I][MBOX] Forwarding Interupt SDH (33) to D0 (0x58008bbc)
[I][MBOX] Forwarding Interupt GPIO (60) to D0 (0x58008d0e)
[I][MAIN] Running...
[I][MBOX] Mailbox IRQ Stats:
[I][MBOX] Peripheral SDH (33): 0
[I][MBOX] Peripheral GPIO (60): 0
[I][MBOX] Unhandled Interupts: 0 Unhandled Signals 0
[I][MBOX] ====================================
[I][MBOX] Mailbox IRQ Stats:
[I][MBOX] Peripheral SDH (33): 0
[I][MBOX] Peripheral GPIO (60): 0
[I][MBOX] Unhandled Interupts: 0 Unhandled Signals 0
[I][MBOX] ====================================
[I][MBOX] Mailbox IRQ Stats:
[I][MBOX] Peripheral SDH (33): 0
[I][MBOX] Peripheral GPIO (60): 0
[I][MBOX] Unhandled Interupts: 0 Unhandled Signals 0
[I][MBOX] ====================================
This is promising and nice to see firmware giving us a channel for further debugging in the future.
I wrote the example sd card image onto a card and tried to boot and got nothing.
In the instructions I spotted that the uart for u-boot/linux was on gpio 16/17 on the other side of the board to where I had soldered headers (I only had enough right angled headers to do one side). Luckily, I soldered straight headers onto all pins of the second Ox64 I got for working on this.
With those connected I still got nothing, I tried all the uarts on the pinout in the wiki. Dropping to 115200 for the baud because some disconnected messages in the discord suggested that might be the issue. I verified that the baud should be 2000000 in the DTB files.
I gave in, joined the Pine64 discord and asked what might be up. Arne said that Devcube 1.8.4 has some konwn issues and I should try Devcube 1.8.3 . This version didn't like uploading the bl808-firmware.bin file, it was unable to establish a connection to the firmware and gave me the same errors I saw before when I was using the USB uart adapter.
I tried fruitlessly many combinations of boot switches and power cycles, but I couldn't get the firmware to write.
Not sure what to try next I connected up to the system uart to see if I could get any further hints and instead I got a booting Linux:
[I][]
[I][] ____ ____ __ __ _
[I][] / __ \ | _ \ / _|/ _| | |
[I][] | | | |_ __ ___ _ __ | |_) | ___ _ _| |_| |_ __ _| | ___
[I][] | | | | '_ \ / _ \ '_ \| _ < / _ \| | | | _| _/ _` | |/ _ \
[I][] | |__| | |_) | __/ | | | |_) | (_) | |_| | | | || (_| | | (_) |
[I][] \____/| .__/ \___|_| |_|____/ \___/ \__,_|_| |_| \__,_|_|\___/
[I][] | |
[I][] |_|
[I][]
[I][] Powered by BouffaloLab
[I][] Build:11:52:04,Mar 6 2023
[I][] Copyright (c) 2023 OpenBouffalo team
[I][] Copyright (c) 2022 Bouffalolab team
[I][] dynamic memory init success,heap s[I][LowLoad] D0 start...
[I][LowLoad] low_load start...
[I][LowLoad] Header at 0x5d5ff000
[I][LowLoad] Section dtb(1) - Start 0x5d5ff100, Size 14314
[I][LowLoad] Copying DTB to 0x51ff8000...0x51ffb7ea
[I][LowLoad] Done!
[I][LowLoad] Section OpenSBI(2) - Start 0x5d60f100, Size 109864
[I][LowLoad] Copying OpenSBI to 0x3ef80000...0x3ef9ad28
[I][LowLoad] Done!
[I][LowLoad] Section Kernel(3) - Start 0x5d62f100, Size 315597
[I][LowLoad] Uncompressing Kernel to 0x50000000...
[I][LowLoad] Done!
[I][LowLoad] CRC: 00000000
[I][LowLoad] load time: 61311 us
[I][LowLoad] Setting PMP
[I][LowLoad] Booting OpenSBI at 0x000000003ef80000 with DTB at 0x51ff8000
OpenSBI v1.2
____ _____ ____ _____
/ __ \ / ____| _ \_ _|
| | | |_ __ ___ _ __ | (___ | |_) || |
| | | | '_ \ / _ \ '_ \ \___ \| _ < | |
| |__| | |_) | __/ | | |____) | |_) || |_
\____/| .__/ \___|_| |_|_____/|____/_____|
| |
|_|
Platform Name : Pine64 Ox64 (D0)
Platform Features : medeleg
Platform HART Count : 1
Platform IPI Device : aclint-mswi
Platform Timer Device : aclint-mtimer @ 1000000Hz
Platform Console Device : bflb_uart
Platform HSM Device : ---
Platform PMU Device : ---
Platform Reboot Device : ---
Platform Shutdown Device : ---
Firmware Base : 0x3ef80000
Firmware Size : 200 KB
Runtime SBI Version : 1.0
Domain0 Name : root
Domain0 Boot HART : 0
Domain0 HARTs : 0*
Domain0 Region00 : 0x00000000e4008000-0x00000000e400bfff (I)
Domain0 Region01 : 0x00000000e4000000-0x00000000e4007fff (I)
Domain0 Region02 : 0x000000003ef80000-0x000000003efbffff ()
Domain0 Region03 : 0x0000000000000000-0xffffffffffffffff (R,W,X)
Domain0 Next Address : 0x0000000050000000
Domain0 Next Arg1 : 0x0000000051ff8000
Domain0 Next Mode : S-mode
Domain0 SysReset : yes
Boot HART ID : 0
Boot HART Domain : root
Boot HART Priv Version : v1.11
Boot HART Base ISA : rv64imafdcvx
Boot HART ISA Extensions : time
Boot HART PMP Count : 8
Boot HART PMP Granularity : 4096
Boot HART PMP Address Bits: 38
Boot HART MHPM Count : 8
Boot HART MIDELEG : 0x0000000000000222
Boot HART MEDELEG : 0x000000000000b109
U-Boot 2023.04-rc2 (Mar 06 2023 - 11:48:40 +0000)
DRAM: 64 MiB
Core: 36 devices, 17 uclasses, devicetree: board
MMC: mmc@20060000: 0
Loading Environment from FAT... Unable to read "uboot.env" from mmc0:2...
Loading Environment from nowhere... OK
In: serial@30002000
Out: serial@30002000
Err: serial@30002000
Net:
Warning: emac@20070000 (eth0) using random MAC address - fa:6f:53:9f:5c:36
eth0: emac@20070000
Hit any key to stop autoboot: 0
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:2...
Found /extlinux/extlinux.conf
Retrieving file: /extlinux/extlinux.conf
Select the boot mode
1: Pine64 0X64 Kernel
2: Sipeed M1SDock Kernel
Enter choice: 1
1: Pine64 0X64 Kernel
Retrieving file: /extlinux/../Image
append: root=PARTLABEL=rootfs rootwait rw rootfstype=ext4 console=ttyS0,2000000 loglevel=8 earlycon=sbi
Retrieving file: /extlinux/../bl808-pine64-ox64.dtb
## Flattened Device Tree blob at 51ff8000
Booting using the fdt blob at 0x51ff8000
Working FDT set to 51ff8000
Loading Device Tree to 0000000053f22000, end 0000000053f25fab ... OK
Working FDT set to 53f22000
Starting kernel ...
[ 0.000000] Linux version 6.2.0 (runner@fv-az587-938) (riscv64-unknown-linux-gnu-gcc (Xuantie-900 linux-5.10.4 glibc gcc Toolchain V2.6.1 B-20220906) 10.2.0, GNU ld (GNU Binutils) 2.35) #1 Mon Mar 6 11:17:27 UTC 2023
[ 0.000000] OF: fdt: Ignoring memory range 0x50000000 - 0x50200000
[ 0.000000] Machine model: Pine64 Ox64
[ 0.000000] earlycon: sbi0 at I/O port 0x0 (options '')
[ 0.000000] printk: bootconsole [sbi0] enabled
[ 0.000000] efi: UEFI not found.
[ 0.000000] Zone ranges:
[ 0.000000] DMA32 [mem 0x0000000050200000-0x0000000053ffffff]
[ 0.000000] Normal empty
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000050200000-0x0000000053ffffff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000050200000-0x0000000053ffffff]
[ 0.000000] SBI specification v1.0 detected
[ 0.000000] SBI implementation ID=0x1 Version=0x10002
[ 0.000000] SBI TIME extension detected
[ 0.000000] SBI IPI extension detected
[ 0.000000] SBI RFENCE extension detected
[ 0.000000] riscv: base ISA extensions acdfim
[ 0.000000] riscv: ELF capabilities acdfim
[ 0.000000] pcpu-alloc: s0 r0 d32768 u32768 alloc=1*32768
[ 0.000000] pcpu-alloc: [0] 0
[ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 15655
[ 0.000000] Kernel command line: root=PARTLABEL=rootfs rootwait rw rootfstype=ext4 console=ttyS0,2000000 loglevel=8 earlycon=sbi
[ 0.000000] Dentry cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[ 0.000000] Inode-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)
[ 0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[ 0.000000] Memory: 48084K/63488K available (3941K kernel code, 4584K rwdata, 2048K rodata, 2118K init, 301K bss, 15404K reserved, 0K cma-reserved)
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[ 0.000000] riscv-intc: 64 local interrupts mapped
[ 0.000000] plic: interrupt-controller@e0000000: mapped 64 interrupts with 1 handlers for 2 contexts.
[ 0.000000] riscv-timer: riscv_timer_init_dt: Registering clocksource cpuid [0] hartid [0]
[ 0.000000] clocksource: riscv_clocksource: mask: 0xffffffffffffffff max_cycles: 0x1d854df40, max_idle_ns: 3526361616960 ns
[ 0.000003] sched_clock: 64 bits at 1000kHz, resolution 1000ns, wraps every 2199023255500ns
[ 0.000920] Console: colour dummy device 80x25
[ 0.001166] Calibrating delay loop (skipped), value calculated using timer frequency.. 2.00 BogoMIPS (lpj=4000)
[ 0.001687] pid_max: default: 32768 minimum: 301
[ 0.002408] Mount-cache hash table entries: 512 (order: 0, 4096 bytes, linear)
[ 0.002749] Mountpoint-cache hash table entries: 512 (order: 0, 4096 bytes, linear)
[ 0.007710] cblist_init_generic: Setting adjustable number of callback queues.
[ 0.008028] cblist_init_generic: Setting shift to 0 and lim to 1.
[ 0.008940] ASID allocator using 16 bits (65536 entries)
[ 0.010160] EFI services will not be available.
[ 0.011191] devtmpfs: initialized
[ 0.014879] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[ 0.015375] futex hash table entries: 256 (order: 0, 6144 bytes, linear)
[ 0.016003] pinctrl core: initialized pinctrl subsystem
[ 0.018536] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[ 0.019435] DMA: preallocated 128 KiB GFP_KERNEL pool for atomic allocations
[ 0.019823] DMA: preallocated 128 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[ 0.025441] bflb-ipc 30005000.mailbox: Bouffalo Lab IPC mailbox interrupt controller
[ 0.030626] SCSI subsystem initialized
[ 0.033265] clocksource: Switched to clocksource riscv_clocksource
[ 0.061833] NET: Registered PF_INET protocol family
[ 0.062497] IP idents hash table entries: 2048 (order: 2, 16384 bytes, linear)
[ 0.064593] tcp_listen_portaddr_hash hash table entries: 512 (order: 0, 4096 bytes, linear)
[ 0.065029] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)
[ 0.065544] TCP established hash table entries: 512 (order: 0, 4096 bytes, linear)
[ 0.065953] TCP bind hash table entries: 512 (order: 1, 8192 bytes, linear)
[ 0.066387] TCP: Hash tables configured (established 512 bind 512)
[ 0.067071] UDP hash table entries: 256 (order: 1, 8192 bytes, linear)
[ 0.067407] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes, linear)
[ 0.068175] NET: Registered PF_UNIX/PF_LOCAL protocol family
[ 0.070709] workingset: timestamp_bits=62 max_order=14 bucket_order=0
[ 0.073019] NET: Registered PF_ALG protocol family
[ 0.073455] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
[ 0.073801] io scheduler mq-deadline registered
[ 0.074056] io scheduler kyber registered
[ 0.074387] io scheduler bfq registered
[ 0.075570] bflb-gpio-pinctrl 200008c4.pinctrl: No cache defaults, reading back from HW
[ 0.077954] bflb-gpio-pinctrl 200008c4.pinctrl: Bouffalo Lab pinctrl+GPIO(+interrupt) controller - Registered 32 function(s) for 46 pin(s)
[ 0.083806] 30002000.serial: ttyS0 at MMIO 0x30002000 (irq = 2, base_baud = 2500000) is a BFLB UART
[ 0.084288] printk: console [ttyS0] enabled
[ 0.084288] printk: console [ttyS0] enabled
[ 0.084735] printk: bootconsole [sbi0] disabled
[ 0.084735] printk: bootconsole [sbi0] disabled
[ 0.086406] 2000aa00.serial: ttyS1 at MMIO 0x2000aa00 (irq = 3, base_baud = 2500000) is a BFLB UART
[ 0.115172] brd: module loaded
[ 0.131893] loop: module loaded
[ 0.133183] physmap-flash 58500000.xip_flash: physmap platform flash device: [mem 0x58500000-0x588fffff]
[ 0.139081] wireguard: WireGuard 1.0.0 loaded. See www.wireguard.com for information.
[ 0.139577] wireguard: Copyright (C) 2015-2019 Jason A. Donenfeld <Jason@zx2c4.com>. All Rights Reserved.
[ 0.140210] PPP generic driver version 2.4.2
[ 0.140910] PPP BSD Compression module registered
[ 0.141223] PPP Deflate Compression module registered
[ 0.142332] sdhci: Secure Digital Host Controller Interface driver
[ 0.142723] sdhci: Copyright(c) Pierre Ossman
[ 0.142993] sdhci-pltfm: SDHCI platform and OF driver helper
[ 0.144330] mmc0 bounce up to 128 segments into one, max segment size 65536 bytes
[ 0.145494] ledtrig-cpu: registered to indicate activity on CPUs
[ 0.146299] bflb-seceng 20004000.seceng: No cache defaults, reading back from HW
[ 0.149836] random: crng init done
[ 0.150084] bflb-seceng 20004000.seceng: Bouffalo Lab Secure Engine
[ 0.151045] riscv-pmu-sbi: SBI PMU extension is available
[ 0.151453] riscv-pmu-sbi: 16 firmware and 10 hardware counters
[ 0.153359] NET: Registered PF_INET6 protocol family
[ 0.156148] Segment Routing with IPv6
[ 0.156517] In-situ OAM (IOAM) with IPv6
[ 0.156953] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[ 0.158856] NET: Registered PF_PACKET protocol family
[ 0.159229] Key type dns_resolver registered
[ 0.183293] debug_vm_pgtable: [debug_vm_pgtable ]: Validating architecture page table helpers
[ 0.197399] mmc0: SDHCI controller on 20060000.sdhci [20060000.sdhci] using DMA
[ 0.198161] Waiting for root device PARTLABEL=rootfs...
[ 0.489067] mmc0: new high speed SDHC card at address b368
[ 0.491548] mmcblk0: mmc0:b368 NCard 29.1 GiB
[ 0.510192] mmcblk0: p1 p2 p3
[ 0.551381] EXT4-fs (mmcblk0p3): mounted filesystem c0eebeb6-58dd-4946-bc48-e8c2c43098cc with ordered data mode. Quota mode: disabled.
[ 0.552227] VFS: Mounted root (ext4 filesystem) on device 179:3.
[ 0.559988] devtmpfs: mounted
[ 0.568251] Freeing unused kernel image (initmem) memory: 2116K
[ 0.568740] Run /sbin/init as init process
[ 0.569018] with arguments:
[ 0.569212] /sbin/init
[ 0.569447] with environment:
[ 0.569654] HOME=/
[ 0.569812] TERM=linux
[ 0.841528] EXT4-fs (mmcblk0p3): re-mounted c0eebeb6-58dd-4946-bc48-e8c2c43098cc. Quota mode: disabled.
[ 1.647489] Adding 1048572k swap on /dev/mmcblk0p1. Priority:-2 extents:1 across:1048572k SS
Root Partition Already resized
Starting syslogd: OK
Starting klogd: OK
Running sysctl: OK
Starting mdev... OK
Initializing random number generator: OK
Saving random seed: OK
Starting rpcbind: OK
Starting iptables: OK
Starting network: OK
Starting dhcpcd...
dhcpcd-9.4.1 starting
dhcp_vendor: Invalid argument
forked to background, child pid 103
Starting sntp: sntp 4.2.8p15@1.3728-o Mon Mar 6 10:45:15 UTC 2023 (1)
pool.ntp.org lookup error Temporary failure in name resolution
FAIL
Starting ntpd: OK
Starting dropbear sshd: OK
Welcome to Buildroot
ox64 login:
Cool, that is Linux going, with only a lot of hassle. I wonder what I could try next?