Pimera Project: FreeBSD

2019-08-27

Based on my suspension of the KubeHat project and initial efforts on my new Pimera project, my first step is a stab at getting FreeBSD running on one of the Pi Zeros!

Goals

The following are my goals for getting FreeBSD working on one of the Raspberry Pi Zeros in my Cluster HAT setup:

Don't impact operation of the rest of the cluster.
Latest current FreeBSD (13-CURRENT is the only version that fully worked for me)
Feature parity with Raspbian, including bridged networking, SSH server, mDNS discovery.
All configured to come up automatically on boot

I'm going to be using the FreeBSD Pi wiki page as a starting point, and going from there.

Disk Image

First, I brought down the existing p4 Pi Zero from the controller, leaving the rest of the cluster running:

pi@controller$ clusterhat off p4

Once that completed, the status light for the fourth cluster device went dark, and I could remove the SD card and plug it into my laptop.

Next, I downloaded the latest snapshot disk image from https://download.freebsd.org/ftp/snapshots/arm/armv6/ISO-IMAGES/

Once downloaded, from my Linux laptop, I unzipped the image:

peter@localhost$ unxz FreeBSD-13.0-CURRENT-arm-armv6-RPI-B-20190822-r351363.img.xz

Next, I wrote the image to my SD card:

peter@localhost$ sudo dd if=FreeBSD-13.0-CURRENT-arm-armv6-RPI-B-20190822-r351363.img of=/dev/mmcblk0 bs=4M conv=fsync

When that completed, I removed the SD card from my laptop, re-inserted it into the fourth Pi Zero slot

Running FreeBSD

Next, I brought the fourth Pi Zero device back online in the cluster from the controller:

pi@controller$ clusterhat on p4

From there, running dmesg I can see eventually the FreeBSD device come live, and register the virtual serial console:

pi@controller$ dmesg | tail -n 8
[861430.624826] usb 1-1.2.1: new high-speed USB device number 10 using dwc_otg
[861435.764830] usb 1-1.2.1: device descriptor read/64, error -110
[861436.017185] usb 1-1.2.1: New USB device found, idVendor=16c0, idProduct=27dd, bcdDevice= 1.00
[861436.017211] usb 1-1.2.1: New USB device strings: Mfr=2, Product=3, SerialNumber=4
[861436.017223] usb 1-1.2.1: Product: Virtual serial port
[861436.017235] usb 1-1.2.1: Manufacturer: The FreeBSD Project (https://www.FreeBSD.org)
[861436.017247] usb 1-1.2.1: SerialNumber: FreeBSD1
[861436.089370] cdc_acm 1-1.2.1:1.0: ttyACM3: USB ACM device

Logging into FreeBSD

Using the tio utility (or screen, if you prefer), you can log into FreeBSD on the Pi Zero over the virtual serial console:

pi@controller$ tio /dev/ttyACM0
[tio 18:58:58] tio v1.32
[tio 18:58:58] Press ctrl-t q to quit
[tio 18:58:58] Connected


FreeBSD/arm (p4) (ttyU0)

login:

If using tio, you may need to hit Enter once to get the login prompt to appear. You can then log in using the default root:root combo from the base image.

USB Device Mode Network Device

In order to access the FreeBSD Pi over SSH, we'll need to get FreeBSD to create a virtual ethernet device via USB device mode that the main controller Pi will bridge to the physical ethernet adapter on that device.

Taking inspiration from the instructions from the FreeBSD Handbook, well edit /boot/loader.conf, using vi to do so (which is part of base) to do the following:

Change the USB template mode to 8 so we get both serial console and networking. See usb_template(4).
Make sure the if_cdce driver is loaded at boot, so the network device can load.

You should end up with the following lines in that file:

if_cdce_load="YES"
hw.usb.template=8

Then reboot FreeBSD:

root# reboot

Once the device reboots, you should see a new line in the dmesg output:

pi@controller$ dmesg | tail -n 1
[864885.946765] cdc_ether 1-1.2.1:1.2 usb0: register 'cdc_ether' at usb-3f980000.usb-1.2.1, CDC Ethernet Device, 2a:02:03:04:05:06

Unfortunatly, the MAC address that FreeBSD will use for the USB ethernet device will not match the ones originally configured in the Cluster HAT Raspbian images, and so the newly created device on the controller side will be usb0, and not properly bridged automatically:

pi@controller$ ifconfig usb0
usb0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether 2a:02:03:04:05:06  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

To fix this, we'll note the MAC address, e.g. 2a:02:03:04:05:06 and update the file /etc/udev/rules.d/90-clusterhat.rules with that MAC address, Since I'm using p4 for my FreeBSD instance, I'll update the line like so:

SUBSYSTEM=="net", ATTR{address}=="2a:02:03:04:05:06", NAME="ethpi4"

Now, rebooting the FreeBSD instance from the serial console should show the following dmesg output from the controller:

[81069.650887] usb 1-1.2.1: new high-speed USB device number 22 using dwc_otg
[81074.780880] usb 1-1.2.1: device descriptor read/64, error -110
[81075.033585] usb 1-1.2.1: New USB device found, idVendor=16c0, idProduct=05dc, bcdDevice= 1.00
[81075.033603] usb 1-1.2.1: New USB device strings: Mfr=6, Product=7, SerialNumber=8
[81075.033615] usb 1-1.2.1: Product: Serial/Ethernet device
[81075.033628] usb 1-1.2.1: Manufacturer: The FreeBSD Project (https://www.FreeBSD.org)
[81075.033639] usb 1-1.2.1: SerialNumber: FreeBSD1
[81075.116046] cdc_acm 1-1.2.1:1.0: ttyACM0: USB ACM device
[81075.121932] cdc_ether 1-1.2.1:1.2 usb0: register 'cdc_ether' at usb-3f980000.usb-1.2.1, CDC Ethernet Device, 2a:02:03:04:05:06
[81075.200713] cdc_ether 1-1.2.1:1.2 ethpi4: renamed from usb0
[81075.555493] br0: port 3(ethpi4) entered blocking state
[81075.555506] br0: port 3(ethpi4) entered disabled state
[81075.556209] device ethpi4 entered promiscuous mode
[81075.620942] br0: port 3(ethpi4) entered blocking state
[81075.620957] br0: port 3(ethpi4) entered forwarding state

Now that the controller side is configured, we'll set up DHCP inside of FreeBSD, via the serial console. Edit /etc/rc.conf to add the following line:

ifconfig_ue0="SYNCDHCP"

Where ue0 is the name of the USB mode ethernet device that FreeBSD creates. Once that is done, run the following from FreeBSD:

root# service netif restart
dhclient not running? (check /var/run/dhclient/dhclient.ue0.pid).
Stopping Network: lo0 ue0.
lo0: flags=8048<LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ue0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 2a:02:03:04:05:f9
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Starting dhclient.
DHCPREQUEST on ue0 to 255.255.255.255 port 67
DHCPACK from 192.168.1.1
bound to 192.168.1.212 -- renewal in 43200 seconds.
Starting Network: lo0 ue0.
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ue0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 2a:02:03:04:05:f9
        inet 192.168.1.212 netmask 0xffffff00 broadcast 192.168.1.255
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

After that, you should have working network access from the FreeBSD Pi:

root# ping google.com
PING google.com (172.217.10.238): 56 data bytes
64 bytes from lga25s59-in-f14.1e100.net (172.217.10.238): icmp_seq=0 ttl=56 time=36.768 ms
64 bytes from lga25s59-in-f14.1e100.net (172.217.10.238): icmp_seq=1 ttl=56 time=35.297 ms

Passwords, Hostname, NTP

Next, you should update the passwords for root and freebsd users:

root# passwd
Changing local password for root
New Password:
Retype New Password:
root# passwd freebsd
Changing local password for freebsd
New Password:
Retype New Password:

You should update the hostname in /etc/rc.conf to match the numbered Pi Zero, e.g. p4:

hostname="p4"

Because there is not persistent real time clock on the Pi Zero, you'll want to enable ntpd in /etc/rc.conf as well so you have accurate time on the device:

ntpd_enable="YES"
ntpd_sync_on_start="YES"

Once enabled, reboot your FreeBSD Pi once more for all the changes to take effect.

Using pkg for package management

See the Packages and Ports handbook section for setting up pkg and adding packages like neovim, tmux, etc.

Multicast DNS (i.e. ZeroConf) Configuration

I haven't yet been able to get mDNS working for the FreeBSD instance. In particular, the mDNSResponderPosix keeps segfaulting after starting. Perhaps at a later date I will recompile with debug symbols and dig into the details.

End Result

FreeBSD SSH Login on Raspberry Pi Zero

Pimera Progress

So far, we've got the following completed for the Primera Project

Pimera Progress

Next Step: Alpine Linux

NetBSD I'm not sure in fact supports the "USB Gadget" functionality, so I'm going to try to get Alpine Linux going on p2 next!