Textplain

The Source Will Be With You, Always

Tutorials

Cross Compiling FreeBSD Ports for the BeagleBone Black

published

I've been playing with FreeBSD on embedded devices recently, namely the Raspberry Pi and Beagle Bone Black. Installing the OS is easy enough. You just download the image that you want and write to the SD card:

bzcat FreeBSD-10.1-RELEASE-arm-armv6-BEAGLEBONE.img.bz2 | dd of=/dev/da0 bs=1M

The image is even nice enough to automatically expand the filesystem to take up the full SD card during first boot and disable some services (like sendmail) that you won't want.

Installing ports is more difficult. The default pkg server at pkg.freebsd.org only provides packages for i386 and amd64. I imagine the reason being that there are no default flags that would make everyone happy when you're dealing with unusual or embedded architectures.

So you'll need to build your own. In my first attempt at this, I had the beagle NFS mount /usr/ports/ from a different machine. This meant I didn't have to store the ports tree or compile on an SD card, which was good. But this stopped being a good solution as soon as I needed a more than a handful of packages due to the insanely long build times. Time for something better.

Enter poudriere. Poudriere is an amazing tool for building and testing packages in bulk. It's what produces the pkg repos mentioned above, among other things. Recently, it gained the ability to cross compile for ARM, with the aid of QEMU.

I had great difficulty researching this. Most of what I found was several years old and was no longer relevant. Some recent posts about it had incorrect commands or didn't describe the whole process. This is exactly what I did to make it work, spelled out clearly.

Installing the Tools

If possible, use a server for this. It can be anything with some horsepower that you can leave running for a long time (i.e. not your laptop).

  1. The first step is to install the development version of poudriere. As far as I know, the ability to build for ARM has not been merged into the stable version yet.

    cd /usr/ports/ports-mgmt/poudriere-devel
    make config

    Because you're cross compiling for ARM, you need to enable the QEMU flag to build qemu-user-static.

    make install clean
  2. Edit /usr/local/etc/poudriere.conf and read everything. You need to tell it whether to use ZFS or not, define an FTP server, and can set a bunch of other options.

Build the Environment

  1. Before you can do anything more with ARM stuff, you need to tell the kernel what to do with ARM binaries, because you obviously can't run them natively.

    binmiscctl add armv6 --interpreter "/usr/local/bin/qemu-arm-static" --magic "\x7f\x45\x4c\x46\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00" --mask "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff" --size 20 --set-enabled

    This tells your machine to use the QEMU ARM emulator whenever the magic header of an ELF binary matches the armv6 architecture. This allows you to run ARM binaries transparently on your build server.

  2. Now it's time to have poudriere create its build jail. I strongly recommend reading the full man page before continuing. Generally, you can use either FTP (binary) or SVN (source) to create the jail. But because this is ARM, you have to use SVN.

    poudriere jail -c -j 101armv6 -a arm.armv6 -m svn -v releng/10.1

    You have some time to kill while poudriere builds the jail. I chose to use the 10.1-RELEASE branch, but you can grab whichever one you want. Just make sure the major version is the same as what you're running on the beagle, or the ports you build won't be ABI compatible.

  3. Tell poudriere to fetch a ports tree.

    poudriere ports -c

Start the Build

  1. Now you just need a list of ports for poudriere to build. You don't want it to build everything -- it will take forever, and not everything will compile for ARM anyway. Just make a file with the ports that you need, ignoring dependencies. Here's mine.

    sysutils/tmux
    sysutils/screen
    sysutils/usbutils
    net/vnstat
    multimedia/webcamd
    multimedia/ffmpeg
    benchmarks/iperf
    devel/git
    multimedia/v4l-utils
  2. The last step before we actually start the build is to set options for each port. You're building this for an embedded device, so go though these and trim off as many dependencies as you can get away with. You don't want to accidentally pull in Xorg or ruby or something big that you don't need.

    poudriere options -f /usr/home/randy/common/bbb-pkg -j 101armv6
  3. Now start the real build.

    poudriere bulk -f /usr/home/randy/common/bbb-pkg -j 101armv6

Watch it Happen

  1. Poudriere dumps log and status files in such a way that you can start a webserver and watch the build in a browser. We'll also need this to serve packages later. Install nginx and add the following server definition (based on the one here) to your /usr/local/etc/nginx/nginx.conf

    server {
            listen       0.0.0.0:8080;
            #server_name  build.example.com;
            root         /usr/local/share/poudriere/html;
            # Allow caching static resources
            location ~* ^.+\.(jpg|jpeg|gif|png|ico|svg|woff|css|js|html)$ {
                    add_header Cache-Control "public";
                    expires 2d;
            }
            location /data {
                    alias /usr/local/poudriere/data/logs/bulk;
                    # Allow caching dynamic files but ensure they get rechecked
                    location ~* ^.+\.(log|txz|tbz|bz2|gz)$ {
                            add_header Cache-Control "public, must-revalidate, proxy-revalidate";
                    }
                    # Don't log json requests as they come in frequently and ensure
                    # caching works as expected
                    location ~* ^.+\.(json)$ {
                            add_header Cache-Control "public, must-revalidate, proxy-revalidate";
                            access_log off;
                            log_not_found off;
                    }
                    # Allow indexing only in log dirs
                    location ~ /data/?.*/(logs|latest-per-pkg)/ {
                            autoindex on;
                    }
                    break;
            }
            location /repo {
                    alias /usr/local/poudriere/data/packages;
                    autoindex on;
            }
    }
  2. Now you can run service nginx onestart and hit the build server in your browser. You'll see a bunch of build statistics and can just leave the auto-refreshing page up while the build runs. If something fails to build, you're kind of on your own. If it's a dependency you don't need, get rid of it. If not, try to figure out the problem and file a PR.

Using Your New Repo from the Beagle

Now that everything is built, you need to know how to use it. Fortunately, this is the easiest part. The file /etc/pkg/FreeBSD.conf is what defines your pkg repo location. You need to override that by creating a local version on the beagle.

  1. Disable the default repo.

    mkdir -p /usr/local/etc/pkg/repos
    echo "FreeBSD: { enabled: no }" > /usr/local/etc/pkg/repos/FreeBSD.conf
  2. Add your own repo definition to /usr/local/etc/pkg/repos/myrepo.conf

    myrepo: {
      url: "pkg+http://build.example.com:8080/repo/101armv6-default/.latest",
      mirror_type: "srv",
      enabled: yes
    }

    I didn't add any authentication because this is only for my use, but you're welcome to do so.

  3. Now you can run pkg update and it will bootstrap pkg from your new repo -- no ports tree is required on the beagle at all. pkg search and pkg install will work as expected.

Conclusion

This was my first time using poudriere and also my first time cross compiling FreeBSD ports. Finding the information was difficult, but this is actually a straightforward task. Poudriere is a very impressive tool.

After seeing how easy this is, I'll never compile on an underpowered machine again. Whenever I need to change an option, I just hop over to my build server to change the flag and rebuild, then pkg update from my embedded device. It's easy to maintain different versions on the same server, too.

Now if you're like me, you'll start walking around thinking, "Hmm, maybe this random thing could be running BSD..."

References

I collected this information from many sites, but these were some of the more helpful ones (unordered).


published 2015-06-03