View on GitHub

Build Gear

A lightweight embedded firmware build tool

Build Gear Handbook

Build Gear Handbook

Version 0.9.26

This work is licensed under the Creative Commons Attribution-ShareAlike 3.0 License. To view a copy of the license, visit http://creativecommons.org/licenses/by-sa/3.0

2016-05-20


1.  Introduction

1.1.  What is Build Gear?

Build Gear is a lightweight build tool for building embedded firmware. Its target audience is experienced GNU/Linux users and developers.

The primary focus of this build tool is to keep building simple which is reflected in a straightforward command line interface and support for easy readable and maintainable build files. The secondary focus is build performance and build integrity. Build Gear is easy to use and well suited for rapid prototyping and product development of GNU/Linux firmware to be deployed in small to medium sized embedded systems.

1.2.  Why use Build Gear?

In short, use Build Gear because it is simple and because it enables full control of how you build your firmware. By design, it operates in simple ways and offers a high level of transparency in regards to how things are built. This way it becomes easy to create and customize build files which in turn helps ensure a high level of software quality. Also, other popular build tools often delivers firmware build files (meta data) which is permeated with all sorts of noisy and often unwanted configuration code. In contrast, Build Gear is designed to allow for quickly and concisely putting together build files from scratch which include configuration code and options only relevant for the end customer. This way the end customer will get a clean cut firmware meta data delivery which is much easier to navigate and work with.

1.3.  History

Build Gear started out as an experimental build tool used for rapid prototyping of firmware for various GNU/Linux based systems.

The more complexity introduced in the build meta data the more prone the firmware build becomes to build errors and increased maintenance. Also, with high complexity it becomes hard to maintain a high level of software quality. This cost is overlooked by many existing build tools.

In response to these issues, Build Gear is an attempt to create a build tool which does things as simple as possible and with as little infrastructure as possible in order to perform the specific job of building fully tailored embedded firmware images for prototypes or products.

1.4.  License

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

1.5.  Terminology

Build system

The machine you are building on (typically x86).

Host system

The machine you are building for (eg. arm, powerpc, etc.).

Build

The buildable unit represented by a Buildfile.

Build space

The repository within which the build tool operates.

2.  Installing Build Gear

2.1.  Supported GNU/Linux Distributions

Build Gear is tested with the latest Ubuntu release which is currently Ubuntu 14.10 (64-bit). Installation on other GNU/Linux distributions is not officially supported. However, installing and building on other distributions might work, especially if they are based on Debian - but do so on your own risk.

2.2.  Installing From Source

To build the Build Gear tool successfully you need to install the following build dependecies:

$ sudo apt-get install build-essential libcurl4-gnutls-dev libncurses5-dev unzip

Then build and install Build Gear using standard configure and make options. For example:

$ wget http://buildgear.io/release/buildgear-0.9.26.tar.xz
$ tar xf buildgear-0.9.26.tar.xz
$ cd buildgear-0.9.26
$ ./configure --prefix=$HOME/opt/buildgear
$ make
$ make install

The following two steps are not be necessary if you install Build Gear system wide (ie. --prefix=/usr).

To add Build Gear to your path, do eg.:

$ echo 'export PATH=$HOME/opt/buildgear/bin:$PATH' >> ~/.bashrc

To enable Build Gear bash auto completion, do eg.:

$ echo 'source $HOME/opt/buildgear/etc/bash_completion.d/buildgear-bash-completion.sh' >>~/.bashrc

2.3. Installing From Ubuntu PPA

Build Gear can be installed via Ubuntu PPA:

$ sudo apt-add-repository ppa:team-buildgear/buildgear
$ sudo apt-get update
$ sudo apt-get install buildgear

In case you do not have Ubuntu installed or do not wish to install Ubuntu as your system OS, then it is recommended to instead use Build Gear with Ubuntu installed in a virtual machine such as VirtualBox, Qemu, etc..

3.  Build Gear Basics

3.1.  Getting Started

The easiest way to get started learning Build Gear is by cloning an existing build files repository and start building stuff. For example, to get started with the Mini Distribution for the Hummingboard simply do:

$ git clone --recursive git://github.com/mini-distribution/hummingboard.git

The Mini Distribution serves as a reference distribution and example on how to create firmware using Build Gear.

Note

The --recursive option makes git recursively checkout the submodules of the git repository. Most Build Gear build repositories make use of submodules. If one forgets to use this option it is possible to force an update of the submodules by running the command git submodule update --init.

Next, enter the repository, also known as the build space:

$ cd hummingboard

Please take a careful look at the README included in the board repository. It usually includes various useful informantion such as known issues and what build system dependencies are needed to be installed in order to build successfully:

$ buildgear show --readme

In case of the Hummingboard, you will find the following useful section in the README:

Build system requirements:

   Tested on Ubuntu 14.10 (64-bit x86) with the following additional packages
   installed:

   $ sudo apt-get install unzip groff bison flex gperf texinfo xsltproc \
                          libtool libncurses5-dev gawk libexpat1-dev \
                          gettext u-boot-tools libglib2.0-dev intltool \
                          libxml2-utils docbook-xml cmake g++ pixz lzop \
                          u-boot-tools


   Also requires reconfiguring "/bin/sh" to use bash:

   $ sudo dpkg-reconfigure dash

   Fact: Many software distributions fail to build using the dash shell.

Note

Please take care to reconfigure your default shell to use bash instead of dash. If you don't you will experience build errors.

After installing the required dependencies and reconfiguring your default shell to use bash, then you are ready to start building. Also found in the README is usually a short description of the main builds of interest:

Main builds:

   fs      - Filesystem
   linux   - Linux kernel
   u-boot  - Bootloader

Now, you can go ahead and build the filesystem:

   $ buildgear build fs

Note

In case of the Hummingboard, the fs build automatically builds the linux kernel because the kernel depends on the Linux kernel modules to be installed in the filesystem.

Next, build the bootloader:

   $ buildgear build u-boot

The resulting output files from these commands are found in the build/output directory:

   fs.tar.xz  SPL  u-boot.img

Finish by installing the output files on a SD card and you are ready to boot the Mini Distribution on the Hummingboard. Usually the README contains details on how to create and format the partitions on the SD Card and how to install the files.

3.2.  Build Space Layout

When you enter a build repository directory you also enter the repository build space. All Build Gear commands relate to the build space you are currently in. A typical build space layout looks like this (Hummingboard example):

hummingboard
├── build
│   ├── build.log
│   ├── output
│   ├── package
│   └── source
├── buildfiles
│   ├── cross
│   │   ├── cross-core
│   │   ├── cross-extra
│   │   ├── cross-freescale
│   │   ├── cross-opt
│   │   └── cross-hummingboard
│   └── native
│       └── native-hummingboard
├── config
│   └── config
└── README

The buildfiles directory contains the build files which is split into cross and native type build file directories. Each of these directories contains various build files submodules. cross-core, cross-opt, and cross-extra are all submodules containing generic build files from the Mini Distribution. In this case, the directories native-hummingboard and cross-hummingboard are actually not submodules but are simply directories belonging to the top level repository - they contain board specific build files.

Note

The Mini Distribution is a reference distribution. You can decide to use its build files as they are or fork and modify the repositories as you like. If needed, individual build files can be overridden by your own custom build files using layers.

The build files are arranged in eg. cross-core like so:

cross-core
├── acl
├── attr
├── bash
├── core
├── coreutils
├── dbus
├── expat
├── findutils
├── fs-skeleton
...
├── shadow
├── sysfsutils
├── systemd
├── sysvinit
├── tcp_wrappers
├── usbutils
├── util-linux
├── wget
└── zlib

In case of the Mini Distribution, the cross-core directory contains the the build files which represent the "core" functionality. Likewise, the cross-opt and cross-extra directories contains the build files with "optional" and "extra" functionality respectively.

The config directory contains the global build files configuration file config.

The build directory is the temporary build directory which is automatically created when building - it contains build log, build output, internal build packages, and downloaded build source. The build log file is build/build.log. The build output files are located in build/output. The internal build package files are located in build/package. The downloaded build source is located in build/source.

During build the build directory also contains a work directory in which things are built. This directory is automatically removed after after a successful build. However, if a build error occurs this directory is kept so that one can inspect the failing build data.

3.3.  Buildfile

Each buildable unit is represented by a build file named Buildfile. A typical build file is structured like so (buildfiles/cross/cross-core/attr/Buildfile example):

# Description: Commands for Manipulating Filesystem Extended Attributes
# URL: http://savannah.nongnu.org/projects/attr
# License: GPLv2

name=attr
version=2.4.47
release=1
source=(http://mirrors.zerg.biz/nongnu/attr/$name-$version.src.tar.gz)

build() {
   cd $name-$version

   ./configure --build=$BUILD \
               --host=$HOST \
               --prefix=/usr \
               --disable-static \
               --disable-nls

   make -j $JOBS
   make DIST_ROOT=$PKG install install-lib install-dev

   rm -rf $PKG/usr/share/{doc,locale}

   # Fix libtool files
   fix_la_files $PKG
}

check() {
   # Test for preinstalled msgfmt tool
   check_tool msgfmt "Please install msgfmt"
}

A Buildfile contains the following variables and functions:

Variable Description Type
name Build name. Mandatory
version Build version. Mandatory
release Package release number. Optional
source A list of sources - these sources can be local files or remote files, ie. URLs beginning with http(s):// or ftp(s)://. Optional
depends A list of build dependencies. To keep things simple, notice that there is no distinction between build time and run time dependencies. Optional
options A list of special options. Supported options are:
buildlock

Locks the build so that it will be the only one building at any given time (when parallel_builds > 1).

nostrip

Disables automatic strip of executables, shared libraries (.so), and static archives (.a).

Optional
build() Build function. Optional
check() Check function. Optional
SRC Source directory - this variable refers to the location of the files defined in the source array. Compressed source files are automatically extracted to this location. The following types of compressed files are supported: .tar.gz, .tar.xz, .tgz, .txz, .zip. Provided
PKG Package directory - files copied to this location will end up in an internal package file. Provided
OUTPUT Output directory - files copied to this location will end up in the output directory (build/output). Provided

Some variables are mandatory and must be defined while others are automatically provided by Build Gear and some can simply be optionally defined.

Note

By convention, the directory containing the Buildfile should have the exact same name as the build name defined in the same Buildfile.

3.4.  Build Mechanism

The build mechanism in Build Gear is quite simple. Whenever the build command is used such as buildgear build fs or buildgear build zlib etc., the following steps are performed by the Build Gear tool:

  1. Searches for available build files in the current build space
  2. Loads the found build files
  3. Resolves build order from dependency information
  4. Runs build system checks (calls check() function in build files)
  5. Downloads any missing source files
  6. Detects build system type
  7. Verifies configured host system type
  8. Cleans out old work directory (if any)
  9. Starts building in resolved build order (calls build() function in all resolved build files)

The steps above a directly reflected in the terminal output when running the tool, eg. running buildgear build fs outputs the following:

$ buildgear build fs

Searching for build files..     Done (246 files)
Loading build files..           Done
Resolving dependencies..        Done (87 dependencies)
Running build system checks..   Done
Downloading sources..           Done
Detecting BUILD system type..   x86_64-unknown-linux-gnu
Configured HOST system type..   arm-cortex_a9-linux-gnueabi
Configured parallel builds..    4
Cleaning old work directory..   Done

Building 'cross/fs'..
 / Building   'native/linux-source'
   ...

3.5.  Build Environment & Configuration

The bash environment in which the build() function of each build file is executed is constituted by the following environments added in the following order:

User shell environment + Global build files config + Buildfile

This means that variables and functions defined in the Buildfile override the ones defined in the global configuration file which in turn overrides the users shell environment.

Note

All build actions are performed in a fakeroot session. Using fakeroot allow builds to create special device files and files with custom ownership. This makes it possible to maintain consistent footprints across build machines. It is possible to disable the use of fakeroot by using the --no-fakeroot option but it is normally ill advised to do so.

3.5.1.  Global Buildfiles Configuration

The global build files configuration is located in the config/config file. It defines the global configuration for the particular set of build files located in the build space. It is global because it can be used to define bash variables and functions which are available for all build files. It also contains some configuration variables which are required by the build tool.

With the global configuration file you can create a fully configurable software distribution if needed. Meaning, you can introduce all sorts of configuration variables here that build files can use and react on accordingly. Though, it is recommended to carefully keep the amount of this type of configuration variables to a minimum to keep the configuration sample space low and minimize maintenance.

A minimal global build files configuration file would typically look something like this (Hummingboard example):

# Build files configuration

# Set host system name
HOST="arm-cortex_a9-linux-gnueabi"

# Make all cross builds depend on cross toolchain
CROSS_DEPENDS=native/crosstool-ng

# Set cross sysroot path
CROSS_SYSROOT="$NATIVE_SYSROOT/$HOST/sysroot"

# Set download mirror
DOWNLOAD_MIRROR="http://buildgear.io/mirror"

# Add cross toolchain to build environment
PATH="$NATIVE_SYSROOT/bin:$PATH"
PATH="$NATIVE_SYSROOT/usr/bin:$PATH"

# Set number of parallel make jobs
JOBS=`cat /proc/cpuinfo | grep processor | wc -l`

# Pkg-config cross configuration
if [ "$BUILD_TYPE" = "cross" ]; then
   export PKG_CONFIG_DIR=
   export PKG_CONFIG_PATH=
   export PKG_CONFIG_LIBDIR="$CROSS_SYSROOT/usr/lib/pkgconfig"
   export PKG_CONFIG_LIBDIR+=":$CROSS_SYSROOT/usr/share/pkgconfig"
   export PKG_CONFIG_SYSROOT_DIR="$CROSS_SYSROOT"
fi

# Pkg-config native configuration
if [ "$BUILD_TYPE" = "native" ]; then
   export PKG_CONFIG_LIBDIR="$NATIVE_SYSROOT/usr/lib/pkgconfig"
   export PKG_CONFIG_LIBDIR+=":`pkg-config --variable pc_path pkg-config`"
fi

# Helper functions

fix_la_files()
{
   find $1 -name "*.la" \
           -exec sed -i -e "/dependency_libs/s; /usr; `readlink -f $SYSROOT`/usr;g" {} \; \
           -exec sed -i -e "/libdir/s;'/usr;'`readlink -f $SYSROOT`/usr;g" {} \;
}

check_tool()
{
   type $1 &> /dev/null
   if [ $? != 0 ]; then
      echo -e "\n"
      echo "Failed ($1 is not found)"
      echo "$2"
      exit 1
   fi
}

check_exist()
{
   test -e $1 &> /dev/null
   if [ $? != 0 ]; then
      echo -e "\n"
      echo "Failed ($1 is not found)"
      echo "$2"
      exit 1
   fi
}

At a first glance it migh look a bit strange but in reality it is actually quite simple. To understand the global build files configuration, lets go through the individual parts.

The configuration above starts by defining the HOST variable which expresses the host system type (eg. arm-cortex_a9-linux-gnueabi). This variable is used in the configure line (eg. configure --host=$HOST of virtually all cross build files.

Secondly, the CROSS_DEPENDS variable is defined - this variable tells the build tool that all cross builds should depend on this build. This will typically be the Crosstool-NG toolchain build which is of course of the native type because it builds and runs on the build machine.

Next, the build tool needs the CROSS_SYSROOT which defines where the cross sysroot is located. Using standard Crosstool-NG configuration, this will typically be inside the toolchain package added to the native sysroot. Both the native sysroot and cross sysroot act as staging areas - build packages are added here accordingly during the build process.

The download super mirror for your specific set of build files is defined by the DOWNLOAD_MIRROR variable.

The toolchain is added to the environment using the PATH variable.

The JOBS variable is used by the build files to define how many concurrent jobs that make should support (eg. make -j $JOBS).

Next, the pkgconfig tool needs two configurations, one for native building, and a second for cross building. The two are distinguised using the BUILD_TYPE variable which is automatically provided.

This simple fix_la_files() function is a necessary evil to bypass a long standing problem with libtool - namely it's inability to support cross building.

The check_tool() and check_exist() are helper functions which are used in the check() function of build files which rely on certain tools or libraries to be preinstalled on the build system.

All of the bash functions are custom and can be defined and used for your specific needs. The following variables are special and have specific meaning:

Variable Description Type
BUILD Build system triplet (eg. "x86_64-unknown-linux-gnu"). Provided
HOST Host system triplet (eg. "arm-cortex_a8-linux-gnueabi"). Mandatory
CROSS_DEPENDS A list of builds that all cross builds depends on. This variable typically contains a single element, namely the name of the toolchain build (eg. native/crosstool-ng) which of course every cross build should depend on. Mandatory
CROSS_SYSROOT Defines the sysroot for cross builds. Mandatory
SYSROOT Defines the sysroot. In case of cross building it points to the cross sysroot. In case of native building it points to the native sysroot. Provided
NATIVE_SYSROOT Defines the sysroot for native builds. Provided
BUILD_TYPE Defines the build type. Value is either "cross" or "native". Provided
DOWNLOAD_MIRROR Defines the download super mirror. If a source file fails downloading from the source URL then Build Gear will attempt to download the same file from the super mirror. Optional
LAYERS Defines build layer names and priority. Layers are prioritized in the order they are listed. The layer name listed first is highest priority, the last listed is lowest priority. If the 'default' layer is not listed it will automatically assume lowest priority. Optional

Some variables are "mandatory" and must be defined while others are "provided" by Build Gear and some are simply "optional".

4. Using Build Gear

The following chapter describes the mostly used Build Gear features by command line examples.

4.1.  Help

To list all available Build Gear commands and options simply type:

$ buildgear --help

To ask for help on a specific command type eg.:

$ buildgear help build

4.2.  Building

To build for example the "fs" build simply type:

$ buildgear build fs

This will automatically download all sources, resolve all dependencies and start building the "fs" build.

4.3.  Building With Load Chart Enabled

To build and have Build Gear produce a load chart containing CPU and memory usage during the build process simply do eg.:

$ buildgear build fs --load-chart

After a successful "fs" build the load chart will be available in build/output/load-chart.fs.svg.

4.4.  Show Log

To have Build Gear show the full log after a build has finished simply type:

$ buildgear show --log

Alternatively, when Build Gear is building you can follow (tail) the build log live by simply typing:

$ buildgear show --log-tail

This features is quite useful - it enables you to have eg. one terminal window open where you launch the actual build commands while in another window you follow the log as it progresses during the build. It will keep tailing the log across builds.

4.5.  Show Build Order

To have Build Gear show the build order of a particular build, simply type eg.:

$ buildgear show --build-order fs

This will produce a build order list. In case of parallel building, builds marked with the same group number is built concurrently.

4.6.  Show Build Dependency

Build Gear can produce a graphical overview of the dependencies for a specific build. For example:

$ buildgear show --dependency fs

After resolving the dependencies successfully for the "fs" build the dependency chart will be available in build/output/dependency.fs.svg.

4.7.  Update Footprint

To update the footprint for a build simply do eg.:

$ buildgear build zlib --update-footprint

Likewise, to update the footprints for the build and all its dependecies simply do eg.:

$ buildgear build fs --update-footprint --all

4.8.  Update Checksum

To update the checksum for a build simply do eg.:

$ buildgear build zlib --update-checksum

Likewise, to update the checksums for the build and all its dependecies simply do eg.:

$ buildgear build fs --update-checksum --all

4.9.  Show Footprint

To show the footprint for a build simply do eg.:

$ buildgear show --footprint zlib

4.10.  Show Checksum

To show the checksum for a build simply do eg.:

$ buildgear show --checksum zlib

4.11.  Download Source

To download the source of a build simply do eg.:

$ buildgear download zlib

Likewise, to download the source of a build including the source of all its dependencies simply do eg.:

$ buildgear download --all zlib

To download the source of all available builds simply do:

$ buildgear download --all

4.12.  Cleaning

To clean (delete) the internal build package of a build simply do eg.:

$ buildgear clean connman

This will of course force a rebuild the next time you build zlib.

Likewise, to clean a build and all of its dependencies simply do eg.:

$ buildgear clean --all connman

To clean all builds simply do:

$ buildgear clean --all

Please be careful, the above command will remove everything in build/package. Meaning everything will rebuild from scratch in your next build command.

4.13.  Tool Configuration

You can use the Build Gear tool configuration command to set eg. the number of parallel builds in your current build space:

$ buildgear config parallel_builds 4

Likewise, to set the same configuration globally simply do:

$ buildgear config --global parallel_builds 4

To list the available tool configuration settings simple do:

$ buildgear config --list

Note

Tool configuration settings can be applied to either a specific build space (local) or to all build spaces (global). Local settings always override global settings. If settings are not set they simply retain their default value.

4.14.  Layer Configuration

Using build layers it is possible to override build files with build files of a higher priority. This feature is quite useful in case some existing build files do not suit your specific needs and you want to override with your own variants. Layers are defined in the global configuration file (config/config) like so:

# Layer configuration
LAYERS=(layer0 layer1 layer2 default)

Layers with higher priority overrides layers with lower prority. The layers are prioritized according to the order in which they are listed. Meaning, the first layer in the list (layer0) is the highest priority and the layer listed last (default) is the lowest priority. The default layer name can be omitted in which case it will automatically assume lowest priority.

Note

Please notice that the layer name default is a reserved keyword for the layer name that a buildfile belongs to by default if no layer is defined for the buildfile.

Buildfiles are automatically assigned to <layer> when placed in:

buildfiles/cross/cross-<layer>
buildfiles/native/native-<layer>

If <layer> is not defined in the LAYERS list then build files in these locations will automatically be assigned to the default layer.

Alternatively, it is also possible to assign a single buildfile to a specific layer by adding the layer variable to the buildfile, eg.:

layer=layer0

Note

Please notice that the preferred method for assigning layers is by Buildfile path location (eg. cross-<layer>) and not by using the layer variable.

4.15.  Strip Configuration

By default the output object files of a build are automatically stripped (symbols discarded). The stripped files include executables, shared libraries (.so), and static archives (.a).

It is possible to disable this strip step entirely by simply adding the nostrip option to the special options array in the buildfile like so:

options=(nostrip)

Alternatively, it is also possible to select which files will not be automatically stripped by adding a nostrip array to the buildfile matching the files you don't want to be stripped. For example:

nostrip=(/usr/lib/libtest.so /usr/lib/libtest.a)

Entries in the nostrip array are treated as regular expressions (as used by grep) making it possible to do file pattern matching.

Note

Please notice that the nostrip array will have no effect if the nostrip options is defined in the special options array.

4.16.  Force Non-Parallel Building

When building in parallel mode (parallel_builds > 1) it is possible to force a specific build to be built in non-parallel by adding a build lock. To do so simple add the following option to the special options field in the relevant buildfile like so:

options=(buildlock)

5. FAQ

Q:

Why is Build Gear not written in Python or Make like other popular build tools?

A:

Most of the core functionality of the tool is written in C/C++ to achieve best build performance. Some of the tool functionality is written in a bash helper script. Also, all build files are written in bash which makes perfect sense since most install actions and descriptions are bash centric.

Q:

Why does the Mini Distribution rely on so many Ubuntu packages to be installed to do a successful build?

A:

By choice, the Mini Distribution relies on many preinstalled Ubuntu packages in order to reduce the total build time. The alternative is to build virtually every single native build that your cross builds need. This will add significantly to the total build time and in most cases not worth the effort. That being said, anyone can create their own distribution which include all required native builds. Such a distribution will be likely to work with just about any build system different from Ubuntu.