Version 0.9.26
Copyright © 2014 Martin Lund
This work is licensed under the Creative Commons Attribution-ShareAlike 3.0 License. To view a copy of the license, visit http://creativecommons.org/licenses/by-sa/3.0
2016-05-20
Table of Contents
- 1. Introduction
- 2. Installing Build Gear
- 3. Build Gear Basics
- 4. Using Build Gear
- 4.1. Help
- 4.2. Building
- 4.3. Building With Load Chart Enabled
- 4.4. Show Log
- 4.5. Show Build Order
- 4.6. Show Build Dependency
- 4.7. Update Footprint
- 4.8. Update Checksum
- 4.9. Show Footprint
- 4.10. Show Checksum
- 4.11. Download Source
- 4.12. Cleaning
- 4.13. Tool Configuration
- 4.14. Layer Configuration
- 4.15. Strip Configuration
- 4.16. Force Non-Parallel Building
- 5. FAQ
Build Gear is a lightweight build tool for building embedded firmware. Its target audience is experienced GNU/Linux users and developers.
The primary focus of this build tool is to keep building simple which is reflected in a straightforward command line interface and support for easy readable and maintainable build files. The secondary focus is build performance and build integrity. Build Gear is easy to use and well suited for rapid prototyping and product development of GNU/Linux firmware to be deployed in small to medium sized embedded systems.
In short, use Build Gear because it is simple and because it enables full control of how you build your firmware. By design, it operates in simple ways and offers a high level of transparency in regards to how things are built. This way it becomes easy to create and customize build files which in turn helps ensure a high level of software quality. Also, other popular build tools often delivers firmware build files (meta data) which is permeated with all sorts of noisy and often unwanted configuration code. In contrast, Build Gear is designed to allow for quickly and concisely putting together build files from scratch which include configuration code and options only relevant for the end customer. This way the end customer will get a clean cut firmware meta data delivery which is much easier to navigate and work with.
Build Gear started out as an experimental build tool used for rapid prototyping of firmware for various GNU/Linux based systems.
The more complexity introduced in the build meta data the more prone the firmware build becomes to build errors and increased maintenance. Also, with high complexity it becomes hard to maintain a high level of software quality. This cost is overlooked by many existing build tools.
In response to these issues, Build Gear is an attempt to create a build tool which does things as simple as possible and with as little infrastructure as possible in order to perform the specific job of building fully tailored embedded firmware images for prototypes or products.
Build Gear is tested with the latest Ubuntu release which is currently Ubuntu 14.10 (64-bit). Installation on other GNU/Linux distributions is not officially supported. However, installing and building on other distributions might work, especially if they are based on Debian - but do so on your own risk.
To build the Build Gear tool successfully you need to install the following build dependecies:
$ sudo apt-get install build-essential libcurl4-gnutls-dev libncurses5-dev unzip
Then build and install Build Gear using standard configure and make options. For example:
$ wget http://buildgear.io/release/buildgear-0.9.26.tar.xz $ tar xf buildgear-0.9.26.tar.xz $ cd buildgear-0.9.26 $ ./configure --prefix=$HOME/opt/buildgear $ make $ make install
The following two steps are not be necessary if you install Build Gear system wide (ie. --prefix=/usr).
To add Build Gear to your path, do eg.:
$ echo 'export PATH=$HOME/opt/buildgear/bin:$PATH' >> ~/.bashrc
To enable Build Gear bash auto completion, do eg.:
$ echo 'source $HOME/opt/buildgear/etc/bash_completion.d/buildgear-bash-completion.sh' >>~/.bashrc
Build Gear can be installed via Ubuntu PPA:
$ sudo apt-add-repository ppa:team-buildgear/buildgear $ sudo apt-get update $ sudo apt-get install buildgear
In case you do not have Ubuntu installed or do not wish to install Ubuntu as your system OS, then it is recommended to instead use Build Gear with Ubuntu installed in a virtual machine such as VirtualBox, Qemu, etc..
The easiest way to get started learning Build Gear is by cloning an existing build files repository and start building stuff. For example, to get started with the Mini Distribution for the Hummingboard simply do:
$ git clone --recursive git://github.com/mini-distribution/hummingboard.git
The Mini Distribution serves as a reference distribution and example on how to create firmware using Build Gear.
Note
The--recursive
option makes git recursively checkout
the submodules of the git repository. Most Build Gear build
repositories make use of submodules. If one forgets to use this
option it is possible to force an update of the submodules by running
the command git submodule update --init
.
Next, enter the repository, also known as the build space:
$ cd hummingboard
Please take a careful look at the README included in the board repository. It usually includes various useful informantion such as known issues and what build system dependencies are needed to be installed in order to build successfully:
$ buildgear show --readme
In case of the Hummingboard, you will find the following useful section in the README:
Build system requirements: Tested on Ubuntu 14.10 (64-bit x86) with the following additional packages installed: $ sudo apt-get install unzip groff bison flex gperf texinfo xsltproc \ libtool libncurses5-dev gawk libexpat1-dev \ gettext u-boot-tools libglib2.0-dev intltool \ libxml2-utils docbook-xml cmake g++ pixz lzop \ u-boot-tools Also requires reconfiguring "/bin/sh" to use bash: $ sudo dpkg-reconfigure dash Fact: Many software distributions fail to build using the dash shell.
Note
Please take care to reconfigure your default shell to use bash instead of dash. If you don't you will experience build errors.After installing the required dependencies and reconfiguring your default shell to use bash, then you are ready to start building. Also found in the README is usually a short description of the main builds of interest:
Main builds: fs - Filesystem linux - Linux kernel u-boot - Bootloader
Now, you can go ahead and build the filesystem:
$ buildgear build fs
Note
In case of the Hummingboard, thefs
build automatically
builds the linux kernel because the kernel depends on the Linux kernel
modules to be installed in the filesystem.
Next, build the bootloader:
$ buildgear build u-boot
The resulting output files from these commands are found in the
build/output
directory:
fs.tar.xz SPL u-boot.img
Finish by installing the output files on a SD card and you are ready to boot the Mini Distribution on the Hummingboard. Usually the README contains details on how to create and format the partitions on the SD Card and how to install the files.
When you enter a build repository directory you also enter the repository build space. All Build Gear commands relate to the build space you are currently in. A typical build space layout looks like this (Hummingboard example):
hummingboard ├── build │ ├── build.log │ ├── output │ ├── package │ └── source ├── buildfiles │ ├── cross │ │ ├── cross-core │ │ ├── cross-extra │ │ ├── cross-freescale │ │ ├── cross-opt │ │ └── cross-hummingboard │ └── native │ └── native-hummingboard ├── config │ └── config └── README
The buildfiles
directory contains the build files which
is split into cross and native type build file directories. Each of
these directories contains various build files submodules.
cross-core
, cross-opt
, and
cross-extra
are all submodules containing generic
build files from the Mini Distribution. In this case, the directories
native-hummingboard
and cross-hummingboard
are actually not submodules but are simply directories belonging to
the top level repository - they contain board specific build files.
Note
The Mini Distribution is a reference distribution. You can decide to use its build files as they are or fork and modify the repositories as you like. If needed, individual build files can be overridden by your own custom build files using layers.
The build files are arranged in eg. cross-core
like so:
cross-core ├── acl ├── attr ├── bash ├── core ├── coreutils ├── dbus ├── expat ├── findutils ├── fs-skeleton ... ├── shadow ├── sysfsutils ├── systemd ├── sysvinit ├── tcp_wrappers ├── usbutils ├── util-linux ├── wget └── zlib
In case of the Mini Distribution, the cross-core
directory contains the the build files which represent the "core"
functionality. Likewise, the cross-opt
and
cross-extra
directories contains the build files with
"optional" and "extra" functionality respectively.
The config
directory contains the global build files
configuration file config
.
The build
directory is the temporary build directory
which is automatically created when building - it contains build log,
build output, internal build packages, and downloaded build source.
The build log file is build/build.log
. The build output
files are located in build/output
. The internal build
package files are located in build/package
. The
downloaded build source is located in build/source
.
During build the build
directory also contains a
work
directory in which things are built. This directory
is automatically removed after after a successful build. However, if a
build error occurs this directory is kept so that one can inspect the
failing build data.
Each buildable unit is represented by a build file named
Buildfile
. A typical build file is structured like so
(buildfiles/cross/cross-core/attr/Buildfile
example):
# Description: Commands for Manipulating Filesystem Extended Attributes # URL: http://savannah.nongnu.org/projects/attr # License: GPLv2 name=attr version=2.4.47 release=1 source=(http://mirrors.zerg.biz/nongnu/attr/$name-$version.src.tar.gz) build() { cd $name-$version ./configure --build=$BUILD \ --host=$HOST \ --prefix=/usr \ --disable-static \ --disable-nls make -j $JOBS make DIST_ROOT=$PKG install install-lib install-dev rm -rf $PKG/usr/share/{doc,locale} # Fix libtool files fix_la_files $PKG } check() { # Test for preinstalled msgfmt tool check_tool msgfmt "Please install msgfmt" }
A Buildfile
contains the following variables and functions:
Variable | Description | Type |
---|---|---|
name
| Build name. | Mandatory |
version
| Build version. | Mandatory |
release
| Package release number. | Optional |
source
| A list of sources - these sources can be local files or remote files, ie. URLs beginning with http(s):// or ftp(s)://. | Optional |
depends
| A list of build dependencies. To keep things simple, notice that there is no distinction between build time and run time dependencies. | Optional |
options
|
A list of special options. Supported options are:
| Optional |
build()
| Build function. | Optional |
check()
| Check function. | Optional |
SRC
|
Source directory - this variable refers to the location of the files
defined in the source array. Compressed source
files are automatically extracted to this location. The following
types of compressed files are supported: .tar.gz, .tar.xz, .tgz,
.txz, .zip.
| Provided |
PKG
| Package directory - files copied to this location will end up in an internal package file. | Provided |
OUTPUT
|
Output directory - files copied to this location will end up in the
output directory (build/output ).
| Provided |
Some variables are mandatory and must be defined while others are automatically provided by Build Gear and some can simply be optionally defined.
Note
By convention, the directory containing the Buildfile should have the exact same name as the build name defined in the same Buildfile.
The build mechanism in Build Gear is quite simple. Whenever the build
command is used such as buildgear build fs
or
buildgear build zlib
etc., the following steps are
performed by the Build Gear tool:
- Searches for available build files in the current build space
- Loads the found build files
- Resolves build order from dependency information
-
Runs build system checks (calls
check()
function in build files) - Downloads any missing source files
- Detects build system type
- Verifies configured host system type
- Cleans out old work directory (if any)
-
Starts building in resolved build order (calls
build()
function in all resolved build files)
The steps above a directly reflected in the terminal output when running
the tool, eg. running buildgear build fs
outputs the
following:
$ buildgear build fs Searching for build files.. Done (246 files) Loading build files.. Done Resolving dependencies.. Done (87 dependencies) Running build system checks.. Done Downloading sources.. Done Detecting BUILD system type.. x86_64-unknown-linux-gnu Configured HOST system type.. arm-cortex_a9-linux-gnueabi Configured parallel builds.. 4 Cleaning old work directory.. Done Building 'cross/fs'.. / Building 'native/linux-source' ...
The bash environment in which the build()
function of
each build file is executed is constituted by the following
environments added in the following order:
User shell environment + Global build files config + Buildfile
This means that variables and functions defined in the Buildfile override the ones defined in the global configuration file which in turn overrides the users shell environment.
Note
All build actions are performed in a fakeroot session. Using fakeroot allow builds to create special device files and files with custom ownership. This makes it possible to maintain consistent footprints across build machines. It is possible to disable the use of fakeroot by using the--no-fakeroot
option but it is normally ill
advised to do so.
The global build files configuration is located in the
config/config
file. It defines the global configuration
for the particular set of build files located in the build space. It
is global because it can be used to define bash variables and
functions which are available for all build files. It also contains
some configuration variables which are required by the build tool.
With the global configuration file you can create a fully configurable software distribution if needed. Meaning, you can introduce all sorts of configuration variables here that build files can use and react on accordingly. Though, it is recommended to carefully keep the amount of this type of configuration variables to a minimum to keep the configuration sample space low and minimize maintenance.
A minimal global build files configuration file would typically look something like this (Hummingboard example):
# Build files configuration # Set host system name HOST="arm-cortex_a9-linux-gnueabi" # Make all cross builds depend on cross toolchain CROSS_DEPENDS=native/crosstool-ng # Set cross sysroot path CROSS_SYSROOT="$NATIVE_SYSROOT/$HOST/sysroot" # Set download mirror DOWNLOAD_MIRROR="http://buildgear.io/mirror" # Add cross toolchain to build environment PATH="$NATIVE_SYSROOT/bin:$PATH" PATH="$NATIVE_SYSROOT/usr/bin:$PATH" # Set number of parallel make jobs JOBS=`cat /proc/cpuinfo | grep processor | wc -l` # Pkg-config cross configuration if [ "$BUILD_TYPE" = "cross" ]; then export PKG_CONFIG_DIR= export PKG_CONFIG_PATH= export PKG_CONFIG_LIBDIR="$CROSS_SYSROOT/usr/lib/pkgconfig" export PKG_CONFIG_LIBDIR+=":$CROSS_SYSROOT/usr/share/pkgconfig" export PKG_CONFIG_SYSROOT_DIR="$CROSS_SYSROOT" fi # Pkg-config native configuration if [ "$BUILD_TYPE" = "native" ]; then export PKG_CONFIG_LIBDIR="$NATIVE_SYSROOT/usr/lib/pkgconfig" export PKG_CONFIG_LIBDIR+=":`pkg-config --variable pc_path pkg-config`" fi # Helper functions fix_la_files() { find $1 -name "*.la" \ -exec sed -i -e "/dependency_libs/s; /usr; `readlink -f $SYSROOT`/usr;g" {} \; \ -exec sed -i -e "/libdir/s;'/usr;'`readlink -f $SYSROOT`/usr;g" {} \; } check_tool() { type $1 &> /dev/null if [ $? != 0 ]; then echo -e "\n" echo "Failed ($1 is not found)" echo "$2" exit 1 fi } check_exist() { test -e $1 &> /dev/null if [ $? != 0 ]; then echo -e "\n" echo "Failed ($1 is not found)" echo "$2" exit 1 fi }
At a first glance it migh look a bit strange but in reality it is actually quite simple. To understand the global build files configuration, lets go through the individual parts.
The configuration above starts by defining the HOST
variable which expresses the host system type (eg.
arm-cortex_a9-linux-gnueabi
). This variable is used in
the configure line (eg. configure --host=$HOST
of
virtually all cross build files.
Secondly, the CROSS_DEPENDS
variable is defined - this
variable tells the build tool that all cross builds should depend on
this build. This will typically be the Crosstool-NG toolchain build
which is of course of the native type because it builds and runs on
the build machine.
Next, the build tool needs the CROSS_SYSROOT
which
defines where the cross sysroot is located. Using standard
Crosstool-NG configuration, this will typically be inside the
toolchain package added to the native sysroot. Both the native sysroot
and cross sysroot act as staging areas - build packages are added here
accordingly during the build process.
The download super mirror for your specific set of build files is
defined by the DOWNLOAD_MIRROR
variable.
The toolchain is added to the environment using the PATH
variable.
The JOBS
variable is used by the build files to define
how many concurrent jobs that make should support (eg. make -j
$JOBS
).
Next, the pkgconfig tool needs two configurations, one for native
building, and a second for cross building. The two are distinguised
using the BUILD_TYPE
variable which is automatically
provided.
This simple fix_la_files()
function is a necessary evil
to bypass a long standing problem with libtool - namely it's inability
to support cross building.
The check_tool()
and check_exist()
are helper
functions which are used in the check()
function of build
files which rely on certain tools or libraries to be preinstalled on
the build system.
All of the bash functions are custom and can be defined and used for your specific needs. The following variables are special and have specific meaning:
Variable | Description | Type |
---|---|---|
BUILD
| Build system triplet (eg. "x86_64-unknown-linux-gnu"). | Provided |
HOST
| Host system triplet (eg. "arm-cortex_a8-linux-gnueabi"). | Mandatory |
CROSS_DEPENDS
| A list of builds that all cross builds depends on. This variable typically contains a single element, namely the name of the toolchain build (eg. native/crosstool-ng) which of course every cross build should depend on. | Mandatory |
CROSS_SYSROOT
| Defines the sysroot for cross builds. | Mandatory |
SYSROOT
| Defines the sysroot. In case of cross building it points to the cross sysroot. In case of native building it points to the native sysroot. | Provided |
NATIVE_SYSROOT
| Defines the sysroot for native builds. | Provided |
BUILD_TYPE
| Defines the build type. Value is either "cross" or "native". | Provided |
DOWNLOAD_MIRROR
| Defines the download super mirror. If a source file fails downloading from the source URL then Build Gear will attempt to download the same file from the super mirror. | Optional |
LAYERS
| Defines build layer names and priority. Layers are prioritized in the order they are listed. The layer name listed first is highest priority, the last listed is lowest priority. If the 'default' layer is not listed it will automatically assume lowest priority. | Optional |
Some variables are "mandatory" and must be defined while others are "provided" by Build Gear and some are simply "optional".
The following chapter describes the mostly used Build Gear features by command line examples.
To list all available Build Gear commands and options simply type:
$ buildgear --help
To ask for help on a specific command type eg.:
$ buildgear help build
To build for example the "fs" build simply type:
$ buildgear build fs
This will automatically download all sources, resolve all dependencies and start building the "fs" build.
To build and have Build Gear produce a load chart containing CPU and memory usage during the build process simply do eg.:
$ buildgear build fs --load-chart
After a successful "fs" build the load chart will be available in
build/output/load-chart.fs.svg
.
To have Build Gear show the full log after a build has finished simply type:
$ buildgear show --log
Alternatively, when Build Gear is building you can follow (tail) the build log live by simply typing:
$ buildgear show --log-tail
This features is quite useful - it enables you to have eg. one terminal window open where you launch the actual build commands while in another window you follow the log as it progresses during the build. It will keep tailing the log across builds.
To have Build Gear show the build order of a particular build, simply type eg.:
$ buildgear show --build-order fs
This will produce a build order list. In case of parallel building, builds marked with the same group number is built concurrently.
Build Gear can produce a graphical overview of the dependencies for a specific build. For example:
$ buildgear show --dependency fs
After resolving the dependencies successfully for the "fs" build the
dependency chart will be available in
build/output/dependency.fs.svg
.
To update the footprint for a build simply do eg.:
$ buildgear build zlib --update-footprint
Likewise, to update the footprints for the build and all its dependecies simply do eg.:
$ buildgear build fs --update-footprint --all
To update the checksum for a build simply do eg.:
$ buildgear build zlib --update-checksum
Likewise, to update the checksums for the build and all its dependecies simply do eg.:
$ buildgear build fs --update-checksum --all
To show the footprint for a build simply do eg.:
$ buildgear show --footprint zlib
To show the checksum for a build simply do eg.:
$ buildgear show --checksum zlib
To download the source of a build simply do eg.:
$ buildgear download zlib
Likewise, to download the source of a build including the source of all its dependencies simply do eg.:
$ buildgear download --all zlib
To download the source of all available builds simply do:
$ buildgear download --all
To clean (delete) the internal build package of a build simply do eg.:
$ buildgear clean connman
This will of course force a rebuild the next time you build zlib.
Likewise, to clean a build and all of its dependencies simply do eg.:
$ buildgear clean --all connman
To clean all builds simply do:
$ buildgear clean --all
Please be careful, the above command will remove everything in
build/package
. Meaning everything will rebuild from
scratch in your next build command.
You can use the Build Gear tool configuration command to set eg. the number of parallel builds in your current build space:
$ buildgear config parallel_builds 4
Likewise, to set the same configuration globally simply do:
$ buildgear config --global parallel_builds 4
To list the available tool configuration settings simple do:
$ buildgear config --list
Note
Tool configuration settings can be applied to either a specific build space (local) or to all build spaces (global). Local settings always override global settings. If settings are not set they simply retain their default value.
Using build layers it is possible to override build files with build
files of a higher priority. This feature is quite useful in case some
existing build files do not suit your specific needs and you want to
override with your own variants. Layers are defined in the global
configuration file (config/config
) like so:
# Layer configuration LAYERS=(layer0 layer1 layer2 default)
Layers with higher priority overrides layers with lower prority. The
layers are prioritized according to the order in which they are
listed. Meaning, the first layer in the list (layer0
) is
the highest priority and the layer listed last (default
)
is the lowest priority. The default
layer name can be
omitted in which case it will automatically assume lowest priority.
Note
Please notice that the layer namedefault
is a reserved
keyword for the layer name that a buildfile belongs to by default if
no layer is defined for the buildfile.
Buildfiles are automatically assigned to <layer> when placed in:
buildfiles/cross/cross-<layer> buildfiles/native/native-<layer>
If <layer> is not defined in the LAYERS list then build files in
these locations will automatically be assigned to the
default
layer.
Alternatively, it is also possible to assign a single buildfile to a
specific layer by adding the layer
variable to the
buildfile, eg.:
layer=layer0
Note
Please notice that the preferred method for assigning layers is by Buildfile path location (eg. cross-<layer>) and not by using thelayer
variable.
By default the output object files of a build are automatically stripped (symbols discarded). The stripped files include executables, shared libraries (.so), and static archives (.a).
It is possible to disable this strip step entirely by simply adding
the nostrip
option to the special options array in the
buildfile like so:
options=(nostrip)
Alternatively, it is also possible to select which files will not be
automatically stripped by adding a nostrip
array to the
buildfile matching the files you don't want to be stripped. For
example:
nostrip=(/usr/lib/libtest.so /usr/lib/libtest.a)
Entries in the nostrip array are treated as regular expressions (as used by grep) making it possible to do file pattern matching.
Note
Please notice that the nostrip array will have no effect if the nostrip options is defined in the special options array.When building in parallel mode (parallel_builds > 1) it is possible to force a specific build to be built in non-parallel by adding a build lock. To do so simple add the following option to the special options field in the relevant buildfile like so:
options=(buildlock)