How to install BuildBot on Ubuntu 16.04

Introduction

Buildbot is a Python based continuous system integration (CI) for automating software build, testing, and release process. It uses Python, the Twisted library to handle asynchronous communication between a buildmaster and one or more workers, to facilitate testing across multiple platforms. Buildbot is highly customizable and makes several assumptions about how the build process should work, making it suitable for complex processes or projects that require tools to grow with the unique needs of a build project.

In this tutorial, we will install and configure BuildBot master and worker on the same machine.

Prerequisites

To complete this tutorial, you will need:

  • Ubuntu server 16.04 with at least 1 GB of RAMconfigured with a non-root sudo user and a firewall as described in the Ubuntu 16.04 Initial Server Setup Guide.

With the server set up, you’re ready to follow along.

Step 1 – Installing BuildBot

The Buildbot project recommends using the Python Package Index, pip, and installing BuildBot in order to get the most recent version, which often has several releases upstream of the newer Ubuntu packages.

We’ll start as a sudo user and use apt-get update to provide the latest package listing:

sudo apt-get update

Then we’ll install pip:

sudo apt-get install python-pip

After pip is installed, we use it to install the BuildBot package, which includes master and worker, as well as other dependencies, including those that require a web interface. Pip creates .cache files in the home directory of the user who runs it. We’ll use sudo with the -H flag to put these files in the right place:

sudo -H pip install 'buildbot[bundle]'

Depending on the speed of your server, it may take a little time before the process ends. The final exit from a successful installation should look like this:

Output

Successfully installed Automat-0.5.0 Jinja2-2.9.6 MarkupSafe-1.0 
PyJWT-1.5.0 Tempita-0.5.2 Twisted-17.1.0 attrs-16.3.0 autobahn-17.5.1 
buildbot-0.9.6 buildbot-console-view-0.9.6 buildbot-waterfall-view-0.9.6 
buildbot-worker-0.9.6 buildbot-www-0.9.6 constantly-15.1.0 
decorator-4.0.11 future-0.16.0 incremental-16.10.1 pbr-3.0.0 
python-dateutil-2.6.0 six-1.10.0 sqlalchemy-1.1.9 
sqlalchemy-migrate-0.11.0 sqlparse-0.2.3 txaio-2.7.1 zope.interface-4.4.0

It can also display a recommendation for pip upgrade:

Output

. . .
You are using pip version 8.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

While this won’t affect our BuildBot installation, we’ll take a moment to upgrade to the latest pip version:

sudo -H pip install --upgrade pip

Output

Collecting pip
 Downloading pip-9.0.1-py2.py3-none-any.whl (1.3MB)
   100% |--------------------------------| 1.3MB 768kB/s
Installing collected packages: pip
 Found existing installation: pip 8.1.1
   Not uninstalling pip at /usr/lib/python2.7/dist-packages, outside environment /usr
Successfully installed pip-9.0.1

Finally, we’ll check the BuildBot installation:

buildbot --version

Output

Buildbot version: 0.9.6
Twisted version: 17.1.0

In this article, we configured the UFW firewall to only allow SSH traffic. We will check the status:

sudo ufw status

Output

Status: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere
OpenSSH (v6)               ALLOW       Anywhere (v6)

Buildbot uses port 8010 for the web interface, which is not currently running, so we will open it now.

sudo ufw allow 8010

Then we’ll add a dedicated system user and group to run BuildBot:

sudo addgroup --system buildbot
sudo adduser buildbot --system --ingroup buildbot --shell /bin/bash

Finally, we’ll log in as our new user to install BuildBot:

sudo --login --user buildbot

This will register us as the buildbot user and place us in the / home / buildbot directory, where we will set up our master and worker:

Step 2 – Setting up the Master

We’ll use BuildBot with the create-master command followed by the base directory value:

buildbot create-master master

Output

mkdir /home/buildbot/master
creating /home/buildbot/master/master.cfg.sample
creating database (sqlite:///state.sqlite)
buildmaster configured in /home/buildbot/master

Next, we’ll copy master.cfg.sample to master.cfg and leave the original where it was for reference:

cp ~/master/master.cfg.sample ~/master/master.cfg

We will then edit the file to allow us to access the web interface from the local computer.

nano ~/master/master.cfg

In order to access the web interface from a computer or other device, we will change the buildbotURL from localhost to the IP address or domain name of the server. Other important configuration parameters are set in master.cfg.

At the bottom of the file, find the buildbotURL line and replace localhost with the IP address or domain name of your site:

~ /master/master.cfg

c['buildbotURL'] = "http://<span class="highlight">IP_or_site_domain</span>:8010/"

Note: master.cfg also pre-defines the worker in the “Workers” section. ~ /master/master.cfg

. . .
####### WORKERS

# The 'workers' list defines the set of recognized workers. Each element is
# a Worker object, specifying a unique worker name and password.  The same
# worker name and password must be configured on the worker.
c['workers'] = [worker.Worker("example-worker", "pass")]
. . .

Later in the article, we will create a worker with these credentials.

When you’ve finished modifying ‘buildbotURL’, save and close the file, and then start the wizard:

buildbot start ~/master

Output

Following twistd.log until startup finished..
The buildmaster appears to have (re)started correctly.

Finally, let’s visit the site in a web browser on port 8010 at the buildbot URL that we configured:

http: // IP_or_site_domain: 8010 /

Now that we have a worker master, we will create an example worker.

Step 3 – configuring the worker

The relationship between master and worker is established when the worker’s name and password in the master.cfg file match the worker name and password configured for use by master.

In this step, we will create and configure a worker to invoke the buildbot-worker’s create-worker command and consists of four settings:

  • worker: this is the name of the directory where the settings for the worker will be stored
  • localhost: this is the address where the worker master is running
  • example-worker: This is the name of the worker and should uniquely identify the worker in the ~ / master / master.cfg file.
  • pass: the worker’s password and this password must match the value of ~ master / master.cfg.
buildbot-worker create-worker worker localhost example-worker pass

Output

mkdir /home/buildbot/worker
mkdir /home/buildbot/worker/info
Creating info/admin, you need to edit it appropriately.
Creating info/host, you need to edit it appropriately.
Not creating info/access_uri - add it if you wish
Please edit the files in /home/buildbot/worker/info appropriately.
worker configured in /home/buildbot/worker

When the worker first connects, it will send files to the info directory on the buildmaster where it is running. They will be displayed in the web interface to give developers more information on test failures.

We will set them up now. First, open the file containing the admin email, remove the example line: Your Name Here and replace it with your name and email address.

nano ~/worker/info/admin

~ / worker / info / admin

AndreyEx <[email protected]>

When you’re done, save and close the file.

The info / host file, by convention, provides the OS, version, memory size, processor speed, versions of the respective libraries installed, and finally the Buildbot version runs on worker.

Open the file and paste the information on the appropriate line, update the content as needed for your system:

nano ~/worker/info/host

Update the information you use to reflect the specifics of your system:

~ / worker / info / host

Ubuntu 16.04.2 2GB - Buildbot version: 0.9.6 - Twisted version: 17.1.0

When you’re done, save and exit. Finally, start worker:

buildbot-worker start worker

Output

Following twistd.log until startup finished..
The buildbot-worker appears to have (re)started correctly.

Now, the master and worker are up and running. we will now perform a test build.

Step 4 – Running the Build Test

In order to run the test build, we will open the Builds menu in the web interface, then select Workers. We set the worker example and information to info / admin and info / host. From here, we can click on the default worker, “runtests”, to start the creation.

Screenshot of the workspace screen in BuildBot

The “runtests” screen will have little information until the first build request is made. We’ll force it now by clicking the force button in the upper right corner of the screen:

Screenshot shows the power button

This will bring up a dialog box that allows you to enter information about the forced build.

Screenshot of Build force popup in BuildBot

For this build test, we will leave the fields blank and click the “Start Build” button in the pop-up window. Please note that if you enter a value in the “Your name” field, then it must contain a valid email address.

After a few seconds, the build should complete successfully:

Screenshot shows a successful build

You can explore the details of each step in the assembly by clicking on the number or arrow next to its name:

enter image description here

You may have noticed that we didn’t have to log in to complete this build. By default, anyone can access the administrative functions, so before we finish, we take a moment to log out and create a user account. You can read more about the available module options at http://docs.buildbot.net/current/developer/authz.html.

Open the master.cfg file again:

nano ~/master/master.cfg

At the bottom of the file, add the following lines, changing your username and password.

File: ~ / master / master.cfg

. . .
c['www']['authz'] = util.Authz(
       allowRules = [
           util.AnyEndpointMatcher(role="admins")
       ],
       roleMatchers = [
           util.RolesFromUsername(roles=['admins'], usernames=['AndreyEx'])
       ]
)
c['www']['auth'] = util.UserPasswordAuth([('AndreyEx','Password')])

buildbot restart ~/master

When we reload the web interface, the link should be displayed in the upper right corner, where it indicates that anonymous access to administrative functions is no longer available.

enter image description here

We will verify the credentials, we just added by clicking the “Anonymous” button, after which the login window will open, where you can enter the username and password that we have configured. When we logged in, we should see that “Anonymous” no longer has access, to start building, our user “AndreyEx” can.

enter image description here

At this point, our BuildBot installation is complete, and we have taken a minimal step to customize the interface. The username and password, however, are transmitted in plain text. We recommend that as a next step and before using BuildBot in earnest, you set up your web interface with a reverse proxy server.

Output

In this article, we have installed and configured master BuildBot and worker BuildBot on the same machine.

Sidebar