Buildbot is a Python based continuous system integration (CI) for automating software build, testing, and release process. It uses Python, the Twisted library to handle asynchronous communication between a buildmaster and one or more workers, to facilitate testing across multiple platforms. Buildbot is highly customizable and makes several assumptions about how the build process should work, making it suitable for complex processes or projects that require tools to grow with the unique needs of a build project.
In this tutorial, we will install and configure BuildBot master and worker on the same machine.
To complete this tutorial, you will need:
- Ubuntu server 16.04 with at least 1 GB of RAM configured with a non-root sudo user and a firewall as described in the Ubuntu 16.04 Initial Server Setup Guide.
With the server set up, you’re ready to follow along.
Step 1 – Installing BuildBot
The Buildbot project recommends using the Python Package Index, pip, and installing BuildBot in order to get the most recent version, which often has several releases upstream of the newer Ubuntu packages.
We’ll start as a sudo user and use apt-get update to provide the latest package listing:
sudo apt-get update
Then we’ll install pip:
sudo apt-get install python-pip
After pip is installed, we use it to install the BuildBot package, which includes master and worker, as well as other dependencies, including those that require a web interface. Pip creates .cache files in the home directory of the user who runs it. We’ll use sudo with the -H flag to put these files in the right place:
sudo -H pip install 'buildbot[bundle]'
Depending on the speed of your server, it may take a little time before the process ends. The final exit from a successful installation should look like this:
Successfully installed Automat-0.5.0 Jinja2-2.9.6 MarkupSafe-1.0 PyJWT-1.5.0 Tempita-0.5.2 Twisted-17.1.0 attrs-16.3.0 autobahn-17.5.1 buildbot-0.9.6 buildbot-console-view-0.9.6 buildbot-waterfall-view-0.9.6 buildbot-worker-0.9.6 buildbot-www-0.9.6 constantly-15.1.0 decorator-4.0.11 future-0.16.0 incremental-16.10.1 pbr-3.0.0 python-dateutil-2.6.0 six-1.10.0 sqlalchemy-1.1.9 sqlalchemy-migrate-0.11.0 sqlparse-0.2.3 txaio-2.7.1 zope.interface-4.4.0
It can also display a recommendation for pip upgrade:
. . . You are using pip version 8.1.1, however version 9.0.1 is available. You should consider upgrading via the 'pip install --upgrade pip' command.
While this won’t affect our BuildBot installation, we’ll take a moment to upgrade to the latest pip version:
sudo -H pip install --upgrade pip
Collecting pip Downloading pip-9.0.1-py2.py3-none-any.whl (1.3MB) 100% |--------------------------------| 1.3MB 768kB/s Installing collected packages: pip Found existing installation: pip 8.1.1 Not uninstalling pip at /usr/lib/python2.7/dist-packages, outside environment /usr Successfully installed pip-9.0.1
Finally, we’ll check the BuildBot installation:
Buildbot version: 0.9.6 Twisted version: 17.1.0
In this article, we configured the UFW firewall to only allow SSH traffic. We will check the status:
sudo ufw status
Status: active To Action From -- ------ ---- OpenSSH ALLOW Anywhere OpenSSH (v6) ALLOW Anywhere (v6)
Buildbot uses port 8010 for the web interface, which is not currently running, so we will open it now.
sudo ufw allow 8010
Then we’ll add a dedicated system user and group to run BuildBot:
sudo addgroup --system buildbot sudo adduser buildbot --system --ingroup buildbot --shell /bin/bash
Finally, we’ll log in as our new user to install BuildBot:
sudo --login --user buildbot
This will register us as the buildbot user and place us in the / home / buildbot directory, where we will set up our master and worker:
Step 2 – Setting up the Master
We’ll use BuildBot with the create-master command followed by the base directory value:
buildbot create-master master
mkdir /home/buildbot/master creating /home/buildbot/master/master.cfg.sample creating database (sqlite:///state.sqlite) buildmaster configured in /home/buildbot/master
Next, we’ll copy master.cfg.sample to master.cfg and leave the original where it was for reference:
cp ~/master/master.cfg.sample ~/master/master.cfg
We will then edit the file to allow us to access the web interface from the local computer.
In order to access the web interface from a computer or other device, we will change the buildbotURL from localhost to the IP address or domain name of the server. Other important configuration parameters are set in master.cfg.
At the bottom of the file, find the buildbotURL line and replace localhost with the IP address or domain name of your site:
c['buildbotURL'] = "https://<span class="highlight">IP_or_site_domain</span>:8010/"
Note: master.cfg also pre-defines the worker in the “Workers” section. ~ /master/master.cfg
. . . ####### WORKERS # The 'workers' list defines the set of recognized workers. Each element is # a Worker object, specifying a unique worker name and password. The same # worker name and password must be configured on the worker. c['workers'] = [worker.Worker("example-worker", "pass")] . . .
Later in the article, we will create a worker with these credentials.
When you’ve finished modifying ‘buildbotURL’, save and close the file, and then start the wizard:
buildbot start ~/master
Following twistd.log until startup finished.. The buildmaster appears to have (re)started correctly.
Finally, let’s visit the site in a web browser on port 8010 at the buildbot URL that we configured:
http: // IP_or_site_domain: 8010 /
Now that we have a worker master, we will create an example worker.
Step 3 – configuring the worker
The relationship between master and worker is established when the worker’s name and password in the master.cfg file match the worker name and password configured for use by master.
In this step, we will create and configure a worker to invoke the buildbot-worker’s create-worker command and consists of four settings:
- worker: this is the name of the directory where the settings for the worker will be stored
- localhost: this is the address where the worker master is running
- example-worker: This is the name of the worker and should uniquely identify the worker in the ~ / master / master.cfg file.
- pass: the worker’s password and this password must match the value of ~ master / master.cfg.
buildbot-worker create-worker worker localhost example-worker pass
mkdir /home/buildbot/worker mkdir /home/buildbot/worker/info Creating info/admin, you need to edit it appropriately. Creating info/host, you need to edit it appropriately. Not creating info/access_uri - add it if you wish Please edit the files in /home/buildbot/worker/info appropriately. worker configured in /home/buildbot/worker
When the worker first connects, it will send files to the info directory on the buildmaster where it is running. They will be displayed in the web interface to give developers more information on test failures.
We will set them up now. First, open the file containing the admin email, remove the example line: Your Name Here
~ / worker / info / admin
AndreyEx <[email protected]>
When you’re done, save and close the file.
The info / host file, by convention, provides the OS, version, memory size, processor speed, versions of the respective libraries installed, and finally the Buildbot version runs on worker.
Open the file and paste the information on the appropriate line, update the content as needed for your system:
Update the information you use to reflect the specifics of your system:
~ / worker / info / host
Ubuntu 16.04.2 2GB - Buildbot version: 0.9.6 - Twisted version: 17.1.0
When you’re done, save and exit. Finally, start worker:
buildbot-worker start worker
Following twistd.log until startup finished.. The buildbot-worker appears to have (re)started correctly.
Now, the master and worker are up and running. we will now perform a test build.
Step 4 – Running the Build Test
In order to run the test build, we will open the Builds menu in the web interface, then select Workers. We set the worker example and information to info / admin and info / host. From here, we can click on the default worker, “runtests”, to start the creation.
The “runtests” screen will have little information until the first build request is made. We’ll force it now by clicking the force button in the upper right corner of the screen:
This will bring up a dialog box that allows you to enter information about the forced build.
For this build test, we will leave the fields blank and click the “Start Build” button in the pop-up window. Please note that if you enter a value in the “Your name” field, then it must contain a valid email address.
After a few seconds, the build should complete successfully:
You can explore the details of each step in the assembly by clicking on the number or arrow next to its name:
You may have noticed that we didn’t have to log in to complete this build. By default, anyone can access the administrative functions, so before we finish, we take a moment to log out and create a user account. You can read more about the available module options at https://docs.buildbot.net/current/developer/authz.html.
Open the master.cfg file again:
At the bottom of the file, add the following lines, changing your username and password.
File: ~ / master / master.cfg
. . . c['www']['authz'] = util.Authz( allowRules = [ util.AnyEndpointMatcher(role="admins") ], roleMatchers = [ util.RolesFromUsername(roles=['admins'], usernames=['AndreyEx']) ] ) c['www']['auth'] = util.UserPasswordAuth([('AndreyEx','Password')])
buildbot restart ~/master
When we reload the web interface, the link should be displayed in the upper right corner, where it indicates that anonymous access to administrative functions is no longer available.
We will verify the credentials, we just added by clicking the “Anonymous” button, after which the login window will open, where you can enter the username and password that we have configured. When we logged in, we should see that “Anonymous” no longer has access, to start building, our user “AndreyEx” can.
At this point, our BuildBot installation is complete, and we have taken a minimal step to customize the interface. The username and password, however, are transmitted in plain text. We recommend that as a next step and before using BuildBot in earnest, you set up your web interface with a reverse proxy server.
In this article, we have installed and configured master BuildBot and worker BuildBot on the same machine.