How to build Android disks on Ubuntu 16.04

Android is the most popular operating system in the world. Hundreds of different original equipment manufacturers or OEMs choose to install it on their devices because it is free, open source, and has a large ecosystem of applications and services built around it. Unfortunately, many manufacturers do not provide regular updates (OTAs) to Android. And other manufacturers only provide updates for a limited period of time after launch for the device. In addition, manufacturers tend to customize Android extensively to ensure that their devices have a unique look and feel. Their settings include alternative launchers, themed user interfaces, systems, and pre-installed applications.

If you want to remove all settings, or if you want to run the latest version of pure Android on your device, you can create a new firmware for it yourself. In the Android modding community, these are commonly referred to as ROMs, or Read Only Memory for short.

In this article, we will be building an Android Oreo ROM based on the Android Open Source Project, or AOSP for short. To keep this article device independent and generic, we’ll only target the AOSP emulator, but you can apply the same techniques to real devices.

Prerequisites

In order to be able to complete all the points, you need:

  • One Ubuntu 16.04 x64 server with at least 16GB of RAM, 4 CPUs and 120GB of disk space to install, following the Ubuntu 16.04 initial server setup guide including non-root sudo user and firewall. The compilation process requires a lot of RAM, and a large number of processors will speed up compilation times. Moreover, the files that you will be downloading are quite large.
  • Installed Git.

Step 1 – Starting a Screen Session

Some of the commands you run in this tutorial can potentially run for hours. If the SSH connection between the computer and the server is interrupted while the commands are running, they will be terminated abruptly. To avoid this situation, use the screen utility, which allows you to start multiple console sessions in the same terminal. With screen, you can disconnect from a running session and attach to it later.

Start a new session screen.

screen

When you launch the screen for the first time, you will be presented with a license agreement. Click on Enter to accept the license.

From now on, when you are unable to connect to SSH, your long running commands will continue to run in the background. After you re-establish your SSH connection, you can resume your session by running screen -r.

Next, let’s install the components, we need to compile Android.

Step 2 – Installing Dependencies

The AOSP source code is distributed through several different Git repositories. In order to make it more user-friendly, download all of these repositories, the AOSP community has created a command line tool called repo.

We will download the latest version of the program using wget and save it in the ~ / bin directory. First, you need to create a ~ / bin directory:

mkdir -p ~/bin

Then download the repo script:

wget 'https://storage.googleapis.com/git-repo-downloads/repo' -P ~/bin

Note: If you are worried about the safety of executing a script on your computer that you downloaded from another site, check the contents of the script:

less ~/bin/repo

Once you have checked the contents of the script, continue with this tutorial.

Use chmod to grant permission for the current user to run the repo.

chmod +x ~/bin/repo

The repo tool uses Git and requires you to create a Git configuration with your username and email address. Run the following commands to do this:

git config --global user.name "ваше_имя"

git config --global user.email "ваша_почта@ваш_домен.ru"

Android source code mainly consists of Java, C ++ and XML files. To compile the source code, you need to install OpenJDK 8, GNU C and the C ++ compiler, XML parsing library, ImageMagick, and several other related packages. Fortunately, you can install all of them with apt. Before doing this, make sure to update your server’s package list.

sudo apt-get update

After updating the lists, install the dependencies:

sudo apt-get install openjdk-8-jdk android-tools-adb bc bison build-essential curl flex g++-multilib gcc-multilib gnupg gperf imagemagick lib32ncurses5-dev lib32readline-dev lib32z1-dev libesd0-dev liblz4-tool libncurses5-dev libsdl1.2-dev libssl-dev libwxgtk3.0-dev libxml2 libxml2-utils lzop pngcrush rsync schedtool squashfs-tools xsltproc yasm zip zlib1g-dev

After downloading the dependencies, we can use the repo script to get the Android sources.

Step 3 – Downloading the source code

We will be using the repo script to do a few things, we need to prepare our workspace. Create a new directory to hold the Android sources you are about to download:

mkdir -p ~/aosp/oreo

You will be working in this directory for the rest of this tutorial, so head over to it now:

cd ~/aosp/oreo

The directory must be initialized with the AOSP repository manifest, this is a special Git repository containing an XML file named default.xml that defines the paths of all other Git repositories that together form the AOSP code.

Working with the complete AOSP tree code can be cumbersome. Therefore, you must additionally provide the name of the specific revision or industry in which you are interested. In this article, we are building an Oreo ROM, we will be using android-8.0.0_r33 whose build has the ID OPD1.170816.025. You can get a list of all available build IDs and branch names from the AOSP official page https://source.android.com/setup/build-numbers.

Plus, you won’t need the entire code history of the commit tree for this article. You can save time and space by storing a 1-depth piece of history.

Accordingly, use the repo init command to initialize the directory and specify the following parameters:

repo init -u https://android.googlesource.com/platform/manifest -b android-8.0.0_r33 --depth=1

When prompted to turn on the color display, press Y and then Enter

Finally, download the actual AOSP files from various repositories by running the repo sync command:

repo sync

The above command will download over 30GB of data. Once that happens, we’ll create a cache to speed up compilation.

Step 4 – Preparing the Compile Cache

To speed up assemblies, you can use the compiler cache. As the name suggests, the compiler cache avoids recompiling portions of ROMs that have already been fetched.

To enable the use of the compiler cache, let’s set an environment variable named USE_CCACHE.

export USE_CCACHE=1

If you have a lot of free disk space and don’t want the cache to grow too large, you can limit its size. if you are building a ROM for a single device, you can limit it to 15GB. To do this, use the ccache command.

prebuilts/misc/linux-x86/ccache/ccache -M 15G

You will see output that confirms you made this change:

Output

Set cache size limit to 15.0 Gbytes

There is one more optimization we need to do before we can build. Let’s do this in the next paragraph.

Step 5 – Configuring Jack

The Jack server, which is responsible for generating most Java from chunks of ROM, requires a lot of memory. To avoid memory allocation errors, you can use an environment variable named ANDROID_JACK_VM_ARGS to specify how much memory Jack is allowed to use. Usually, about 50% of your server’s RAM is allocated. This environment variable also determines other compilation options.

Run the following command to allocate 8GB of RAM to the Jack server and keep the default compilation options:

export ANDROID_JACK_VM_ARGS="-Xmx8g -Dfile.encoding=UTF-8 -XX:+TieredCompilation"

You are now ready to build your Android ROM.

Step 6 – Running Build

The AOSP code tree contains a script named envsetup.sh that has several assemblies associated with helper functions. While many of the helper functions such as mm, mma, and mmm serve as shortcuts to the make command, others such as the lunch set have important environment variables that, among other things, decide the processor architecture in the ROM, and the type of build.

The Source script will have access to the helper functions.

source build/envsetup.sh

Output

including device/asus/fugu/vendorsetup.sh
including device/generic/car/car-arm64/vendorsetup.sh
including device/generic/car/car-armv7-a-neon/vendorsetup.sh
including device/generic/car/car-x86_64/vendorsetup.sh
including device/generic/car/car-x86/vendorsetup.sh
including device/generic/mini-emulator-arm64/vendorsetup.sh
including device/generic/mini-emulator-armv7-a-neon/vendorsetup.sh
including device/generic/mini-emulator-mips64/vendorsetup.sh
including device/generic/mini-emulator-mips/vendorsetup.sh
including device/generic/mini-emulator-x86_64/vendorsetup.sh
including device/generic/mini-emulator-x86/vendorsetup.sh
including device/google/dragon/vendorsetup.sh
including device/google/marlin/vendorsetup.sh
including device/google/muskie/vendorsetup.sh
including device/google/taimen/vendorsetup.sh
including device/huawei/angler/vendorsetup.sh
including device/lge/bullhead/vendorsetup.sh
including device/linaro/hikey/vendorsetup.sh
including sdk/bash_completion/adb.bash

Then run lunch and pass your device’s codename to it, with a build type suffix, which can be either eng, userdebug, or user. While eng and userdebug build types result in ROMs that are best for testing purposes, a custom build type is recommended for production use.

To create a test disk that can run on the AOSP ARM emulator, pass aosp_arm-eng in the lunch command:

lunch aosp_arm-eng

You will see this output showing the environment settings:

Output

============================================
PLATFORM_VERSION_CODENAME=REL
PLATFORM_VERSION=8.0.0
TARGET_PRODUCT=aosp_arm
TARGET_BUILD_VARIANT=eng
TARGET_BUILD_TYPE=release
TARGET_PLATFORM_VERSION=OPD1
TARGET_BUILD_APPS=
TARGET_ARCH=arm
TARGET_ARCH_VARIANT=armv7-a
TARGET_CPU_VARIANT=generic
TARGET_2ND_ARCH=
TARGET_2ND_ARCH_VARIANT=
TARGET_2ND_CPU_VARIANT=
HOST_ARCH=x86_64
HOST_2ND_ARCH=x86
HOST_OS=linux
HOST_OS_EXTRA=Linux-4.4.0-104-generic-x86_64-with-Ubuntu-16.04-xenial
HOST_CROSS_OS=windows
HOST_CROSS_ARCH=x86
HOST_CROSS_2ND_ARCH=x86_64
HOST_BUILD_TYPE=release
BUILD_ID=OPD1.170816.025
OUT_DIR=out
AUX_OS_VARIANT_LIST=
============================================

Finally, run make to start building. make supports parallel jobs, so you can speed up the build significantly by using the -j option to set the number of parallel jobs equal to the number of processors available on the server.

Use the nproc command to see how many processors you have:

nproc

The command returns the number of processors:

Output

8

You can use this number with make to indicate parallel execution:

make -j8

Even with 8 processors, you will have to wait more than an hour for the build to complete, assuming there are no other CPU-intensive processes active on the server. The build time is directly proportional to the amount of RAM and the number of processors you have.

Note: You will see many warning messages generated during the build process. You can safely ignore them.

After the disk is ready, you should see a message that the creation completed successfully. You will also be able to see the exact duration of the build.

Output

...
Creating filesystem with parameters:
    Size: 2147483648
    Block size: 4096
    Blocks per group: 32768
    Inodes per group: 8192
    Inode size: 256
    Journal blocks: 8192
    Label: system
    Blocks: 524288
    Block groups: 16
    Reserved block group size: 127
Created filesystem with 2266/131072 inodes and 178244/524288 blocks
[100% 63193/63193] Install system fs i... out/target/product/generic/system.img
out/target/product/generic/system.img+ maxsize=2192446080 blocksize=2112 total=2147483648 reserve=22146432

#### make completed successfully (01:05:44 (hh:mm:ss)) ####

Let’s check that everything is built correctly.

Step 7 – Checking build

The build output is composed of multiple filesystem images, which together form a ROM. You will find them in the out / target / product / generic / directory.

ls -l out/target/product/generic/*.img

Output

-rw-r--r-- 1 sammy sammy   69206016 Jan  5 18:51 out/target/product/generic/cache.img
-rw-rw-r-- 1 sammy sammy    1699731 Jan  5 19:09 out/target/product/generic/ramdisk.img
-rw-r--r-- 1 sammy sammy 2147483648 Jan  5 19:10 out/target/product/generic/system.img
-rw-r--r-- 1 sammy sammy  576716800 Jan  5 19:09 out/target/product/generic/userdata.img

To check the ROM, you can try to boot the emulator with it by running the emulator command. If you are in an environment without a graphical user interface, make sure you pass the -no-window and -noaudio flags to it.

emulator -no-window -noaudio > /dev/null 2>&1 &

To test if the emulator can boot successfully, wait a minute and use the Android Debug Bridge tool, adb, to open a shell on the emulator.

adb shell

If the disk has no problems, you will see a prompt from the shell running on the emulator.

Output

* daemon not running; starting now at tcp:5037
* daemon started successfully
generic:/ #

Exit this shell by typing exit and pressing ENTER, or by pressing CTRL + D.

Note: If you try to open a shell before starting the emulator, you will see an error message informing that the emulator is missing. Wait a while and try again.

Fixing problems

If the build fails, the most likely cause is insufficient memory. To fix this, first kill the Jack server by running the following command:

jack-admin kill-server

Then start building again, but with fewer parallel jobs. For example, here’s how you can reduce the number of parallel jobs for 2:

make -j2

If the build fails due to insufficient disk space, you will probably try to build multiple times without cleaning up the results of previous builds. To reverse the results of previous builds, you can run the following command:

make clobber

Output

In this article, you have successfully built an AOSP based ROM for Android Oreo. The methods you learned today apply to all AOSPs too, such as Lineage OS and Resurrection Remix OS.

Related Posts