How to backup with Duplicity on Ubuntu 20.04

If you are a system administrator and responsible for managing the server, backing up data is one of your most important tasks. Therefore, you must know enough about backup tools to make your task easier.

Duplicity is a free, open source advanced command-line backup utility based on librsync and GnuPG. It generates digitally signed, version controlled, and encrypted tar volumes for storage on local or remote computers. Duplicity supports many protocols for connecting to file servers, including ssh/scp, rsync, ftp, DropBox, Amazon S3, Google Docs, Google Drive, local file system, OneDrive, WebDAV, etc.

In this tutorial, we will show you how to install and configure Duplicity to perform manual and automatic backups on an Ubuntu 20.04 server.

prerequisites

• Two servers running Ubuntu 20.04. • The static IP address 104.245.32.161 is configured on the Duplicity server, and 104.245.32.194 is configured on the backup server. • The root password is configured in the server.

getting Started

Before you start, you need to update the software package to the latest version. You can update all these files with the following command:

apt-get update -y

After updating all packages, you also need to install some necessary dependencies on the server. You can install all of them by running the following command:

apt-get install ncftp python3-paramiko python-pycryptopp lftp python3-boto python3-dev librsync-dev -y

Once all packages are installed, you can proceed to install Duplicity.

Installation repeatability

By default, Duplicity is available in the Ubuntu 20.04 default repository. You can install it by running the following command:

apt-get install duplicity -y

After the installation is complete, use the following command to verify the installed version of Duplicity:

duplicity --version

You should see the following output:

duplicity 0.8.12

Configure SSH password less authentication

Next, you will need to generate an SSH key with RSA 2048-bit encryption and use it to securely authenticate the remote system without providing a password.

First, use the following command to generate an RSA key:

ssh-keygen -t rsa -m PEM

You will be asked to specify the location of the key and password as shown below:

Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa
Your public key has been saved in /root/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:BwKSLtXT+WHhPqhu3MIH07QEb0c5Zh/f0XolMRyQaQE [email protected]
The key's randomart image is:
+---[RSA 3072]----+
|  .o.. ..oE.o*++ |
|  o.+.o.O . + +.o|
| o   +.*o+ + . +.|
|. .   =+o.. . o .|
| .   =.oS .    . |
|    o.o  o       |
|   o.+           |
|   .= o          |
|   ..o           |
+----[SHA256]-----+

Next, use the following command to copy the generated RSA key to the remote backup server:

ssh-copy-id [email protected]

The system will ask you to provide the remote server password as shown below:

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
[email protected]'s password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh '[email protected]'"
and check to make sure that only the key(s) you wanted were added.

Generate GPG key

Next, before transferring the data to the remote backup server, you also need to generate a GPG key to encrypt the data. GPG keys are used to provide an additional layer of security and encryption for data transmitted across servers.

You can use the GPG key on the Duplicity server with the following command:

gpg --gen-key

During the key generation process, the system will ask you to provide the password of the GPG key, as shown below:

gpg (GnuPG) 2.2.19; Copyright (C) 2019 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

gpg: directory '/root/.gnupg' created
gpg: keybox '/root/.gnupg/pubring.kbx' created
Note: Use "gpg --full-generate-key" for a full featured key generation dialog.

GnuPG needs to construct a user ID to identify your key.

Real name: Hitesh Jethva
Email address: [email protected]
You selected this USER-ID:
    "Hitesh Jethva "

Change (N)ame, (E)mail, or (O)kay/(Q)uit? O
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
gpg: /root/.gnupg/trustdb.gpg: trustdb created
gpg: key E8A334EB747BAD6A marked as ultimately trusted
gpg: directory '/root/.gnupg/openpgp-revocs.d' created
gpg: revocation certificate stored as '/root/.gnupg/openpgp-revocs.d/6683A9D4720C5895BF7A7AE9E8A334EB747BAD6A.rev'
public and secret key created and signed.

pub   rsa3072 2020-07-30 [SC] [expires: 2022-07-30]
      6683A9D4720C5895BF7A7AE9E8A334EB747BAD6A
uid                      Hitesh Jethva 
sub   rsa3072 2020-07-30 [E] [expires: 2022-07-30]

The above command will generate the GPG key and save it in the /root/.gnupg/ directory.

If you forget the GPG key, you can find it with the following command:

gpg --list-keys

You should get the following output:

gpg: checking the trustdb
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: next trustdb check due at 2022-07-30
/root/.gnupg/pubring.kbx
------------------------
pub   rsa3072 2020-07-30 [SC] [expires: 2022-07-30]
      6683A9D4720C5895BF7A7AE9E8A334EB747BAD6A
uid           [ultimate] Hitesh Jethva 
sub   rsa3072 2020-07-30 [E] [expires: 2022-07-30]

Create a backup directory

Next, you will need to create a directory on the backup server to store the backups.

mkdir -p /remotebackup/Duplicity/

Once completed, you can proceed to the next step.

Perform a manual backup

At this point, Duplicity will be installed and configured. Now, let’s perform a manual backup on the Duplicity server.

Run the following command to back up the directory named /etc to the remote backup server (104.245.32.194) in the /remotebackup/Duplicity/ directory:

duplicity /etc/ sftp://[email protected]//remotebackup/Duplicity/

You need to provide the GPG key password to perform the backup operation, as shown below:

Local and Remote metadata are synchronized, no sync needed.
Last full backup date: none
GnuPG passphrase for decryption: 
Retype passphrase for decryption to confirm: 
No signatures found, switching to full backup.
--------------[ Backup Statistics ]--------------
StartTime 1596110093.56 (Thu Jul 30 11:54:53 2020)
EndTime 1596110095.24 (Thu Jul 30 11:54:55 2020)
ElapsedTime 1.67 (1.67 seconds)
SourceFiles 1413
SourceFileSize 2698964 (2.57 MB)
NewFiles 1413
NewFileSize 2698964 (2.57 MB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 1413
RawDeltaSize 1857098 (1.77 MB)
TotalDestinationSizeChange 493579 (482 KB)
Errors 0
-------------------------------------------------

Now, log in to the remote backup server and verify the backup using the following command:

ls /remotebackup/Duplicity/

You should see the following output:

duplicity-full.20200730T115821Z.manifest.gpg      duplicity-full-signatures.20200730T115821Z.sigtar.gpg
duplicity-full.20200730T115821Z.vol1.difftar.gpg

Use cron to automate backups

You can also schedule backups every day or every week by setting the cron job to run automatically at specified time intervals.

First, you need to create a passphrase file in the /root directory to automatically pass the passphrase without interactive prompting for the phrase.

You can create it with the following command:

nano /root/.passphrase

Add your GPG key password as shown below:

PASSPHRASE="admin"

Save and close the file, then use the following command to protect the file:

chmod 600 /root/.passphrase

Now, create a daily backup script in the /etc/cron.daily/ directory, which will run every day:

nano /etc/cron.daily/duplicity.inc

Add the following line:

#!/bin/sh

test -x $(which duplicity) || exit 0
. /root/.passphrase

export PASSPHRASE
$(which duplicity) --encrypt-key 6683A9D4720C5895BF7A7AE9E8A334EB747BAD6A --exclude /var --exclude /home --exclude /swapfile --exclude /proc --exclude /sys --exclude /tmp --exclude /usr / sftp://[email protected]//remotebackup/Duplicity/

Save and close the file when you are done. Then, use the following command to grant executable permissions to the script:

chmod +x /etc/cron.daily/duplicity.inc

The above script will backup the entire root server, excluding some folders.

Now test the script to make sure everything is ok.

/etc/cron.daily/duplicity.inc

You should see the following output:

Local and Remote metadata are synchronized, no sync needed.
Last full backup left a partial set, restarting.
Last full backup date: Thu Jul 30 12:06:52 2020
RESTART: The first volume failed to upload before termination.
         Restart is impossible...starting backup from beginning.
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: none
No signatures found, switching to full backup.
--------------[ Backup Statistics ]--------------
StartTime 1596110992.57 (Thu Jul 30 12:09:52 2020)
EndTime 1596111062.63 (Thu Jul 30 12:11:02 2020)
ElapsedTime 70.06 (1 minute 10.06 seconds)
SourceFiles 13313
SourceFileSize 777994890 (742 MB)
NewFiles 13313
NewFileSize 777994890 (742 MB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 13313
RawDeltaSize 772067520 (736 MB)
TotalDestinationSizeChange 318263295 (304 MB)
Errors 0
-------------------------------------------------

Now you can check the status of the backup with the following command:

PASSPHRASE="admin" duplicity --encrypt-key 6683A9D4720C5895BF7A7AE9E8A334EB747BAD6A collection-status  sftp://[email protected]45.32.194//remotebackup/Duplicity/

You should get the following output:

Last full backup date: Thu Jul 30 12:14:17 2020
Collection Status
-----------------
Connecting with backend: BackendWrapper
Archive dir: /root/.cache/duplicity/b2af509c9fab82d874aa492a933aaf70

Found 0 secondary backup chains.

Found primary backup chain with matching signature chain:
-------------------------
Chain start time: Thu Jul 30 12:14:17 2020
Chain end time: Thu Jul 30 12:14:17 2020
Number of contained backup sets: 1
Total number of contained volumes: 2
 Type of backup set:                            Time:      Num volumes:
                Full         Thu Jul 30 12:14:17 2020                 2
-------------------------
No orphaned or incomplete backup sets found.

If you want to make a full backup, run the following command:

PASSPHRASE="admin" duplicity full  --encrypt-key 6683A9D4720C5895BF7A7AE9E8A334EB747BAD6A remove-all-but-n-full 3 --force sftp://[email protected]//remotebackup/Duplicity/

You can also use the option remove-all-but-n-full to clear old backup files. For example, if you want to delete all unnecessary old backup files and keep only 2 backups and their corresponding incremental backups, run the following command:

PASSPHRASE="admin" duplicity --encrypt-key 6683A9D4720C5895BF7A7AE9E8A334EB747BAD6A remove-all-but-n-full 2 --force  sftp://[email protected]//remotebackup/Duplicity/

Restore backup

You can also easily restore files and directories from a remote backup server. You can use the –file-to-restore option to restore files and directories.

Before performing the restore process, you can use the following command to list all available files and directories on the backup server:

duplicity --encrypt-key 6683A9D4720C5895BF7A7AE9E8A334EB747BAD6A list-current-files  sftp://[email protected]//remotebackup/Duplicity/ | less

You should see the following output:

Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Thu Jul 30 12:14:17 2020
Thu Jul 30 12:04:04 2020 .
Thu May 14 03:28:54 2020 bin
Thu May 14 03:35:45 2020 boot
Mon Apr 20 16:33:49 2020 boot/System.map-5.4.0-26-generic
Wed Apr 29 13:34:16 2020 boot/System.map-5.4.0-29-generic
Mon Apr 20 16:33:49 2020 boot/config-5.4.0-26-generic
Wed Apr 29 13:34:16 2020 boot/config-5.4.0-29-generic
Thu May 14 03:37:26 2020 boot/grub
Thu May 14 03:31:36 2020 boot/grub/fonts
Thu May 14 03:31:36 2020 boot/grub/fonts/unicode.pf2
Thu May 14 03:31:29 2020 boot/grub/gfxblacklist.txt
Thu May 14 03:37:26 2020 boot/grub/grub.cfg
Wed Jul 29 10:05:41 2020 boot/grub/grubenv
Thu May 14 03:31:37 2020 boot/grub/i386-pc
Thu May 14 03:31:36 2020 boot/grub/i386-pc/915resolution.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/acpi.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/adler32.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/affs.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/afs.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/ahci.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/all_video.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/aout.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/archelp.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/at_keyboard.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/ata.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/backtrace.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/bfs.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/biosdisk.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/bitmap.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/bitmap_scale.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/blocklist.mod
Thu May 14 03:31:37 2020 boot/grub/i386-pc/boot.img
Thu May 14 03:31:36 2020 boot/grub/i386-pc/boot.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/bsd.mod
Thu May 14 03:31:35 2020 boot/grub/i386-pc/bswap_test.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/btrfs.mod

Now, restore the directory named “opt” from the backup server to the /opt directory by running the following command:

PASSPHRASE="admin" duplicity --encrypt-key 6683A9D4720C5895BF7A7AE9E8A334EB747BAD6A --file-to-restore opt sftp://[email protected]//remotebackup/Duplicity/ /opt

Your restore process has completed successfully, and you should see the following output:

Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Thu Jul 30 12:14:17 2020

in conclusion

In the above tutorial, we learned how to use Duplicity to backup and restore files and directories on an Ubuntu 20.04 server. We also learned how to automate the backup process by using Cron to schedule the backup process. Hope this article can help you save a lot of time.