If you are a system administrator and responsible for managing the server, backing up data is one of your most important tasks. Therefore, you must know enough about backup tools to make your task easier.
Duplicity is a free, open source advanced command-line backup utility based on librsync and GnuPG. It generates digitally signed, version controlled, and encrypted tar volumes for storage on local or remote computers. Duplicity supports many protocols for connecting to file servers, including ssh/scp, rsync, ftp, DropBox, Amazon S3, Google Docs, Google Drive, local file system, OneDrive, WebDAV, etc.
In this tutorial, we will show you how to install and configure Duplicity to perform manual and automatic backups on an Ubuntu 20.04 server.
prerequisites
• Two servers running Ubuntu 20.04. • The static IP address 104.245.32.161 is configured on the Duplicity server, and 104.245.32.194 is configured on the backup server. • The root password is configured in the server.
getting Started
Before you start, you need to update the software package to the latest version. You can update all these files with the following command:
apt-get update -y
After updating all packages, you also need to install some necessary dependencies on the server. You can install all of them by running the following command:
apt-get install ncftp python3-paramiko python-pycryptopp lftp python3-boto python3-dev librsync-dev -y
Once all packages are installed, you can proceed to install Duplicity.
Installation repeatability
By default, Duplicity is available in the Ubuntu 20.04 default repository. You can install it by running the following command:
apt-get install duplicity -y
After the installation is complete, use the following command to verify the installed version of Duplicity:
duplicity --version
You should see the following output:
duplicity 0.8.12
Configure SSH password less authentication
Next, you will need to generate an SSH key with RSA 2048-bit encryption and use it to securely authenticate the remote system without providing a password.
First, use the following command to generate an RSA key:
ssh-keygen -t rsa -m PEM
You will be asked to specify the location of the key and password as shown below:
Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa Your public key has been saved in /root/.ssh/id_rsa.pub The key fingerprint is: SHA256:BwKSLtXT+WHhPqhu3MIH07QEb0c5Zh/f0XolMRyQaQE [email protected] The key's randomart image is: +---[RSA 3072]----+ | .o.. ..oE.o*++ | | o.+.o.O . + +.o| | o +.*o+ + . +.| |. . =+o.. . o .| | . =.oS . . | | o.o o | | o.+ | | .= o | | ..o | +----[SHA256]-----+
Next, use the following command to copy the generated RSA key to the remote backup server:
ssh-copy-id [email protected]
The system will ask you to provide the remote server password as shown below:
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub" /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys [email protected]'s password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh '[email protected]'" and check to make sure that only the key(s) you wanted were added.
Generate GPG key
Next, before transferring the data to the remote backup server, you also need to generate a GPG key to encrypt the data. GPG keys are used to provide an additional layer of security and encryption for data transmitted across servers.
You can use the GPG key on the Duplicity server with the following command:
gpg --gen-key
During the key generation process, the system will ask you to provide the password of the GPG key, as shown below:
gpg (GnuPG) 2.2.19; Copyright (C) 2019 Free Software Foundation, Inc. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. gpg: directory '/root/.gnupg' created gpg: keybox '/root/.gnupg/pubring.kbx' created Note: Use "gpg --full-generate-key" for a full featured key generation dialog. GnuPG needs to construct a user ID to identify your key. Real name: Hitesh Jethva Email address: [email protected] You selected this USER-ID: "Hitesh Jethva" Change (N)ame, (E)mail, or (O)kay/(Q)uit? O We need to generate a lot of random bytes. It is a good idea to perform some other action (type on the keyboard, move the mouse, utilize the disks) during the prime generation; this gives the random number generator a better chance to gain enough entropy. We need to generate a lot of random bytes. It is a good idea to perform some other action (type on the keyboard, move the mouse, utilize the disks) during the prime generation; this gives the random number generator a better chance to gain enough entropy. gpg: /root/.gnupg/trustdb.gpg: trustdb created gpg: key E8A334EB747BAD6A marked as ultimately trusted gpg: directory '/root/.gnupg/openpgp-revocs.d' created gpg: revocation certificate stored as '/root/.gnupg/openpgp-revocs.d/6683A9D4720C5895BF7A7AE9E8A334EB747BAD6A.rev' public and secret key created and signed. pub rsa3072 2020-07-30 [SC] [expires: 2022-07-30] 6683A9D4720C5895BF7A7AE9E8A334EB747BAD6A uid Hitesh Jethva sub rsa3072 2020-07-30 [E] [expires: 2022-07-30]
The above command will generate the GPG key and save it in the /root/.gnupg/ directory.
If you forget the GPG key, you can find it with the following command:
gpg --list-keys
You should get the following output:
gpg: checking the trustdb
gpg: marginals needed: 3 completes needed: 1 trust model: pgp
gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: next trustdb check due at 2022-07-30
/root/.gnupg/pubring.kbx
------------------------
pub rsa3072 2020-07-30 [SC] [expires: 2022-07-30]
6683A9D4720C5895BF7A7AE9E8A334EB747BAD6A
uid [ultimate] Hitesh Jethva
sub rsa3072 2020-07-30 [E] [expires: 2022-07-30]
Create a backup directory
Next, you will need to create a directory on the backup server to store the backups.
Log in to the backup server and use the following command to create a backup directory:
mkdir -p /remotebackup/Duplicity/
Once completed, you can proceed to the next step.
Perform a manual backup
At this point, Duplicity will be installed and configured. Now, let’s perform a manual backup on the Duplicity server.
Run the following command to back up the directory named /etc to the remote backup server (104.245.32.194) in the /remotebackup/Duplicity/ directory:
duplicity /etc/ sftp://[email protected]//remotebackup/Duplicity/
You need to provide the GPG key password to perform the backup operation, as shown below:
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: none
GnuPG passphrase for decryption:
Retype passphrase for decryption to confirm:
No signatures found, switching to full backup.
--------------[ Backup Statistics ]--------------
StartTime 1596110093.56 (Thu Jul 30 11:54:53 2020)
EndTime 1596110095.24 (Thu Jul 30 11:54:55 2020)
ElapsedTime 1.67 (1.67 seconds)
SourceFiles 1413
SourceFileSize 2698964 (2.57 MB)
NewFiles 1413
NewFileSize 2698964 (2.57 MB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 1413
RawDeltaSize 1857098 (1.77 MB)
TotalDestinationSizeChange 493579 (482 KB)
Errors 0
-------------------------------------------------
Now, log in to the remote backup server and verify the backup using the following command:
ls /remotebackup/Duplicity/
You should see the following output:
duplicity-full.20200730T115821Z.manifest.gpg duplicity-full-signatures.20200730T115821Z.sigtar.gpg
duplicity-full.20200730T115821Z.vol1.difftar.gpg
Use cron to automate backups
You can also schedule backups every day or every week by setting the cron job to run automatically at specified time intervals.
First, you need to create a passphrase file in the /root directory to automatically pass the passphrase without interactive prompting for the phrase.
You can create it with the following command:
nano /root/.passphrase
Add your GPG key password as shown below:
PASSPHRASE="admin"
Save and close the file, then use the following command to protect the file:
chmod 600 /root/.passphrase
Now, create a daily backup script in the /etc/cron.daily/ directory, which will run every day:
nano /etc/cron.daily/duplicity.inc
Add the following line:
#!/bin/sh test -x $(which duplicity) || exit 0 . /root/.passphrase export PASSPHRASE $(which duplicity) --encrypt-key 6683A9D4720C5895BF7A7AE9E8A334EB747BAD6A --exclude /var --exclude /home --exclude /swapfile --exclude /proc --exclude /sys --exclude /tmp --exclude /usr / sftp://[email protected]//remotebackup/Duplicity/
Save and close the file when you are done. Then, use the following command to grant executable permissions to the script:
chmod +x /etc/cron.daily/duplicity.inc
The above script will backup the entire root server, excluding some folders.
Now test the script to make sure everything is ok.
/etc/cron.daily/duplicity.inc
You should see the following output:
Local and Remote metadata are synchronized, no sync needed.
Last full backup left a partial set, restarting.
Last full backup date: Thu Jul 30 12:06:52 2020
RESTART: The first volume failed to upload before termination.
Restart is impossible...starting backup from beginning.
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: none
No signatures found, switching to full backup.
--------------[ Backup Statistics ]--------------
StartTime 1596110992.57 (Thu Jul 30 12:09:52 2020)
EndTime 1596111062.63 (Thu Jul 30 12:11:02 2020)
ElapsedTime 70.06 (1 minute 10.06 seconds)
SourceFiles 13313
SourceFileSize 777994890 (742 MB)
NewFiles 13313
NewFileSize 777994890 (742 MB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 13313
RawDeltaSize 772067520 (736 MB)
TotalDestinationSizeChange 318263295 (304 MB)
Errors 0
-------------------------------------------------
Now you can check the status of the backup with the following command:
PASSPHRASE="admin" duplicity --encrypt-key 6683A9D4720C5895BF7A7AE9E8A334EB747BAD6A collection-status sftp://[email protected]//remotebackup/Duplicity/
You should get the following output:
Last full backup date: Thu Jul 30 12:14:17 2020
Collection Status
-----------------
Connecting with backend: BackendWrapper
Archive dir: /root/.cache/duplicity/b2af509c9fab82d874aa492a933aaf70
Found 0 secondary backup chains.
Found primary backup chain with matching signature chain:
-------------------------
Chain start time: Thu Jul 30 12:14:17 2020
Chain end time: Thu Jul 30 12:14:17 2020
Number of contained backup sets: 1
Total number of contained volumes: 2
Type of backup set: Time: Num volumes:
Full Thu Jul 30 12:14:17 2020 2
-------------------------
No orphaned or incomplete backup sets found.
If you want to make a full backup, run the following command:
PASSPHRASE="admin" duplicity full --encrypt-key 6683A9D4720C5895BF7A7AE9E8A334EB747BAD6A remove-all-but-n-full 3 --force sftp://[email protected]//remotebackup/Duplicity/
You can also use the option remove-all-but-n-full to clear old backup files. For example, if you want to delete all unnecessary old backup files and keep only 2 backups and their corresponding incremental backups, run the following command:
PASSPHRASE="admin" duplicity --encrypt-key 6683A9D4720C5895BF7A7AE9E8A334EB747BAD6A remove-all-but-n-full 2 --force sftp://[email protected]//remotebackup/Duplicity/
Restore backup
You can also easily restore files and directories from a remote backup server. You can use the –file-to-restore option to restore files and directories.
Before performing the restore process, you can use the following command to list all available files and directories on the backup server:
duplicity --encrypt-key 6683A9D4720C5895BF7A7AE9E8A334EB747BAD6A list-current-files sftp://[email protected]//remotebackup/Duplicity/ | less
You should see the following output:
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Thu Jul 30 12:14:17 2020
Thu Jul 30 12:04:04 2020 .
Thu May 14 03:28:54 2020 bin
Thu May 14 03:35:45 2020 boot
Mon Apr 20 16:33:49 2020 boot/System.map-5.4.0-26-generic
Wed Apr 29 13:34:16 2020 boot/System.map-5.4.0-29-generic
Mon Apr 20 16:33:49 2020 boot/config-5.4.0-26-generic
Wed Apr 29 13:34:16 2020 boot/config-5.4.0-29-generic
Thu May 14 03:37:26 2020 boot/grub
Thu May 14 03:31:36 2020 boot/grub/fonts
Thu May 14 03:31:36 2020 boot/grub/fonts/unicode.pf2
Thu May 14 03:31:29 2020 boot/grub/gfxblacklist.txt
Thu May 14 03:37:26 2020 boot/grub/grub.cfg
Wed Jul 29 10:05:41 2020 boot/grub/grubenv
Thu May 14 03:31:37 2020 boot/grub/i386-pc
Thu May 14 03:31:36 2020 boot/grub/i386-pc/915resolution.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/acpi.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/adler32.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/affs.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/afs.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/ahci.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/all_video.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/aout.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/archelp.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/at_keyboard.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/ata.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/backtrace.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/bfs.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/biosdisk.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/bitmap.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/bitmap_scale.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/blocklist.mod
Thu May 14 03:31:37 2020 boot/grub/i386-pc/boot.img
Thu May 14 03:31:36 2020 boot/grub/i386-pc/boot.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/bsd.mod
Thu May 14 03:31:35 2020 boot/grub/i386-pc/bswap_test.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/btrfs.mod
Now, restore the directory named “opt” from the backup server to the /opt directory by running the following command:
PASSPHRASE="admin" duplicity --encrypt-key 6683A9D4720C5895BF7A7AE9E8A334EB747BAD6A --file-to-restore opt sftp://[email protected]//remotebackup/Duplicity/ /opt
Your restore process has completed successfully, and you should see the following output:
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Thu Jul 30 12:14:17 2020
in conclusion
In the above tutorial, we learned how to use Duplicity to backup and restore files and directories on an Ubuntu 20.04 server. We also learned how to automate the backup process by using Cron to schedule the backup process. Hope this article can help you save a lot of time.
