Installing and configuring rclone. Backing up to s3 or cloud.mail.ru

Rclone is a cloud storage file sync software.

Due to the fact that Yandex is tightening the screws in its services, due to which there are interruptions with Yandex Disk, more precisely with the protocol webdav, which I used to store a backup of the site, I had to look for other options. The choice fell on the program rclone and storage from mail.ru.

Installing rclone

Downloading rclone and unpack it

$ curl -O https://downloads.rclone.org/rclone-current-linux-amd64.zip
$ unzip rclone-current-linux-amd64.zip
$ cd rclone-v1.50.2-linux-amd64/

Change the location of the executable file and configure the rights

$ sudo cp rclone /usr/sbin/
$ sudo chown root:root /usr/sbin/rclone
$ sudo chmod 755 /usr/sbin/rclone

Installing a manual for rclone

$ sudo mkdir -p /usr/local/share/man/man1
$ sudo cp rclone.1 /usr/local/share/man/man1/
$ sudo mandb

This completes the installation.

Configuring rclone for s3 from mail.ru

First you need to register on the mcs.mail.ru portal. Create an account in object storage, where it will be generated Access Key ID and Secret key… This data must be saved and then created Bucket – a logical entity that helps to organize the storage of objects (i.e., roughly speaking, the directory where our files will be located).

Run the command to create a configuration for storage

$ rclone config
n/s/q> n

# Укажите имя подключения к удаленному хранилищу
name> mailru-s3
...
4 / Amazon S3 Compliant Storage Provider
...

# Выберите тип хранилища, указываем s3:
Storage> 4
...
10 / Any other S3 compatible provider
...

# Выберите поставщика услуг, указываем Other:
provider> 10

# Выберите тип ввода учетных данных (вручную или из переменных окружения)
# Для ввода данных вручную укажите false:
env_auth> 1

# Вводим Access Key ID
access_key_id> ...

# Вводим Secret Key
secret_access_key> ...

# Вводим регион, я указал 1
region> 1

# Вводим Endpoint Можно узнать на портале: mcs.mail.ru - объектное хранилище
endpoint> hb.bizmrg.com

# Укажите регион создания бакета
# оставьте поле пустым и нажмите Enter:
location_constraint>

# Установите права доступа ACL, private:
acl> 1

# После этого можно отказаться от расширенной конфигурации:
Edit advanced config? (y/n)
y) Yes
n) No
y/n> n

Next, the configuration that we created will be displayed, save it and exit

y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q

Storage Commands

The list of commands and supported flags can be found on the official website of the application rcolne

View the list of buckets:

$ rclone lsd mailru-s3:

Creating a new bucket

$ rclone mkdir mailru-s3:itdraft-testbucket

Viewing a list of files in a bucket

$ rclone ls mailru-s3:itdraft-bucket

Copying Files from Local Machine to Storage

$ rclone copy /mnt/storage/itdraft.ru/backup mailru-s3:itdraft-bucket

Synchronizing files on the local machine and in storage

$ rclone sync /mnt/storage/itdraft.ru/backup mailru-s3:itdraft-bucket

Copying files from storage to local machine

$ rclone copy mailru-s3:itdraft-bucket/backup_itdraft_ru.sql.gz /home/test

Don’t sync files less than 1 day old

$ rclone --min-age 1d --delete-excluded sync /mnt/storage/itdraft.ru/backup mailru-s3:itdraft-bucket

Don’t sync files older than 7 days

$ rclone --max-age 7d --delete-excluded sync /mnt/storage/itdraft.ru/backup mailru-s3:itdraft-bucket

Delete files less than 7 days old

$ rclone --max-age 7d delete mailru-s3:itdraft-bucket

Delete files older than 7 days

$ rclone --min-age 7d delete mailru-s3:itdraft-bucket

Configuring rclone to work with cloud.mail.ru

Run the command to create a configuration for storage

$ rclone config
e/n/d/r/c/s/q> n
name> cloud-mailru
Storage> mailru
user> [email protected]
Password
y) Yes type in my own password
g) Generate random password
y/g> y
Enter the password:
password:
Confirm the password:
password:
speedup_enable> 1
Edit advanced config? (y/n)
y) Yes
n) No
y/n> n

Next, the configuration that we created will be displayed, save it and exit

y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:

Name                 Type
====                 ====
cloud-mailru         mailru
mailru-s3            s3

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q

Example script for backing up a MySQL database dump to the repository

The logic of the backup script is as follows:

  • backups of the database are stored locally for 8 days (since they are small)
  • backups of the database are stored in the cloud for 1 month
  • information about the number of local backups and which ones were deleted is written to the log

The script is executed via crontab daily

#!/bin/sh

# Переменные
SRC=/mnt/storage/itdraft.ru/backup
DEST=cloud-mailru:/itdraft-backup

# Дата-Врем
DATETIME="$(date +%Y%m%d_%H%M%S)"

# Ящики
[email protected]
[email protected]
[email protected]

# БД
mysqluser=%user%
mysqlpassword=%pass%
mysqlbase=%database%

# Логи
LOG_FILE=$SRC/backup_mysql.log

# Хранение
DAYS=8
DAYSDEST=1M

# Создаем каталог, если его еще нет
mkdir -p $SRC

# Время запуска скрипта в логах
echo "$DATETIME INFO: Start" | tee "$LOG_FILE"

# Создаем дамп базы локально и архивируем
mysqldump -u $mysqluser -p$mysqlpassword $mysqlbase | gzip > $SRC/$DATETIME.backup_itdraft_ru.sql.gz

# Копируем в облако
rclone copy $SRC $DEST --log-file "$LOG_FILE"

# Удаляем в облаке файлы старше 1 месяца
rclone --min-age $DAYSDEST delete $DEST --log-file "$LOG_FILE"

# Информация для лог-файла: Find local MYSQL dump
echo "Local MYSQL dump" | tee -a "$LOG_FILE"
du -csh --time $SRC/*.sql.gz | tee -a "$LOG_FILE"
echo "Deleted" | tee -a "$LOG_FILE"
find $SRC/*.sql.gz -type f -mtime +$DAYS -print | tee -a "$LOG_FILE"

# Удаляем локальные копии, старше 8 дней
find $SRC/*.sql.gz -type f -mtime +$DAYS -exec rm -f {} ;

# Время завершения скрипта в логах
echo "$DATETIME INFO: End" | tee -a "$LOG_FILE"

# Отправляем лог на почту
echo "itdraft.ru: $DATETIME | deleted files" | mail -v -A yandex -s "itdraft.ru: $DATETIME | deleted files" -a "$LOG_FILE" -r $mail_from $mail_to

To back up the site itself, I run a similar script once a week, but due to a different launch frequency:

  • 1 full backup of site files is stored locally (since this backup weighs several orders of magnitude more than the base)
  • the cloud stores backups for a month (i.e. 4 pcs.)

UPD 13.12.2019

If you configured rclone from one user, and the backup script is run as the root user, you need to copy the rclone configuration file

$ sudo mkdir -p /root/.config/rclone/
$ sudo cp /home/%user%/.config/rclone/rclone.conf /root/.config/rclone/rclone.conf
Sidebar