How to synchronize local and remote directories in Linux

This tutorial describes how to sync local and remote directories on Linux using Lsyncd (Live Syncing Daemon). Using Lsyncd, we can mirror local directories and remote directories easily. This will be very useful if you want to sync data between catalogs that are frequently updated with new content. Lsyncd is designed to synchronize a local directory tree with a low profile of expected changes to a remote mirror. This is especially useful for syncing data from a protected area to a less secure location.

Synchronizing local and remote directories in Linux using Lsyncd

Installing Lsyncd

On Debian and derivatives like Ubuntu, Linux Mint, run the following command to install Lsyncd:

sudo apt-get install lsyncd

On RHEL, CentOS, you need to enable the EPEL repository first.

sudo yum install epel-release

Then install Lsyncd using the command:

sudo yum install lsyncd

Lsyncd configuration on Debian and Ubuntu and derivatives

Lsyncd does not provide a default configuration file on DEB based systems. We need to create it manually. However, Lsyncd includes some sample configuration files that are useful to get a general idea of ​​how to do synchronization. Configuration example files will be found under the directory “/ Usr / share / doc / lsyncd – * / examples”

ls /usr/share/doc/lsyncd/examples/

Output example:

lbash.lua lgforce.lua lpostcmd.lua lrsyncssh.lua
lecho.lua limagemagic.lua lrsync.lua

As you can see in the above output, each config file is with a .lua extension. This is because the lsync configuration file is written using the Lua programming language. Let’s take a look at a sample configuration file.

cat /usr/share/doc/lsyncd/examples/lrsync.lua

The Lrsync configuration file looks like this:

----
-- User configuration file for lsyncd.
--
-- Simple example for default rsync.
--
settings = {
 statusFile = "/tmp/lsyncd.stat",
 statusInterval = 1,
}

sync{
 default.rsync,
 source="src",
 target="trg",
}

Lsyncd configuration on RHEL and CentOS

On RHEL and CentOS systems, the Lsyncd configuration file will be created automatically. You can view the contents of this file with the command:

cat /etc/lsyncd.conf

And, sample configuration files are available in the following locations.

ls /usr/share/doc/lsyncd-2.1.5/examples/

Synchronizing local directories on Debian, Ubuntu

We will now sync the two local directories.

Let’s create two directories with some content, for example:

sudo mkdir source_dir
sudo mkdir dest_dir

Let’s create random files in the source_dir directory:

sudo touch source_dir/file{1..10}

Optionally, you can create multiple log files for Lysyncd. This is only optional. If you don’t want the log files, skip this step.

sudo mkdir /var/log/lsyncd
sudo touch /var/log/lsyncd/lsyncd.{log,status}

Then create a config file for Lsyncd:

sudo mkdir /etc/lsyncd
sudo nano /etc/lsyncd/lsyncd.conf.lua

Edit / change the following lines:

settings {
        logfile = "/var/log/lsyncd/lsyncd.log",
        statusFile = "/var/log/lsyncd/lsyncd.status"
}

sync {
        default.rsync,
        source = "/home/sk/source_dir",
        target = "/home/sk/dest_dir",
}

Replace the source and destination directories in the path with your own values. Save and close the file.

Enable, start or restart the Lsyncd service using the commands:

sudo systemctl enable lsyncd
sudo systemctl restart lsyncd

Now, check the contents of the source_dir and dest_dir directories.

ls source_dir/

Output example:

file1 file10 file2 file3 file4 file5 file6 file7 file8 file9

Check the contents of the dest_dir directory:

ls dest_dir/

Output example:

file1 file10 file2 file3 file4 file5 file6 file7 file8 file9

Remember that we only created files in the source_dir directory and the contents of this directory were automatically synchronized with the dest_dir directory.

In addition, you can view the log files to check if replication has completed or not.

tail -10 /var/log/lsyncd/lsyncd.log

Output example:

Sat Apr 15 13:14:12 2017 Normal: recursive startup rsync: /home/sk/source_dir/ -> /home/sk/dest_dir/
Sat Apr 15 13:14:12 2017 Normal: Startup of "/home/sk/source_dir/" finished.

Check the Lsyncd status file:

more /var/log/lsyncd/lsyncd.status

Output example:

Lsyncd status report at Sat Apr 15 13:14:12 2017

Sync1 source=/home/sk/source_dir/
There are 0 delays
Excluding:
 nothing.


Inotify watching 1 directories
 1: /home/sk/source_dir/

Synchronization of local directories on RHEL, CentOS

Editing the Lsyncd config file:

sudo nano /etc/lsyncd.conf

Edit and change the following lines:

----
-- User configuration file for lsyncd.
--
-- Simple example for default rsync, but executing moves through on the target.
--
-- For more examples, see /usr/share/doc/lsyncd*/examples/
-- 
settings {
 logfile = "/var/log/lsyncd.log",
 statusFile = "/var/log/lsyncd.status"
}

sync {
 default.rsync,
 source = "/root/source_dir",
 target = "/root/dest_dir",

Replace the source and target directory paths with your own values. Save and close the file.

Enable and start the Lsyncd service using the commands:

sudo systemctl enable lsyncd
sudo systemctl start lsyncd

Now, check the contents of the source_dir and dest_dir directories.

Ls source_dir /

Output example:

file10 file1 file2 file3 file4 file5 file6 file7 file8 file9

Check the contents of the dest_dir directory:

Ls dest_dir /

Output example:

file10 file1 file2 file3 file4 file5 file6 file7 file8 file9

Fine! The content of source_dir was successfully synchronized with the dest_dir directory.

In addition, you can view the log files to check if replication has completed or not.

tail -10 /var/log/lsyncd.log

Check the Lsyncd status file:

more /var/log/lsyncd.status

Synchronize remote directories on Debian, Ubuntu systems

We need to set up a login and password for SSH. This is because Lsyncd automatically copies the contents of the local directory to the remote directory without user intervention.

sudo su
ssh-keygen -t rsa

Do not enter a password. Just press ENTER to accept the defaults.

Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/home/sk/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:dHc2aKk5F2wBq7CIc5sVHJwYNjoPdwgxDKb1KuBBkgs [email protected]
The key's randomart image is:
+---[RSA 2048]----+
|o*=.++.. ... |
|E..*.+o. o + |
|+o+ + = . o O + |
|+ .B o = o * + . |
|..+ + o S + . |
| . o + o |
| o |
| |
| |
+----[SHA256]-----+

Now, copy the SSH public key to your remote system:

ssh-copy-id [email protected]

Output example:

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host '192.168.10.150 (192.168.10.150)' can't be established.
ECDSA key fingerprint is SHA256:U7I0O1OOzzbHFlhIG0HoGDr1usHzLBju6Jmr6bUB9Es.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
[email protected]'s password: 

Number of key(s) added: 1

Now try logging into the machine, with: "ssh '[email protected]'"
and check to make sure that only the key(s) you wanted were added.

Here 192.168.10.150 is the IP address of the remote system.

Now you will be able to SSH into your remote system without entering a password:

ssh [email protected]

Create the target directory on the remote system using the command:

mkdir remotesync

Log out of the remote system:

exit

Then edit the Lsyncd config file on the source machine:

sudo nano /etc/lsyncd/lsyncd.conf.lua

Edit / change the following lines:

settings {
 logfile = "/var/log/lsyncd/lsyncd.log",
 statusFile = "/var/log/lsyncd/lsyncd.status"
}

sync {
 default.rsync,
 source = "/home/sk/source_dir",
 host = "192.168.10.150",
 targetdir = "/root/remotesync",
}

Here 192.168.10.150 is the IP address of my remote host. And, the destination directory is / root / remotesync / on my remote system. Replace the values ​​with your own. Save and close the file.

Restart the Lsyncd service to begin replicating the contents of the source directory to the destination.

sudo systemctl restart lsyncd

Now, SSH into your remote system:

SSH [email protected]

And, check the contents of the target directory (i.e., in our case, / root / remotesync /). You will see all files from the source directory from the local system have been played to the target directory on the target system.

[email protected] ~]# ls remotesync/
file1 file10 file2 file3 file4 file5 file6 file7 file8 file9

This procedure is the same for RHEL and CentOS systems. Set up passwordless SSH login and make sure you change the correct source and destination paths in the main Lsyncd config file (/etc/lsyncd.conf) on RHEL or CentOS system, and the rest of the steps are the same.

Now you know how to sync local and remote directories on Linux using Lsyncd. As you can see, this is a very easy and straightforward method.

Resources:

Sidebar