Compress and deduplicate storage using Virtual Data Optimizer (VDO)
In this guide, you will learn how to use the Virtual Data Optimizer (VDO) to compress and deduplicate data on storage devices to ensure optimal storage space. RHEL / CentOS and Fedora Linux distributions have an installable Virtual Data Optimizer (VDO) Linux device mapper.
VDO optimizes the data footprint on block devices by reducing disk space usage on block devices and minimizing data replication, saving disk space and even increasing data throughput. VDO contains two kernel modules:
- v Module-transparently control data compression,
- Uz Module-handles deduplication.
The VDO layer is placed on top of existing block storage devices (such as local disks, RAID devices, encryption devices). Storage layers such as file systems and LVM logical volumes are then placed on top of VDO devices.
Step 1: Install the Virtual Data Optimizer (VDO)
For RHEL and CentOS Linux distributions, you can easily install the Virtual Data Optimizer (VDO) Linux Device Mapper by running the following command.
sudo yum -y install vdo kmod-kvdoWait for the installation to complete.
Loaded plugins: fastestmirror
Determining fastest mirrors
* base: centos.mirror.liquidtelecom.com
* extras: centos.mirror.liquidtelecom.com
* updates: centos.mirror.liquidtelecom.com
base | 3.6 kB 00:00
extras | 2.9 kB 00:00
updates | 2.9 kB 00:00
(1/2): extras/7/x86_64/primary_db | 159 kB 00:06
(2/2): updates/7/x86_64/primary_db | 6.7 MB 00:40
Resolving Dependencies
--> Running transaction check
---> Package kmod-kvdo.x86_64 0:6.1.2.41-5.el7 will be installed
---> Package vdo.x86_64 0:6.1.2.41-4.el7 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
===========================================================================
Package Arch Version Repository Size
===========================================================================
Installing:
kmod-kvdo x86_64 6.1.2.41-5.el7 base 317 k
vdo x86_64 6.1.2.41-4.el7 base 624 k
Transaction Summary
===========================================================================
Install 2 Packages
Total download size: 941 k
Installed size: 4.5 M
Downloading packages:
(1/2): kmod-kvdo-6.1.2.41-5.el7.x86_64.rpm | 317 kB 00:01
(2/2): vdo-6.1.2.41-4.el7.x86_64.rpm | 624 kB 00:24
---------------------------------------------------------------------------
Total 38 kB/s | 941 kB 00:24
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : kmod-kvdo-6.1.2.41-5.el7.x86_64 1/2
Installing : vdo-6.1.2.41-4.el7.x86_64 2/2
Verifying : kmod-kvdo-6.1.2.41-5.el7.x86_64 1/2
Verifying : vdo-6.1.2.41-4.el7.x86_64 2/2
Installed:
kmod-kvdo.x86_64 0:6.1.2.41-5.el7 vdo.x86_64 0:6.1.2.41-4.el7
Complete!Step 2: Create a VDO volume
VDO volume It is a logical device created using VDO. They are like disk partitions. You just need to format it with a file system and then you can mount VDO volumes like a regular file system. If you prefer LVM, you can use VDO volumes as LVM physical volumes.
I have a 10 GB disk that will be used for this exercise.
$ lsblk /dev/sdb
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdb 8:16 0 10G 0 disk This is how to create a VDO volume:
$ sudo vdo create --name myvdo --device /dev/sdb --vdoLogicalSize 5G
Creating VDO myvdo
Starting VDO myvdo
Starting compression on VDO myvdo
VDO instance 0 volume is ready at /dev/mapper/myvdowhere:
- Myvdo Is the name of the logical device provided to the user by VDO.
- / dev / sdb Is the block device to be used by the VDO volume
- 5G Is the logical size of the VDO volume. This is optional and can be larger than the physical size of the actual block device.
Displays a list of started and unstarted volumes.
$ sudo vdo list --all
myvdoRun by vdo status Command to analyze the volume.
$ sudo vdo status -n myvdo
VDO status:
Date: '2020-02-12 12:42:04+03:00'
Node: repo-server-01.safaricom.net
Kernel module:
Loaded: true
Name: kvdo
Version information:
kvdo version: 6.1.2.41
Configuration:
File: /etc/vdoconf.yml
Last modified: '2020-02-12 12:39:36'
VDOs:
myvdo:
Acknowledgement threads: 1
Activate: enabled
Bio rotation interval: 64
Bio submission threads: 4
Block map cache size: 128M
Block map period: 16380
.......Compression and deduplication should be enabled.
$ sudo vdo status -n myvdo | egrep 'Compression|Deduplication'
Compression: enabled
Deduplication: enabled
You can use the command vdo growLogical to expand an existing volume. I will increase the capacity to 10GB.
sudo vdo growLogical -n myvdo --vdoLogicalSize 10Gconfirm:
$ sudo vdo status -n myvdo | grep size
Block map cache size: 128M
Block size: 4096
Logical size: 10G
Physical size: 10G
Read cache size: 0M
Slab size: 2G
block map cache size: 134217728
block size: 4096Step 3: Format the VDO volume using the file system.
You can format VDO volumes with your chosen file system type or create PV, VG and LV from them.
$ sudo mkfs.xfs /dev/mapper/myvdoFor LVM creation:
# Create PV
$ sudo pvcreate /dev/mapper/myvdo
Physical volume "/dev/mapper/myvdo" successfully created.
# Create VG
$ sudo vgcreate vg01 /dev/mapper/myvdo
Volume group "vg01" successfully created
# Create LV
$ sudo lvcreate -n lv01 -l+100%FREE vg01
Logical volume "lv01" created.
# Create a file system
$ sudo mkfs -t xfs /dev/mapper/vg01-lv01
meta-data=/dev/mapper/vg01-lv01 isize=512 agcount=4, agsize=655104 blks
= sectsz=4096 attr=2, projid32bit=1
= crc=1 finobt=0, sparse=0
data = bsize=4096 blocks=2620416, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=4096 sunit=1 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0You can now register and install new devices.
sudo udevadm settle
sudo mkdir /myvdo
--- For standard VDO volume ---
$ sudo mount /dev/mapper/myvdo1 /myvdo
--- For LVM ---
$ sudo mount /dev/mapper/vg01-lv01 /myvdo
For a persistent installation, configure / etc / fstab file.
UUID=XXXXX /myvdo xfs defaults,x-systemd.requires=vdo.service 0 0You can also display statistics in an easy-to-understand format.
$ sudo vdostats --human-readable
Device Size Used Available Use% Space saving%
/dev/mapper/myvdo 10.0G 4.0G 6.0G 40% 98%
Step 4: Test deduplication:
I will download an ISO file to test deduplication.
wget http://mirror.centos.org/centos/7/os/x86_64/images/boot.isoCopy the files to the / myvdo directory.
sudo cp boot.iso /myvdo/boot1.isoCheck storage statistics.
--- Before copy ---
$ sudo vdostats --human-readable
Device Size Used Available Use% Space saving%
/dev/mapper/myvdo 10.0G 4.0G 6.0G 40% 98%
--- After copy ---
$ sudo vdostats --human-readable
Device Size Used Available Use% Space saving%
/dev/mapper/myvdo 10.0G 4.2G 5.8G 42% 7%
You will notice used The field was increased from 4.0G to 4.2G because we copied the files to a volume that took up some space.
Let’s copy the same file.
sudo cp boot.iso /myvdo/boot2.isoCheck the volume statistics again.
$ sudo vdostats --human-readable
Device Size Used Available Use% Space saving%
/dev/mapper/myvdo 10.0G 4.2G 5.8G 42% 52%
You can see that the volume space used has not changed. But save space The increase to 52% proves that deduplication has occurred to reduce the space consumption of redundant copies of the same file.
More guides:
Configure user password aging / expiration policy in Linux
How to use tuned-adm to optimize Linux system performance
Retain system journal records with persistent storage
Understanding the Linux file system hierarchy
How to create hard and soft (symbolic) links in Linux