How to Manage Log Files with LogRotate on Ubuntu 16.04

Introduction

Logrotate is a system utility that manages the automatic splitting and compression of log files. If the log files are not rotated, compressed, and periodically truncated, they could ultimately consume all the free disk space on the system.

Logrotate is installed by default on Ubuntu 16.04, and is configured to handle the need to rotate the logs of all installed packages, including rsyslog, the default syslog processor.

In this article, we’ll go over the default LogRotate configuration and then set up log rotation for fictional custom applications.

Prerequisites

This tutorial assumes you have an Ubuntu 16.04 server, with a SUDO user with non-root user support enabled, as described in the initial server setup with Ubuntu 16.04.

Logrotate is available on many other Linux distributions, but the default settings can be quite different. The other sections of this guide will still apply as long as your version of LogRotate is similar to Ubuntu 16.04. Follow step 1 to determine the version of LogRotate.

Log in to your server as SUDO user.

Determining the LogRotate version

If you are using a non-Ubuntu server, first make sure Logrotate is installed by asking for its version information:

logrotate --version

Output

logrotate 3.8.7

If Logrotate is not installed, you will receive an error message. Please install the software using your Linux distribution’s package manager.

If Logrotate is installed, but the version number is significantly different, then there may be problems with the configuration described in this manual. Check the documentation for your specific version of LogRotate by reading the man page:

man logrotate

Next, we will look at the default LogRotate configuration structure on Ubuntu.

Exploring the Logrotate Configuration

LogRotate configuration information in general can be found in two places on Ubuntu:

  • /etc/logrotate.conf: This file contains some defaults and sets rotations for a few logs that are not owned by any package system. It also uses an include statement to load configuration from any file in the /etc/logrotate.d directory.
  • /etc/logrotate.d/: This is where all installable packages that need help rotating the log place the Logrotate configuration. On a standard installation, you should already have files here for basic system tools like apt, dpkg, rsyslog, and so on.

By default, the logrotate.conf file is configured to rotate the logs weekly, with the log files owned by the user root and group system log (su root syslog), with four log files saved (rotate 4), and new empty log files are created after only one rotates (create),

Let’s take a look at the configuration file for the LogRotate package, in the /etc/logrotate.d file. cat for the apt package utility:

cat /etc/logrotate.d/apt

Output

/var/log/apt/term.log {
  rotate 12
  monthly
  compress
  missingok
  notifempty
}

/var/log/apt/history.log {
  rotate 12
  monthly
  compress
  missingok
  notifempty
}

This file contains configuration blocks for two different log files in the / var / log / apt / directory: term.log and history.log. They both have the same parameters. Any options not set in these configuration blocks inherit the defaults or those in /etc/logrotate.conf. The kit for the apt log options are:

  • rotate 12: Store twelve old log files.
  • monthly: Rotate once a month.
  • compress: Compress rotated files. It uses gzip by default and results in files ending with the .gz extension. The compress command can be changed using the compresscmd option.
  • missingok: Do not write an error message if the log file is missing.
  • notifempty: Do not rotate the log file if it is empty.

There are many configuration options available. You can read about all of them by typing man logrotate at the command line to bring up the LogRotate man page.

Next, we’ll create a log handling configuration file for the fictitious service.

Configuration example

To manage the log files for third-party service applications on prepackaged and pre-configured systems, we have two options:

  1. Create a new Logrotate configuration file and place it in the /etc/logrotate.d/ directory. It will work daily as a user root along with all other standard LogRotate jobs.
  2. Create a new config file and run it with the default LogRotate settings in Ubuntu. This is only really necessary if you need to run LogRotate as a non-root user, or if you want to rotate the logs more often than daily (the hourly configuration in /etc/logrotate.d/ will be ineffective as the Logrotate system installation only runs once a day).

Let’s walk through these two options with some setup examples.

Adding configuration to /etc/logrotate.d/ directory

We want to set up a rotation of logs for a fictional web server that puts access.log and error.log in the / var / log / example-app / directory. It works like www-data as user and group.

To add the configuration to the /etc/logrotate.d/ directory, first open a new file:

sudo nano /etc/logrotate.d/example-app

Here’s an example of a config file that can handle these logs:

/etc/logrotate.d/example-app

/var/log/example-app/*.log {
    daily
    missingok
    rotate 14
    compress
    notifempty
    create 0640 www-data www-data
    sharedscripts
    postrotate
        systemctl reload example-app
    endscript
}

Some of the new configuration directives in this file:

  • create 0640 www-data www-data: This command will create a new empty log file after rotation with the given permissions (0640), owner (www-data) and group (www-data).
  • sharedscripts: This flag means that any scripts added to the configuration are executed only once, not to rotate each file. Since this configuration will correspond to two log files in the example-app directory, the script specified in postrotate will run twice without this option.
  • Between postrotate and endscript: This block contains the script to run after the log file is rotated. In this case, we reload the sample application. Sometimes it is necessary for the application to switch to the new log file. Note that the build is done before the logs are compressed. Compression can take a long time and the software should immediately switch to the new log file. For tasks that need to be done after shrinking the logs, use the lastaction block.

After configuring according to your needs, save it in /etc/logrotate.d file, you can check it by doing the following:

sudo logrotate /etc/logrotate.conf --debug

This calls logrotate, points to the standard configuration file, and turns on debug mode.

Information will be printed out about which log files Logrotate is processing and what it would do with them. If everything looks good, everything will end well. The standard Logrotate job will run once a day and include the new configuration.

Next, we will try to install the installer, which does not use the default Ubuntu configuration at all.

Creating a LogRotate configuration

In this example we have an application that runs as a user Andreyex, generating logs that are stored in the / home / andreyex / logs / directory. We want to rotate these logs on an hourly basis, so we have to install it outside of the /etc/logrotate.d structure provided in Ubuntu.

First, we will create a config file in our directory. Open it in a text editor:

nano /home/andreyex/logrotate.conf

Then paste in the following configuration:

/home/andreyex/logrotate.conf

/home/andreyex/logs/*.log {
    hourly
    missingok
    rotate 24
    compress
    create
}

Save and close the file. We’ve seen all of these options in the previous steps, but let’s summarize: this configuration will rotate files hourly, compressing and saving twenty-four old logs and creating a new log file to replace the rotated one.

You need to tweak the configuration to suit your application, but this is a good start.

To test that it works, let’s make a log file:

cd ~
mkdir logs
touch logs/access.log

Now that we have an empty log file in the right place, let’s run the logrotate command.

Since the magazines belong AndreyEx we don’t need to use sudo. However, we need to specify a state file. This file records what logrotate saw and did last time, so it knows what to do next time it starts. This is handled for us when using the Ubuntu Logrotate installation (it can be found in / var / lib / logrotate / status), but we need to do it manually.

We’ll ask Logrotate to put the state file directly in our home directory for this example. We can specify anywhere that is available and convenient:

logrotate /home/andreyex/logrotate.conf --state /home/andreyex/logrotate-state --verbose

Output

reading config file /home/andreyex/logrotate.conf

Handling 1 logs

rotating pattern: /home/andreyex/logs/*.log  hourly (24 rotations)
empty log files are rotated, old logs are removed
considering log /home/andreyex/logs/access.log
  log does not need rotating

–Verbose will print out detailed information about what Logrotate is doing. In this case, it looks like it didn’t do anything. This is the first time LogRotate sees this log file, as the file is known to be zero hours old and should not be rotated.

If we look at the state file, we can see that Logrotate has recorded the startup information:

cat /home/andreyex/logrotate-state

Output

logrotate state -- version 2
"/home/andreyex/logs/access.log" 2017-11-7-19:0:0

Logrotate noted that he saw the logs and when he last viewed their rotation. If you run the same command one hour later, the log will be rotated as expected.

If you want to force LogRotate to rotate the log file, then use the –force flag:

logrotate /home/andreyex/logrotate.conf --state /home/andreyex/logrotate-state --verbose --force

This is useful when testing postrotate and other scenarios.

Finally, we need to set up a cron job to run Logrotate every hour. Open the user’s crontab:

crontab -e

A text file will open. There may already be some comments in the file explaining the expected basic syntax. Move the cursor to a new blank line at the end of the file and add the following:

crontab

14 * * * * /usr/sbin/logrotate /home/andreyex/logrotate.conf --state /home/andreyex/logrotate-state

This task will be performed at the 14th minute of every hour, every day. Basically the same logrotate commands that we ran into work earlier, although we extended logrotate, its full path is / usr / sbin / logrotate, just for security. It is good practice to be as explicit as possible when writing a cron job.

Save the file and exit. This will install cron and our task will run according to the specified schedule.

If we re-enter our directory after about an hour, we should find the compressed log file access.log.1.gz (or, .2.gz if you ran LogRotate with the –force flag).

Output

In this article, we checked our version of LogRotate, examined the default Ubuntu LogRotate configuration, and set up two different types of custom configurations. To learn more about the command line and configuration options available for LogRotate, you can read the man page by running man logrotate in a terminal.

Sidebar