How to deal with huge (and growing) log files in Linux

If you have been running a Linux server for a long time, you are familiar with the problem of log files. Sometimes they can be hard enough to even find them, and then you sometimes come across a file hundreds of MB (or even GB) in size. Searching through it is a pain, and they may even end up eating up your storage space.

This article will show you how to deal with this problem. We are going to use the built-in Linux syslog package that already exists. This means you don’t need to install new packages! All you need to know is to add configuration for your own log files.

How Linux logs rotate with logrotate

There are several system log files in the following location:

/var/log/

In the screenshot below, we will list all the files in this directory and show the file size in MB:

As you can see, there are many large log files that are dated. They are generated by default “logrotate”. Looking at the timestamps, we can see that they are generated every seven days. So for every log file here, logrotate does the following:

  1. Renames a log file with a time stamp
  2. Creates a new empty log file with the same name as before

Checking the default Logrotate configuration

Logrotate checks the following file for its configuration:

/etc/logrotate.conf

In this we see that he receives all the important information about:

  1. How often should he rotate the magazines
  2. How many backups to keep
  3. What suffix should he add to old magazines
  4. Do they need to squeeze them or not
  5. etc…

Here is a screenshot of the logrotate config file:

How to deal with huge (and growing) log files in Linux

By default, old log files are not compressed. You can change this by simply uncommenting the “compress” directive as shown above.

Adding files for logrotate

How does logrotate know which logs to work with? Files in this directory:

/etc/logrotate.d

For example, in our “secure” log we saw in the first screenshot, we can figure out which file in the logrotate.d file contains it with a simple grep command, as detailed in our advanced grep guide:

grep -r secure /etc/logrotate.d

The screenshot below shows which file is responsible for the “secure” log:

How to deal with huge (and growing) log files in Linux

Opening “syslog”, we see that it tells logrotate to work on multiple files:

How to deal with huge (and growing) log files in Linux

They are all located in the / var / log / path. Therefore, each file contains:

  1. The names of the log files that it should rotate
  2. Directives specific to this particular file
  3. In the absence of directives, the default values ​​from logrotate.conf will be used

Creating your own Logrotate file for your logs

We can follow the pattern above to create an entry with logrotate that tells us that it handles our own files in the same way.

Get owner and group for your directory

To make sure all permissions are set, we need to run logrotate with the appropriate permissions. Change to the directory containing the log file (s) you want to process and get the owner as well as the group using this command:

ls -l -d

For example, the / var / log directory has the owner as “root” and the group “root”:

How to deal with huge (and growing) log files in Linux

Creating a Logrotate Configuration File

Then create a new file in /etc/logrotate.d/ and paste the following configuration into it:

/var/log/newlogfile {

size 10M

compress

delaycompress

su root root

}

Replace / var / log / newlogfile with the location of your own log file. Also, replace su root root with the owner and group name you got in the previous step. Here is a list of logrotate directives that you can use. The delaycompress directive tells logrotate to only compress files with two versions, old or older. This way you get quick and easy access to the last compressed log file.

You can also use the following lines in the curly brace block to run scripts before and after your log files are rotated:

postrotate
// скрипты для ротации журналов
endscript

As you can see Linux has a complete solution to rotate the log file. All you have to do is add a few lines of code to the existing structure, and your own log files can be rotated without any problem!

Please disable your ad blocker or whitelist this site!

Related Posts