Retain system journal records with persistent storage

In this guide, we will study how to persist Systemd Journals logs on the server. When problems continue, it is useful to keep your system logs to troubleshoot your service. At the end of this article, you should be able to configure the system log to preserve event records when the server is rebooted.

System Journal Basics

By default, Systemd stores system logs in / Run / log / news table of Contents. As we discussed in Understanding the Linux File System Hierarchy, everything in the / run directory will be cleared and the content will be recreated on reboot. This means that the log will be cleared when the system is rebooted.

We can adjust System daily Services for /etc/systemd/journald.conf File so that the log persists after a restart. Open the file to see its contents:

$ sudo vim /etc/systemd/journald.conf

These are common options that can be configured-most options in most systems are commented out so you can adjust accordingly.

[Journal]
#Storage=auto
#Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitInterval=30s
#RateLimitBurst=1000
#SystemMaxUse=
#SystemKeepFree=
#SystemMaxFileSize=
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#MaxRetentionSec=
#MaxFileSec=1month
#ForwardToSyslog=yes
#ForwardToKMsg=no
#ForwardToConsole=no
#ForwardToWall=yes
#TTYPath=/dev/console
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info
#MaxLevelWall=emerg
#LineMax=48K

Configure permanent system journal

To configure the systemd-journald service to keep system logs permanently after a restart, you need to set Storage to persistent:

Other values ​​that can be set for the Storage parameter are:

  • persistent: Store journal in/var/log/journalThe directory still exists after the restart. Directory creation (if it does not exist)
  • volatile: Store journals in volatile/run/log/journaltable of Contents. This will not cause the system to restart.
  • auto: Rsyslog will determine whether to use persistent storage or volatile storage. in case/var/log/journalIf the directory exists, rsyslog uses persistent storage, otherwise it uses volatile storage.

For permanent journal storage, set it to:

[Journal]
Storage=persistent

After committing the changes, restart the systemd-journald service for the configuration changes to take effect.

sudo systemctl restart systemd-journald

Confirm that the service is restarted and running:

$ systemctl status systemd-journald
● systemd-journald.service - Journal Service
   Loaded: loaded (/usr/lib/systemd/system/systemd-journald.service; static; vendor preset: disabled)
   Active: active (running) since Sun 2020-02-09 10:59:36 UTC; 8s ago
     Docs: man:systemd-journald.service(8)
           man:journald.conf(5)
 Main PID: 5299 (systemd-journal)
   Status: "Processing requests..."
   CGroup: /system.slice/systemd-journald.service
           └─5299 /usr/lib/systemd/systemd-journald

Feb 09 10:59:36 cent7.novalocal systemd-journal[5299]: Permanent journal is using 8.0M (max allowed 1.9G, trying to leave 2.9G free of 16.9G a…it 1.9G).
Feb 09 10:59:36 cent7.novalocal systemd-journal[5299]: Permanent journal is using 8.0M (max allowed 1.9G, trying to leave 2.9G free of 16.9G a…it 1.9G).
Feb 09 10:59:36 cent7.novalocal systemd-journal[5299]: Time spent on flushing to /var is 68.506ms for 7962 entries.
Feb 09 10:59:36 cent7.novalocal systemd-journal[5299]: Journal started
Hint: Some lines were ellipsized, use -l to show in full.

The / var / log / journal directory should be created.

$ ls /var/log/journal
65113b1a0d2f6087d515e6a8cd0ee7ef

Subdirectories /var/log/journal Hex characters in long names and*.journalfile.

$ ls /var/log/journal/65113b1a0d2f6087d515e6a8cd0ee7ef/
system.journal

of*.journalA file is a binary file that stores structured and indexed journal entries.

Adjust the maximum storage usage of the journal

You can set the maximum size of the permanent log by uncommenting and changing the following:

SystemMaxUse=500M

The default size limit is set to 10% of the base file system size, but the upper limit is 4 GiB.

sudo systemctl restart systemd-journald

Understanding the Linux file system hierarchy

How to create hard and soft (symbolic) links in Linux

Sidebar