In this guide, we will study how to persist Systemd Journals logs on the server. When problems continue, it is useful to keep your system logs to troubleshoot your service. At the end of this article, you should be able to configure the system log to preserve event records when the server is rebooted.
System Journal Basics
By default, Systemd stores system logs in / Run / log / news table of Contents. As we discussed in Understanding the Linux File System Hierarchy, everything in the / run directory will be cleared and the content will be recreated on reboot. This means that the log will be cleared when the system is rebooted.
We can adjust System daily Services for
/etc/systemd/journald.conf File so that the log persists after a restart. Open the file to see its contents:
$ sudo vim /etc/systemd/journald.conf
These are common options that can be configured-most options in most systems are commented out so you can adjust accordingly.
[Journal] #Storage=auto #Compress=yes #Seal=yes #SplitMode=uid #SyncIntervalSec=5m #RateLimitInterval=30s #RateLimitBurst=1000 #SystemMaxUse= #SystemKeepFree= #SystemMaxFileSize= #RuntimeMaxUse= #RuntimeKeepFree= #RuntimeMaxFileSize= #MaxRetentionSec= #MaxFileSec=1month #ForwardToSyslog=yes #ForwardToKMsg=no #ForwardToConsole=no #ForwardToWall=yes #TTYPath=/dev/console #MaxLevelStore=debug #MaxLevelSyslog=debug #MaxLevelKMsg=notice #MaxLevelConsole=info #MaxLevelWall=emerg #LineMax=48K
Configure permanent system journal
To configure the systemd-journald service to keep system logs permanently after a restart, you need to set Storage to persistent:
Other values that can be set for the Storage parameter are:
persistent: Store journal in
/var/log/journalThe directory still exists after the restart. Directory creation (if it does not exist)
volatile: Store journals in volatile
/run/log/journaltable of Contents. This will not cause the system to restart.
auto: Rsyslog will determine whether to use persistent storage or volatile storage. in case
/var/log/journalIf the directory exists, rsyslog uses persistent storage, otherwise it uses volatile storage.
For permanent journal storage, set it to:
After committing the changes, restart the systemd-journald service for the configuration changes to take effect.
sudo systemctl restart systemd-journald
Confirm that the service is restarted and running:
$ systemctl status systemd-journald ● systemd-journald.service - Journal Service Loaded: loaded (/usr/lib/systemd/system/systemd-journald.service; static; vendor preset: disabled) Active: active (running) since Sun 2020-02-09 10:59:36 UTC; 8s ago Docs: man:systemd-journald.service(8) man:journald.conf(5) Main PID: 5299 (systemd-journal) Status: "Processing requests..." CGroup: /system.slice/systemd-journald.service └─5299 /usr/lib/systemd/systemd-journald Feb 09 10:59:36 cent7.novalocal systemd-journal: Permanent journal is using 8.0M (max allowed 1.9G, trying to leave 2.9G free of 16.9G a…it 1.9G). Feb 09 10:59:36 cent7.novalocal systemd-journal: Permanent journal is using 8.0M (max allowed 1.9G, trying to leave 2.9G free of 16.9G a…it 1.9G). Feb 09 10:59:36 cent7.novalocal systemd-journal: Time spent on flushing to /var is 68.506ms for 7962 entries. Feb 09 10:59:36 cent7.novalocal systemd-journal: Journal started Hint: Some lines were ellipsized, use -l to show in full.
The / var / log / journal directory should be created.
$ ls /var/log/journal 65113b1a0d2f6087d515e6a8cd0ee7ef
/var/log/journal Hex characters in long names and
$ ls /var/log/journal/65113b1a0d2f6087d515e6a8cd0ee7ef/ system.journal
*.journalA file is a binary file that stores structured and indexed journal entries.
Adjust the maximum storage usage of the journal
You can set the maximum size of the permanent log by uncommenting and changing the following:
The default size limit is set to 10% of the base file system size, but the upper limit is 4 GiB.
sudo systemctl restart systemd-journald
Understanding the Linux file system hierarchy
How to create hard and soft (symbolic) links in Linux