108.2 System Logging

Introduction

This lesson is about how Linux records what happened on the system.

The main question is:

How do you read logs, store them safely, rotate them, and understand how journald and rsyslog work together?

What you should be able to do after this lesson:

• Read log data with journalctl.
• Understand the roles of systemd-journald, rsyslog, and logrotate.
• Recognize facilities, priorities, and actions in syslog-style logging.
• Send a test message with logger.
• Understand local vs central log collection.

Big Idea: Linux Logging Is a Pipeline

For beginners, logging becomes much simpler if you picture it like this:

programs create messages -> the system collects them -> messages are stored -> old logs are rotated

On many modern systems:

• systemd-journald collects log messages
• rsyslog can read from the journal and write classic log files in /var/log/
• logrotate keeps those files from growing forever

That is the big picture you want to remember.

systemd-journald

The journal is a structured log database used by systemd systems.

Show recent logs:

journalctl

Follow new messages live:

journalctl -f

Show logs from the current boot:

journalctl -b

Filter by service:

journalctl -u ssh.service

Filter by priority:

journalctl -p err

Filter by time:

journalctl --since today
journalctl --since '2026-03-29 09:00:00'

These filters are extremely useful because beginners often drown in too much log output.

Persistent Journal Storage

Some systems keep journal data only in memory unless persistent storage is enabled.

Common configuration file:

/etc/systemd/journald.conf

Persistent journal files are often stored in:

/var/log/journal/

Useful idea:

• no persistent storage = logs may vanish after reboot
• persistent storage = you can inspect older boots

Cleaning Old Journal Data

If the journal becomes too large, you can remove old data with size or time limits.

Examples:

sudo journalctl --vacuum-size=200M
sudo journalctl --vacuum-time=14d

This is safer than randomly deleting journal files by hand.

Reading Logs from Another System Copy

Sometimes you are debugging a disk mounted from a rescue system or another machine.

Useful pattern:

journalctl --directory=/mnt/var/log/journal

That lets you read journal data from a copied or mounted filesystem.

rsyslog

rsyslog is a classic logging daemon that writes text log files and can forward logs to other hosts.

Common configuration file:

/etc/rsyslog.conf

Common log directory:

/var/log/

Facilities, priorities, actions

In syslog-style configuration, a rule usually says:

facility.priority -> action

Examples of facilities:

• auth
• mail
• kern
• daemon
• user

Examples of priorities:

• debug
• info
• notice
• warning
• err
• crit

An action can be:

• write to a file
• send to another host
• send to a user

Simple example idea:

authpriv.*    /var/log/secure

You do not need deep rsyslog syntax at this level. You do need to recognize what the rule is trying to do.

Sending a Test Log Entry

The logger command is a very practical tool.

Example:

logger -p user.notice 'LPIC logging practice message'

Then look for it:

journalctl --since '5 minutes ago' | grep 'LPIC logging practice'

That gives you a safe lab workflow:

• create a known message
• search for it
• confirm the logging path works

If you want to send command output into the journal through systemd-aware tooling, also recognize:

echo 'journal test' | systemd-cat

Central Log Servers

LPIC expects awareness that a logging daemon can:

• send log data to a central log server
• receive log data as a central log server

You do not need to build a full logging architecture here. Just remember that central logging helps teams collect logs from many machines in one place.

Log Rotation

Plain text logs can grow forever unless you rotate them.

This is the job of logrotate.

Main config:

/etc/logrotate.conf

Extra config snippets:

/etc/logrotate.d/

Typical logrotate jobs:

• rotate old files
• compress them
• keep only a fixed number
• recreate a fresh empty log file

Practice Step by Step

1) Inspect journal data

1. Run: journalctl -b | head -n 20
2. Then filter by priority: journalctl -p warning -b
3. If a known service exists, inspect it: journalctl -u NetworkManager

2) Send and find a test message

1. Run: logger -p user.notice 'LPIC logging practice message'
2. Search for it: journalctl --since '5 minutes ago' | grep 'LPIC logging practice'
3. This confirms that user-generated log messages are visible.

3) Check whether journal storage is persistent

1. List the directory: ls -ld /var/log/journal
2. Open the config if it exists: sudo sed -n '1,120p' /etc/systemd/journald.conf
3. Look for storage-related settings.

4) Inspect log rotation config

1. Run: sed -n '1,120p' /etc/logrotate.conf
2. Then list: ls /etc/logrotate.d/
3. Notice how log rotation is configured as small rule files.

Cheat Sheet

journalctl
journalctl -b
journalctl -f
journalctl -u ssh.service
journalctl -p err
journalctl --since today
logger -p user.notice 'test message'
echo 'journal test' | systemd-cat
sudo journalctl --vacuum-size=200M
sudo journalctl --vacuum-time=14d
ls /var/log/
sed -n '1,120p' /etc/rsyslog.conf
sed -n '1,120p' /etc/logrotate.conf
ls /etc/logrotate.d/