Part II · Your system
26.2 Where logs go: journald, syslog and /var/log
also/var/logvar/logrsyslogsyslog-ngsyslogjournald.confpersistent journalvolatile journalauth.log/var/log/syslog/var/log/messageslog files
TL;DR
- journald receives the logs. It keeps them on disk in
/var/log/journal/if that folder exists, otherwise in RAM under/run/log/journal/, lost at reboot. - A syslog daemon (rsyslog) may also copy them to text files:
/var/log/syslog(Debian, Ubuntu) or/var/log/messages(older RHEL), andauth.logfor logins. - Some programs write their own files:
/var/log/nginx/,/var/log/apt/,/var/log/pacman.log.
sudo mkdir -p /var/log/journal && sudo systemctl restart systemd-journald
ls -lt /var/log | head
sudo tail -f /var/log/nginx/error.log
On your machine
$ atfq 26.2 Journal on disk (/var/log/journal): kept across reboots · 300 B in 2 files Settings SystemMaxUse=500M Syslog rsyslog also writes text files in /var/log Rotation 2 rules in /etc/logrotate.d Biggest /var/log/syslog 4.9 KiB · /var/log/nginx/access.log 2.9 KiB · /var/log/auth.log 1.2 KiB
An example: Alice's laptop, one of the test machines. In your terminal, this block shows your own machine, read live.
Pitfalls
- Logs from before a crash need a persistent journal: create
/var/log/journal(or setStorage=persistent) before you need them. - In containers there's often no journald: logs go to the container's
output (
docker logs). /var/log/wtmpandbtmpare binary login records: read them withlastandlastb.
Deep dive 2 questions
Why both journald and rsyslog?
journald arrived with systemd (2011); rsyslog was already everywhere, with remote logging, text files that tools expected, and filtering rules admins had written. Many distros kept both, journald forwarding to rsyslog. Recent ones increasingly drop rsyslog and rely on the journal alone, which can send logs to another machine itself.
How do logs reach journald?
Three ways: a service's standard output and error, captured by systemd;
the syslog socket (/dev/log) that the syslog() call writes to; and
journald's own API with structured fields. The kernel's messages are read
from /dev/kmsg.