Part II · Your system
26.4 Logging from your own programs and services
alsologgersystemd-catlogginglog levelsstructured loggingstdout loggingapplication logsjson logssd_journal
TL;DR
- Under systemd, a service only has to print to standard output or error: journald stores each line with the time, unit and PID.
- Prefix a line with
<3>(error) to<7>(debug) to set its priority;journalctl -pcan then filter it. - From a shell script,
loggersends a message to the journal;systemd-catsends a whole command's output.
logger -p user.err "backup failed: disk full"
systemd-cat -t backup ./backup.sh
journalctl -t backup # by tag
Pitfalls
- Writing your own log file from a service means handling rotation, permissions and disk space yourself: standard output is simpler.
- Never log secrets: passwords, tokens and full request bodies end up in files many people and tools can read.
- Logging every request at full detail can cost more than the work: keep debug logs off in production, behind a level setting.
- Python's output to a pipe is buffered: set
PYTHONUNBUFFERED=1, or lines reach the journal late.
Deep dive 2 questions
Plain text or structured logs?
Plain lines are easy to read and grep. Structured logs (JSON, or the
journal's own fields) are easy to filter and count: user_id=42,
duration_ms=180. Many services print JSON for a log collector to index;
the journal can also take extra fields directly through its API.
Which level for what?
Error: something failed and someone should look. Warning: unexpected but handled. Info: what happened, at a level an operator follows. Debug: details for the developer, off by default. If every request logs a warning, warnings stop meaning anything.