Part II · Your system

21.4 syscalls and strace

alsostracesyscallsyscallssystem callsystem callsptraceptrace_scope

TL;DR

Programs don't touch files, the network or devices themselves: they ask the kernel, with system calls (openat, read, connect…).
strace shows every system call a program makes, with its arguments and result. Errors such as ENOENT (no such file) stand out.
-f follows child processes, -e filters, -p attaches to a running process.

strace -f -e trace=openat ./myapp 2>&1 | grep ENOENT
strace -c ls               # count calls, summary at the end
sudo strace -p 1234

On your machine

alice@laptopexample

$ atfq 21.4
  vm.swappiness                    60  how willing the kernel is to swap
  vm.overcommit_memory 0 0 guess, 1 always say yes, 2 strict limit
  kernel.dmesg_restrict             1  1: reading dmesg needs root
  net.ipv4.ip_forward               1  1: this machine routes packets
  kernel.sysrq 176 0 off, 1 all, other: some Alt+SysRq keys
  fs.inotify.max_user_watches  524288  files a user can watch for changes
  Set in    vm.swappiness = 10 (/etc/sysctl.d/90-swap.conf)
            fs.inotify.max_user_watches = 524288
            (/etc/sysctl.d/99-inotify.conf)

An example: Alice's laptop, one of the test machines. In your terminal, this block shows your own machine, read live.

Pitfalls

strace slows the traced program down a lot: don't leave it attached to a busy production service.
strace -p on your own process may fail: with kernel.yama.ptrace_scope = 1 (Ubuntu's default), you may only trace your own child processes. Use sudo.
strace writes to stderr: add 2>&1 before a pipe, or -o trace.txt to write a file.
A failing call isn't always a problem: programs often try several paths and use the first that exists.

Deep dive 2 questions

What is a system call?

The way a program asks the kernel for something it can't do alone: open a file, send a packet, start a process. The program puts the call number and arguments in CPU registers, and a special instruction switches to the kernel, which does the work and returns a result. Library functions such as fopen or printf end up calling them.

How does strace work?

Through ptrace, the same mechanism debuggers use: the kernel stops the traced program at every system call and lets strace read its registers. That's why it's slow. It's also why Yama restricts it: a tool that can trace your programs can read their memory, passwords included.

Commands: strace
See also: 16.2 Signals