Part I · How it works

8.4 Too many open files: limits on processes

alsotoo many open filesulimitulimit -nemfilenofilelimitnofilerlimitrlimitsresource temporarily unavailablepid_maxthreads-maxtasksmax/proc/pid/limitsfork failed

TL;DR

Each process inherits limits from its parent: open files, processes, stack size… Each has a soft value (in force) and a hard ceiling the process may raise it to.
The usual one to hit is open files: soft 1024 by default, while a busy server holds a descriptor per connection and per file.
Services started by systemd take their limits from the unit (LimitNOFILE=), not from your shell or limits.conf.

ulimit -n                     # your soft limit on open files
cat /proc/PID/limits          # a running process's limits
ls /proc/PID/fd | wc -l       # how many files it has open now

On your machine

alice@laptopexample

$ atfq 8.4
  Files     1,024 open per process, can be raised to 524,288
  Processes 62,844 for your user, threads included
  Stack     8.0 MiB per thread, can be raised without limit
  System    14,208 files open on the whole system

  Now       14 threads in 3 processes
  Most      supervisor, PID 4100: 12 · systemd, PID 1: 1 · rsync, PID 5000:
            1
  Ceiling   126,906 threads, PIDs up to 4.2 million

An example: Alice's laptop, one of the test machines. In your terminal, this block shows your own machine, read live.

Pitfalls

"Too many open files" can also be a leak: if the count in /proc/PID/fd keeps growing, a file or socket isn't being closed.
ulimit -n 65536 lasts for that shell and what it starts; for a service, systemctl edit and LimitNOFILE=65536 (22.3).
"fork: Resource temporarily unavailable" means a process or thread limit: ulimit -u, or a cgroup's pids.max (systemd's TasksMax=).
Containers have their own limits: docker run --ulimit nofile=….

Deep dive 2 questions

Why is the default still 1024?

The old select() call can't handle descriptor numbers of 1024 and above. A program using it with more descriptors would corrupt memory, so the soft limit stays at 1024 for safety, while systemd sets the hard limit to 524288: programs that use epoll raise their own soft limit at start.

Where do limits come from at login?

The kernel's defaults, then PAM's pam_limits reads /etc/security/limits.conf and limits.d/ for login sessions, then systemd's settings for your user session. Each process can lower its limits, or raise the soft one up to the hard ceiling; only root raises the ceiling.

Commands: ulimit prlimit ls
Paths: /proc/PID/limits /etc/security/limits.conf
See also: 8.3 Event loops: one thread, thousands of connections
22.3 Unit files and overrides
16.4 cgroups