See beyond
the surface.

Observe Linux workloads, from CPU to disk.
Keep the evidence beside every number.

Discover Omniscient Start observing

ONE TOOL / A SHARED VOCABULARY

Measure the system.
Keep the meaning.

Follow a workload from kernel activity to the time it spends waiting. Profiles, counters and events share one model, with their limits kept visible.

Begin with a question.
Use omn cpu, omn mem or omn latency to start an investigation. Name a source when you need precise control.
See what is available here.
omn list probes this machine. Missing privilege, unsupported hardware and absent collectors come with explicit explanations.
Take the evidence with you.
Export results and profiles. Keep the resolved backend, machine conditions and observation cost alongside the numbers.

NINE FAMILIES / CONNECTED QUESTIONS

Every layer has
a story to tell.

Choose the part of the system you want to understand. Open a family to read its story: the question, its sources, a real run and its limits.

FOLLOW THE SIGNAL / THE SUZANNE SCENARIO

One request.
Every layer in view.

Ask an engine to render Suzanne in 3D and save the result. Follow each handoff across the machine.

Planned Linux capture scenario · this sequence is illustrative. One host; GPU attribution depends on the engine and driver. File writes and durable device completion are distinct.

ENGINEERED FOR EVIDENCE

Visibility is good.
Knowing why is better.

A plausible number is not enough. Know what produced it, what was lost and what supports its attribution.

IDENTITYKEYWAKEUPTEMPORAL

Correlation, with receipts.

A propagated identity and two events close in time are different kinds of evidence. Omniscient tells you which one supports the connection.

Understand the evidence
[ kernel ]
↓
[ omn ]

Direct by design.

eBPF, perf_event_open, procfs, sysfs and netlink. No perf, bpftrace or bcc running underneath.

◴ → ◷

Record. Revisit.

Save a run and replay it in the terminal. Change the view, keep the context.

scope: package
target: process
↳ request refused

Honest boundaries.

If a source cannot answer a question, the validator explains why — instead of answering a different one.

YOUR TERMINAL / MORE DIMENSIONS

Same run.
A different perspective.

Select a request and carry its focus across views. Inspect execution lanes or CPU call stacks — all drawn from one recorded run.

● ● ●omn top / request120 × 40
omn top │ fixture many-requests │ 15.8s                           view: snapshot │ collectors ≥5.21% │ target 0.35% cpu 
 1 system  2 flame  3 ledger  4 request  5 timebar  6 heatmap  7 lanes  8 wake  9 flows  0 process                      
┌ request -- what one request costs ───────────────────────────────────────────────────────────────────────────────────┐
│300 requests served by server (pid 2619831) · slowest a0d7 5.49ms · median 2.18ms · in order of arrival  168 late degr│
│                                                                                                                      │
│  req      span  its life by state, to one scale: the slowest is 5.49ms             mostly                            │
│▶ a001    159µs  ░█                                                                 net in 75.5% · on cpu 24.5%       │
│  a002   2.33ms  ░█▓▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀                                       lock/sleep 89.0%                  │
│  a003   5.00ms  ██▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄        disk 96.2%                        │
│  a004   2.21ms  ░░░░░░░░░░░░░░░░░░░░░░░░░░                                         net in 98.7%                      │
│  a005   56.8µs  ░                                                                  net in 50.5% · on cpu 49.5%       │
│  a006   2.31ms  ░▓▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀                                        lock/sleep 90.6%                  │
│  a007   4.99ms  ░█▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄        disk 95.3%                        │
│  a008   2.17ms  ░░░░░░░░░░░░░░░░░░░░░░░░░░                                         net in 98.7%                      │
│  a009    123µs  ░                                                                  net in 77.7% · on cpu 22.3%       │
│  a00a   2.21ms  █▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀                                         lock/sleep 94.6%                  │
│  a00b   5.00ms  ░█▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄        disk 95.3%  +289 more ↓           │
│  ░░ net in   ██ on cpu   ▓▓ run queue   ▀▀ lock/sleep   ▄▄ disk   ░░ net out                                         │
│                                                                                                                      │
│── a001 · 159µs from its first byte in to its last byte out · thread 2619831 ─────────────────────────────────────────│
│            ┊0                  ┊50.0µs              ┊100µs                                                           │
│  net in    ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░                  120µs before start                    │
│  serving                                                    █████████████████  39.0µs from start to done             │
│  on device                                                                     no block request                      │
│  net out                                                                       gone when done                        │
│  inside `serving` the states are drawn by share: net.request sums them, it does not keep their order                 │
│                                                                                                                      │
│  where its 159µs went               evidence                       its share of each resource                        │
│  ░ net in        120µs   75.5% ░░░           by socket 160254      cpu        39.0µs: 0.07% of the 55.9ms server took│
│  █ on cpu       39.0µs   24.5% █                                              off cpu 0 times, preempted 0           │
│  ▓ run queue         ·                                             disk       no block request issued                │
│  ▀ lock/sleep        ·                                             network    in 120µs · out 0ns                     │
│  ▄ disk              ·                                                        51B read · 59B queued                  │
│  ░ net out           ·                                                                                               │
│  no decomposition in this run: the evidence is not graded                                                            │
│                                                                                                                      │
│                                                                                                                      │
│  what sets it apart from the other 299 requests: ░ net in +93.3µs (theirs 26.6µs)                                    │
└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
q quit  ←/→ tab view  1-9 jump  f clear focus  ? help  ↑/↓ request  s sort  enter focus it, then its lanes  l its thread

A request, in focus — what one request costs, from arrival to reply. Carry its context into the other views.

Snapshots of the many-requests fixture from the view gallery, drawn at 120×40 and held by tests/view_snapshots.rs. Colours mapped onto this page's palette.

REPRODUCIBLE INVESTIGATIONS

Trace as code.

Prototype a measurement in the shell, then keep it in a session file. YAML, JSON and TOML describe the same requests. The program is part of the request: launching it makes it both the target and the clock.

Browse the sessions 
# omn -f examples/mem-rss-run.yml
name: resident memory of a launched program
requests:
  - label: rss
    source: mem.rss
    target:
      launch: [python3, -c, "..."]
    mode: poll@100ms
    emit: [timeline, summary]
    keys: [task, cgroup]

IN PRACTICE / THE EVIDENCE

A problem.
What changed.

Choose a target to read what happened: the problem, the finding and the result. Each story names its evidence and the scope of its validation.

Controlled validation workload

A slow request, traced to the lock and its holder.

Problem
A request can wait in the application, the scheduler or the block layer. A total duration does not say which part delayed it.
Finding
A loopback server declared request identities through USDT and carried them into downstream work. A deliberately injected lock delay appeared in the request’s own row: 40.06 ms, with the holder thread named.
Result
The measurement attributed the wait using propagated identity. The injected delay supplied a known value to check the result against.

A controlled server scenario; this is not a production incident or the planned Suzanne rendering integration.

Omniscient · STATE.md, vertical case · docs/briefs/wave-9-vertical.md

TESTED WITH DISTROPROOF / OCTOBER 2026

Different kernels.
Explicit limits.

The installed releases were tested on 17 distro images and 5 variants, 3 arm64 images and 8 rootless containers, spanning Linux 4.15 to 7.2, glibc and musl. The same binary listed 36 sources usable on Ubuntu 18.04 and 96 on Fedora — with a reason for each refusal.

Availability checked against reality.

The campaign found 27 defects: 13 in the tool and 14 in its tests. Initially, 158 source runs were listed usable but refused at start. After the fixes, none did. Every refusal reason agreed with envprobe’s independent reading of the same guest across all 21 targets.

Explore the distroproof case study →

Features, beyond version numbers.

Real probes catch Debian’s perf policy, BPF maps charged to memlock on 5.10, lockdown restrictions and Alpine’s missing function tracer. RHEL backports are detected as features rather than ruled out by kernel age.

Two release archives.

The glibc archive targets glibc 2.17+. Alpine has a native musl archive with a static engine and a musl preload shim. NVIDIA sources are unavailable in the static build because they need dynamic library loading.

What remains open.

The reference pairs against distro-packaged perf, bpftrace, bcc and other witnesses run on the devutils chain: nine targets, every check holds in the latest runs. A pair skips, with its reason, where a distribution lacks the witness or the witness does not load. The arm64 images are emulated on an x86_64 host.

See the chain’s latest runs → Read the findings and evidence →

OPEN SOURCE / BUILT FOR LINUX

Ask your machine
a better question.

Choose the release for your C library or build from source. Run omn list to see what this machine can serve; eBPF sources need capabilities and suitable kernel settings.

Install Omniscient Start observing
omn top -- ./your-program
x86_64 archives: glibc 2.17+ or musl · Rust 1.82+ to build · availability is probed per source