I was chasing a latency spike in a kernel path and began adding printk() calls. This is the kernel equivalent of looking for a gas leak with a candle: familiar, illuminating and likely to influence the situation under investigation.

Recent kernels include ftrace, with controls under debugfs, normally in /sys/kernel/debug/tracing. It started as a function tracer and has grown useful tracing modes and events. The useful bit is that tracing can often be selected and controlled at runtime rather than compiled into a private forest of diagnostic messages.

The function tracer records kernel function activity in per-CPU ring buffers. Filters can restrict collection to relevant functions, which matters because tracing everything creates enormous output and changes timing. The function-graph tracer adds entry and return relationships, making nested execution and duration easier to see.

For my problem, the scheduler tracing facilities were more revealing than raw function calls. They showed when the task stopped running, which task replaced it and when it returned. The delay I had blamed on one function was mostly time spent waiting to be scheduled. A timestamp around the function could show elapsed time, but not explain where that time went.

Ring buffers are a sensible mechanism for this job. Each CPU can record events with less cross-CPU contention, and old entries can be overwritten when configured as a rolling trace. I can leave tracing ready, reproduce the spike and stop collection soon afterward. The final seconds are usually more valuable than a complete account of the machine booting and becoming bored.

Tracing still has overhead. Function tracing a hot path produces work, filters cost something, and reading the trace while collecting it can perturb results. I begin with the narrowest event set that can answer the question and compare behavior with tracing disabled. A trace is evidence gathered by an instrument, not the unobserved universe itself.

printk() remains appropriate for durable diagnostics that users or administrators need. It is poor as an ad hoc high-volume performance recorder: messages contend for shared facilities, flood logs and require source edits to move the probe. Temporary tracing and permanent logging solve different problems.

For kernel execution and scheduling questions, I try ftrace first. A carefully placed message can fill in semantic information the trace lacks. This is faster and leaves less debris in the source, including my traditional final patch removing forty-seven increasingly desperate print statements.