Linux

I was chasing a latency spike in a kernel path and began adding printk() calls. This is the kernel equivalent of looking for a gas leak with a candle: familiar, illuminating and likely to influence the situation under investigation.

Recent kernels include ftrace, with controls under debugfs, normally in /sys/kernel/debug/tracing. It started as a function tracer and has grown useful tracing modes and events. The useful bit is that tracing can often be selected and controlled at runtime rather than compiled into a private forest of diagnostic messages.

The function tracer records kernel function activity in per-CPU ring buffers. Filters can restrict collection to relevant functions, which matters because tracing everything creates enormous output and changes timing. The function-graph tracer adds entry and return relationships, making nested execution and duration easier to see.

For my problem, the scheduler tracing facilities were more revealing than raw function calls. They showed when the task stopped running, which task replaced it and when it returned. The delay I had blamed on one function was mostly time spent waiting to be scheduled. A timestamp around the function could show elapsed time, but not explain where that time went.

Ring buffers are a sensible mechanism for this job. Each CPU can record events with less cross-CPU contention, and old entries can be overwritten when configured as a rolling trace. I can leave tracing ready, reproduce the spike and stop collection soon afterward. The final seconds are usually more valuable than a complete account of the machine booting and becoming bored.

Tracing still has overhead. Function tracing a hot path produces work, filters cost something, and reading the trace while collecting it can perturb results. I begin with the narrowest event set that can answer the question and compare behavior with tracing disabled. A trace is evidence gathered by an instrument, not the unobserved universe itself.

printk() remains appropriate for durable diagnostics that users or administrators need. It is poor as an ad hoc high-volume performance recorder: messages contend for shared facilities, flood logs and require source edits to move the probe. Temporary tracing and permanent logging solve different problems.

For kernel execution and scheduling questions, I try ftrace first. A carefully placed message can fill in semantic information the trace lacks. This is faster and leaves less debris in the source, including my traditional final patch removing forty-seven increasingly desperate print statements.

I wrote a file server in the most direct way possible: read a block into a user-space buffer, then write that block to the socket. The code was simple and, for small files, entirely satisfactory. With large transfers, the process spent an impressive amount of CPU moving bytes it never inspected.

The ordinary path copies file data from the kernel’s page cache into my buffer and then back into kernel-managed socket buffers. It also crosses the system-call boundary repeatedly. The data needs to travel from disk or cache to the network device, but my temporary ownership adds work without adding information.

Linux offers sendfile() for this case. It transfers data between a file descriptor and a socket while keeping the payload out of the application’s address space. The kernel can connect the file and networking paths more directly, reducing copies and context transitions. “Zero copy” is convenient shorthand, not a sworn statement that no hardware or kernel component ever copies a byte.

The loop still has to handle partial progress. sendfile() can transfer fewer bytes than requested, be interrupted, or return EAGAIN for a nonblocking socket. I keep an offset and remaining length in the connection state, request another writable notification when necessary, and continue later. Replacing read() and write() with one call does not replace flow control.

Headers complicate matters slightly. A protocol header may be generated in memory while the body comes from a file. I can send the small header normally and use sendfile() for the payload. Trying to eliminate the final tiny copy at all costs generally produces more cleverness than performance.

splice() is a more general Linux mechanism for moving data between descriptors through a pipe, and can connect paths that sendfile() does not cover. It also adds another set of constraints and bookkeeping. For serving regular files, sendfile() expresses the intention clearly, so that is where I start.

There are cases where a user-space buffer belongs in the path. Compression, encryption in user space, content transformation and checksums may require examining the bytes. Small responses may show no measurable benefit. Portability also matters; this is an operating-system facility, not ISO C acquiring telepathy.

I strongly prefer the kernel-assisted path when the application is merely forwarding an unchanged large file. The preference comes after profiling, not before. In my test it lowered CPU use while maintaining throughput, which leaves the processor available for actual work. The original loop remains easier to explain, but explaining why a server copies every byte twice is not especially satisfying.

After changing a small server to use an event loop, I began with select(). It is portable, familiar and perfectly adequate for a small number of descriptors. It also requires rebuilding descriptor sets and asking the kernel about the whole set on every pass, even when nearly everything is idle.

Linux has a more suitable interface for this shape of workload: epoll. The application creates an epoll instance, registers descriptors with epoll_ctl(), and waits for ready events with epoll_wait(). The interest list remains in the kernel, so the program changes it only when a connection is added, removed or needs different notifications.

The practical difference is that the returned array contains ready items rather than one bit for every descriptor I might care about. With thousands of quiet sockets and a handful of active ones, work follows activity more closely. This does not make the network faster. It removes repeated bookkeeping around sockets that had nothing to say.

I store a pointer or identifier in epoll_event.data so a returned event can find its connection state without another descriptor lookup. Registration then becomes part of the connection lifecycle. A socket enters the interest list after being made nonblocking and leaves before its state is destroyed. Getting that order wrong creates bugs that appear only when descriptors are reused, which is the computer’s way of encouraging humility.

Level-triggered operation resembles select(): readiness continues to be reported while data remains. Edge-triggered operation reports transitions and can reduce repeated notifications, but handlers must drain reads or writes until they return EAGAIN. Forget one partial operation and the connection may wait forever for an edge that already happened.

I started with edge triggering because it sounded faster. I changed back to level triggering while making the protocol correct, then measured. For this service, clarity won. Edge triggering can be appropriate, but it is not a ceremonial badge awarded to serious network programmers.

Errors and hangups require care too. A readiness event may carry several flags, and readable data can coexist with peer shutdown. The handler should consume valid bytes, handle partial writes and close the connection through one well-defined path. epoll reports conditions; it does not design the state machine.

For many Linux sockets, I strongly prefer epoll to stretching select() beyond its pleasant range. I keep the readiness layer small so another platform mechanism could replace it. The valuable architecture is explicit nonblocking state and ownership. epoll is an efficient Linux engine for that architecture, not the architecture itself.

I wrote a small TCP service using one thread per connection. It was the obvious design: accept a socket, start a thread, perform blocking reads and write the response. Each connection looked like a normal sequential program, and the first version was finished before lunch. This is an excellent property in prototypes and an alarming one in architecture.

The service was fine with a few clients. With many mostly idle connections, however, it accumulated threads whose principal activity was owning stacks and waiting. Shutdown required waking them all, shared statistics needed locks, and one slow operation could hold a mutex needed by several innocent connections. I had converted network concurrency into scheduler concurrency whether the work needed it or not.

An event loop starts from a different observation: most connections are not executing. They are waiting for a socket to become readable or writable, a timer to expire, or another operation to complete. One thread can wait for many such events, dispatch a small handler for each ready source, then wait again.

State instead of stacks

The thread-per-connection version stores progress in the thread’s call stack. A function reads a header, calls another function to read a body, and eventually writes a reply. When a read blocks, the stack quietly remembers where execution should continue.

With an event loop, handlers must not block. The connection’s progress becomes explicit state: reading headers, reading a body, preparing a response, or writing remaining bytes. A read handler consumes whatever is available, updates the state and returns. If the message is incomplete, the loop will call it again after the socket becomes readable.

This looks more complicated because it is more explicit. Partial reads and writes were always possible; blocking calls merely kept the unfinished operation on a private stack. The state machine makes those boundaries visible. That visibility is annoying for the first implementation and useful for every timeout, cancellation and protocol error added later.

A connection object in my revised service holds the socket, input and output buffers, parser state and deadlines. The loop owns the objects and invokes them serially. Because one loop thread mutates this state, most of it needs no locking. Ownership becomes a rule I can state rather than a hope expressed through mutexes.

Readiness is not completion

The operating system reports that an operation is likely to make progress. A readable socket may provide only part of a request. A writable socket may accept only part of a response. Handlers therefore loop until the operation would block, preserve what remains, and return control.

Nonblocking mode is essential. If one handler performs a blocking disk lookup or name resolution, every connection assigned to that loop waits behind it. Event-driven code does not eliminate blocking; it makes accidental blocking more expensive and easier to notice when the entire service pauses at once. Very democratic, if not especially helpful.

Timers belong in the same model. The loop computes the next deadline, limits its wait accordingly, then expires idle connections or retries work. I originally used one timer thread that sent messages to connection threads. Replacing that arrangement with a deadline queue removed both threads and a surprising amount of code.

Where threads still fit

An event loop is not an argument for one thread in the entire program. CPU-heavy work can be sent to a bounded worker pool. Blocking interfaces that cannot be integrated with readiness may need workers too. The important constraint is that results return to the owning loop as events, rather than allowing workers to mutate connection state directly.

Multiple loops can run on multiple processors, each owning a set of connections. This keeps the single-owner rule while using more than one core. Distribution and load balancing then become concerns, but they are explicit and usually less tangled than allowing every connection thread to touch global state.

Threads remain attractive when there are few concurrent operations, the blocking flow closely matches the problem, or the platform’s asynchronous facilities are poor. A thread can also be a good unit of isolation for independent subsystems. I do not want to replace clear blocking code with callbacks merely to win a concurrency argument on the Internet.

The event-driven version has its own failure modes. A handler that performs too much work increases latency for everything behind it. Unbounded input or output buffers merely move resource exhaustion from stacks to heaps. Reentrant callbacks can produce transitions the state machine did not expect. Fairness must be designed: a permanently busy socket should not monopolize one pass through the loop.

My current preference

For a service with many mostly waiting connections, I now strongly prefer an event loop with explicit ownership and a small worker pool for unavoidable blocking or CPU work. It uses threads according to available parallelism, not according to the number of clients. It also gives timeouts, cancellation and backpressure natural places in the design.

That preference is qualified. The state machine must be kept small, protocol parsing should be separated from readiness plumbing, and every handler must have a bounded amount of work. If those rules cannot be maintained, a modest number of threads may be safer than one ingenious loop nobody can debug.

My first threaded service was easier to write. The event-loop version is easier to reason about under load, which is a different and more valuable form of easy. It took me several pages of notes to discover that sleeping threads are still resources. The kernel, rather rudely, knew this already.

One of two batch jobs was eating the workstation, but their worker processes came and went too quickly for top to give me a useful total. Watching individual PIDs was the wrong level. I wanted the CPU bill for each job and all of its children.

The cpuacct cgroup controller does that accounting. I put each launcher in its own group before it forked workers. The children inherited the group, so short-lived compiler and helper processes charged CPU time to the same job without a polling script trying to catch them.

cpuacct.usage gives accumulated CPU time in nanoseconds. Reading it before and after a run gave me a simple total for the whole process tree. cpuacct.stat split the charge into user and system time in clock ticks, which showed that the troublesome job was spending most of its time in user code rather than hammering the kernel.

The counters are cumulative, so I record a starting value or create a fresh group for a run. A single reading is not a percentage. Dividing the delta by elapsed wall time gives a rough average, and a multithreaded job can legitimately consume more than one CPU-second per second on a multicore machine.

This also avoids a trap in process snapshots. If a parent exits after launching workers, summing only its current descendants misses work already completed. The group’s counter keeps the charge after those tasks are gone.

Accounting did not slow either job or reserve a processor. That was fine. It showed that the job I had blamed was innocent and the other one used nearly three times the CPU I expected. Only then did scheduler policy seem worth discussing.

For now I am leaving the groups as meters. A reliable number is already an improvement over glaring at whichever PID happens to be at the top of top.