Blog

A multithreaded parser scaled nicely to two workers and then barely improved. I inspected locks in my code and found no obvious villain. Eventually a profile pointed at allocation and deallocation, which felt unfair because malloc() had not appeared on my list of interesting algorithms.

An allocator manages more than a pointer advancing through memory. It finds suitably sized free regions, records metadata, returns memory to reusable pools and sometimes asks the operating system for more pages. In multiple threads it must also protect shared structures. A program creating millions of short-lived objects can therefore spend substantial time contending inside code it never explicitly called.

The system allocator already uses techniques to reduce this cost. Implementations may maintain size classes, bins and multiple arenas so not every allocation takes one global lock. Alternative allocators such as tcmalloc and jemalloc make different tradeoffs involving per-thread caches, fragmentation and concurrency. Swapping one in can be an excellent experiment, but it is not a substitute for understanding the allocation pattern.

My parser allocated a small object for every token, then freed the whole tree after processing a request. The lifetimes were almost identical. A simple region allocator fit better: reserve larger blocks, hand out aligned pieces by moving a pointer, and release the region at once when the request completes. Individual deallocation disappears because the ownership model says nothing outlives the request.

This improved both speed and clarity. The gain did not come from a magical allocator; it came from expressing lifetime in the data structure. It also introduced a strict rule: pointers into the region cannot escape. Violating that rule turns cleanup into a mass dangling-pointer production line, which is efficient in the wrong direction.

Object pools are useful when objects have uniform shape and repeated lifetimes, but I use them cautiously. A pool retains memory, complicates construction and may conceal an unbounded queue. Per-thread caches can reduce contention while increasing total memory consumption. Peak resident size matters as much as allocations per second on a shared machine.

Measurement must include realistic concurrency and duration. A short benchmark can reward an allocator for keeping every page. A long-running service eventually reveals fragmentation and cache growth. I watch CPU profiles, allocation counts and resident memory rather than treating one throughput number as a complete biography.

I prefer changing ownership and allocation frequency before replacing the general allocator. When patterns are genuinely general and contention remains measurable, testing another mature allocator is reasonable. In this case, the region was the stronger result because it made the program’s lifetime visible. I had gone looking for a faster malloc() and found that asking for less was cheaper.

I wrote a file server in the most direct way possible: read a block into a user-space buffer, then write that block to the socket. The code was simple and, for small files, entirely satisfactory. With large transfers, the process spent an impressive amount of CPU moving bytes it never inspected.

The ordinary path copies file data from the kernel’s page cache into my buffer and then back into kernel-managed socket buffers. It also crosses the system-call boundary repeatedly. The data needs to travel from disk or cache to the network device, but my temporary ownership adds work without adding information.

Linux offers sendfile() for this case. It transfers data between a file descriptor and a socket while keeping the payload out of the application’s address space. The kernel can connect the file and networking paths more directly, reducing copies and context transitions. “Zero copy” is convenient shorthand, not a sworn statement that no hardware or kernel component ever copies a byte.

The loop still has to handle partial progress. sendfile() can transfer fewer bytes than requested, be interrupted, or return EAGAIN for a nonblocking socket. I keep an offset and remaining length in the connection state, request another writable notification when necessary, and continue later. Replacing read() and write() with one call does not replace flow control.

Headers complicate matters slightly. A protocol header may be generated in memory while the body comes from a file. I can send the small header normally and use sendfile() for the payload. Trying to eliminate the final tiny copy at all costs generally produces more cleverness than performance.

splice() is a more general Linux mechanism for moving data between descriptors through a pipe, and can connect paths that sendfile() does not cover. It also adds another set of constraints and bookkeeping. For serving regular files, sendfile() expresses the intention clearly, so that is where I start.

There are cases where a user-space buffer belongs in the path. Compression, encryption in user space, content transformation and checksums may require examining the bytes. Small responses may show no measurable benefit. Portability also matters; this is an operating-system facility, not ISO C acquiring telepathy.

I strongly prefer the kernel-assisted path when the application is merely forwarding an unchanged large file. The preference comes after profiling, not before. In my test it lowered CPU use while maintaining throughput, which leaves the processor available for actual work. The original loop remains easier to explain, but explaining why a server copies every byte twice is not especially satisfying.

After changing a small server to use an event loop, I began with select(). It is portable, familiar and perfectly adequate for a small number of descriptors. It also requires rebuilding descriptor sets and asking the kernel about the whole set on every pass, even when nearly everything is idle.

Linux has a more suitable interface for this shape of workload: epoll. The application creates an epoll instance, registers descriptors with epoll_ctl(), and waits for ready events with epoll_wait(). The interest list remains in the kernel, so the program changes it only when a connection is added, removed or needs different notifications.

The practical difference is that the returned array contains ready items rather than one bit for every descriptor I might care about. With thousands of quiet sockets and a handful of active ones, work follows activity more closely. This does not make the network faster. It removes repeated bookkeeping around sockets that had nothing to say.

I store a pointer or identifier in epoll_event.data so a returned event can find its connection state without another descriptor lookup. Registration then becomes part of the connection lifecycle. A socket enters the interest list after being made nonblocking and leaves before its state is destroyed. Getting that order wrong creates bugs that appear only when descriptors are reused, which is the computer’s way of encouraging humility.

Level-triggered operation resembles select(): readiness continues to be reported while data remains. Edge-triggered operation reports transitions and can reduce repeated notifications, but handlers must drain reads or writes until they return EAGAIN. Forget one partial operation and the connection may wait forever for an edge that already happened.

I started with edge triggering because it sounded faster. I changed back to level triggering while making the protocol correct, then measured. For this service, clarity won. Edge triggering can be appropriate, but it is not a ceremonial badge awarded to serious network programmers.

Errors and hangups require care too. A readiness event may carry several flags, and readable data can coexist with peer shutdown. The handler should consume valid bytes, handle partial writes and close the connection through one well-defined path. epoll reports conditions; it does not design the state machine.

For many Linux sockets, I strongly prefer epoll to stretching select() beyond its pleasant range. I keep the readiness layer small so another platform mechanism could replace it. The valuable architecture is explicit nonblocking state and ownership. epoll is an efficient Linux engine for that architecture, not the architecture itself.

I wrote a small TCP service using one thread per connection. It was the obvious design: accept a socket, start a thread, perform blocking reads and write the response. Each connection looked like a normal sequential program, and the first version was finished before lunch. This is an excellent property in prototypes and an alarming one in architecture.

The service was fine with a few clients. With many mostly idle connections, however, it accumulated threads whose principal activity was owning stacks and waiting. Shutdown required waking them all, shared statistics needed locks, and one slow operation could hold a mutex needed by several innocent connections. I had converted network concurrency into scheduler concurrency whether the work needed it or not.

An event loop starts from a different observation: most connections are not executing. They are waiting for a socket to become readable or writable, a timer to expire, or another operation to complete. One thread can wait for many such events, dispatch a small handler for each ready source, then wait again.

State instead of stacks

The thread-per-connection version stores progress in the thread’s call stack. A function reads a header, calls another function to read a body, and eventually writes a reply. When a read blocks, the stack quietly remembers where execution should continue.

With an event loop, handlers must not block. The connection’s progress becomes explicit state: reading headers, reading a body, preparing a response, or writing remaining bytes. A read handler consumes whatever is available, updates the state and returns. If the message is incomplete, the loop will call it again after the socket becomes readable.

This looks more complicated because it is more explicit. Partial reads and writes were always possible; blocking calls merely kept the unfinished operation on a private stack. The state machine makes those boundaries visible. That visibility is annoying for the first implementation and useful for every timeout, cancellation and protocol error added later.

A connection object in my revised service holds the socket, input and output buffers, parser state and deadlines. The loop owns the objects and invokes them serially. Because one loop thread mutates this state, most of it needs no locking. Ownership becomes a rule I can state rather than a hope expressed through mutexes.

Readiness is not completion

The operating system reports that an operation is likely to make progress. A readable socket may provide only part of a request. A writable socket may accept only part of a response. Handlers therefore loop until the operation would block, preserve what remains, and return control.

Nonblocking mode is essential. If one handler performs a blocking disk lookup or name resolution, every connection assigned to that loop waits behind it. Event-driven code does not eliminate blocking; it makes accidental blocking more expensive and easier to notice when the entire service pauses at once. Very democratic, if not especially helpful.

Timers belong in the same model. The loop computes the next deadline, limits its wait accordingly, then expires idle connections or retries work. I originally used one timer thread that sent messages to connection threads. Replacing that arrangement with a deadline queue removed both threads and a surprising amount of code.

Where threads still fit

An event loop is not an argument for one thread in the entire program. CPU-heavy work can be sent to a bounded worker pool. Blocking interfaces that cannot be integrated with readiness may need workers too. The important constraint is that results return to the owning loop as events, rather than allowing workers to mutate connection state directly.

Multiple loops can run on multiple processors, each owning a set of connections. This keeps the single-owner rule while using more than one core. Distribution and load balancing then become concerns, but they are explicit and usually less tangled than allowing every connection thread to touch global state.

Threads remain attractive when there are few concurrent operations, the blocking flow closely matches the problem, or the platform’s asynchronous facilities are poor. A thread can also be a good unit of isolation for independent subsystems. I do not want to replace clear blocking code with callbacks merely to win a concurrency argument on the Internet.

The event-driven version has its own failure modes. A handler that performs too much work increases latency for everything behind it. Unbounded input or output buffers merely move resource exhaustion from stacks to heaps. Reentrant callbacks can produce transitions the state machine did not expect. Fairness must be designed: a permanently busy socket should not monopolize one pass through the loop.

My current preference

For a service with many mostly waiting connections, I now strongly prefer an event loop with explicit ownership and a small worker pool for unavoidable blocking or CPU work. It uses threads according to available parallelism, not according to the number of clients. It also gives timeouts, cancellation and backpressure natural places in the design.

That preference is qualified. The state machine must be kept small, protocol parsing should be separated from readiness plumbing, and every handler must have a bounded amount of work. If those rules cannot be maintained, a modest number of threads may be safer than one ingenious loop nobody can debug.

My first threaded service was easier to write. The event-loop version is easier to reason about under load, which is a different and more valuable form of easy. It took me several pages of notes to discover that sleeping threads are still resources. The kernel, rather rudely, knew this already.

One of two batch jobs was eating the workstation, but their worker processes came and went too quickly for top to give me a useful total. Watching individual PIDs was the wrong level. I wanted the CPU bill for each job and all of its children.

The cpuacct cgroup controller does that accounting. I put each launcher in its own group before it forked workers. The children inherited the group, so short-lived compiler and helper processes charged CPU time to the same job without a polling script trying to catch them.

cpuacct.usage gives accumulated CPU time in nanoseconds. Reading it before and after a run gave me a simple total for the whole process tree. cpuacct.stat split the charge into user and system time in clock ticks, which showed that the troublesome job was spending most of its time in user code rather than hammering the kernel.

The counters are cumulative, so I record a starting value or create a fresh group for a run. A single reading is not a percentage. Dividing the delta by elapsed wall time gives a rough average, and a multithreaded job can legitimately consume more than one CPU-second per second on a multicore machine.

This also avoids a trap in process snapshots. If a parent exits after launching workers, summing only its current descendants misses work already completed. The group’s counter keeps the charge after those tasks are gone.

Accounting did not slow either job or reserve a processor. That was fine. It showed that the job I had blamed was innocent and the other one used nearly three times the CPU I expected. Only then did scheduler policy seem worth discussing.

For now I am leaving the groups as meters. A reliable number is already an improvement over glaring at whichever PID happens to be at the top of top.