Performance

I wrote a file server in the most direct way possible: read a block into a user-space buffer, then write that block to the socket. The code was simple and, for small files, entirely satisfactory. With large transfers, the process spent an impressive amount of CPU moving bytes it never inspected.

The ordinary path copies file data from the kernel’s page cache into my buffer and then back into kernel-managed socket buffers. It also crosses the system-call boundary repeatedly. The data needs to travel from disk or cache to the network device, but my temporary ownership adds work without adding information.

Linux offers sendfile() for this case. It transfers data between a file descriptor and a socket while keeping the payload out of the application’s address space. The kernel can connect the file and networking paths more directly, reducing copies and context transitions. “Zero copy” is convenient shorthand, not a sworn statement that no hardware or kernel component ever copies a byte.

The loop still has to handle partial progress. sendfile() can transfer fewer bytes than requested, be interrupted, or return EAGAIN for a nonblocking socket. I keep an offset and remaining length in the connection state, request another writable notification when necessary, and continue later. Replacing read() and write() with one call does not replace flow control.

Headers complicate matters slightly. A protocol header may be generated in memory while the body comes from a file. I can send the small header normally and use sendfile() for the payload. Trying to eliminate the final tiny copy at all costs generally produces more cleverness than performance.

splice() is a more general Linux mechanism for moving data between descriptors through a pipe, and can connect paths that sendfile() does not cover. It also adds another set of constraints and bookkeeping. For serving regular files, sendfile() expresses the intention clearly, so that is where I start.

There are cases where a user-space buffer belongs in the path. Compression, encryption in user space, content transformation and checksums may require examining the bytes. Small responses may show no measurable benefit. Portability also matters; this is an operating-system facility, not ISO C acquiring telepathy.

I strongly prefer the kernel-assisted path when the application is merely forwarding an unchanged large file. The preference comes after profiling, not before. In my test it lowered CPU use while maintaining throughput, which leaves the processor available for actual work. The original loop remains easier to explain, but explaining why a server copies every byte twice is not especially satisfying.

My laptop stayed warm with an apparently idle KVM guest running. The guest reported nearly no processor use, and the host showed only small bursts, so I first blamed inaccurate temperature sensors. Sensors are convenient suspects because they cannot write rebuttals.

I stopped services in the guest one by one. A periodic timer user was waking it frequently, but even a minimal guest produced more activity than expected. Virtual timer interrupts, emulated devices and host-side handling all require exits from guest execution. An idle guest is therefore not necessarily an idle host.

Tickless operation helps when the guest kernel can avoid unnecessary periodic ticks and when KVM presents timers accurately enough for longer idle periods. It cannot eliminate device emulation or a program requesting frequent wakeups. The host must also be tickless while idle; saving ticks in one layer and generating them in the other is elaborate bookkeeping, not power management.

After using a tickless guest kernel and disabling a polling monitor, host wakeups dropped and the machine cooled. Timekeeping remained the thing to watch: aggressive experiments are worthless if the guest clock drifts or timers fire late.

My conclusion is to shut down unused guests on battery and to measure host wakeups, not merely guest CPU percentage. For a guest that must remain available, remove polling and use appropriate virtual devices where supported. Virtualization can make a machine appear to be doing nothing in two operating systems simultaneously while the processor is, with admirable professionalism, doing work for both.

I upgraded my workstation to Linux 2.6.23 mainly to try the Completely Fair Scheduler. The machine compiles code, plays music, runs a browser with an unreasonable number of pages, and occasionally hosts a virtual machine. Under a large parallel build, the old setup could make switching desktops feel as though the keyboard had been sent by post.

My first test was not clever. I started a build, dragged windows around and declared the new scheduler smoother. Then I rebooted into the older kernel and it also seemed smooth. Human perception is a wonderful measurement instrument if the desired unit is confidence.

For a better comparison, I repeated the same build with the same compiler jobs, timed it, and introduced an interactive task that periodically needed a small amount of processor time. Total build time changed little. The interesting difference was the delay before the interactive task ran when all cores were busy. Large stalls became less frequent with 2.6.23, though disk activity could still make the desktop miserable for entirely different reasons.

CFS approaches scheduling without the old collection of active and expired priority arrays. It tracks each runnable task’s virtual runtime, an adjusted measure of how much processor service that task has received. The task with the smallest virtual runtime is the one most entitled to run next. Runnable tasks are kept in a red-black tree, making that leftmost choice efficient as tasks arrive, block and wake.

The word “fair” needs care. Equal nice levels should receive roughly equal shares over time; it does not mean every task receives an identical slice at every instant. Nice values weight the virtual runtime, so a task with lower priority accumulates entitlement differently and receives a smaller share. Sleeper behavior emerges because a task that blocks stops consuming processor time. When it wakes, it may be behind its competitors and therefore run promptly, within limits intended to prevent absurd advantages.

My naive mental model was that CFS simply gives tiny slices to interactive programs and large slices to compilers. That is not how it identifies interactivity. There is no reliable label saying “this is the music player.” Instead, the accounting rewards tasks that use little processor time and wake to perform brief work. A music player, terminal or window manager often behaves that way, but so can a less noble process.

The scheduler also has to balance two competing costs. Switching too often improves apparent responsiveness but loses time to context switches and cache disruption. Waiting too long lets a runnable task monopolise a processor. CFS uses a target scheduling latency and derives slices from the number and weights of runnable tasks, subject to a minimum granularity. As the run queue grows, it cannot promise that every task runs within the same tiny interval without turning the processor into a context-switching demonstration.

This explains one of my results. With a modest build load, interactive delays stayed short. With a deliberately ridiculous number of runnable jobs, latency grew again. Fair scheduling cannot manufacture processor time. It can distribute scarcity coherently, which is less exciting but more useful.

Another misleading result came from I/O. During linking, the desktop paused even though CFS was selecting processor tasks correctly. The disk queue was saturated and applications blocked waiting for files. Changing the CPU scheduler did not cure that. Likewise, swapping overwhelmed every subtle scheduler difference. Before blaming CFS, I now check whether the delayed task is actually runnable rather than sleeping on storage.

Nice levels behaved more predictably in my tests than before, but group fairness remains a practical concern. If one user starts one processor-bound task and another starts twenty, scheduling every task independently can grant the second user most of the machine. There is work around grouping tasks, but the default desktop case still mostly exposes per-task fairness. The name should not be read as a constitutional guarantee.

I also tried changing scheduler tunables because numbers invite interference. Reducing latency made window feedback slightly sharper under synthetic load and increased context switches. Increasing it helped throughput by an amount too small to distinguish confidently while making the terminal occasionally sticky. The defaults survived my experiments, an outcome that wounded my pride but spared my configuration files.

After a week, I am keeping 2.6.23. The improvement is not that compilations finish magically sooner. They do not. The improvement is that short, waking tasks more often get service without elaborate interactivity heuristics, and nice values map onto a cleaner accounting model. That makes a loaded workstation feel less capricious.

The practical conclusion is to test CFS with repeatable mixed workloads, not window wiggling alone. Separate CPU delay from disk delay, compare distributions of latency rather than one lucky run, and resist tuning until a real workload demonstrates a problem. CFS is a substantial scheduler change, not a performance patch that makes every number smaller. On my machine it makes contention more orderly. Given what computers usually do under pressure, orderly is a fairly ambitious achievement.

I built a recent kernel with tickless idle enabled because the laptop fan had begun running while the machine did apparently nothing. I expected the new kernel to become silent by virtue of one configuration option. It did not. The fan is apparently not impressed by release notes.

My naive test was to leave a terminal open, wait a minute, and compare the battery estimate. That number wandered by nearly an hour between readings. I then watched processor usage, which stayed close to zero while the fan continued. Neither test told me what was waking the processor.

The periodic timer tick normally interrupts each processor at a fixed frequency so the kernel can account for time and perform scheduling work. During idle periods, those interrupts wake an otherwise sleeping processor for very little purpose. Tickless idle programs the next timer event instead, allowing a longer uninterrupted sleep. It cannot help if an application or driver requests a timer every few milliseconds.

On my machine, an application polling for status updates was doing exactly that. Closing it reduced wakeups dramatically and the fan eventually slowed. A network driver also produced regular activity when the interface was unused. Disabling the interface for a test separated that problem from the timer configuration.

Measuring several idle intervals also mattered. One short sample could be dominated by mail checking, disk flushing or my own impatient mouse movement.

There are limits to the name. Tickless does not mean the kernel never uses timer interrupts, and the current work mainly avoids ticks while a processor is idle. A busy processor still needs scheduling and accounting. Hardware timer support matters too; unreliable timers can turn a power improvement into missed timeouts or clocks that drift.

The practical lesson is that tickless operation removes one systematic source of wakeups. It does not forgive programs that poll constantly, nor drivers that chatter with hardware for entertainment. I am keeping it enabled because idle residency improved and battery measurements over several complete runs are better. But the useful debugging question is now “who wakes the processor?” rather than “is CPU usage low?” Zero percent can be composed of thousands of tiny interruptions, which is a very computer-like definition of resting.