Kernel

My laptop stayed warm with an apparently idle KVM guest running. The guest reported nearly no processor use, and the host showed only small bursts, so I first blamed inaccurate temperature sensors. Sensors are convenient suspects because they cannot write rebuttals.

I stopped services in the guest one by one. A periodic timer user was waking it frequently, but even a minimal guest produced more activity than expected. Virtual timer interrupts, emulated devices and host-side handling all require exits from guest execution. An idle guest is therefore not necessarily an idle host.

Tickless operation helps when the guest kernel can avoid unnecessary periodic ticks and when KVM presents timers accurately enough for longer idle periods. It cannot eliminate device emulation or a program requesting frequent wakeups. The host must also be tickless while idle; saving ticks in one layer and generating them in the other is elaborate bookkeeping, not power management.

After using a tickless guest kernel and disabling a polling monitor, host wakeups dropped and the machine cooled. Timekeeping remained the thing to watch: aggressive experiments are worthless if the guest clock drifts or timers fire late.

My conclusion is to shut down unused guests on battery and to measure host wakeups, not merely guest CPU percentage. For a guest that must remain available, remove polling and use appropriate virtual devices where supported. Virtualization can make a machine appear to be doing nothing in two operating systems simultaneously while the processor is, with admirable professionalism, doing work for both.

I wanted a compile job and its helper processes to stop overwhelming everything else on my test machine. Renicing the shell helped until the build launched processes with behavior I had not accounted for, and it did nothing about memory consumption.

Linux 2.6.24 includes the control groups framework, so I enabled the relevant options and mounted a control-group filesystem for testing. My first attempt was to write the compiler’s process ID into a group after the build had started. Some children were already elsewhere, and the result was inconsistent enough to look supernatural.

A control group is a hierarchy used to classify tasks. Controllers attach particular resource policies or accounting to that hierarchy. The framework itself is not one universal “make this process small” knob. CPU scheduling, processor sets and other controllers expose different files and semantics. A task belongs to a group in a hierarchy, and children normally inherit membership, so classification should happen before launching the workload.

I created a group, moved my test shell into it, and then started the build from that shell. Its descendants appeared in the same group. Applying CPU scheduling controls made the machine more responsive under contention without changing every process individually. Memory control is a separate matter and depends on kernel configuration; enabling it also has overhead that should not be waved away.

The hierarchy is powerful but easy to design badly. Groups may represent users, services or workloads, and those choices do not always nest neatly. Multiple hierarchies can attach different controllers, which provides flexibility at the cost of another opportunity to create an administrative puzzle.

For now I regard cgroups as kernel machinery for building resource-management policy, not as a finished desktop feature. The interface is low-level and sharp-edged. Tools will need to create groups, place tasks early, apply limits coherently and clean up afterward.

My practical conclusion is to start with one measured problem and one controller. Verify membership before interpreting results, and do not confuse accounting with enforcement. Cgroups can organize a family of processes in a way nice alone cannot. They can also organize one’s mistakes into a tidy hierarchy, which is progress of a sort.

I upgraded my workstation to Linux 2.6.23 mainly to try the Completely Fair Scheduler. The machine compiles code, plays music, runs a browser with an unreasonable number of pages, and occasionally hosts a virtual machine. Under a large parallel build, the old setup could make switching desktops feel as though the keyboard had been sent by post.

My first test was not clever. I started a build, dragged windows around and declared the new scheduler smoother. Then I rebooted into the older kernel and it also seemed smooth. Human perception is a wonderful measurement instrument if the desired unit is confidence.

For a better comparison, I repeated the same build with the same compiler jobs, timed it, and introduced an interactive task that periodically needed a small amount of processor time. Total build time changed little. The interesting difference was the delay before the interactive task ran when all cores were busy. Large stalls became less frequent with 2.6.23, though disk activity could still make the desktop miserable for entirely different reasons.

CFS approaches scheduling without the old collection of active and expired priority arrays. It tracks each runnable task’s virtual runtime, an adjusted measure of how much processor service that task has received. The task with the smallest virtual runtime is the one most entitled to run next. Runnable tasks are kept in a red-black tree, making that leftmost choice efficient as tasks arrive, block and wake.

The word “fair” needs care. Equal nice levels should receive roughly equal shares over time; it does not mean every task receives an identical slice at every instant. Nice values weight the virtual runtime, so a task with lower priority accumulates entitlement differently and receives a smaller share. Sleeper behavior emerges because a task that blocks stops consuming processor time. When it wakes, it may be behind its competitors and therefore run promptly, within limits intended to prevent absurd advantages.

My naive mental model was that CFS simply gives tiny slices to interactive programs and large slices to compilers. That is not how it identifies interactivity. There is no reliable label saying “this is the music player.” Instead, the accounting rewards tasks that use little processor time and wake to perform brief work. A music player, terminal or window manager often behaves that way, but so can a less noble process.

The scheduler also has to balance two competing costs. Switching too often improves apparent responsiveness but loses time to context switches and cache disruption. Waiting too long lets a runnable task monopolise a processor. CFS uses a target scheduling latency and derives slices from the number and weights of runnable tasks, subject to a minimum granularity. As the run queue grows, it cannot promise that every task runs within the same tiny interval without turning the processor into a context-switching demonstration.

This explains one of my results. With a modest build load, interactive delays stayed short. With a deliberately ridiculous number of runnable jobs, latency grew again. Fair scheduling cannot manufacture processor time. It can distribute scarcity coherently, which is less exciting but more useful.

Another misleading result came from I/O. During linking, the desktop paused even though CFS was selecting processor tasks correctly. The disk queue was saturated and applications blocked waiting for files. Changing the CPU scheduler did not cure that. Likewise, swapping overwhelmed every subtle scheduler difference. Before blaming CFS, I now check whether the delayed task is actually runnable rather than sleeping on storage.

Nice levels behaved more predictably in my tests than before, but group fairness remains a practical concern. If one user starts one processor-bound task and another starts twenty, scheduling every task independently can grant the second user most of the machine. There is work around grouping tasks, but the default desktop case still mostly exposes per-task fairness. The name should not be read as a constitutional guarantee.

I also tried changing scheduler tunables because numbers invite interference. Reducing latency made window feedback slightly sharper under synthetic load and increased context switches. Increasing it helped throughput by an amount too small to distinguish confidently while making the terminal occasionally sticky. The defaults survived my experiments, an outcome that wounded my pride but spared my configuration files.

After a week, I am keeping 2.6.23. The improvement is not that compilations finish magically sooner. They do not. The improvement is that short, waking tasks more often get service without elaborate interactivity heuristics, and nice values map onto a cleaner accounting model. That makes a loaded workstation feel less capricious.

The practical conclusion is to test CFS with repeatable mixed workloads, not window wiggling alone. Separate CPU delay from disk delay, compare distributions of latency rather than one lucky run, and resist tuning until a real workload demonstrates a problem. CFS is a substantial scheduler change, not a performance patch that makes every number smaller. On my machine it makes contention more orderly. Given what computers usually do under pressure, orderly is a fairly ambitious achievement.

I built a recent kernel with tickless idle enabled because the laptop fan had begun running while the machine did apparently nothing. I expected the new kernel to become silent by virtue of one configuration option. It did not. The fan is apparently not impressed by release notes.

My naive test was to leave a terminal open, wait a minute, and compare the battery estimate. That number wandered by nearly an hour between readings. I then watched processor usage, which stayed close to zero while the fan continued. Neither test told me what was waking the processor.

The periodic timer tick normally interrupts each processor at a fixed frequency so the kernel can account for time and perform scheduling work. During idle periods, those interrupts wake an otherwise sleeping processor for very little purpose. Tickless idle programs the next timer event instead, allowing a longer uninterrupted sleep. It cannot help if an application or driver requests a timer every few milliseconds.

On my machine, an application polling for status updates was doing exactly that. Closing it reduced wakeups dramatically and the fan eventually slowed. A network driver also produced regular activity when the interface was unused. Disabling the interface for a test separated that problem from the timer configuration.

Measuring several idle intervals also mattered. One short sample could be dominated by mail checking, disk flushing or my own impatient mouse movement.

There are limits to the name. Tickless does not mean the kernel never uses timer interrupts, and the current work mainly avoids ticks while a processor is idle. A busy processor still needs scheduling and accounting. Hardware timer support matters too; unreliable timers can turn a power improvement into missed timeouts or clocks that drift.

The practical lesson is that tickless operation removes one systematic source of wakeups. It does not forgive programs that poll constantly, nor drivers that chatter with hardware for entertainment. I am keeping it enabled because idle residency improved and battery measurements over several complete runs are better. But the useful debugging question is now “who wakes the processor?” rather than “is CPU usage low?” Zero percent can be composed of thousands of tiny interruptions, which is a very computer-like definition of resting.

I reviewed a kernel path today that checked a pointer, acquired a spinlock, then used the pointer. It looked economical. It was also wrong.

if (device->queue) {
    spin_lock(&device->lock);
    flush_queue(device->queue);
    spin_unlock(&device->lock);
}

Another CPU can clear and free queue between the check and the lock. The check does not reserve anything; it merely records a fact that may expire immediately.

The simple repair is to protect both observation and use with the same lock:

spin_lock(&device->lock);
if (device->queue)
    flush_queue(device->queue);
spin_unlock(&device->lock);

This is safe only if flush_queue cannot sleep and does not acquire locks in an incompatible order. Spinlocks disable preemption in the relevant context; they are not permission to perform leisurely work. If flushing can block, I need a different design, perhaps detach the queue under the spinlock and process it later with appropriate lifetime ownership.

I prefer establishing ownership under one clearly documented lock rather than scattering “probably still valid” checks. The caveat is contention: a correct giant critical section can turn several processors into an expensive single processor.

The lesson is not “add locks.” It is “name the invariant, then guard every transition that can violate it.” Locks without an invariant are just punctuation with cache-line traffic.