Kernel on Roberto Selbach

Counting Bits Like the Kernel

Tue, 09 Nov 2010 20:39:00 +0000

Kernel code contains algorithms that look unnecessarily peculiar until the constraints are visible. Population count, the number of set bits in a machine word, is a good example.

The obvious C version examines every bit:

unsigned int count_bits(unsigned long word)
{
 unsigned int n = 0;

 while (word) {
 n += word & 1;
 word >>= 1;
 }
 return n;
}

This performs one iteration for every bit position up to the highest set bit. A sparse bitmap still pays for all the zeros in between.

Brian Kernighan’s familiar operation removes the lowest set bit at each iteration:

unsigned int count_bits(unsigned long word)
{
 unsigned int n = 0;

 while (word) {
 word &= word - 1;
 n++;
 }
 return n;
}

Subtracting one flips the lowest set bit to zero and turns lower zeros into ones. The & keeps everything above that bit and clears the changed suffix. The loop therefore runs once per set bit.

That is excellent for sparse words and less impressive for dense ones. Kernels also use table lookups and architecture-specific instructions where available. A byte table has predictable work but adds memory access; a processor instruction can win decisively but needs dispatch or compile-time selection. There is no universally fastest source fragment independent of data and machine.

Word width matters too. Kernel types and helpers make that choice explicit, while a casual userspace benchmark can accidentally compare different widths and announce a victory produced by less work.

I benchmarked the two loops over generated sparse and dense arrays, keeping generation outside the timed section. As expected, the clear-lowest-bit version dominated sparse input. Dense input narrowed the gap considerably. Repeating one constant word produced lovely numbers and measured the compiler more than the routine, so I stopped doing that.

The broader lesson from kernel work is to understand the representation before admiring the trick. Bitmaps pack state compactly and permit word-at-a-time operations, but contention, cache placement, and scan direction can matter more than shaving an instruction from population count.

I like this algorithm because its mechanism fits in one sentence and its performance caveat requires another. Most optimization stories should have both sentences. The ones with only the first usually end in a benchmark that accidentally proves the author’s laptop exists.

perf Turns Counters Into Questions

Tue, 08 Dec 2009 18:53:00 +0000

I had two versions of a parser. One finished faster, so I declared victory and nearly deleted the slower one. Before doing that, I tried the new perf tools included with recent kernels and discovered that my explanation for the improvement was wrong.

The kernel’s performance-counter infrastructure provides a common way to measure hardware and software events. The perf tool can count events for a command, sample execution and report where those samples landed. Instead of beginning with a profiler tied to one processor model, I can ask through one kernel interface and use the events available on this machine.

perf stat is a useful first question. It reports elapsed time along with counts such as cycles, instructions, context switches and page faults. Ratios matter more than isolated totals. Instructions per cycle can suggest whether the processor is retiring useful work efficiently, while cache-related events may explain why an apparently smaller algorithm still stalls.

My faster parser did not execute dramatically fewer instructions. It incurred fewer cache misses because its data was laid out more compactly. I had credited a clever branch change that happened nearby. The benchmark result was real; my story about it was fan fiction.

Sampling answers a different question. perf record periodically captures the current instruction pointer, and perf report aggregates samples by symbol. With suitable symbols, hot functions become visible without instrumenting every call. Sampling has overhead and statistical uncertainty, but it is often much less disruptive than logging entry and exit around a hot path.

Counters are constrained resources. The processor can measure only a limited number simultaneously, so events may be multiplexed. Some events are model-specific, and virtualized or restricted environments may expose less. A cache-miss count without knowing which cache or how the event is defined is an attractive number with an uncertain biography.

I also avoid optimizing from one profile. Workload, input size, compiler options and machine state all matter. I run repeated measurements, preserve the input and compare complete behavior. A ten-percent gain in a microbenchmark is not useful if the changed layout doubles memory for the real service.

I strongly prefer starting performance work with perf stat, then sampling when the totals suggest a question. ftrace remains better for many scheduling and kernel-flow investigations; perf is especially convenient for connecting program hotspots to processor and kernel counters. Neither tool replaces understanding, but both are considerably more reliable than staring at source until one loop begins to look guilty.

KMS Moves the Screen Into the Kernel

Fri, 17 Jul 2009 17:34:00 +0000

My test laptop changes display modes several times while booting: firmware text, a console mode, the X server’s preferred resolution, and sometimes another mode after resume. Each transition blanks the screen long enough to make me wonder whether the machine is considering a different profession.

Kernel mode setting moves responsibility for configuring display modes into the kernel graphics driver. Traditionally the X server performed that setup from user space. With KMS, the kernel knows about connectors, CRTCs, encoders, framebuffers and modes early enough to establish the display before X starts.

That early ownership makes boot transitions smoother, but the architectural benefit is larger. Suspend and resume already require kernel coordination with devices. If display state is also managed there, restoration does not depend on a particular user-space server reconstructing everything after the fact. Panic and virtual-terminal paths can retain a usable framebuffer as well.

The Direct Rendering Manager interface exposes the resources to user space. A display server can discover connectors and available modes, choose how framebuffers map to scanout hardware, and request changes through a common kernel boundary. The kernel arbitrates access to the device rather than allowing unrelated programs to reprogram it optimistically.

KMS also fits the broader memory-management work in graphics drivers. Rendering buffers and scanout buffers share hardware constraints, and transitions between them need synchronization. Putting mode setting beside DRM’s device and buffer management reduces the number of components that must agree through private arrangements.

My Intel machine with a recent kernel now reaches its native mode earlier and switches to X with less flashing. Resume has also become more predictable. This is not universal evidence. Driver support varies, hardware contains creative surprises, and a failed early mode set can replace a merely ugly boot with a completely dark one.

Debugging changes too. Problems that once belonged entirely to the X log may now begin in kernel messages and driver parameters. Distributions need rescue paths and conservative defaults while support matures. The mechanism is better placed, but moving code into the kernel also raises the cost of mistakes.

I strongly prefer KMS as the long-term division of responsibility. Display hardware is a shared device whose state matters before, during and after the graphical session; the kernel is the natural owner. For current machines I enable it where the driver is known to work and keep a boot option that disables it. Architectural conviction is comforting, but a visible console is more immediately useful.

ftrace Before More printk

Fri, 26 Jun 2009 20:51:00 +0000

I was chasing a latency spike in a kernel path and began adding printk() calls. This is the kernel equivalent of looking for a gas leak with a candle: familiar, illuminating and likely to influence the situation under investigation.

Recent kernels include ftrace, with controls under debugfs, normally in /sys/kernel/debug/tracing. It started as a function tracer and has grown useful tracing modes and events. The useful bit is that tracing can often be selected and controlled at runtime rather than compiled into a private forest of diagnostic messages.

The function tracer records kernel function activity in per-CPU ring buffers. Filters can restrict collection to relevant functions, which matters because tracing everything creates enormous output and changes timing. The function-graph tracer adds entry and return relationships, making nested execution and duration easier to see.

For my problem, the scheduler tracing facilities were more revealing than raw function calls. They showed when the task stopped running, which task replaced it and when it returned. The delay I had blamed on one function was mostly time spent waiting to be scheduled. A timestamp around the function could show elapsed time, but not explain where that time went.

Ring buffers are a sensible mechanism for this job. Each CPU can record events with less cross-CPU contention, and old entries can be overwritten when configured as a rolling trace. I can leave tracing ready, reproduce the spike and stop collection soon afterward. The final seconds are usually more valuable than a complete account of the machine booting and becoming bored.

Tracing still has overhead. Function tracing a hot path produces work, filters cost something, and reading the trace while collecting it can perturb results. I begin with the narrowest event set that can answer the question and compare behavior with tracing disabled. A trace is evidence gathered by an instrument, not the unobserved universe itself.

printk() remains appropriate for durable diagnostics that users or administrators need. It is poor as an ad hoc high-volume performance recorder: messages contend for shared facilities, flood logs and require source edits to move the probe. Temporary tracing and permanent logging solve different problems.

For kernel execution and scheduling questions, I try ftrace first. A carefully placed message can fill in semantic information the trace lacks. This is faster and leaves less debris in the source, including my traditional final patch removing forty-seven increasingly desperate print statements.

Cgroups Are Accounting Before Containment

Thu, 22 Jan 2009 20:16:00 +0000

One of two batch jobs was eating the workstation, but their worker processes came and went too quickly for top to give me a useful total. Watching individual PIDs was the wrong level. I wanted the CPU bill for each job and all of its children.

The cpuacct cgroup controller does that accounting. I put each launcher in its own group before it forked workers. The children inherited the group, so short-lived compiler and helper processes charged CPU time to the same job without a polling script trying to catch them.

cpuacct.usage gives accumulated CPU time in nanoseconds. Reading it before and after a run gave me a simple total for the whole process tree. cpuacct.stat split the charge into user and system time in clock ticks, which showed that the troublesome job was spending most of its time in user code rather than hammering the kernel.

The counters are cumulative, so I record a starting value or create a fresh group for a run. A single reading is not a percentage. Dividing the delta by elapsed wall time gives a rough average, and a multithreaded job can legitimately consume more than one CPU-second per second on a multicore machine.

This also avoids a trap in process snapshots. If a parent exits after launching workers, summing only its current descendants misses work already completed. The group’s counter keeps the charge after those tasks are gone.

Accounting did not slow either job or reserve a processor. That was fine. It showed that the job I had blamed was innocent and the other one used nearly three times the CPU I expected. Only then did scheduler policy seem worth discussing.

For now I am leaving the groups as meters. A reliable number is already an improvement over glaring at whichever PID happens to be at the top of top.

Idle Host, Busy Guest

Wed, 25 Jun 2008 20:45:00 +0000

My laptop stayed warm with an apparently idle KVM guest running. The guest reported nearly no processor use, and the host showed only small bursts, so I first blamed inaccurate temperature sensors. Sensors are convenient suspects because they cannot write rebuttals.

I stopped services in the guest one by one. A periodic timer user was waking it frequently, but even a minimal guest produced more activity than expected. Virtual timer interrupts, emulated devices and host-side handling all require exits from guest execution. An idle guest is therefore not necessarily an idle host.

Tickless operation helps when the guest kernel can avoid unnecessary periodic ticks and when KVM presents timers accurately enough for longer idle periods. It cannot eliminate device emulation or a program requesting frequent wakeups. The host must also be tickless while idle; saving ticks in one layer and generating them in the other is elaborate bookkeeping, not power management.

After using a tickless guest kernel and disabling a polling monitor, host wakeups dropped and the machine cooled. Timekeeping remained the thing to watch: aggressive experiments are worthless if the guest clock drifts or timers fire late.

My conclusion is to shut down unused guests on battery and to measure host wakeups, not merely guest CPU percentage. For a guest that must remain available, remove polling and use appropriate virtual devices where supported. Virtualization can make a machine appear to be doing nothing in two operating systems simultaneously while the processor is, with admirable professionalism, doing work for both.

Trying Control Groups After 2.6.24

Fri, 04 Apr 2008 19:15:00 +0000

I wanted a compile job and its helper processes to stop overwhelming everything else on my test machine. Renicing the shell helped until the build launched processes with behavior I had not accounted for, and it did nothing about memory consumption.

Linux 2.6.24 includes the control groups framework, so I enabled the relevant options and mounted a control-group filesystem for testing. My first attempt was to write the compiler’s process ID into a group after the build had started. Some children were already elsewhere, and the result was inconsistent enough to look supernatural.

A control group is a hierarchy used to classify tasks. Controllers attach particular resource policies or accounting to that hierarchy. The framework itself is not one universal “make this process small” knob. CPU scheduling, processor sets and other controllers expose different files and semantics. A task belongs to a group in a hierarchy, and children normally inherit membership, so classification should happen before launching the workload.

I created a group, moved my test shell into it, and then started the build from that shell. Its descendants appeared in the same group. Applying CPU scheduling controls made the machine more responsive under contention without changing every process individually. Memory control is a separate matter and depends on kernel configuration; enabling it also has overhead that should not be waved away.

The hierarchy is powerful but easy to design badly. Groups may represent users, services or workloads, and those choices do not always nest neatly. Multiple hierarchies can attach different controllers, which provides flexibility at the cost of another opportunity to create an administrative puzzle.

For now I regard cgroups as kernel machinery for building resource-management policy, not as a finished desktop feature. The interface is low-level and sharp-edged. Tools will need to create groups, place tasks early, apply limits coherently and clean up afterward.

My practical conclusion is to start with one measured problem and one controller. Verify membership before interpreting results, and do not confuse accounting with enforcement. Cgroups can organize a family of processes in a way nice alone cannot. They can also organize one’s mistakes into a tidy hierarchy, which is progress of a sort.

CFS After the First Week

Sun, 28 Oct 2007 17:35:00 +0000

I upgraded my workstation to Linux 2.6.23 mainly to try the Completely Fair Scheduler. The machine compiles code, plays music, runs a browser with an unreasonable number of pages, and occasionally hosts a virtual machine. Under a large parallel build, the old setup could make switching desktops feel as though the keyboard had been sent by post.

My first test was not clever. I started a build, dragged windows around and declared the new scheduler smoother. Then I rebooted into the older kernel and it also seemed smooth. Human perception is a wonderful measurement instrument if the desired unit is confidence.

For a better comparison, I repeated the same build with the same compiler jobs, timed it, and introduced an interactive task that periodically needed a small amount of processor time. Total build time changed little. The interesting difference was the delay before the interactive task ran when all cores were busy. Large stalls became less frequent with 2.6.23, though disk activity could still make the desktop miserable for entirely different reasons.

CFS approaches scheduling without the old collection of active and expired priority arrays. It tracks each runnable task’s virtual runtime, an adjusted measure of how much processor service that task has received. The task with the smallest virtual runtime is the one most entitled to run next. Runnable tasks are kept in a red-black tree, making that leftmost choice efficient as tasks arrive, block and wake.

The word “fair” needs care. Equal nice levels should receive roughly equal shares over time; it does not mean every task receives an identical slice at every instant. Nice values weight the virtual runtime, so a task with lower priority accumulates entitlement differently and receives a smaller share. Sleeper behavior emerges because a task that blocks stops consuming processor time. When it wakes, it may be behind its competitors and therefore run promptly, within limits intended to prevent absurd advantages.

My naive mental model was that CFS simply gives tiny slices to interactive programs and large slices to compilers. That is not how it identifies interactivity. There is no reliable label saying “this is the music player.” Instead, the accounting rewards tasks that use little processor time and wake to perform brief work. A music player, terminal or window manager often behaves that way, but so can a less noble process.

The scheduler also has to balance two competing costs. Switching too often improves apparent responsiveness but loses time to context switches and cache disruption. Waiting too long lets a runnable task monopolise a processor. CFS uses a target scheduling latency and derives slices from the number and weights of runnable tasks, subject to a minimum granularity. As the run queue grows, it cannot promise that every task runs within the same tiny interval without turning the processor into a context-switching demonstration.

This explains one of my results. With a modest build load, interactive delays stayed short. With a deliberately ridiculous number of runnable jobs, latency grew again. Fair scheduling cannot manufacture processor time. It can distribute scarcity coherently, which is less exciting but more useful.

Another misleading result came from I/O. During linking, the desktop paused even though CFS was selecting processor tasks correctly. The disk queue was saturated and applications blocked waiting for files. Changing the CPU scheduler did not cure that. Likewise, swapping overwhelmed every subtle scheduler difference. Before blaming CFS, I now check whether the delayed task is actually runnable rather than sleeping on storage.

Nice levels behaved more predictably in my tests than before, but group fairness remains a practical concern. If one user starts one processor-bound task and another starts twenty, scheduling every task independently can grant the second user most of the machine. There is work around grouping tasks, but the default desktop case still mostly exposes per-task fairness. The name should not be read as a constitutional guarantee.

I also tried changing scheduler tunables because numbers invite interference. Reducing latency made window feedback slightly sharper under synthetic load and increased context switches. Increasing it helped throughput by an amount too small to distinguish confidently while making the terminal occasionally sticky. The defaults survived my experiments, an outcome that wounded my pride but spared my configuration files.

After a week, I am keeping 2.6.23. The improvement is not that compilations finish magically sooner. They do not. The improvement is that short, waking tasks more often get service without elaborate interactivity heuristics, and nice values map onto a cleaner accounting model. That makes a loaded workstation feel less capricious.

The practical conclusion is to test CFS with repeatable mixed workloads, not window wiggling alone. Separate CPU delay from disk delay, compare distributions of latency rather than one lucky run, and resist tuning until a real workload demonstrates a problem. CFS is a substantial scheduler change, not a performance patch that makes every number smaller. On my machine it makes contention more orderly. Given what computers usually do under pressure, orderly is a fairly ambitious achievement.

Tickless Is Not Sleepless

Mon, 10 Sep 2007 18:45:00 +0000

I built a recent kernel with tickless idle enabled because the laptop fan had begun running while the machine did apparently nothing. I expected the new kernel to become silent by virtue of one configuration option. It did not. The fan is apparently not impressed by release notes.

My naive test was to leave a terminal open, wait a minute, and compare the battery estimate. That number wandered by nearly an hour between readings. I then watched processor usage, which stayed close to zero while the fan continued. Neither test told me what was waking the processor.

The periodic timer tick normally interrupts each processor at a fixed frequency so the kernel can account for time and perform scheduling work. During idle periods, those interrupts wake an otherwise sleeping processor for very little purpose. Tickless idle programs the next timer event instead, allowing a longer uninterrupted sleep. It cannot help if an application or driver requests a timer every few milliseconds.

On my machine, an application polling for status updates was doing exactly that. Closing it reduced wakeups dramatically and the fan eventually slowed. A network driver also produced regular activity when the interface was unused. Disabling the interface for a test separated that problem from the timer configuration.

Measuring several idle intervals also mattered. One short sample could be dominated by mail checking, disk flushing or my own impatient mouse movement.

There are limits to the name. Tickless does not mean the kernel never uses timer interrupts, and the current work mainly avoids ticks while a processor is idle. A busy processor still needs scheduling and accounting. Hardware timer support matters too; unreliable timers can turn a power improvement into missed timeouts or clocks that drift.

The practical lesson is that tickless operation removes one systematic source of wakeups. It does not forgive programs that poll constantly, nor drivers that chatter with hardware for entertainment. I am keeping it enabled because idle residency improved and battery measurements over several complete runs are better. But the useful debugging question is now “who wakes the processor?” rather than “is CPU usage low?” Zero percent can be composed of thousands of tiny interruptions, which is a very computer-like definition of resting.

Lock Before Looking

Fri, 26 May 2006 20:17:00 +0000

I reviewed a kernel path today that checked a pointer, acquired a spinlock, then used the pointer. It looked economical. It was also wrong.

if (device->queue) {
 spin_lock(&device->lock);
 flush_queue(device->queue);
 spin_unlock(&device->lock);
}

Another CPU can clear and free queue between the check and the lock. The check does not reserve anything; it merely records a fact that may expire immediately.

The simple repair is to protect both observation and use with the same lock:

spin_lock(&device->lock);
if (device->queue)
 flush_queue(device->queue);
spin_unlock(&device->lock);

This is safe only if flush_queue cannot sleep and does not acquire locks in an incompatible order. Spinlocks disable preemption in the relevant context; they are not permission to perform leisurely work. If flushing can block, I need a different design, perhaps detach the queue under the spinlock and process it later with appropriate lifetime ownership.

I prefer establishing ownership under one clearly documented lock rather than scattering “probably still valid” checks. The caveat is contention: a correct giant critical section can turn several processors into an expensive single processor.

The lesson is not “add locks.” It is “name the invariant, then guard every transition that can violate it.” Locks without an invariant are just punctuation with cache-line traffic.

What the Desktop Taught Me This Year

Fri, 16 Dec 2005 18:15:00 +0000

I spent much of this year fixing things that I first tried to solve at the wrong layer. A missing button event sent me into desktop settings. A wandering disk name produced a worse shell script. A driver race invited more logging. My instincts have been very energetic, if not consistently useful.

The better method is to follow the mechanism. Kernel devices appear through the device model and sysfs; udev applies naming policy. Input travels from the kernel through X.Org to applications. Qt objects have explicit parent ownership. Deferred kernel work has context and lifetime rules. Git history is a graph of objects, not a remote folder with ceremony.

KDE 3.5 ties the user-facing side together with unusual steadiness. Its maturity is not one clever feature but the accumulated reliability of applications, libraries, and conventions.

My practical conclusion for the year is to identify the boundary before editing the configuration or code. Ask what layer owns the decision, what evidence crosses that boundary, and what lifetime the data has. Then make one change and test it.

This sounds obvious when written in December. In February it apparently required an oops. Education remains committed to memorable examples.

Replacing the Directory Poll With inotify

Fri, 16 Sep 2005 20:05:00 +0000

Now that Linux 2.6.13 includes inotify, I replaced a timer in a small file-watching utility. The timer scanned a directory every second, which was simple, portable, and wasteful. It also introduced the charming choice between delayed updates and more frequent useless work.

My first inotify version assumed each read() returned one complete event. It worked during gentle testing and became nonsense when several files changed quickly. The interface returns a byte stream containing one or more variable-length struct inotify_event records. Correct code walks the buffer by each fixed header plus its len field.

union {
 struct inotify_event event;
 char bytes[4096];
} buffer;
ssize_t count;
size_t offset = 0;

do {
 count = read(fd, buffer.bytes, sizeof(buffer.bytes));
} while (count < 0 && errno == EINTR);

if (count < 0) {
 perror("read");
 exit(EXIT_FAILURE);
}
if (count == 0) {
 fprintf(stderr, "inotify descriptor reached EOF\n");
 exit(EXIT_FAILURE);
}

while (offset < (size_t)count) {
 struct inotify_event *event;
 size_t event_size;

 if ((size_t)count - offset < sizeof(*event)) {
 fprintf(stderr, "short inotify event header\n");
 exit(EXIT_FAILURE);
 }
 event = (struct inotify_event *)(buffer.bytes + offset);
 event_size = sizeof(*event) + event->len;
 if (event_size > (size_t)count - offset) {
 fprintf(stderr, "short inotify event name\n");
 exit(EXIT_FAILURE);
 }
 handle_event(event);
 offset += event_size;
}

The union gives the byte storage suitable alignment for struct inotify_event; a plain character array doesn’t promise that. The code also deals with interruption, read failure, EOF, and truncated records before treating bytes as an event. A watch identifies a pathname to the kernel, and events report changes associated with a watch descriptor.

The largest conceptual correction was learning that notifications are hints about state changes, not a private journal guaranteed to preserve my worldview. Events can arrive in bursts. A queue can overflow. A watched object can move or disappear. On overflow, the safe response is to rescan and rebuild known state rather than guess what was missed.

Renames deserve similar care. Related move events carry a cookie that can help pair them, but the application still needs sensible behavior when only one side is visible. Watching a directory also does not automatically mean recursively watching every directory below it.

I kept one function that performs a complete scan and made event handling update the same internal model. That gives the program a recovery path and keeps startup behavior consistent with overflow recovery. The event loop is an optimization over known state, not the only source of truth.

Compared with polling, inotify is immediate and avoids repeated directory walks when nothing changes. It also exposes more edge cases because the kernel reports actual operations rather than letting a periodic snapshot blur them together.

I prefer the new mechanism, with one reservation: event-driven doesn’t mean infallible. Read complete buffers, parse every record, expect overflow, and keep a rescan path. The kernel says something happened; deciding what the application should now believe is still our job.

Making a Use-After-Free Leave Tracks

Fri, 12 Aug 2005 17:50:00 +0000

A driver crashed several calls after an object had been freed. The final bad pointer looked plausible, and ordinary logs only showed that teardown had happened sometime earlier. I needed the stale access to leave a clearer fingerprint.

I rebuilt the test kernel with CONFIG_DEBUG_SLAB. The slab allocator’s debugging checks and poisoning replace freed storage with a recognizable pattern instead of leaving yesterday’s fields looking valid. The next failure stopped looking like random corruption: the pointer led into an object full of poison bytes.

That didn’t identify the owner by itself. I decoded the complete oops against the matching unstripped vmlinux, then followed the call chain back to a timer callback. The timer retained the private pointer after the remove path had called kfree().

The repair was pleasantly unoriginal: prevent rearming, delete the timer with the synchronizing form, and only then free the object. I ran repeated load, activity, and unload cycles with slab debugging still enabled. I also forced probe failures after each acquisition to make sure partially built objects followed the same lifetime rules.

Poisoning is a diagnostic, not memory safety. It works when the freed bytes haven’t already been reused, and it makes a debug kernel slower. Here it turned a believable stale structure into an unmistakably dead one, which was exactly the clue I needed.

Watching Linux History Move to Git

Fri, 22 Apr 2005 19:35:00 +0000

The interesting part of Git this week is no longer that it exists. Kernel developers are beginning to use it for actual Linux work, which changes the test from “can these objects be written?” to “can people exchange and trust history under pressure?”

I tried to imitate the old centralized habit by treating one repository as the place where truth lived and every other copy as a temporary client. That misses Git’s shape. Each repository can hold commits and their parent relationships. Fetching obtains objects and updates selected references; merging combines lines of development rather than asking a central server to manufacture history.

The kernel workflow makes the reason concrete. Maintainers can collect changes for their subsystem, preserve those commits, and pass a resulting line of history upward. The top-level maintainer can inspect and merge it. Signed e-mail and established review practices still matter; a fast object database does not decide whether a patch is wise.

I also learned not to confuse content identity with human trust. A hash can reveal that an object changed. It cannot tell me whether the author understood locking or whether I fetched from the person I intended. Technical integrity and project trust reinforce one another, but they are not synonyms.

Performance is already part of the appeal. Operations over the kernel tree need to be quick enough that developers use them freely. Cheap local history and branches encourage smaller experiments because recording work does not require a conversation with a remote machine.

That changes behavior as much as speed. A maintainer can inspect and organize work before publishing it instead of using the shared repository as a scratch pad.

The interface remains under construction, and I would not recommend that every project switch during lunch. Linux has an urgent need, unusually capable maintainers, and a willingness to work close to the machinery. That is a special environment, not a universal migration plan.

But adoption by the kernel gives Git a severe and useful proving ground. If it can preserve a large, branching C codebase while maintainers exchange changes independently, the design will have earned attention. I am keeping my test repository now. Calling it comfortable would be generous; calling it merely a toy would already be unfair.

A 2.6 Driver and the Trail of Evidence

Fri, 11 Feb 2005 20:45:00 +0000

I ported a small driver experiment to a newer 2.6 kernel this week. The old code compiled after a few mechanical edits, so I congratulated myself much too early. Loading the module produced an oops during device removal, which is the kernel’s way of reviewing code without softening the language.

My first attempt was to stare at the final faulting line and add a null check. That stopped one crash but left the design wrong. The pointer was not unexpectedly null; it referred to an object whose lifetime had ended. A defensive condition would merely make the race less visible.

The 2.6 device model asks drivers to express relationships among buses, devices, and drivers. Registration is not just bookkeeping. It establishes ownership and determines which callbacks can run as devices appear and disappear. I had treated remove as the reverse spelling of probe, but not as the reverse sequence of acquired resources.

I wrote down each successful step in probe: enable the device, reserve its region, allocate private state, initialize synchronization, and expose the interface. Then I made every failure path unwind only the steps already completed. remove performs the same cleanup in reverse order after preventing new work from entering.

static int sample_probe(struct device *device)
{
 struct sample *sample;
 int error;

 sample = kmalloc(sizeof(*sample), GFP_KERNEL);
 if (!sample)
 return -ENOMEM;

 memset(sample, 0, sizeof(*sample));
 error = reserve_hardware(sample);
 if (error)
 goto free_sample;

 /* Published callbacks must be able to find fully initialized state. */
 dev_set_drvdata(device, sample);

 error = register_interface(sample);
 if (error)
 goto clear_drvdata;

 return 0;

clear_drvdata:
 dev_set_drvdata(device, NULL);
release_hardware:
 release_hardware(sample);
free_sample:
 kfree(sample);
 return error;
}

This is shortened code, but the order matters: private data is attached before register_interface() makes callbacks possible. If publication fails, the failure path clears that pointer before releasing and freeing the state. In C, a linear chain of labels is often clearer than nested cleanup conditionals.

Removal uses the other half of the same rule:

static int sample_remove(struct device *device)
{
 struct sample *sample = dev_get_drvdata(device);

 unregister_interface(sample);
 stop_hardware(sample);
 cancel_deferred_work(sample);
 dev_set_drvdata(device, NULL);
 release_hardware(sample);
 kfree(sample);
 return 0;
}

unregister_interface() first prevents new user callbacks. The hardware is then stopped and outstanding work is drained while drvdata still points to live memory. Only after no callback can use it do I detach and free it.

For evidence, I enabled the kernel debugging options relevant to memory and locking and built a kernel specifically for this work. They cost performance, but a development kernel is a measuring instrument, not a racing bicycle. I also kept the unstripped vmlinux and the exact .config beside the test build.

When the next oops appeared, the symbolic trace identified both the callback and its caller. printk() markers around acquisition and release confirmed that queued work could still run after teardown began. The repair was to stop new requests, cancel or drain deferred activity, and only then free the state it could reference.

The order now looks like this:

block new work
stop device activity
wait for deferred work
remove user-visible interface
release hardware resources
free private state

I tested failure paths too. Returning an artificial error after each acquisition showed whether cleanup restored the previous state. Module load and unload in a loop then exercised the unremarkable path repeatedly. Repetition is valuable because races dislike appointments.

I included reference counts in the review even though the first crash did not mention them. A reference is a claim that an object must remain alive, not merely an integer to increment near a pointer. Every path acquiring one needs a visible release, and teardown must prevent new claims before waiting for old ones. Counting without a shutdown rule only measures how confused the lifetime has become.

Lock choice followed the same reasoning. I did not replace every lock with a larger one to make the warning disappear. I listed which fields formed one invariant and which contexts accessed them. A spinlock suited the short state transition shared with interrupt context; work that could sleep stayed outside it. Narrow critical sections made both the rules and the traces easier to read.

One subtle lesson was to distrust logs as timing-neutral. A printk() can alter scheduling enough to hide a race. I used it to establish broad ordering, then relied on assertions, debug checks, and repeated tests rather than declaring victory when extra logging made the crash vanish.

An oops after testing still required a disciplined capture. I saved the complete report, not only the final address, and matched it to the exact kernel image. A decoded call chain is context: it shows how execution arrived, which often matters more than the instruction that finally touched invalid memory.

The kernel APIs will continue to evolve, and copying a driver from an older tree will continue to be tempting. The durable knowledge is not a particular field name. It is the discipline of checking the current in-tree callers, understanding callback context, defining object lifetime, and making cleanup the exact inverse of setup.

I still prefer user-space debugging, where a bad pointer usually ruins one process rather than the afternoon. The kernel does leave a trail if the build preserves symbols and the code preserves invariants. That’s better than adding null checks until the machine goes quiet.

Sleeping in the Kernel Without Regret

Fri, 28 Jan 2005 22:30:00 +0000

A driver experiment needed to do a little cleanup after an interrupt. I put the cleanup directly in the interrupt handler because it was only a little work. Then that little work called something that could sleep, and the kernel objected with considerably more confidence than I had shown.

The naive model says an interrupt handler is just a function the kernel calls quickly. The missing detail is context. Hard interrupt context cannot sleep: there is no process there to suspend in the normal sense, and holding up interrupt handling harms the rest of the machine.

The useful pattern is to split the operation. The handler acknowledges the device, records the minimal state under the right lock, and schedules deferred work. A workqueue function later runs in process context and may use operations that can sleep.

static void device_work(void *data)
{
 struct sample_device *dev = data;

 /* Slow cleanup is safe in process context. */
 finish_request(dev);
}

The exact initialization APIs deserve a check against the headers for the kernel being built; this area has changed before and surely will again. The invariant matters more than memorizing one macro: know which context executes the callback, what locks are held, and whether every function it calls may sleep.

Allocation flags make the distinction concrete. Code in a context that cannot sleep must not request an allocation mode that may block. Moving slow work to a workqueue lets that callback use normal process-context operations, but it does not make shared state safe automatically.

Teardown is the other half. Before freeing the device state, the driver must stop scheduling work and ensure already queued callbacks have finished. Otherwise the deferred function merely postpones the crash until after the object is gone.

I added an explicit comment at the scheduling point and reduced the handler to the smallest useful unit. Latency improved, but the larger gain was conceptual. Once context is part of the function’s contract, calls that looked harmless become obviously wrong.

Kernel C is not difficult because the syntax is exotic. It is difficult because an ordinary-looking call carries constraints about context, ownership, and concurrency. Ignoring those constraints produces bugs that wait patiently for the least convenient machine. Mine at least had the courtesy to complain immediately.

Following the Device Through sysfs

Fri, 19 Nov 2004 18:40:00 +0000

A USB card reader prompted today’s trip through /sys. I wanted one answer: which physical device sits behind the block-class entry /sys/block/sda?

The useful relationship is the device symlink. /sys/block/sda presents the disk as a member of the block class; its device link points into the physical device hierarchy. Those are two views of one kernel object, not two devices.

I started at the block entry and followed its device link rather than guessing from the node name:

$ readlink -f /sys/block/sda/device
/sys/devices/pci0000:00/0000:00:10.4/usb2/2-2/2-2:1.0/host1/1:0:0:0

The exact target is machine-specific. Here it says that the block disk came through a SCSI host attached to USB interface 2-2:1.0, itself below a PCI USB controller. On another machine the chain will differ, but the link still joins the class view to the device that implements it.

Walking upward from that target lets me inspect the parents that actually own bus attributes. The block entry doesn’t need to copy USB details into its own directory; the parent links already express the relationship.

I don’t parse the punctuation in the resolved path. Directory names and topology are useful while diagnosing a machine, but exported links and attributes are the interface. Hard-coding every component would just trade /dev/sda for a longer accidental name.

This also explains why /dev/sda isn’t a hardware biography. The node names a block interface; sysfs supplies the link back to the device model when I need to ask where that interface came from.

The service corridors are a little dim, but this one symlink answers the question without guesswork.

Watching Latency on Linux 2.6

Thu, 23 Sep 2004 20:10:00 +0000

I have been comparing two Linux 2.6 configurations for a workstation: one with kernel preemption enabled and one without it. Looking only at total build time made them appear nearly identical, while using the machine made them feel different. The missing measurement was latency.

Throughput asks how much work finishes in an interval. Latency asks how long a particular piece of work waits. A kernel build is a reasonable throughput load, but its completion time does not reveal that an audio process waited long enough to miss a deadline or that keyboard input paused for a fraction of a second.

My crude but useful test runs the same build while playing audio, copying a large file, and interacting with KDE. I record build time, note audio underruns reported by the application, and repeat enough times to distinguish a pattern from one lucky run.

time make -j2 bzImage

The O(1) scheduler helps when many tasks are runnable. Its priority arrays and per-CPU run queues keep selection costs predictable, while interactive heuristics favour tasks that sleep and wake frequently. But the scheduler cannot run a task until the current execution context can be switched.

That is where kernel preemption matters. With it enabled, ordinary kernel code can be preempted when no lock or other critical condition forbids it. An interactive task awakened during a long system call may run sooner. Interrupt handlers and protected sections remain non-preemptible, so a poor driver can still dominate worst-case delay.

On my machine, enabling preemption does not materially improve the build time. It does reduce occasional input stalls and audio trouble under mixed I/O load. That is the result I expected: better responsiveness, not newly manufactured processor cycles.

Thread behaviour can alter the picture too. NPTL mutexes stay in userspace when uncontended and use futex operations when a thread must sleep or wake. A heavily contended threaded program may therefore create a very different scheduling load from one with mostly private work. Counting threads alone says little; runnable and blocked states matter.

I also test without excessive parallelism. Starting far more compile jobs than processors can turn the test into a contest between memory pressure, disk traffic, and scheduling. That may resemble a real workload, but it is a poor way to isolate one mechanism.

The caveat is that my observations are not hard real-time guarantees. General-purpose Linux still has sources of unbounded or hardware-dependent delay, and an average hides the worst pause. For desktop tuning, however, repeated underrun counts and interaction under a controlled load tell me more than one impressive jobs-per-minute number.

I am keeping preemption enabled on the workstation and evaluating it separately on servers. A server handling many requests may care about response-time distribution, but a batch machine may prefer the smallest scheduling overhead. There is no universally fast switch. First decide whether the itch is throughput or latency; only then does the benchmark know what question to answer.

Moving a Real Machine from Linux 2.4 to 2.6

Thu, 15 Apr 2004 22:20:00 +0000

I have now moved a machine I actually depend upon from Linux 2.4 to 2.6. Test boxes are excellent for discovering that a kernel boots. A working machine is better at discovering the scanner, firewall rule, and obscure filesystem one forgot to test.

The obvious approach is to install the new kernel and reboot. The correct approach begins with an inventory. I wrote down the storage controller, root filesystem, network devices, sound hardware, mounted filesystems, firewall setup, and every module not supplied by the kernel tree. If a component is required to boot or reach the network, I want its exact name before changing anything.

I then upgraded the required userspace while still running 2.4. module-init-tools understands 2.6 modules and supplies compatibility behaviour for a 2.4 boot. Recent procps tools understand newer /proc details and NPTL process presentation. Filesystem utilities must support the on-disk filesystems regardless of which kernel initiated the repair.

The important point is coexistence. I did not overwrite the old kernel image, old modules, or boot entry. My boot loader contains distinct choices, and I tested selecting 2.4 after installing the new tools. A rollback that has never been attempted is a comforting theory.

Building the first kernel

I used the 2.4 configuration as a reference, not scripture. make oldconfig carries choices forward and asks about new options, but subsystems changed between the series. I read the help for storage, input, sound, device filesystems, networking, and processor options instead of accepting every default at speed.

For the first build, the disk controller and root filesystem went directly into the kernel. This avoids depending on a new module loader and a new initial ramdisk during the same boot. Once the plain configuration worked, I made less essential drivers modular.

make oldconfig
make bzImage
make modules
make modules_install

The exact image installation is distribution and architecture dependent, so I copied it using the same procedure as the existing packaged kernel and created a separate boot-loader stanza. I also saved .config beside the image with a matching name.

Linux 2.6 uses sysfs for its device model. My startup scripts mount it at /sys:

mount -t sysfs none /sys

I kept the existing /dev arrangement for the first migration. Early udev is interesting and I intend to test it, but changing kernel, module format, device discovery, and device-node management in one reboot produces a beautiful failure with too many suspects.

The first boot is only the start

The machine reached a login prompt immediately, which proved almost nothing. I checked dmesg from beginning to end for timeouts, unknown options, and drivers claiming the wrong hardware. I mounted every local filesystem, activated swap, transferred data over each network interface, and exercised packet filtering.

Sound moved to ALSA, now part of the kernel tree. Applications using OSS interfaces can often use ALSA’s compatibility modules, but mixer state and device permissions still need checking. Silence may mean the channel is muted rather than the driver is broken, an ancient audio tradition maintained for compatibility with human suffering.

Threaded programs deserve a run under the new combination of kernel and C library. NPTL relies on facilities available in 2.6 and improves both performance and POSIX behaviour, but old monitoring scripts may interpret the changed process display incorrectly. I checked services by function rather than by expecting the old number of ps lines.

The firewall required careful review. The netfilter framework remains, but kernel configuration symbols and available matches can differ. Loading an old saved rule set is useful only if all required modules exist and the restore command reports success. A script that ignores errors can leave a machine less protected while printing reassuring startup messages.

Out-of-tree modules were the only serious obstacle. Internal kernel interfaces changed, and source written for 2.4 often needs a real port rather than a version-number adjustment. I found replacements in the 2.6 tree where possible. For the remaining hardware, I verified current source before making the migration permanent. Binary-only drivers put the schedule entirely in the vendor’s hands, which is one reason I avoid depending on them.

What changed in use

Under a compile and interactive desktop load, the machine is more responsive. The O(1) scheduler’s per-CPU run queues and priority arrays make task selection predictable as load grows. Kernel preemption, enabled on this workstation, reduces the delay before an awakened interactive task can run when another process is in preemptible kernel code.

This does not mean every benchmark is faster. Responsiveness concerns latency; throughput concerns completed work over time. I measured both and accepted a tiny difference in build time in exchange for fewer audio skips and shorter input pauses.

Sysfs is already useful for diagnosis. Following links from /sys/class to a device and its driver is clearer than assembling the relationship from several /proc files and boot messages. It also provides the structured device information that userspace managers such as udev need.

My migration rule is now simple: preserve 2.4, upgrade userspace first, boot a minimally clever 2.6 configuration, and add new mechanisms one at a time. Linux 2.6 is ready for this machine, but it did not become ready merely because uname -r printed the desired number. It became ready when every required workload passed and the old kernel remained one menu selection away.

What Kernel Preemption Actually Does

Thu, 26 Feb 2004 19:25:00 +0000

I enabled kernel preemption on my workstation and was rewarded with the usual technical diagnosis: the machine feels nicer. That is useful as an observation and useless as an explanation, so I went back to what the option actually changes.

Normal process preemption is not new. The scheduler can stop one userspace task and run another. The relevant 2.6 option allows a task executing kernel code to be preempted too, provided it is not in a critical section where switching would be unsafe.

Suppose a low-priority process enters a long kernel path just before an audio application becomes runnable. Without kernel preemption, the audio task may wait until that path returns to userspace or reaches another scheduling point. With preemption enabled, the scheduler can run the newly awakened task sooner once the current kernel context is in a preemptible state.

The mechanism depends on keeping count of places where preemption must not occur. Spinlocks and explicit preemption disabling protect critical regions. Interrupt context has its own restrictions. The option does not permit the scheduler to interrupt arbitrary code while it holds shared state halfway through an update. That would improve latency in much the same way removing the brakes improves a car’s acceleration.

For a practical comparison I run a kernel build while playing audio, moving windows, and causing disk activity. I watch for skips and input stalls, but I also time the build. Lower worst-case latency is the target; higher total throughput is not guaranteed.

time make -j2 bzImage

On this desktop, preemption reduces the noticeable pauses under mixed load. The difference is most apparent when a background job causes frequent kernel work rather than merely consuming CPU. The O(1) scheduler already decides quickly which task should run; preemption helps the kernel reach a point where that decision can take effect.

I would not enable it automatically on every server. Preemption introduces scheduling opportunities and some overhead, while a throughput-oriented workload may gain nothing from lower interactive latency. Test the actual workload and look at variance, not only an average benchmark.

There are also limits. Long non-preemptible sections, interrupt handlers, slow hardware, and drivers can still cause delays. The option narrows one source of latency; it does not turn a general-purpose kernel into a hard real-time system.

For workstations, audio systems, and other latency-sensitive machines, it is a sensible trade. For a batch server, the default conservative choice may remain better. The nice thing is that the trade is explicit. I can choose responsiveness where I need it without pretending every Linux machine has the same job.

Linux 2.6.0 Arrives

Thu, 18 Dec 2003 23:15:00 +0000

Linux 2.6.0 was released yesterday. After the long 2.5 development series and months of test releases, there is finally a version number administrators can put into a plan without the word “test” in it. This does not mean I am replacing every 2.4 kernel before lunch. It means the migration can become deliberate rather than hypothetical.

The itch for me is desktop latency. My machine compiles, plays audio, runs KDE, and occasionally receives the unreasonable request to respond to the keyboard at the same time. Linux 2.6 has two important mechanisms for that: the O(1) scheduler and optional kernel preemption.

The scheduler keeps per-CPU active and expired priority arrays. Selecting the next task does not require scanning a growing list of runnable processes, and interactive tasks receive dynamic priority treatment based largely on sleep behaviour. Preemption goes a step further. When enabled, kernel code that is not in a critical section can be preempted, reducing the time a newly runnable high-priority task waits for a long kernel path to finish.

Neither feature creates free CPU time. A saturated machine remains saturated, and a bad driver can still ruin the afternoon. What changes is how predictably the system gives urgent work a turn.

Threading is another substantial improvement. The kernel facilities used by NPTL provide proper thread groups, efficient futex-based synchronization, and saner signal and process semantics than LinuxThreads could offer. A POSIX mutex can complete entirely in userspace when uncontended and enter the kernel only to wait or wake when there is contention. This matters both for performance and for making tools show one multithreaded process as something resembling one multithreaded process.

The device model may have the largest administrative consequence. Sysfs exposes devices, buses, classes, and drivers as related kernel objects under /sys. The kernel now has a coherent answer to questions such as “what is this device attached to?” instead of scattering hints across boot messages and /proc.

That structure makes userspace device management practical. Early work on udev can listen for hotplug events, consult sysfs, and create device nodes according to userspace policy. I am not ready to discard a working /dev setup everywhere, but moving naming policy out of the kernel is the right direction. Hardware discovery belongs in the kernel; deciding that a particular camera should have a friendly local name does not.

There are less glamorous changes that may matter more on servers. The block I/O layer and filesystem scalability have received major work. The kernel supports larger systems, more processors, and more memory without treating each as a surprising edge case. Native ALSA support gives sound users a maintained kernel interface, though moving from OSS drivers may require mixer and application adjustments.

The migration is not just a kernel file

My safe upgrade recipe begins by leaving 2.4 intact. I install 2.6 under a distinct name, add a separate boot-loader entry, and verify that selecting the old entry still works. The new kernel image is the easiest part to roll back; userspace and configuration changes require more thought.

Linux 2.6 modules use a different format, so module-init-tools must be installed. The tools are designed to coexist with the programs needed by 2.4, which is exactly what a gradual migration needs. Recent versions of procps, filesystem utilities, and other low-level packages are also worth checking against the distribution’s recommendations.

For a first boot I compile the root disk controller and root filesystem into the kernel. Once the machine boots reliably, I can introduce an initial ramdisk and modularise things. Debugging one new mechanism at a time is slower only until the first failure.

I also mount sysfs explicitly if the startup scripts do not:

mount -t sysfs none /sys

Then I test the boring list: all filesystems, swap, networking, firewall rules, sound, removable devices, suspend if the machine uses it, and clean shutdown. I compare dmesg for errors rather than merely noticing that a login prompt appeared. A successful boot proves the boot path, not the rest of the computer.

Out-of-tree drivers are the largest caveat. The internal kernel interfaces have changed, sometimes substantially, and a driver compiling on 2.4 says nothing about 2.6. Binary-only modules are especially dependent on their vendor producing a matching build. Before an upgrade, inventory these modules with the same seriousness as storage hardware.

Applications should generally continue to run. This is not a new userspace ABI. Problems are more likely to come from programs that inspect /proc, rely on old thread display details, expect OSS device behaviour, or invoke module utilities directly.

My opinion is that 2.6 is a stronger foundation, especially for interactive systems and machines with many tasks or processors. But .0 is a milestone, not a papal declaration of perfection. Production systems deserve a soak period, distribution patches, and testing against their actual workload.

I will install it quickly on my workstation and slowly on servers. This is not inconsistency. The workstation can inconvenience me; the server can introduce me to several irritated people at once. Linux 2.6.0 is ready for serious testing today, and with cautious migration it should become the ordinary kernel soon enough.

Reading the Device Tree in sysfs

Thu, 06 Nov 2003 17:55:00 +0000

I needed to answer a simple question on a 2.6 test machine: which driver owns this network device? My old habit was to rummage through boot messages, /proc, and module output until the answer surrendered. The new device model gives us a less archaeological method.

First, mount sysfs if the installation has not done it already:

mount -t sysfs none /sys

The interesting part is not that /sys contains files. Linux can turn almost anything into files if left unattended. The useful part is the structure. Devices appear according to their physical or logical parentage under /sys/devices. Buses, such as PCI and USB, have views under /sys/bus, while /sys/class groups devices by function, such as network interfaces or block devices.

These are views of the same kernel objects, joined by symbolic links. For an Ethernet interface named eth0, this is a useful start:

readlink /sys/class/net/eth0/device
ls -l /sys/class/net/eth0/device/driver

The first link leads back into the device hierarchy. The driver link, when present, identifies the bound driver. Following parent links tells me which PCI device contains the interface and where that device sits.

This differs from /proc. Procfs grew as a place for process information and gradually acquired assorted kernel knobs and device facts. Sysfs is an exported representation of the kernel’s device model. One attribute per file is the usual convention, which makes simple scripts possible without parsing a decorative essay from the kernel.

I am deliberately cautious about those scripts. The 2.6 series is not final yet, and sysfs details can change as interfaces settle. Scripts should consume documented attributes, not depend on directory ordering or infer meaning from every internal-looking name. A readable tree is still an interface, not an invitation to marry the first implementation detail one encounters.

The larger implication is device management in userspace. If the kernel exposes device identity and events cleanly, a userspace program can create names and permissions according to policy rather than a huge static /dev or a kernel naming scheme. That work is still young, but sysfs supplies the map it needs.

For now, it has already replaced several minutes of log-diving with two symbolic links. I consider that progress, even if it has added another virtual filesystem to my vocabulary.

Why the O(1) Scheduler Feels Different

Thu, 07 Aug 2003 18:20:00 +0000

I have been testing the new kernel on a machine that does two incompatible jobs: compile software and remain pleasant enough to use while compiling software. Under load, the 2.6 test kernel feels less sticky than my usual 2.4 setup. The O(1) scheduler is a large part of that.

The name describes a property, not a performance promise. Choosing the next runnable task takes constant time rather than getting more expensive as the run queue grows. The scheduler keeps arrays of queues indexed by priority, plus a bitmap showing which priorities contain runnable tasks. Finding work becomes a matter of locating the first set bit and taking a task from that queue.

There are actually two priority arrays per processor: active and expired. A normal task runs for its time slice and then moves to the expired array. When active becomes empty, the kernel swaps the arrays. No grand tour through every process is required.

That design also avoids one global run queue becoming a lock everybody fights over on an SMP machine. Each processor has its own run queue, and load balancing periodically moves work when the queues become uneven. Per-processor data makes the common path cheaper, although balancing is still real work and certainly not magic.

Interactive behaviour comes from dynamic priorities. A task that sleeps often, as an editor or terminal usually does while waiting for me, receives favourable treatment when it wakes. A compiler chewing CPU continuously tends to use its slice and wait in the expired array. The scheduler is estimating interactivity from sleep behaviour; applications do not announce that they are important because a human is glaring at them.

There is a caveat. Heuristics can be fooled, and “feels faster” is not a benchmark. A database server and a desktop have very different ideas of fairness. I am watching throughput as well as latency and checking for tasks that receive odd priority treatment.

Still, the mechanism is refreshingly concrete. Constant-time selection, per-CPU queues, and explicit expired work explain the observed behaviour without fairy dust. The desktop remains responsive during a build, the build still finishes, and neither needs a special incantation. That is a scheduler improvement I can actually notice.

Getting Ready for Linux 2.6-test1

Thu, 17 Jul 2003 21:35:00 +0000

Linus released 2.6.0-test1 this week, which means the 2.5 development series has finally put on a tie and is pretending to be suitable for polite company. It is still a test kernel. I would not install it on the machine that pays the bills, but I do want to know what will break before 2.6 becomes ordinary.

The obvious approach is to keep the current 2.4 kernel installed, add 2.6 beside it, and make the boot loader offer both. Do not replace the known-good entry. This sounds insultingly basic until the new kernel cannot find its root filesystem and the old entry is the difference between five minutes and a rescue disk.

My first checklist is mostly userspace:

module-init-tools
recent modutils-compatible configuration
procps that understands the newer /proc output
filesystem tools matching every filesystem in use
working boot loader configuration

The module change is the one most likely to surprise people. Linux 2.6 uses a new module format and the module-init-tools programs. They can coexist with the old tools, so one installation can boot both 2.4 and 2.6. Check this before rebooting, not while staring at an emergency shell.

I also build the drivers needed to reach the root filesystem directly into the kernel for the first attempt. That means the disk controller, root filesystem, and anything else between the kernel and /sbin/init. An initial ramdisk is useful, but removing it from the first experiment makes failures much easier to understand.

The device model has changed substantially. A mounted sysfs, normally at /sys, exposes devices and drivers in a structured tree. Old scripts that scrape /proc or assume particular module names deserve suspicion. Sound users should note that ALSA is now in the kernel tree; OSS compatibility exists, but that does not make every mixer setup identical.

Finally, save the exact .config and the complete build output. make oldconfig is a convenient starting point, not proof that old choices still mean the same thing. Read every new question that affects storage, filesystems, networking, or the console.

The kernel has many attractive improvements: preemption, the O(1) scheduler, better threading support, and a much saner device model. None of them will comfort you if the machine does not boot. My plan is boring: test hardware, test userspace, test rollback, then become adventurous. Adventure is much nicer with a working fallback kernel.

Keeping Work Out of a Network Interrupt

Tue, 27 May 2003 05:47:36 +0000

I put packet bookkeeping into a network device’s interrupt handler because the information was conveniently available there. Under load, the machine reminded me that convenience inside interrupt context is borrowed at unreasonable interest.

The interrupt handler should acknowledge the device, collect enough status to know what happened, move completed descriptors along, and arrange deferred processing where appropriate. It cannot sleep, and time spent there delays other interrupt work. Long parsing, allocation loops, or diagnostic printing are especially poor guests.

Linux 2.4 offers deferred mechanisms including tasklets and the network receive path’s softirq processing. A driver can prepare received skbs and pass them to netif_rx(), allowing the protocol stack to process packets outside the hard interrupt handler. For driver-specific deferred work, I put the tasklet in the device’s private state:

struct sample_dev {
 struct tasklet_struct rx_tasklet;
 /* rings, locks, and device state */
};

static void sample_tasklet(unsigned long data)
{
 struct sample_dev *dev = (struct sample_dev *)data;

 service_completed_packets(dev);
}

static void sample_init_tasklet(struct sample_dev *dev)
{
 tasklet_init(&dev->rx_tasklet, sample_tasklet,
 (unsigned long)dev);
}

sample_init_tasklet() runs after dev is allocated and before the interrupt can schedule dev->rx_tasklet. Teardown disables that interrupt source, calls tasklet_kill(&dev->rx_tasklet), and only then frees dev. The callback therefore never receives the null value that a static DECLARE_TASKLET(..., 0) would have supplied.

Shared rings and status fields need protection across the interrupt, deferred function, transmit entry point, and process-context control operations. I use spinlocks where the data can be touched in interrupt context and choose the interrupt-disabling variant when the same processor could otherwise interrupt a lock holder. I keep the protected section short and never call a function that can sleep while holding it.

There is a balance. Deferring every tiny operation adds overhead and can complicate ordering, while doing everything in the handler harms latency. I measure under sustained receive and transmit load, watch dropped packets, and check that another device’s interrupts are not starved.

Interrupt acknowledgement order is device-specific and worth documenting beside the handler. A level-triggered interrupt that remains asserted can immediately enter again; acknowledging too early on other hardware may lose status that has not been copied. I read the device manual rather than deriving this order from whichever experiment happened not to hang.

Shutdown receives the same scrutiny. The driver disables the device’s interrupt source, prevents new deferred work, frees the IRQ with the required synchronization, and only then releases rings and private data. Otherwise a late tasklet can faithfully process memory that has already acquired a new purpose.

I want the hard interrupt path to be visibly bounded. One caveat: printk can completely distort timing here. A hundred helpful messages per second soon become the performance problem, a promotion few debug statements deserve.

Module Parameters in Linux 2.4

Thu, 16 Jan 2003 08:22:47 +0000

I was rebuilding a test module merely to change its debug level. That is a fine way to turn a ten-second experiment into a small ritual. Linux 2.4 module parameters are a better fit.

For a simple integer, the old module parameter declaration is compact:

static int debug;
MODULE_PARM(debug, "i");
MODULE_PARM_DESC(debug, "enable diagnostic messages");

I can then set it while loading:

insmod sample.o debug=1

The type string tells the module loader how to interpret the value. Other forms support strings and arrays, but I avoid elaborate configuration here. Parameters are useful for hardware choices and diagnostics needed at initialization, not as a private replacement for a proper user interface.

Validation still belongs in the initialization function. A parameter arriving from insmod is input, not a promise. If a buffer count must be positive and bounded, the module should reject nonsense with -EINVAL before allocating resources.

I also print the effective non-default setting at load time when it materially changes behavior. That makes dmesg useful when somebody reports that the driver behaves differently on one machine.

My preference is a quiet default and one obvious parameter for debugging. Five boolean switches rapidly become a combination lock whose correct setting nobody remembers. Recompiling was tedious, but at least it did not accept debug=banana with a straight face.

Reading an skb Without Regretting It

Fri, 20 Dec 2002 20:14:39 +0000

I added a packet check to a Linux 2.4 network path and initially treated skb->data as if it always began with the header I wanted. It did in my first test. Networking code has a generous way of rewarding that assumption just long enough to become embarrassing.

The sk_buff carries packet data plus bookkeeping used as the packet moves through layers. Its head, data, tail, and end pointers describe the allocated area and the currently occupied bytes. Protocol processing can pull a consumed header from the front or push space for a new one without copying the whole packet.

Before reading a header, I check that the required bytes are present and use the header position appropriate to that point in the stack. In a simple driver receive path, the driver allocates an skb, reserves alignment if needed, copies or maps packet bytes, sets the protocol, and hands it upward:

skb = dev_alloc_skb(length + 2);
if (!skb)
 return -ENOMEM;

skb_reserve(skb, 2);
memcpy(skb_put(skb, length), packet, length);
skb->dev = dev;
skb->protocol = eth_type_trans(skb, dev);
netif_rx(skb);

Calling skb_put() extends the valid data at the tail and returns the start of the added area. Writing beyond that area corrupts memory, so the allocation and length must agree. eth_type_trans() interprets the Ethernet header, sets the protocol, and adjusts the skb for the network layer.

Ownership changes at netif_rx(). After handing the skb to the network stack, the driver must not free or inspect it as though it still belongs to the receive routine. Error paths before that handoff must free what they allocated.

Transmit has the opposite pressure. The driver’s hard_start_xmit receives an skb and must obey the device queue rules. If hardware resources are exhausted, stopping the queue and waking it when descriptors become available is better than quietly dropping ownership conventions. Interrupt and bottom-half paths also require locking suited to their contexts; sleeping locks are not available there.

I prefer using skb helpers over pointer arithmetic, even when the arithmetic looks obvious. The helpers document whether code is adding, removing, or reserving bytes and retain their checks in debugging builds. The caveat is that not every skb is necessarily laid out as one convenient linear region at every layer, so code must respect the guarantees of its exact call site. Packets are simple on diagrams. The diagrams do not have cache lines.

Building a Module for the Running Kernel

Tue, 03 Dec 2002 08:41:27 +0000

I compiled a small module against the headers in /usr/include/linux, loaded it, and received a cheerful list of unresolved symbols. The mistake was treating user-space headers as a kernel build environment.

A module must be built for the kernel that will load it. I first identify that kernel and locate its configured source tree:

uname -r
ls -l /lib/modules/`uname -r`/build

On a packaged system the build link commonly points to the matching kernel sources or prepared headers. The tree must have the configuration and generated headers corresponding to the installed kernel. Merely unpacking a similar 2.4 source archive is not enough.

For a tiny experiment, the compile command shows the important ingredients:

gcc -D__KERNEL__ -DMODULE -O2 -Wall \
 -I/lib/modules/`uname -r`/build/include \
 -c sample.c -o sample.o

Real module makefiles should take their flags from the target kernel’s build setup rather than accumulating copied guesses. Processor options, SMP support, and symbol versioning can all affect the expected object. If the kernel was built with module versions, the build also needs the matching version definitions; disabling the complaint does not make incompatible structures compatible.

I compare the module and running kernel before loading, then inspect the actual diagnostic:

insmod sample.o
dmesg

An unresolved symbol may mean the providing facility is not configured, is itself a module not yet loaded, was not exported, or has a different version. depmod -a and modprobe handle dependency information for installed modules, while insmod is useful when I deliberately want the direct error from one test object.

The important mechanism is that a loadable module links against symbols exported by the running kernel and other loaded modules. It is not a normal program with its own copy of kernel interfaces. Configuration differences therefore matter even when function names look identical.

Optimization is not optional decoration here. Kernel headers use inline functions and assumptions expected by the normal build flags, so an unusual hand-written command can expose failures absent from an ordinary kernel build. I use the short command only to understand the pieces, then put repeatable work in a makefile tied to the configured tree.

Before installing, I test removal as well as insertion and confirm that no old object with the same module name is already loaded. Debugging yesterday’s module while reading today’s source is a surprisingly convincing experience.

I prefer building against the exact configured tree and letting version checks fail loudly. The caveat is that matching uname -r is necessary but not magical; a locally rebuilt kernel with the same release string can still differ. Names are labels, not fingerprints, despite what some build directories would like me to believe.

Following a File Through the Linux 2.4 VFS

Tue, 12 Nov 2002 14:56:03 +0000

I was reading a small filesystem and kept confusing dentries, inodes, and open files. They all seemed to point at roughly the same object until a rename and two simultaneous opens proved otherwise. I finally followed one pathname through the Linux 2.4 VFS and wrote down what each structure represents.

The VFS is the layer that gives user programs the usual operations such as open, read, and stat while allowing ext2, NFS, procfs, and other filesystems to implement different storage rules. It does this with common objects and tables of function pointers. The useful mental model is not one structure per file. It is several structures per role.

A filesystem implementation first registers a file_system_type:

static DECLARE_FSTYPE(sample_type, "samplefs", sample_read_super,
 FS_REQUIRES_DEV);

static int __init sample_init(void)
{
 return register_filesystem(&sample_type);
}

In 2.4, mounting leads to the filesystem’s read_super function. It validates or reads the on-disk superblock, fills the VFS super_block, installs superblock operations, and supplies a root dentry. The superblock represents one mounted filesystem instance, not the filesystem format in general. Two mounted ext2 partitions therefore have separate superblocks while sharing ext2’s implementation code.

The inode represents a filesystem object and its metadata: type, mode, owner, times, size, and operation tables. A filesystem usually embeds or associates its private inode information with the VFS inode. The inode number identifies it within that filesystem. Hard links make the distinction important: several directory names can refer to the same inode.

The dentry represents a name in a directory and connects pathname lookup to an inode. It belongs to the dentry cache, remembers parent and child relationships, and may even be negative, meaning that a lookup established that a name does not currently exist. A dentry is therefore about a pathname component and cached lookup state, not merely another spelling of inode.

Suppose a process opens /mnt/notes/todo. Pathname lookup starts from a root or current directory and walks components. For each component, the VFS uses cached dentries when possible and asks the parent inode’s lookup operation when necessary. By the end, it has a dentry for todo and, if the file exists, an associated inode.

Opening then creates a file object for that particular open instance. It records the current offset, access mode, flags, and file operations. Two processes opening the same inode receive distinct file objects and can have different offsets. After fork, file descriptors may instead refer to the same open file object, which explains why their offsets can move together.

The operation tables divide responsibility. Directory inode operations include tasks such as lookup, create, link, and rename. File operations include read, write, llseek, open, and release. Superblock operations deal with the mounted instance, including writing metadata and releasing it. The VFS calls through these tables without needing ext2 rules in generic pathname code.

A tiny read method illustrates another boundary:

static ssize_t sample_read(struct file *file, char *buf,
 size_t count, loff_t *ppos)
{
 const char text[] = "hello\n";
 size_t left;

 if (*ppos >= sizeof(text) - 1)
 return 0;

 left = sizeof(text) - 1 - *ppos;
 if (count > left)
 count = left;
 if (copy_to_user(buf, text + *ppos, count))
 return -EFAULT;

 *ppos += count;
 return count;
}

The method receives the open file, not just the inode, because position and open flags belong to the open instance. It uses copy_to_user() because a user pointer cannot be treated as an ordinary kernel pointer. Returning zero at end of file is part of the read contract, not an error.

Caching makes lifetime rules less obvious. Closing the last file descriptor does not imply that the inode or dentry disappears immediately; the caches may retain them. Conversely, unlinking a name removes a directory entry but an already open file can remain usable until its final reference is released. Reference counts and VFS helper functions, rather than guesses about user-visible names, govern these lifetimes.

Locking also follows roles. Operations changing a directory need the directory inode’s semaphore according to the VFS calling rules, while filesystem-private data may need its own protection. I do not add locks at random around every pointer. I first determine which VFS locks are already held for that operation and what shared state remains uncovered. Too little locking corrupts data; too much can deadlock, which is corruption with better manners.

My preferred debugging method is to trace one operation and print object addresses and names carefully: mount, lookup, open, read, release, and unmount. A toy in-memory filesystem is especially educational because there is no disk format to distract from the object relationships.

The concise version is: a superblock is a mount, an inode is an object, a dentry is a cached name, and a file is an open instance. It is not the whole VFS, but it is enough to stop asking an inode for an open offset. That alone made the source considerably less mysterious.

Waking a Process in Linux 2.4

Wed, 16 Oct 2002 06:38:21 +0000

I had a character driver polling a flag in a loop. It worked, if one defines working as consuming a processor while waiting for a byte that might arrive next Tuesday. A wait queue is the proper mechanism.

The process prepares to sleep on a queue, checks the condition, and asks the scheduler to run something else. The producer changes the condition and wakes sleepers. In 2.4 code, the shape can be written with the wait-event helpers:

static DECLARE_WAIT_QUEUE_HEAD(read_wait);
static int data_ready;

if (wait_event_interruptible(read_wait, data_ready))
 return -ERESTARTSYS;

When data becomes available:

data_ready = 1;
wake_up_interruptible(&read_wait);

The condition is the important part. Wakeups are hints to run and check state, not ownership of an event. A process may wake because of a signal, another reader may consume the data first, or the condition may change again before this process runs. The helper loops around the test, which avoids treating a wakeup as a guarantee.

For interruptible sleep, a pending signal returns control to the caller. Driver code should propagate the interruption rather than quietly restarting a complicated operation itself. User space can then decide what to do.

The producer and consumer still need proper locking around shared state. A wait queue arranges sleeping and waking; it does not make data_ready and the associated buffer safe on a multiprocessor machine. I protect the state with the lock appropriate to all its callers, taking care not to sleep while holding a spinlock.

Underneath, the scheduler tracks each process in a task_struct, including its state. A sleeping task is not selected to run until it is made runnable again. This is why sleeping is so much cheaper than repeatedly checking a flag, and why calling a sleeping function from interrupt context is forbidden: there is no process context there to put to sleep.

Nonblocking reads need a parallel path. If no data is ready and the file was opened with O_NONBLOCK, I return -EAGAIN instead of joining the wait queue. The same readiness condition should drive poll so programs using select or poll observe exactly what a blocking read would observe. Three slightly different definitions of “ready” produce wonderfully intermittent programs.

I prefer wait queues over homemade polling even when the first version seems simpler. The caveat is to make the condition explicit and recheck it under the same synchronization used by the producer. Otherwise the code gains a pleasant nap and an occasional missed wakeup, usually during demonstrations.

A Linux 2.4 Module with a Clean Exit

Fri, 27 Sep 2002 19:25:34 +0000

I unloaded a test module and discovered that its timer was still quite enthusiastic. The machine survived, but only because the callback lost its race with removal. That is not a synchronization strategy I want to defend.

A small 2.4 module has an initialization function and an exit function:

#include <linux/init.h>
#include <linux/module.h>

static int __init sample_init(void)
{
 printk(KERN_INFO "sample: loaded\n");
 return 0;
}

static void __exit sample_exit(void)
{
 printk(KERN_INFO "sample: unloaded\n");
}

module_init(sample_init);
module_exit(sample_exit);
MODULE_LICENSE("GPL");

Initialization should acquire resources in a known order. If the third registration fails, it must release the first two before returning an error. The exit path then releases every successfully acquired resource in the reverse order. This includes interrupt handlers, timers, task queues, proc entries, device numbers, and allocated memory.

The subtle part is activity that may still enter the module. Before freeing data, I stop new work and wait or synchronize with work already in progress. For a timer, del_timer_sync() is preferable when the callback might be running on another processor. For an interrupt, I disable the device’s source as appropriate and call free_irq(), which synchronizes with handlers before returning.

Module use counts provide another guard in 2.4 code. An open file operation commonly increments the count and release decrements it, preventing removal while file operations can still call module text. The exact helper depends on the code and kernel version, but the principle is stable: every path that lends out an active reference must take and return it symmetrically.

Registration order can reduce cleanup work. I initialize private locks and memory before exposing a device entry that another process can open. The externally visible registration comes last. During removal I reverse that idea: withdraw the entry first so no new caller arrives, then drain existing work and release private state. An object should not become public while it is only half constructed.

I keep a small state table while reviewing the code: which resources exist after each possible failure, and which callbacks can still run. If two exits require different cleanup, explicit labels in reverse order are clearer than one flag for every allocation.

I test the unhappy path too. Repeated insmod and rmmod, an intentional allocation failure, and unloading while a user process has the device open find more defects than one successful load at boot.

My preference is an exit routine so boring that each line is the mirror of initialization. The caveat is that reversal alone does not solve concurrency; first make callbacks impossible, then free what they used. Memory is patient. A callback into unloaded text is less so.