Kernel

A driver experiment needed to do a little cleanup after an interrupt. I put the cleanup directly in the interrupt handler because it was only a little work. Then that little work called something that could sleep, and the kernel objected with considerably more confidence than I had shown.

The naive model says an interrupt handler is just a function the kernel calls quickly. The missing detail is context. Hard interrupt context cannot sleep: there is no process there to suspend in the normal sense, and holding up interrupt handling harms the rest of the machine.

The useful pattern is to split the operation. The handler acknowledges the device, records the minimal state under the right lock, and schedules deferred work. A workqueue function later runs in process context and may use operations that can sleep.

static void device_work(void *data)
{
        struct sample_device *dev = data;

        /* Slow cleanup is safe in process context. */
        finish_request(dev);
}

The exact initialization APIs deserve a check against the headers for the kernel being built; this area has changed before and surely will again. The invariant matters more than memorizing one macro: know which context executes the callback, what locks are held, and whether every function it calls may sleep.

Allocation flags make the distinction concrete. Code in a context that cannot sleep must not request an allocation mode that may block. Moving slow work to a workqueue lets that callback use normal process-context operations, but it does not make shared state safe automatically.

Teardown is the other half. Before freeing the device state, the driver must stop scheduling work and ensure already queued callbacks have finished. Otherwise the deferred function merely postpones the crash until after the object is gone.

I added an explicit comment at the scheduling point and reduced the handler to the smallest useful unit. Latency improved, but the larger gain was conceptual. Once context is part of the function’s contract, calls that looked harmless become obviously wrong.

Kernel C is not difficult because the syntax is exotic. It is difficult because an ordinary-looking call carries constraints about context, ownership, and concurrency. Ignoring those constraints produces bugs that wait patiently for the least convenient machine. Mine at least had the courtesy to complain immediately.

A USB card reader prompted today’s trip through /sys. I wanted one answer: which physical device sits behind the block-class entry /sys/block/sda?

The useful relationship is the device symlink. /sys/block/sda presents the disk as a member of the block class; its device link points into the physical device hierarchy. Those are two views of one kernel object, not two devices.

I started at the block entry and followed its device link rather than guessing from the node name:

$ readlink -f /sys/block/sda/device
/sys/devices/pci0000:00/0000:00:10.4/usb2/2-2/2-2:1.0/host1/1:0:0:0

The exact target is machine-specific. Here it says that the block disk came through a SCSI host attached to USB interface 2-2:1.0, itself below a PCI USB controller. On another machine the chain will differ, but the link still joins the class view to the device that implements it.

Walking upward from that target lets me inspect the parents that actually own bus attributes. The block entry doesn’t need to copy USB details into its own directory; the parent links already express the relationship.

I don’t parse the punctuation in the resolved path. Directory names and topology are useful while diagnosing a machine, but exported links and attributes are the interface. Hard-coding every component would just trade /dev/sda for a longer accidental name.

This also explains why /dev/sda isn’t a hardware biography. The node names a block interface; sysfs supplies the link back to the device model when I need to ask where that interface came from.

The service corridors are a little dim, but this one symlink answers the question without guesswork.

I have been comparing two Linux 2.6 configurations for a workstation: one with kernel preemption enabled and one without it. Looking only at total build time made them appear nearly identical, while using the machine made them feel different. The missing measurement was latency.

Throughput asks how much work finishes in an interval. Latency asks how long a particular piece of work waits. A kernel build is a reasonable throughput load, but its completion time does not reveal that an audio process waited long enough to miss a deadline or that keyboard input paused for a fraction of a second.

My crude but useful test runs the same build while playing audio, copying a large file, and interacting with KDE. I record build time, note audio underruns reported by the application, and repeat enough times to distinguish a pattern from one lucky run.

time make -j2 bzImage

The O(1) scheduler helps when many tasks are runnable. Its priority arrays and per-CPU run queues keep selection costs predictable, while interactive heuristics favour tasks that sleep and wake frequently. But the scheduler cannot run a task until the current execution context can be switched.

That is where kernel preemption matters. With it enabled, ordinary kernel code can be preempted when no lock or other critical condition forbids it. An interactive task awakened during a long system call may run sooner. Interrupt handlers and protected sections remain non-preemptible, so a poor driver can still dominate worst-case delay.

On my machine, enabling preemption does not materially improve the build time. It does reduce occasional input stalls and audio trouble under mixed I/O load. That is the result I expected: better responsiveness, not newly manufactured processor cycles.

Thread behaviour can alter the picture too. NPTL mutexes stay in userspace when uncontended and use futex operations when a thread must sleep or wake. A heavily contended threaded program may therefore create a very different scheduling load from one with mostly private work. Counting threads alone says little; runnable and blocked states matter.

I also test without excessive parallelism. Starting far more compile jobs than processors can turn the test into a contest between memory pressure, disk traffic, and scheduling. That may resemble a real workload, but it is a poor way to isolate one mechanism.

The caveat is that my observations are not hard real-time guarantees. General-purpose Linux still has sources of unbounded or hardware-dependent delay, and an average hides the worst pause. For desktop tuning, however, repeated underrun counts and interaction under a controlled load tell me more than one impressive jobs-per-minute number.

I am keeping preemption enabled on the workstation and evaluating it separately on servers. A server handling many requests may care about response-time distribution, but a batch machine may prefer the smallest scheduling overhead. There is no universally fast switch. First decide whether the itch is throughput or latency; only then does the benchmark know what question to answer.

I have now moved a machine I actually depend upon from Linux 2.4 to 2.6. Test boxes are excellent for discovering that a kernel boots. A working machine is better at discovering the scanner, firewall rule, and obscure filesystem one forgot to test.

The obvious approach is to install the new kernel and reboot. The correct approach begins with an inventory. I wrote down the storage controller, root filesystem, network devices, sound hardware, mounted filesystems, firewall setup, and every module not supplied by the kernel tree. If a component is required to boot or reach the network, I want its exact name before changing anything.

I then upgraded the required userspace while still running 2.4. module-init-tools understands 2.6 modules and supplies compatibility behaviour for a 2.4 boot. Recent procps tools understand newer /proc details and NPTL process presentation. Filesystem utilities must support the on-disk filesystems regardless of which kernel initiated the repair.

The important point is coexistence. I did not overwrite the old kernel image, old modules, or boot entry. My boot loader contains distinct choices, and I tested selecting 2.4 after installing the new tools. A rollback that has never been attempted is a comforting theory.

Building the first kernel

I used the 2.4 configuration as a reference, not scripture. make oldconfig carries choices forward and asks about new options, but subsystems changed between the series. I read the help for storage, input, sound, device filesystems, networking, and processor options instead of accepting every default at speed.

For the first build, the disk controller and root filesystem went directly into the kernel. This avoids depending on a new module loader and a new initial ramdisk during the same boot. Once the plain configuration worked, I made less essential drivers modular.

make oldconfig
make bzImage
make modules
make modules_install

The exact image installation is distribution and architecture dependent, so I copied it using the same procedure as the existing packaged kernel and created a separate boot-loader stanza. I also saved .config beside the image with a matching name.

Linux 2.6 uses sysfs for its device model. My startup scripts mount it at /sys:

mount -t sysfs none /sys

I kept the existing /dev arrangement for the first migration. Early udev is interesting and I intend to test it, but changing kernel, module format, device discovery, and device-node management in one reboot produces a beautiful failure with too many suspects.

The first boot is only the start

The machine reached a login prompt immediately, which proved almost nothing. I checked dmesg from beginning to end for timeouts, unknown options, and drivers claiming the wrong hardware. I mounted every local filesystem, activated swap, transferred data over each network interface, and exercised packet filtering.

Sound moved to ALSA, now part of the kernel tree. Applications using OSS interfaces can often use ALSA’s compatibility modules, but mixer state and device permissions still need checking. Silence may mean the channel is muted rather than the driver is broken, an ancient audio tradition maintained for compatibility with human suffering.

Threaded programs deserve a run under the new combination of kernel and C library. NPTL relies on facilities available in 2.6 and improves both performance and POSIX behaviour, but old monitoring scripts may interpret the changed process display incorrectly. I checked services by function rather than by expecting the old number of ps lines.

The firewall required careful review. The netfilter framework remains, but kernel configuration symbols and available matches can differ. Loading an old saved rule set is useful only if all required modules exist and the restore command reports success. A script that ignores errors can leave a machine less protected while printing reassuring startup messages.

Out-of-tree modules were the only serious obstacle. Internal kernel interfaces changed, and source written for 2.4 often needs a real port rather than a version-number adjustment. I found replacements in the 2.6 tree where possible. For the remaining hardware, I verified current source before making the migration permanent. Binary-only drivers put the schedule entirely in the vendor’s hands, which is one reason I avoid depending on them.

What changed in use

Under a compile and interactive desktop load, the machine is more responsive. The O(1) scheduler’s per-CPU run queues and priority arrays make task selection predictable as load grows. Kernel preemption, enabled on this workstation, reduces the delay before an awakened interactive task can run when another process is in preemptible kernel code.

This does not mean every benchmark is faster. Responsiveness concerns latency; throughput concerns completed work over time. I measured both and accepted a tiny difference in build time in exchange for fewer audio skips and shorter input pauses.

Sysfs is already useful for diagnosis. Following links from /sys/class to a device and its driver is clearer than assembling the relationship from several /proc files and boot messages. It also provides the structured device information that userspace managers such as udev need.

My migration rule is now simple: preserve 2.4, upgrade userspace first, boot a minimally clever 2.6 configuration, and add new mechanisms one at a time. Linux 2.6 is ready for this machine, but it did not become ready merely because uname -r printed the desired number. It became ready when every required workload passed and the old kernel remained one menu selection away.

I enabled kernel preemption on my workstation and was rewarded with the usual technical diagnosis: the machine feels nicer. That is useful as an observation and useless as an explanation, so I went back to what the option actually changes.

Normal process preemption is not new. The scheduler can stop one userspace task and run another. The relevant 2.6 option allows a task executing kernel code to be preempted too, provided it is not in a critical section where switching would be unsafe.

Suppose a low-priority process enters a long kernel path just before an audio application becomes runnable. Without kernel preemption, the audio task may wait until that path returns to userspace or reaches another scheduling point. With preemption enabled, the scheduler can run the newly awakened task sooner once the current kernel context is in a preemptible state.

The mechanism depends on keeping count of places where preemption must not occur. Spinlocks and explicit preemption disabling protect critical regions. Interrupt context has its own restrictions. The option does not permit the scheduler to interrupt arbitrary code while it holds shared state halfway through an update. That would improve latency in much the same way removing the brakes improves a car’s acceleration.

For a practical comparison I run a kernel build while playing audio, moving windows, and causing disk activity. I watch for skips and input stalls, but I also time the build. Lower worst-case latency is the target; higher total throughput is not guaranteed.

time make -j2 bzImage

On this desktop, preemption reduces the noticeable pauses under mixed load. The difference is most apparent when a background job causes frequent kernel work rather than merely consuming CPU. The O(1) scheduler already decides quickly which task should run; preemption helps the kernel reach a point where that decision can take effect.

I would not enable it automatically on every server. Preemption introduces scheduling opportunities and some overhead, while a throughput-oriented workload may gain nothing from lower interactive latency. Test the actual workload and look at variance, not only an average benchmark.

There are also limits. Long non-preemptible sections, interrupt handlers, slow hardware, and drivers can still cause delays. The option narrows one source of latency; it does not turn a general-purpose kernel into a hard real-time system.

For workstations, audio systems, and other latency-sensitive machines, it is a sensible trade. For a batch server, the default conservative choice may remain better. The nice thing is that the trade is explicit. I can choose responsiveness where I need it without pretending every Linux machine has the same job.