Kernel

Linux 2.6.0 was released yesterday. After the long 2.5 development series and months of test releases, there is finally a version number administrators can put into a plan without the word “test” in it. This does not mean I am replacing every 2.4 kernel before lunch. It means the migration can become deliberate rather than hypothetical.

The itch for me is desktop latency. My machine compiles, plays audio, runs KDE, and occasionally receives the unreasonable request to respond to the keyboard at the same time. Linux 2.6 has two important mechanisms for that: the O(1) scheduler and optional kernel preemption.

The scheduler keeps per-CPU active and expired priority arrays. Selecting the next task does not require scanning a growing list of runnable processes, and interactive tasks receive dynamic priority treatment based largely on sleep behaviour. Preemption goes a step further. When enabled, kernel code that is not in a critical section can be preempted, reducing the time a newly runnable high-priority task waits for a long kernel path to finish.

Neither feature creates free CPU time. A saturated machine remains saturated, and a bad driver can still ruin the afternoon. What changes is how predictably the system gives urgent work a turn.

Threading is another substantial improvement. The kernel facilities used by NPTL provide proper thread groups, efficient futex-based synchronization, and saner signal and process semantics than LinuxThreads could offer. A POSIX mutex can complete entirely in userspace when uncontended and enter the kernel only to wait or wake when there is contention. This matters both for performance and for making tools show one multithreaded process as something resembling one multithreaded process.

The device model may have the largest administrative consequence. Sysfs exposes devices, buses, classes, and drivers as related kernel objects under /sys. The kernel now has a coherent answer to questions such as “what is this device attached to?” instead of scattering hints across boot messages and /proc.

That structure makes userspace device management practical. Early work on udev can listen for hotplug events, consult sysfs, and create device nodes according to userspace policy. I am not ready to discard a working /dev setup everywhere, but moving naming policy out of the kernel is the right direction. Hardware discovery belongs in the kernel; deciding that a particular camera should have a friendly local name does not.

There are less glamorous changes that may matter more on servers. The block I/O layer and filesystem scalability have received major work. The kernel supports larger systems, more processors, and more memory without treating each as a surprising edge case. Native ALSA support gives sound users a maintained kernel interface, though moving from OSS drivers may require mixer and application adjustments.

The migration is not just a kernel file

My safe upgrade recipe begins by leaving 2.4 intact. I install 2.6 under a distinct name, add a separate boot-loader entry, and verify that selecting the old entry still works. The new kernel image is the easiest part to roll back; userspace and configuration changes require more thought.

Linux 2.6 modules use a different format, so module-init-tools must be installed. The tools are designed to coexist with the programs needed by 2.4, which is exactly what a gradual migration needs. Recent versions of procps, filesystem utilities, and other low-level packages are also worth checking against the distribution’s recommendations.

For a first boot I compile the root disk controller and root filesystem into the kernel. Once the machine boots reliably, I can introduce an initial ramdisk and modularise things. Debugging one new mechanism at a time is slower only until the first failure.

I also mount sysfs explicitly if the startup scripts do not:

mount -t sysfs none /sys

Then I test the boring list: all filesystems, swap, networking, firewall rules, sound, removable devices, suspend if the machine uses it, and clean shutdown. I compare dmesg for errors rather than merely noticing that a login prompt appeared. A successful boot proves the boot path, not the rest of the computer.

Out-of-tree drivers are the largest caveat. The internal kernel interfaces have changed, sometimes substantially, and a driver compiling on 2.4 says nothing about 2.6. Binary-only modules are especially dependent on their vendor producing a matching build. Before an upgrade, inventory these modules with the same seriousness as storage hardware.

Applications should generally continue to run. This is not a new userspace ABI. Problems are more likely to come from programs that inspect /proc, rely on old thread display details, expect OSS device behaviour, or invoke module utilities directly.

My opinion is that 2.6 is a stronger foundation, especially for interactive systems and machines with many tasks or processors. But .0 is a milestone, not a papal declaration of perfection. Production systems deserve a soak period, distribution patches, and testing against their actual workload.

I will install it quickly on my workstation and slowly on servers. This is not inconsistency. The workstation can inconvenience me; the server can introduce me to several irritated people at once. Linux 2.6.0 is ready for serious testing today, and with cautious migration it should become the ordinary kernel soon enough.

I needed to answer a simple question on a 2.6 test machine: which driver owns this network device? My old habit was to rummage through boot messages, /proc, and module output until the answer surrendered. The new device model gives us a less archaeological method.

First, mount sysfs if the installation has not done it already:

mount -t sysfs none /sys

The interesting part is not that /sys contains files. Linux can turn almost anything into files if left unattended. The useful part is the structure. Devices appear according to their physical or logical parentage under /sys/devices. Buses, such as PCI and USB, have views under /sys/bus, while /sys/class groups devices by function, such as network interfaces or block devices.

These are views of the same kernel objects, joined by symbolic links. For an Ethernet interface named eth0, this is a useful start:

readlink /sys/class/net/eth0/device
ls -l /sys/class/net/eth0/device/driver

The first link leads back into the device hierarchy. The driver link, when present, identifies the bound driver. Following parent links tells me which PCI device contains the interface and where that device sits.

This differs from /proc. Procfs grew as a place for process information and gradually acquired assorted kernel knobs and device facts. Sysfs is an exported representation of the kernel’s device model. One attribute per file is the usual convention, which makes simple scripts possible without parsing a decorative essay from the kernel.

I am deliberately cautious about those scripts. The 2.6 series is not final yet, and sysfs details can change as interfaces settle. Scripts should consume documented attributes, not depend on directory ordering or infer meaning from every internal-looking name. A readable tree is still an interface, not an invitation to marry the first implementation detail one encounters.

The larger implication is device management in userspace. If the kernel exposes device identity and events cleanly, a userspace program can create names and permissions according to policy rather than a huge static /dev or a kernel naming scheme. That work is still young, but sysfs supplies the map it needs.

For now, it has already replaced several minutes of log-diving with two symbolic links. I consider that progress, even if it has added another virtual filesystem to my vocabulary.

I have been testing the new kernel on a machine that does two incompatible jobs: compile software and remain pleasant enough to use while compiling software. Under load, the 2.6 test kernel feels less sticky than my usual 2.4 setup. The O(1) scheduler is a large part of that.

The name describes a property, not a performance promise. Choosing the next runnable task takes constant time rather than getting more expensive as the run queue grows. The scheduler keeps arrays of queues indexed by priority, plus a bitmap showing which priorities contain runnable tasks. Finding work becomes a matter of locating the first set bit and taking a task from that queue.

There are actually two priority arrays per processor: active and expired. A normal task runs for its time slice and then moves to the expired array. When active becomes empty, the kernel swaps the arrays. No grand tour through every process is required.

That design also avoids one global run queue becoming a lock everybody fights over on an SMP machine. Each processor has its own run queue, and load balancing periodically moves work when the queues become uneven. Per-processor data makes the common path cheaper, although balancing is still real work and certainly not magic.

Interactive behaviour comes from dynamic priorities. A task that sleeps often, as an editor or terminal usually does while waiting for me, receives favourable treatment when it wakes. A compiler chewing CPU continuously tends to use its slice and wait in the expired array. The scheduler is estimating interactivity from sleep behaviour; applications do not announce that they are important because a human is glaring at them.

There is a caveat. Heuristics can be fooled, and “feels faster” is not a benchmark. A database server and a desktop have very different ideas of fairness. I am watching throughput as well as latency and checking for tasks that receive odd priority treatment.

Still, the mechanism is refreshingly concrete. Constant-time selection, per-CPU queues, and explicit expired work explain the observed behaviour without fairy dust. The desktop remains responsive during a build, the build still finishes, and neither needs a special incantation. That is a scheduler improvement I can actually notice.

Linus released 2.6.0-test1 this week, which means the 2.5 development series has finally put on a tie and is pretending to be suitable for polite company. It is still a test kernel. I would not install it on the machine that pays the bills, but I do want to know what will break before 2.6 becomes ordinary.

The obvious approach is to keep the current 2.4 kernel installed, add 2.6 beside it, and make the boot loader offer both. Do not replace the known-good entry. This sounds insultingly basic until the new kernel cannot find its root filesystem and the old entry is the difference between five minutes and a rescue disk.

My first checklist is mostly userspace:

module-init-tools
recent modutils-compatible configuration
procps that understands the newer /proc output
filesystem tools matching every filesystem in use
working boot loader configuration

The module change is the one most likely to surprise people. Linux 2.6 uses a new module format and the module-init-tools programs. They can coexist with the old tools, so one installation can boot both 2.4 and 2.6. Check this before rebooting, not while staring at an emergency shell.

I also build the drivers needed to reach the root filesystem directly into the kernel for the first attempt. That means the disk controller, root filesystem, and anything else between the kernel and /sbin/init. An initial ramdisk is useful, but removing it from the first experiment makes failures much easier to understand.

The device model has changed substantially. A mounted sysfs, normally at /sys, exposes devices and drivers in a structured tree. Old scripts that scrape /proc or assume particular module names deserve suspicion. Sound users should note that ALSA is now in the kernel tree; OSS compatibility exists, but that does not make every mixer setup identical.

Finally, save the exact .config and the complete build output. make oldconfig is a convenient starting point, not proof that old choices still mean the same thing. Read every new question that affects storage, filesystems, networking, or the console.

The kernel has many attractive improvements: preemption, the O(1) scheduler, better threading support, and a much saner device model. None of them will comfort you if the machine does not boot. My plan is boring: test hardware, test userspace, test rollback, then become adventurous. Adventure is much nicer with a working fallback kernel.

I put packet bookkeeping into a network device’s interrupt handler because the information was conveniently available there. Under load, the machine reminded me that convenience inside interrupt context is borrowed at unreasonable interest.

The interrupt handler should acknowledge the device, collect enough status to know what happened, move completed descriptors along, and arrange deferred processing where appropriate. It cannot sleep, and time spent there delays other interrupt work. Long parsing, allocation loops, or diagnostic printing are especially poor guests.

Linux 2.4 offers deferred mechanisms including tasklets and the network receive path’s softirq processing. A driver can prepare received skbs and pass them to netif_rx(), allowing the protocol stack to process packets outside the hard interrupt handler. For driver-specific deferred work, I put the tasklet in the device’s private state:

struct sample_dev {
        struct tasklet_struct rx_tasklet;
        /* rings, locks, and device state */
};

static void sample_tasklet(unsigned long data)
{
        struct sample_dev *dev = (struct sample_dev *)data;

        service_completed_packets(dev);
}

static void sample_init_tasklet(struct sample_dev *dev)
{
        tasklet_init(&dev->rx_tasklet, sample_tasklet,
                     (unsigned long)dev);
}

sample_init_tasklet() runs after dev is allocated and before the interrupt can schedule dev->rx_tasklet. Teardown disables that interrupt source, calls tasklet_kill(&dev->rx_tasklet), and only then frees dev. The callback therefore never receives the null value that a static DECLARE_TASKLET(..., 0) would have supplied.

Shared rings and status fields need protection across the interrupt, deferred function, transmit entry point, and process-context control operations. I use spinlocks where the data can be touched in interrupt context and choose the interrupt-disabling variant when the same processor could otherwise interrupt a lock holder. I keep the protected section short and never call a function that can sleep while holding it.

There is a balance. Deferring every tiny operation adds overhead and can complicate ordering, while doing everything in the handler harms latency. I measure under sustained receive and transmit load, watch dropped packets, and check that another device’s interrupts are not starved.

Interrupt acknowledgement order is device-specific and worth documenting beside the handler. A level-triggered interrupt that remains asserted can immediately enter again; acknowledging too early on other hardware may lose status that has not been copied. I read the device manual rather than deriving this order from whichever experiment happened not to hang.

Shutdown receives the same scrutiny. The driver disables the device’s interrupt source, prevents new deferred work, frees the IRQ with the required synchronization, and only then releases rings and private data. Otherwise a late tasklet can faithfully process memory that has already acquired a new purpose.

I want the hard interrupt path to be visibly bounded. One caveat: printk can completely distort timing here. A hundred helpful messages per second soon become the performance problem, a promotion few debug statements deserve.