Administration on Roberto Selbach

Moving a Real Machine from Linux 2.4 to 2.6

Thu, 15 Apr 2004 22:20:00 +0000

I have now moved a machine I actually depend upon from Linux 2.4 to 2.6. Test boxes are excellent for discovering that a kernel boots. A working machine is better at discovering the scanner, firewall rule, and obscure filesystem one forgot to test.

The obvious approach is to install the new kernel and reboot. The correct approach begins with an inventory. I wrote down the storage controller, root filesystem, network devices, sound hardware, mounted filesystems, firewall setup, and every module not supplied by the kernel tree. If a component is required to boot or reach the network, I want its exact name before changing anything.

I then upgraded the required userspace while still running 2.4. module-init-tools understands 2.6 modules and supplies compatibility behaviour for a 2.4 boot. Recent procps tools understand newer /proc details and NPTL process presentation. Filesystem utilities must support the on-disk filesystems regardless of which kernel initiated the repair.

The important point is coexistence. I did not overwrite the old kernel image, old modules, or boot entry. My boot loader contains distinct choices, and I tested selecting 2.4 after installing the new tools. A rollback that has never been attempted is a comforting theory.

Building the first kernel

I used the 2.4 configuration as a reference, not scripture. make oldconfig carries choices forward and asks about new options, but subsystems changed between the series. I read the help for storage, input, sound, device filesystems, networking, and processor options instead of accepting every default at speed.

For the first build, the disk controller and root filesystem went directly into the kernel. This avoids depending on a new module loader and a new initial ramdisk during the same boot. Once the plain configuration worked, I made less essential drivers modular.

make oldconfig
make bzImage
make modules
make modules_install

The exact image installation is distribution and architecture dependent, so I copied it using the same procedure as the existing packaged kernel and created a separate boot-loader stanza. I also saved .config beside the image with a matching name.

Linux 2.6 uses sysfs for its device model. My startup scripts mount it at /sys:

mount -t sysfs none /sys

I kept the existing /dev arrangement for the first migration. Early udev is interesting and I intend to test it, but changing kernel, module format, device discovery, and device-node management in one reboot produces a beautiful failure with too many suspects.

The first boot is only the start

The machine reached a login prompt immediately, which proved almost nothing. I checked dmesg from beginning to end for timeouts, unknown options, and drivers claiming the wrong hardware. I mounted every local filesystem, activated swap, transferred data over each network interface, and exercised packet filtering.

Sound moved to ALSA, now part of the kernel tree. Applications using OSS interfaces can often use ALSA’s compatibility modules, but mixer state and device permissions still need checking. Silence may mean the channel is muted rather than the driver is broken, an ancient audio tradition maintained for compatibility with human suffering.

Threaded programs deserve a run under the new combination of kernel and C library. NPTL relies on facilities available in 2.6 and improves both performance and POSIX behaviour, but old monitoring scripts may interpret the changed process display incorrectly. I checked services by function rather than by expecting the old number of ps lines.

The firewall required careful review. The netfilter framework remains, but kernel configuration symbols and available matches can differ. Loading an old saved rule set is useful only if all required modules exist and the restore command reports success. A script that ignores errors can leave a machine less protected while printing reassuring startup messages.

Out-of-tree modules were the only serious obstacle. Internal kernel interfaces changed, and source written for 2.4 often needs a real port rather than a version-number adjustment. I found replacements in the 2.6 tree where possible. For the remaining hardware, I verified current source before making the migration permanent. Binary-only drivers put the schedule entirely in the vendor’s hands, which is one reason I avoid depending on them.

What changed in use

Under a compile and interactive desktop load, the machine is more responsive. The O(1) scheduler’s per-CPU run queues and priority arrays make task selection predictable as load grows. Kernel preemption, enabled on this workstation, reduces the delay before an awakened interactive task can run when another process is in preemptible kernel code.

This does not mean every benchmark is faster. Responsiveness concerns latency; throughput concerns completed work over time. I measured both and accepted a tiny difference in build time in exchange for fewer audio skips and shorter input pauses.

Sysfs is already useful for diagnosis. Following links from /sys/class to a device and its driver is clearer than assembling the relationship from several /proc files and boot messages. It also provides the structured device information that userspace managers such as udev need.

My migration rule is now simple: preserve 2.4, upgrade userspace first, boot a minimally clever 2.6 configuration, and add new mechanisms one at a time. Linux 2.6 is ready for this machine, but it did not become ready merely because uname -r printed the desired number. It became ready when every required workload passed and the old kernel remained one menu selection away.

sysfs Is Not a New proc

Thu, 08 Jan 2004 18:40:00 +0000

I keep hearing /sys described as “the new /proc.” This is close enough to create the wrong expectation.

Procfs primarily exposes processes, plus a historical collection of kernel information and controls that accumulated there because it was available. Sysfs exposes the kernel device model. Its directories represent devices, buses, drivers, and classes, with symbolic links expressing how those objects relate.

For example, /sys/class/net/eth0 is the network-class view of an interface. Its device link leads to the underlying hardware in /sys/devices, and that device may have a driver link identifying what controls it.

readlink /sys/class/net/eth0/device
readlink /sys/class/net/eth0/device/driver

That relationship is the useful bit. I no longer need to combine a line from lspci, a module name from lsmod, and a hopeful guess.

Sysfs attributes are generally small files containing one value. This is much easier to consume than parsing a pretty table, but I would not make scripts depend on every path visible today. Use the stable, documented attributes provided for the job. The filesystem mirrors kernel objects, and not every implementation detail is a promise.

The distinction also explains why sysfs matters to udev. Userspace receives a device event, follows the structured information in /sys, and applies naming or permission policy. /proc was never a clean database for that purpose.

So I mount both. /proc answers process and assorted kernel questions; /sys answers device-model questions. Linux has not replaced one miscellaneous cupboard with another. It has, at last, labelled a new set of shelves.

Linux 2.6.0 Arrives

Thu, 18 Dec 2003 23:15:00 +0000

Linux 2.6.0 was released yesterday. After the long 2.5 development series and months of test releases, there is finally a version number administrators can put into a plan without the word “test” in it. This does not mean I am replacing every 2.4 kernel before lunch. It means the migration can become deliberate rather than hypothetical.

The itch for me is desktop latency. My machine compiles, plays audio, runs KDE, and occasionally receives the unreasonable request to respond to the keyboard at the same time. Linux 2.6 has two important mechanisms for that: the O(1) scheduler and optional kernel preemption.

The scheduler keeps per-CPU active and expired priority arrays. Selecting the next task does not require scanning a growing list of runnable processes, and interactive tasks receive dynamic priority treatment based largely on sleep behaviour. Preemption goes a step further. When enabled, kernel code that is not in a critical section can be preempted, reducing the time a newly runnable high-priority task waits for a long kernel path to finish.

Neither feature creates free CPU time. A saturated machine remains saturated, and a bad driver can still ruin the afternoon. What changes is how predictably the system gives urgent work a turn.

Threading is another substantial improvement. The kernel facilities used by NPTL provide proper thread groups, efficient futex-based synchronization, and saner signal and process semantics than LinuxThreads could offer. A POSIX mutex can complete entirely in userspace when uncontended and enter the kernel only to wait or wake when there is contention. This matters both for performance and for making tools show one multithreaded process as something resembling one multithreaded process.

The device model may have the largest administrative consequence. Sysfs exposes devices, buses, classes, and drivers as related kernel objects under /sys. The kernel now has a coherent answer to questions such as “what is this device attached to?” instead of scattering hints across boot messages and /proc.

That structure makes userspace device management practical. Early work on udev can listen for hotplug events, consult sysfs, and create device nodes according to userspace policy. I am not ready to discard a working /dev setup everywhere, but moving naming policy out of the kernel is the right direction. Hardware discovery belongs in the kernel; deciding that a particular camera should have a friendly local name does not.

There are less glamorous changes that may matter more on servers. The block I/O layer and filesystem scalability have received major work. The kernel supports larger systems, more processors, and more memory without treating each as a surprising edge case. Native ALSA support gives sound users a maintained kernel interface, though moving from OSS drivers may require mixer and application adjustments.

The migration is not just a kernel file

My safe upgrade recipe begins by leaving 2.4 intact. I install 2.6 under a distinct name, add a separate boot-loader entry, and verify that selecting the old entry still works. The new kernel image is the easiest part to roll back; userspace and configuration changes require more thought.

Linux 2.6 modules use a different format, so module-init-tools must be installed. The tools are designed to coexist with the programs needed by 2.4, which is exactly what a gradual migration needs. Recent versions of procps, filesystem utilities, and other low-level packages are also worth checking against the distribution’s recommendations.

For a first boot I compile the root disk controller and root filesystem into the kernel. Once the machine boots reliably, I can introduce an initial ramdisk and modularise things. Debugging one new mechanism at a time is slower only until the first failure.

I also mount sysfs explicitly if the startup scripts do not:

mount -t sysfs none /sys

Then I test the boring list: all filesystems, swap, networking, firewall rules, sound, removable devices, suspend if the machine uses it, and clean shutdown. I compare dmesg for errors rather than merely noticing that a login prompt appeared. A successful boot proves the boot path, not the rest of the computer.

Out-of-tree drivers are the largest caveat. The internal kernel interfaces have changed, sometimes substantially, and a driver compiling on 2.4 says nothing about 2.6. Binary-only modules are especially dependent on their vendor producing a matching build. Before an upgrade, inventory these modules with the same seriousness as storage hardware.

Applications should generally continue to run. This is not a new userspace ABI. Problems are more likely to come from programs that inspect /proc, rely on old thread display details, expect OSS device behaviour, or invoke module utilities directly.

My opinion is that 2.6 is a stronger foundation, especially for interactive systems and machines with many tasks or processors. But .0 is a milestone, not a papal declaration of perfection. Production systems deserve a soak period, distribution patches, and testing against their actual workload.

I will install it quickly on my workstation and slowly on servers. This is not inconsistency. The workstation can inconvenience me; the server can introduce me to several irritated people at once. Linux 2.6.0 is ready for serious testing today, and with cautious migration it should become the ordinary kernel soon enough.

Getting Ready for Linux 2.6-test1

Thu, 17 Jul 2003 21:35:00 +0000

Linus released 2.6.0-test1 this week, which means the 2.5 development series has finally put on a tie and is pretending to be suitable for polite company. It is still a test kernel. I would not install it on the machine that pays the bills, but I do want to know what will break before 2.6 becomes ordinary.

The obvious approach is to keep the current 2.4 kernel installed, add 2.6 beside it, and make the boot loader offer both. Do not replace the known-good entry. This sounds insultingly basic until the new kernel cannot find its root filesystem and the old entry is the difference between five minutes and a rescue disk.

My first checklist is mostly userspace:

module-init-tools
recent modutils-compatible configuration
procps that understands the newer /proc output
filesystem tools matching every filesystem in use
working boot loader configuration

The module change is the one most likely to surprise people. Linux 2.6 uses a new module format and the module-init-tools programs. They can coexist with the old tools, so one installation can boot both 2.4 and 2.6. Check this before rebooting, not while staring at an emergency shell.

I also build the drivers needed to reach the root filesystem directly into the kernel for the first attempt. That means the disk controller, root filesystem, and anything else between the kernel and /sbin/init. An initial ramdisk is useful, but removing it from the first experiment makes failures much easier to understand.

The device model has changed substantially. A mounted sysfs, normally at /sys, exposes devices and drivers in a structured tree. Old scripts that scrape /proc or assume particular module names deserve suspicion. Sound users should note that ALSA is now in the kernel tree; OSS compatibility exists, but that does not make every mixer setup identical.

Finally, save the exact .config and the complete build output. make oldconfig is a convenient starting point, not proof that old choices still mean the same thing. Read every new question that affects storage, filesystems, networking, or the console.

The kernel has many attractive improvements: preemption, the O(1) scheduler, better threading support, and a much saner device model. None of them will comfort you if the machine does not boot. My plan is boring: test hardware, test userspace, test rollback, then become adventurous. Adventure is much nicer with a working fallback kernel.