Blog

A program I was debugging had six POSIX threads, ps showed one process, and /proc showed several task directories. This produced the inevitable question: are Linux threads implemented in userspace or in the kernel? The unhelpful but accurate answer is both.

The pthread library provides the interface the application sees: pthread_create, mutexes, condition variables, thread-local state, and POSIX rules. With NPTL, creating a thread eventually uses the kernel’s clone() facility to create another schedulable task that shares selected resources with its siblings.

The kernel does not need a completely different object named “thread.” It schedules tasks. Flags passed to clone() determine whether tasks share an address space, file descriptor table, signal handlers, and other state. Tasks in the same thread group form what userspace presents as one process.

This is visible under procfs:

ls /proc/$(pidof myprogram)/task

Each entry is a task ID. The main thread’s task ID is also the process ID normally shown by tools. Updated ps can display either the process-oriented view or individual threads, depending on its options. Neither view is lying; they answer different questions.

Synchronization also crosses the boundary selectively. An uncontended NPTL mutex is normally handled with atomic operations in userspace. Entering the kernel for every lock and unlock would be expensive. When a thread cannot acquire the lock, a futex operation lets it sleep in the kernel; an unlock can wake a waiter when necessary.

This “fast userspace, kernel on contention” pattern explains why futex means fast userspace mutex. It does not mean the kernel knows nothing about waiting threads. The kernel supplies the queueing and scheduling that userspace cannot safely improvise.

Signal handling is where abstractions become less tidy. Some signals target the process or thread group, while others target a particular task. The library and kernel cooperate to provide POSIX behaviour. Code should use pthread signal operations instead of guessing that numeric task IDs are interchangeable with old Linux process IDs.

The same warning applies to debugging. If one thread blocks, inspect the task view; if the question concerns resource ownership, remember that file descriptors and memory may be shared. Killing a random task because it looks like a child process is not thread debugging. It is experimental program termination.

I find the model easier once I stop asking whether a thread is “really” a process. In Linux it is a schedulable task with a particular set of shared resources, grouped so userspace can provide process and pthread semantics. That is a mechanism, not a philosophical position, and it matches what the tools are showing.

I enabled kernel preemption on my workstation and was rewarded with the usual technical diagnosis: the machine feels nicer. That is useful as an observation and useless as an explanation, so I went back to what the option actually changes.

Normal process preemption is not new. The scheduler can stop one userspace task and run another. The relevant 2.6 option allows a task executing kernel code to be preempted too, provided it is not in a critical section where switching would be unsafe.

Suppose a low-priority process enters a long kernel path just before an audio application becomes runnable. Without kernel preemption, the audio task may wait until that path returns to userspace or reaches another scheduling point. With preemption enabled, the scheduler can run the newly awakened task sooner once the current kernel context is in a preemptible state.

The mechanism depends on keeping count of places where preemption must not occur. Spinlocks and explicit preemption disabling protect critical regions. Interrupt context has its own restrictions. The option does not permit the scheduler to interrupt arbitrary code while it holds shared state halfway through an update. That would improve latency in much the same way removing the brakes improves a car’s acceleration.

For a practical comparison I run a kernel build while playing audio, moving windows, and causing disk activity. I watch for skips and input stalls, but I also time the build. Lower worst-case latency is the target; higher total throughput is not guaranteed.

time make -j2 bzImage

On this desktop, preemption reduces the noticeable pauses under mixed load. The difference is most apparent when a background job causes frequent kernel work rather than merely consuming CPU. The O(1) scheduler already decides quickly which task should run; preemption helps the kernel reach a point where that decision can take effect.

I would not enable it automatically on every server. Preemption introduces scheduling opportunities and some overhead, while a throughput-oriented workload may gain nothing from lower interactive latency. Test the actual workload and look at variance, not only an average benchmark.

There are also limits. Long non-preemptible sections, interrupt handlers, slow hardware, and drivers can still cause delays. The option narrows one source of latency; it does not turn a general-purpose kernel into a hard real-time system.

For workstations, audio systems, and other latency-sensitive machines, it is a sensible trade. For a batch server, the default conservative choice may remain better. The nice thing is that the trade is explicit. I can choose responsiveness where I need it without pretending every Linux machine has the same job.

KDE 3.2 was released this week, and I have moved my normal account from 3.1 to the new version. The upgrade was painless on this machine, but I still copied .kde first. Confidence is a fine emotion; it is not a backup strategy.

The feature I notice most is Kontact. Having KMail, KOrganizer, the address book, and related tools in one window sounds cosmetic until one uses it all day. The components remain useful separately, but the shared shell reduces window hunting and makes the PIM suite feel like one piece of software rather than neighbours who happen to share a fence.

Konqueror feels more polished as both browser and file manager. KHTML handles more of the troublesome pages I encounter, although “works in Safari” still does not guarantee “works in Konqueror.” They share ancestry and code, not every platform integration or every patch. Web developers should test both instead of converting the relationship into a new browser-detection shortcut.

I also like the many small improvements more than any single headline. Configuration pages are easier to navigate, applications agree more often about behaviour, and the visual result is cleaner. KDE already had enough knobs to operate a modest submarine. The useful work now is choosing good defaults and making the common paths obvious.

My post-upgrade check is deliberately mundane:

send and receive mail
open old calendar entries
print from two applications
test file associations
visit SSL sites
check keyboard shortcuts
log out and back in

That last step catches session restoration and startup problems that a quick tour misses. I also removed third-party panel applets before upgrading and added them back one at a time. Plugins compiled against an older set of libraries are not where I want to begin debugging the desktop itself.

Performance is difficult to judge scientifically from a chair, but the desktop remains responsive and startup is reasonable. The larger applications naturally consume memory; integration does not repeal arithmetic. On a modest machine I would choose which Kontact components to load rather than assuming the complete suite is free.

KDE 3.2 is a substantial release, but it succeeds because the desktop feels more connected, not because the version contains a longer feature list. I can do the same work with fewer interruptions. That is not glamorous, and it is exactly what I want from a desktop.

I keep hearing /sys described as “the new /proc.” This is close enough to create the wrong expectation.

Procfs primarily exposes processes, plus a historical collection of kernel information and controls that accumulated there because it was available. Sysfs exposes the kernel device model. Its directories represent devices, buses, drivers, and classes, with symbolic links expressing how those objects relate.

For example, /sys/class/net/eth0 is the network-class view of an interface. Its device link leads to the underlying hardware in /sys/devices, and that device may have a driver link identifying what controls it.

readlink /sys/class/net/eth0/device
readlink /sys/class/net/eth0/device/driver

That relationship is the useful bit. I no longer need to combine a line from lspci, a module name from lsmod, and a hopeful guess.

Sysfs attributes are generally small files containing one value. This is much easier to consume than parsing a pretty table, but I would not make scripts depend on every path visible today. Use the stable, documented attributes provided for the job. The filesystem mirrors kernel objects, and not every implementation detail is a promise.

The distinction also explains why sysfs matters to udev. Userspace receives a device event, follows the structured information in /sys, and applies naming or permission policy. /proc was never a clean database for that purpose.

So I mount both. /proc answers process and assorted kernel questions; /sys answers device-model questions. Linux has not replaced one miscellaneous cupboard with another. It has, at last, labelled a new set of shelves.

Linux 2.6.0 was released yesterday. After the long 2.5 development series and months of test releases, there is finally a version number administrators can put into a plan without the word “test” in it. This does not mean I am replacing every 2.4 kernel before lunch. It means the migration can become deliberate rather than hypothetical.

The itch for me is desktop latency. My machine compiles, plays audio, runs KDE, and occasionally receives the unreasonable request to respond to the keyboard at the same time. Linux 2.6 has two important mechanisms for that: the O(1) scheduler and optional kernel preemption.

The scheduler keeps per-CPU active and expired priority arrays. Selecting the next task does not require scanning a growing list of runnable processes, and interactive tasks receive dynamic priority treatment based largely on sleep behaviour. Preemption goes a step further. When enabled, kernel code that is not in a critical section can be preempted, reducing the time a newly runnable high-priority task waits for a long kernel path to finish.

Neither feature creates free CPU time. A saturated machine remains saturated, and a bad driver can still ruin the afternoon. What changes is how predictably the system gives urgent work a turn.

Threading is another substantial improvement. The kernel facilities used by NPTL provide proper thread groups, efficient futex-based synchronization, and saner signal and process semantics than LinuxThreads could offer. A POSIX mutex can complete entirely in userspace when uncontended and enter the kernel only to wait or wake when there is contention. This matters both for performance and for making tools show one multithreaded process as something resembling one multithreaded process.

The device model may have the largest administrative consequence. Sysfs exposes devices, buses, classes, and drivers as related kernel objects under /sys. The kernel now has a coherent answer to questions such as “what is this device attached to?” instead of scattering hints across boot messages and /proc.

That structure makes userspace device management practical. Early work on udev can listen for hotplug events, consult sysfs, and create device nodes according to userspace policy. I am not ready to discard a working /dev setup everywhere, but moving naming policy out of the kernel is the right direction. Hardware discovery belongs in the kernel; deciding that a particular camera should have a friendly local name does not.

There are less glamorous changes that may matter more on servers. The block I/O layer and filesystem scalability have received major work. The kernel supports larger systems, more processors, and more memory without treating each as a surprising edge case. Native ALSA support gives sound users a maintained kernel interface, though moving from OSS drivers may require mixer and application adjustments.

The migration is not just a kernel file

My safe upgrade recipe begins by leaving 2.4 intact. I install 2.6 under a distinct name, add a separate boot-loader entry, and verify that selecting the old entry still works. The new kernel image is the easiest part to roll back; userspace and configuration changes require more thought.

Linux 2.6 modules use a different format, so module-init-tools must be installed. The tools are designed to coexist with the programs needed by 2.4, which is exactly what a gradual migration needs. Recent versions of procps, filesystem utilities, and other low-level packages are also worth checking against the distribution’s recommendations.

For a first boot I compile the root disk controller and root filesystem into the kernel. Once the machine boots reliably, I can introduce an initial ramdisk and modularise things. Debugging one new mechanism at a time is slower only until the first failure.

I also mount sysfs explicitly if the startup scripts do not:

mount -t sysfs none /sys

Then I test the boring list: all filesystems, swap, networking, firewall rules, sound, removable devices, suspend if the machine uses it, and clean shutdown. I compare dmesg for errors rather than merely noticing that a login prompt appeared. A successful boot proves the boot path, not the rest of the computer.

Out-of-tree drivers are the largest caveat. The internal kernel interfaces have changed, sometimes substantially, and a driver compiling on 2.4 says nothing about 2.6. Binary-only modules are especially dependent on their vendor producing a matching build. Before an upgrade, inventory these modules with the same seriousness as storage hardware.

Applications should generally continue to run. This is not a new userspace ABI. Problems are more likely to come from programs that inspect /proc, rely on old thread display details, expect OSS device behaviour, or invoke module utilities directly.

My opinion is that 2.6 is a stronger foundation, especially for interactive systems and machines with many tasks or processors. But .0 is a milestone, not a papal declaration of perfection. Production systems deserve a soak period, distribution patches, and testing against their actual workload.

I will install it quickly on my workstation and slowly on servers. This is not inconsistency. The workstation can inconvenience me; the server can introduce me to several irritated people at once. Linux 2.6.0 is ready for serious testing today, and with cautious migration it should become the ordinary kernel soon enough.