Blog

I compiled a small module against the headers in /usr/include/linux, loaded it, and received a cheerful list of unresolved symbols. The mistake was treating user-space headers as a kernel build environment.

A module must be built for the kernel that will load it. I first identify that kernel and locate its configured source tree:

uname -r
ls -l /lib/modules/`uname -r`/build

On a packaged system the build link commonly points to the matching kernel sources or prepared headers. The tree must have the configuration and generated headers corresponding to the installed kernel. Merely unpacking a similar 2.4 source archive is not enough.

For a tiny experiment, the compile command shows the important ingredients:

gcc -D__KERNEL__ -DMODULE -O2 -Wall \
    -I/lib/modules/`uname -r`/build/include \
    -c sample.c -o sample.o

Real module makefiles should take their flags from the target kernel’s build setup rather than accumulating copied guesses. Processor options, SMP support, and symbol versioning can all affect the expected object. If the kernel was built with module versions, the build also needs the matching version definitions; disabling the complaint does not make incompatible structures compatible.

I compare the module and running kernel before loading, then inspect the actual diagnostic:

insmod sample.o
dmesg

An unresolved symbol may mean the providing facility is not configured, is itself a module not yet loaded, was not exported, or has a different version. depmod -a and modprobe handle dependency information for installed modules, while insmod is useful when I deliberately want the direct error from one test object.

The important mechanism is that a loadable module links against symbols exported by the running kernel and other loaded modules. It is not a normal program with its own copy of kernel interfaces. Configuration differences therefore matter even when function names look identical.

Optimization is not optional decoration here. Kernel headers use inline functions and assumptions expected by the normal build flags, so an unusual hand-written command can expose failures absent from an ordinary kernel build. I use the short command only to understand the pieces, then put repeatable work in a makefile tied to the configured tree.

Before installing, I test removal as well as insertion and confirm that no old object with the same module name is already loaded. Debugging yesterday’s module while reading today’s source is a surprisingly convincing experience.

I prefer building against the exact configured tree and letting version checks fail loudly. The caveat is that matching uname -r is necessary but not magical; a locally rebuilt kernel with the same release string can still differ. Names are labels, not fingerprints, despite what some build directories would like me to believe.

I was reading a small filesystem and kept confusing dentries, inodes, and open files. They all seemed to point at roughly the same object until a rename and two simultaneous opens proved otherwise. I finally followed one pathname through the Linux 2.4 VFS and wrote down what each structure represents.

The VFS is the layer that gives user programs the usual operations such as open, read, and stat while allowing ext2, NFS, procfs, and other filesystems to implement different storage rules. It does this with common objects and tables of function pointers. The useful mental model is not one structure per file. It is several structures per role.

A filesystem implementation first registers a file_system_type:

static DECLARE_FSTYPE(sample_type, "samplefs", sample_read_super,
                      FS_REQUIRES_DEV);

static int __init sample_init(void)
{
    return register_filesystem(&sample_type);
}

In 2.4, mounting leads to the filesystem’s read_super function. It validates or reads the on-disk superblock, fills the VFS super_block, installs superblock operations, and supplies a root dentry. The superblock represents one mounted filesystem instance, not the filesystem format in general. Two mounted ext2 partitions therefore have separate superblocks while sharing ext2’s implementation code.

The inode represents a filesystem object and its metadata: type, mode, owner, times, size, and operation tables. A filesystem usually embeds or associates its private inode information with the VFS inode. The inode number identifies it within that filesystem. Hard links make the distinction important: several directory names can refer to the same inode.

The dentry represents a name in a directory and connects pathname lookup to an inode. It belongs to the dentry cache, remembers parent and child relationships, and may even be negative, meaning that a lookup established that a name does not currently exist. A dentry is therefore about a pathname component and cached lookup state, not merely another spelling of inode.

Suppose a process opens /mnt/notes/todo. Pathname lookup starts from a root or current directory and walks components. For each component, the VFS uses cached dentries when possible and asks the parent inode’s lookup operation when necessary. By the end, it has a dentry for todo and, if the file exists, an associated inode.

Opening then creates a file object for that particular open instance. It records the current offset, access mode, flags, and file operations. Two processes opening the same inode receive distinct file objects and can have different offsets. After fork, file descriptors may instead refer to the same open file object, which explains why their offsets can move together.

The operation tables divide responsibility. Directory inode operations include tasks such as lookup, create, link, and rename. File operations include read, write, llseek, open, and release. Superblock operations deal with the mounted instance, including writing metadata and releasing it. The VFS calls through these tables without needing ext2 rules in generic pathname code.

A tiny read method illustrates another boundary:

static ssize_t sample_read(struct file *file, char *buf,
                           size_t count, loff_t *ppos)
{
    const char text[] = "hello\n";
    size_t left;

    if (*ppos >= sizeof(text) - 1)
        return 0;

    left = sizeof(text) - 1 - *ppos;
    if (count > left)
        count = left;
    if (copy_to_user(buf, text + *ppos, count))
        return -EFAULT;

    *ppos += count;
    return count;
}

The method receives the open file, not just the inode, because position and open flags belong to the open instance. It uses copy_to_user() because a user pointer cannot be treated as an ordinary kernel pointer. Returning zero at end of file is part of the read contract, not an error.

Caching makes lifetime rules less obvious. Closing the last file descriptor does not imply that the inode or dentry disappears immediately; the caches may retain them. Conversely, unlinking a name removes a directory entry but an already open file can remain usable until its final reference is released. Reference counts and VFS helper functions, rather than guesses about user-visible names, govern these lifetimes.

Locking also follows roles. Operations changing a directory need the directory inode’s semaphore according to the VFS calling rules, while filesystem-private data may need its own protection. I do not add locks at random around every pointer. I first determine which VFS locks are already held for that operation and what shared state remains uncovered. Too little locking corrupts data; too much can deadlock, which is corruption with better manners.

My preferred debugging method is to trace one operation and print object addresses and names carefully: mount, lookup, open, read, release, and unmount. A toy in-memory filesystem is especially educational because there is no disk format to distract from the object relationships.

The concise version is: a superblock is a mount, an inode is an object, a dentry is a cached name, and a file is an open instance. It is not the whole VFS, but it is enough to stop asking an inode for an open offset. That alone made the source considerably less mysterious.

I had a character driver polling a flag in a loop. It worked, if one defines working as consuming a processor while waiting for a byte that might arrive next Tuesday. A wait queue is the proper mechanism.

The process prepares to sleep on a queue, checks the condition, and asks the scheduler to run something else. The producer changes the condition and wakes sleepers. In 2.4 code, the shape can be written with the wait-event helpers:

static DECLARE_WAIT_QUEUE_HEAD(read_wait);
static int data_ready;

if (wait_event_interruptible(read_wait, data_ready))
    return -ERESTARTSYS;

When data becomes available:

data_ready = 1;
wake_up_interruptible(&read_wait);

The condition is the important part. Wakeups are hints to run and check state, not ownership of an event. A process may wake because of a signal, another reader may consume the data first, or the condition may change again before this process runs. The helper loops around the test, which avoids treating a wakeup as a guarantee.

For interruptible sleep, a pending signal returns control to the caller. Driver code should propagate the interruption rather than quietly restarting a complicated operation itself. User space can then decide what to do.

The producer and consumer still need proper locking around shared state. A wait queue arranges sleeping and waking; it does not make data_ready and the associated buffer safe on a multiprocessor machine. I protect the state with the lock appropriate to all its callers, taking care not to sleep while holding a spinlock.

Underneath, the scheduler tracks each process in a task_struct, including its state. A sleeping task is not selected to run until it is made runnable again. This is why sleeping is so much cheaper than repeatedly checking a flag, and why calling a sleeping function from interrupt context is forbidden: there is no process context there to put to sleep.

Nonblocking reads need a parallel path. If no data is ready and the file was opened with O_NONBLOCK, I return -EAGAIN instead of joining the wait queue. The same readiness condition should drive poll so programs using select or poll observe exactly what a blocking read would observe. Three slightly different definitions of “ready” produce wonderfully intermittent programs.

I prefer wait queues over homemade polling even when the first version seems simpler. The caveat is to make the condition explicit and recheck it under the same synchronization used by the producer. Otherwise the code gains a pleasant nap and an occasional missed wakeup, usually during demonstrations.

I unloaded a test module and discovered that its timer was still quite enthusiastic. The machine survived, but only because the callback lost its race with removal. That is not a synchronization strategy I want to defend.

A small 2.4 module has an initialization function and an exit function:

#include <linux/init.h>
#include <linux/module.h>

static int __init sample_init(void)
{
    printk(KERN_INFO "sample: loaded\n");
    return 0;
}

static void __exit sample_exit(void)
{
    printk(KERN_INFO "sample: unloaded\n");
}

module_init(sample_init);
module_exit(sample_exit);
MODULE_LICENSE("GPL");

Initialization should acquire resources in a known order. If the third registration fails, it must release the first two before returning an error. The exit path then releases every successfully acquired resource in the reverse order. This includes interrupt handlers, timers, task queues, proc entries, device numbers, and allocated memory.

The subtle part is activity that may still enter the module. Before freeing data, I stop new work and wait or synchronize with work already in progress. For a timer, del_timer_sync() is preferable when the callback might be running on another processor. For an interrupt, I disable the device’s source as appropriate and call free_irq(), which synchronizes with handlers before returning.

Module use counts provide another guard in 2.4 code. An open file operation commonly increments the count and release decrements it, preventing removal while file operations can still call module text. The exact helper depends on the code and kernel version, but the principle is stable: every path that lends out an active reference must take and return it symmetrically.

Registration order can reduce cleanup work. I initialize private locks and memory before exposing a device entry that another process can open. The externally visible registration comes last. During removal I reverse that idea: withdraw the entry first so no new caller arrives, then drain existing work and release private state. An object should not become public while it is only half constructed.

I keep a small state table while reviewing the code: which resources exist after each possible failure, and which callbacks can still run. If two exits require different cleanup, explicit labels in reverse order are clearer than one flag for every allocation.

I test the unhappy path too. Repeated insmod and rmmod, an intentional allocation failure, and unloading while a user process has the device open find more defects than one successful load at boot.

My preference is an exit routine so boring that each line is the mirror of initialization. The caveat is that reversal alone does not solve concurrency; first make callbacks impossible, then free what they used. Memory is patient. A callback into unloaded text is less so.

I added history.cpp to an application, ran make, and spent several minutes wondering why none of my mistakes appeared in the compiler output. The build was not unusually forgiving. I had neglected to tell it that the file existed.

In a typical KDE source tree, the immediate place to look is Makefile.am. The target’s source list must contain both implementation and relevant headers:

bin_PROGRAMS = notebook

notebook_SOURCES = main.cpp notebook.cpp history.cpp \
                   notebook.h history.h

notebook_LDADD = $(LIB_KDEUI)

Automake uses this description to generate the lower-level make rules. KDE’s admin machinery and configure input then arrange compiler flags, library checks, translations, installation paths, and moc handling. Editing the generated Makefile is temporary theatre; the next regeneration removes the change with admirable confidence.

After changing the build description, I regenerate from the source root:

make -f Makefile.cvs
./configure --prefix="$KDEDIR" --enable-debug
make

For an out-of-tree build I run configure from the build directory after regenerating the source-side files. If I have added a new subdirectory, it also needs to appear in the parent’s SUBDIRS, and its makefile must be included by the configure machinery. A source file alone is simpler and normally needs only the target list.

Headers with Q_OBJECT are worth checking after the first build. If moc output is absent, I confirm the header is listed and that the macro is visible in the class declaration. Stale dependencies can confuse matters, so I prefer a fresh build directory before rewriting build rules that may already be correct.

Installation lists are a separate concern from compilation lists. A desktop file belongs in the appropriate data variable with its destination, translations belong in their directory, and public headers need an installation rule only if they truly form an interface. Putting every file into EXTRA_DIST may make an archive complete, but it does not make installation correct.

I check both operations before sending a change:

make dist
make install DESTDIR=/tmp/notebook-install

The staged installation makes missing icons and resources visible without scattering test files through the real prefix. I inspect the archive too, because a local build can use a file that never entered the distribution. Generated files and source files have different rules, and guessing which will be recreated on another machine is avoidable.

I keep custom shell commands out of Makefile.am unless the normal primary and source variables cannot express the job. Automake syntax can feel indirect, but the standard forms understand clean builds, distribution archives, and dependency tracking. A handcrafted shortcut often understands only the exact machine on which it was written, which is a rather narrow audience.