Kernel

Module Parameters in Linux 2.4

I was rebuilding a test module merely to change its debug level. That is a fine way to turn a ten-second experiment into a small ritual. Linux 2.4 module parameters are a better fit.

For a simple integer, the old module parameter declaration is compact:

static int debug;
MODULE_PARM(debug, "i");
MODULE_PARM_DESC(debug, "enable diagnostic messages");

I can then set it while loading:

insmod sample.o debug=1

The type string tells the module loader how to interpret the value. Other forms support strings and arrays, but I avoid elaborate configuration here. Parameters are useful for hardware choices and diagnostics needed at initialization, not as a private replacement for a proper user interface.

Validation still belongs in the initialization function. A parameter arriving from insmod is input, not a promise. If a buffer count must be positive and bounded, the module should reject nonsense with -EINVAL before allocating resources.

I also print the effective non-default setting at load time when it materially changes behavior. That makes dmesg useful when somebody reports that the driver behaves differently on one machine.

My preference is a quiet default and one obvious parameter for debugging. Five boolean switches rapidly become a combination lock whose correct setting nobody remembers. Recompiling was tedious, but at least it did not accept debug=banana with a straight face.

Reading an skb Without Regretting It

I added a packet check to a Linux 2.4 network path and initially treated skb->data as if it always began with the header I wanted. It did in my first test. Networking code has a generous way of rewarding that assumption just long enough to become embarrassing.

The sk_buff carries packet data plus bookkeeping used as the packet moves through layers. Its head, data, tail, and end pointers describe the allocated area and the currently occupied bytes. Protocol processing can pull a consumed header from the front or push space for a new one without copying the whole packet.

Before reading a header, I check that the required bytes are present and use the header position appropriate to that point in the stack. In a simple driver receive path, the driver allocates an skb, reserves alignment if needed, copies or maps packet bytes, sets the protocol, and hands it upward:

skb = dev_alloc_skb(length + 2);
if (!skb)
    return -ENOMEM;

skb_reserve(skb, 2);
memcpy(skb_put(skb, length), packet, length);
skb->dev = dev;
skb->protocol = eth_type_trans(skb, dev);
netif_rx(skb);

Calling skb_put() extends the valid data at the tail and returns the start of the added area. Writing beyond that area corrupts memory, so the allocation and length must agree. eth_type_trans() interprets the Ethernet header, sets the protocol, and adjusts the skb for the network layer.

Ownership changes at netif_rx(). After handing the skb to the network stack, the driver must not free or inspect it as though it still belongs to the receive routine. Error paths before that handoff must free what they allocated.

Transmit has the opposite pressure. The driver’s hard_start_xmit receives an skb and must obey the device queue rules. If hardware resources are exhausted, stopping the queue and waking it when descriptors become available is better than quietly dropping ownership conventions. Interrupt and bottom-half paths also require locking suited to their contexts; sleeping locks are not available there.

I prefer using skb helpers over pointer arithmetic, even when the arithmetic looks obvious. The helpers document whether code is adding, removing, or reserving bytes and retain their checks in debugging builds. The caveat is that not every skb is necessarily laid out as one convenient linear region at every layer, so code must respect the guarantees of its exact call site. Packets are simple on diagrams. The diagrams do not have cache lines.

Building a Module for the Running Kernel

I compiled a small module against the headers in /usr/include/linux, loaded it, and received a cheerful list of unresolved symbols. The mistake was treating user-space headers as a kernel build environment.

A module must be built for the kernel that will load it. I first identify that kernel and locate its configured source tree:

uname -r
ls -l /lib/modules/`uname -r`/build

On a packaged system the build link commonly points to the matching kernel sources or prepared headers. The tree must have the configuration and generated headers corresponding to the installed kernel. Merely unpacking a similar 2.4 source archive is not enough.

For a tiny experiment, the compile command shows the important ingredients:

gcc -D__KERNEL__ -DMODULE -O2 -Wall \
    -I/lib/modules/`uname -r`/build/include \
    -c sample.c -o sample.o

Real module makefiles should take their flags from the target kernel’s build setup rather than accumulating copied guesses. Processor options, SMP support, and symbol versioning can all affect the expected object. If the kernel was built with module versions, the build also needs the matching version definitions; disabling the complaint does not make incompatible structures compatible.

I compare the module and running kernel before loading, then inspect the actual diagnostic:

insmod sample.o
dmesg

An unresolved symbol may mean the providing facility is not configured, is itself a module not yet loaded, was not exported, or has a different version. depmod -a and modprobe handle dependency information for installed modules, while insmod is useful when I deliberately want the direct error from one test object.

The important mechanism is that a loadable module links against symbols exported by the running kernel and other loaded modules. It is not a normal program with its own copy of kernel interfaces. Configuration differences therefore matter even when function names look identical.

Optimization is not optional decoration here. Kernel headers use inline functions and assumptions expected by the normal build flags, so an unusual hand-written command can expose failures absent from an ordinary kernel build. I use the short command only to understand the pieces, then put repeatable work in a makefile tied to the configured tree.

Before installing, I test removal as well as insertion and confirm that no old object with the same module name is already loaded. Debugging yesterday’s module while reading today’s source is a surprisingly convincing experience.

I prefer building against the exact configured tree and letting version checks fail loudly. The caveat is that matching uname -r is necessary but not magical; a locally rebuilt kernel with the same release string can still differ. Names are labels, not fingerprints, despite what some build directories would like me to believe.

Following a File Through the Linux 2.4 VFS

I was reading a small filesystem and kept confusing dentries, inodes, and open files. They all seemed to point at roughly the same object until a rename and two simultaneous opens proved otherwise. I finally followed one pathname through the Linux 2.4 VFS and wrote down what each structure represents.

The VFS is the layer that gives user programs the usual operations such as open, read, and stat while allowing ext2, NFS, procfs, and other filesystems to implement different storage rules. It does this with common objects and tables of function pointers. The useful mental model is not one structure per file. It is several structures per role.

A filesystem implementation first registers a file_system_type:

static DECLARE_FSTYPE(sample_type, "samplefs", sample_read_super,
                      FS_REQUIRES_DEV);

static int __init sample_init(void)
{
    return register_filesystem(&sample_type);
}

In 2.4, mounting leads to the filesystem’s read_super function. It validates or reads the on-disk superblock, fills the VFS super_block, installs superblock operations, and supplies a root dentry. The superblock represents one mounted filesystem instance, not the filesystem format in general. Two mounted ext2 partitions therefore have separate superblocks while sharing ext2’s implementation code.

The inode represents a filesystem object and its metadata: type, mode, owner, times, size, and operation tables. A filesystem usually embeds or associates its private inode information with the VFS inode. The inode number identifies it within that filesystem. Hard links make the distinction important: several directory names can refer to the same inode.

The dentry represents a name in a directory and connects pathname lookup to an inode. It belongs to the dentry cache, remembers parent and child relationships, and may even be negative, meaning that a lookup established that a name does not currently exist. A dentry is therefore about a pathname component and cached lookup state, not merely another spelling of inode.

Suppose a process opens /mnt/notes/todo. Pathname lookup starts from a root or current directory and walks components. For each component, the VFS uses cached dentries when possible and asks the parent inode’s lookup operation when necessary. By the end, it has a dentry for todo and, if the file exists, an associated inode.

Opening then creates a file object for that particular open instance. It records the current offset, access mode, flags, and file operations. Two processes opening the same inode receive distinct file objects and can have different offsets. After fork, file descriptors may instead refer to the same open file object, which explains why their offsets can move together.

The operation tables divide responsibility. Directory inode operations include tasks such as lookup, create, link, and rename. File operations include read, write, llseek, open, and release. Superblock operations deal with the mounted instance, including writing metadata and releasing it. The VFS calls through these tables without needing ext2 rules in generic pathname code.

A tiny read method illustrates another boundary:

static ssize_t sample_read(struct file *file, char *buf,
                           size_t count, loff_t *ppos)
{
    const char text[] = "hello\n";
    size_t left;

    if (*ppos >= sizeof(text) - 1)
        return 0;

    left = sizeof(text) - 1 - *ppos;
    if (count > left)
        count = left;
    if (copy_to_user(buf, text + *ppos, count))
        return -EFAULT;

    *ppos += count;
    return count;
}

The method receives the open file, not just the inode, because position and open flags belong to the open instance. It uses copy_to_user() because a user pointer cannot be treated as an ordinary kernel pointer. Returning zero at end of file is part of the read contract, not an error.

Caching makes lifetime rules less obvious. Closing the last file descriptor does not imply that the inode or dentry disappears immediately; the caches may retain them. Conversely, unlinking a name removes a directory entry but an already open file can remain usable until its final reference is released. Reference counts and VFS helper functions, rather than guesses about user-visible names, govern these lifetimes.

Locking also follows roles. Operations changing a directory need the directory inode’s semaphore according to the VFS calling rules, while filesystem-private data may need its own protection. I do not add locks at random around every pointer. I first determine which VFS locks are already held for that operation and what shared state remains uncovered. Too little locking corrupts data; too much can deadlock, which is corruption with better manners.

My preferred debugging method is to trace one operation and print object addresses and names carefully: mount, lookup, open, read, release, and unmount. A toy in-memory filesystem is especially educational because there is no disk format to distract from the object relationships.

The concise version is: a superblock is a mount, an inode is an object, a dentry is a cached name, and a file is an open instance. It is not the whole VFS, but it is enough to stop asking an inode for an open offset. That alone made the source considerably less mysterious.

Waking a Process in Linux 2.4

I had a character driver polling a flag in a loop. It worked, if one defines working as consuming a processor while waiting for a byte that might arrive next Tuesday. A wait queue is the proper mechanism.

The process prepares to sleep on a queue, checks the condition, and asks the scheduler to run something else. The producer changes the condition and wakes sleepers. In 2.4 code, the shape can be written with the wait-event helpers:

static DECLARE_WAIT_QUEUE_HEAD(read_wait);
static int data_ready;

if (wait_event_interruptible(read_wait, data_ready))
    return -ERESTARTSYS;

When data becomes available:

data_ready = 1;
wake_up_interruptible(&read_wait);

The condition is the important part. Wakeups are hints to run and check state, not ownership of an event. A process may wake because of a signal, another reader may consume the data first, or the condition may change again before this process runs. The helper loops around the test, which avoids treating a wakeup as a guarantee.

For interruptible sleep, a pending signal returns control to the caller. Driver code should propagate the interruption rather than quietly restarting a complicated operation itself. User space can then decide what to do.

The producer and consumer still need proper locking around shared state. A wait queue arranges sleeping and waking; it does not make data_ready and the associated buffer safe on a multiprocessor machine. I protect the state with the lock appropriate to all its callers, taking care not to sleep while holding a spinlock.

Underneath, the scheduler tracks each process in a task_struct, including its state. A sleeping task is not selected to run until it is made runnable again. This is why sleeping is so much cheaper than repeatedly checking a flag, and why calling a sleeping function from interrupt context is forbidden: there is no process context there to put to sleep.

Nonblocking reads need a parallel path. If no data is ready and the file was opened with O_NONBLOCK, I return -EAGAIN instead of joining the wait queue. The same readiness condition should drive poll so programs using select or poll observe exactly what a blocking read would observe. Three slightly different definitions of “ready” produce wonderfully intermittent programs.

I prefer wait queues over homemade polling even when the first version seems simpler. The caveat is to make the condition explicit and recheck it under the same synchronization used by the producer. Otherwise the code gains a pleasant nap and an occasional missed wakeup, usually during demonstrations.