Networking

I put packet bookkeeping into a network device’s interrupt handler because the information was conveniently available there. Under load, the machine reminded me that convenience inside interrupt context is borrowed at unreasonable interest.

The interrupt handler should acknowledge the device, collect enough status to know what happened, move completed descriptors along, and arrange deferred processing where appropriate. It cannot sleep, and time spent there delays other interrupt work. Long parsing, allocation loops, or diagnostic printing are especially poor guests.

Linux 2.4 offers deferred mechanisms including tasklets and the network receive path’s softirq processing. A driver can prepare received skbs and pass them to netif_rx(), allowing the protocol stack to process packets outside the hard interrupt handler. For driver-specific deferred work, I put the tasklet in the device’s private state:

struct sample_dev {
        struct tasklet_struct rx_tasklet;
        /* rings, locks, and device state */
};

static void sample_tasklet(unsigned long data)
{
        struct sample_dev *dev = (struct sample_dev *)data;

        service_completed_packets(dev);
}

static void sample_init_tasklet(struct sample_dev *dev)
{
        tasklet_init(&dev->rx_tasklet, sample_tasklet,
                     (unsigned long)dev);
}

sample_init_tasklet() runs after dev is allocated and before the interrupt can schedule dev->rx_tasklet. Teardown disables that interrupt source, calls tasklet_kill(&dev->rx_tasklet), and only then frees dev. The callback therefore never receives the null value that a static DECLARE_TASKLET(..., 0) would have supplied.

Shared rings and status fields need protection across the interrupt, deferred function, transmit entry point, and process-context control operations. I use spinlocks where the data can be touched in interrupt context and choose the interrupt-disabling variant when the same processor could otherwise interrupt a lock holder. I keep the protected section short and never call a function that can sleep while holding it.

There is a balance. Deferring every tiny operation adds overhead and can complicate ordering, while doing everything in the handler harms latency. I measure under sustained receive and transmit load, watch dropped packets, and check that another device’s interrupts are not starved.

Interrupt acknowledgement order is device-specific and worth documenting beside the handler. A level-triggered interrupt that remains asserted can immediately enter again; acknowledging too early on other hardware may lose status that has not been copied. I read the device manual rather than deriving this order from whichever experiment happened not to hang.

Shutdown receives the same scrutiny. The driver disables the device’s interrupt source, prevents new deferred work, frees the IRQ with the required synchronization, and only then releases rings and private data. Otherwise a late tasklet can faithfully process memory that has already acquired a new purpose.

I want the hard interrupt path to be visibly bounded. One caveat: printk can completely distort timing here. A hundred helpful messages per second soon become the performance problem, a promotion few debug statements deserve.

I had an application talking to a small read-only service and was about to put the protocol code directly into its file dialog path. A KIO slave turned out to be a cleaner experiment: teach KDE one URL scheme, then let Konqueror and any KIO-aware application use it.

The slave is a separate process driven through KIO’s protocol. A class derived from KIO::SlaveBase implements operations such as get, stat, or listDir, depending on what the scheme supports. For a tiny read-only resource, get is enough to prove the mechanism:

void NoteProtocol::get(const KURL &url)
{
    QByteArray bytes = loadNote(url.path());

    mimeType("text/plain");
    totalSize(bytes.size());
    data(bytes);
    data(QByteArray());
    finished();
}

Real code must report failures with error() rather than returning silently. It should also send data in useful chunks for large objects instead of reading everything into memory as this demonstration does. The empty data block marks the end of data before finished() completes the operation.

KIO discovers the slave through an installed protocol description and service information. Those files state the scheme, executable, supported operations, and whether the protocol works with files, directories, reading, or writing. If Konqueror says the protocol is unknown, I check installation paths and run kbuildsycoca before blaming the C++.

The process boundary is the interesting part. The client starts a KIO job. KIO selects and communicates with the slave, and the slave translates operations into the remote or special resource’s rules. A failure in the protocol handler need not take the client application down with it, and authentication or progress can use KDE’s common machinery.

I am careful not to claim operations the protocol cannot honor. Advertising rename for an immutable service only moves the error from an absent menu item to an annoyed user. stat results must also be consistent with get; applications rely on metadata for decisions before fetching.

I test the scheme directly in Konqueror and with a tiny KIO client, including a missing object, an empty object, and a path containing escaped characters. URL decoding must happen at the protocol boundary exactly once. Decoding twice turns a perfectly legal percent sign into an accidental instruction.

My preference is a KIO slave when a resource genuinely behaves like a hierarchy of files or directories and should be available across KDE. For one private command, DCOP or an application-specific class is smaller. Giving every service a URL is attractive, but some services are verbs wearing a hat.

I added a packet check to a Linux 2.4 network path and initially treated skb->data as if it always began with the header I wanted. It did in my first test. Networking code has a generous way of rewarding that assumption just long enough to become embarrassing.

The sk_buff carries packet data plus bookkeeping used as the packet moves through layers. Its head, data, tail, and end pointers describe the allocated area and the currently occupied bytes. Protocol processing can pull a consumed header from the front or push space for a new one without copying the whole packet.

Before reading a header, I check that the required bytes are present and use the header position appropriate to that point in the stack. In a simple driver receive path, the driver allocates an skb, reserves alignment if needed, copies or maps packet bytes, sets the protocol, and hands it upward:

skb = dev_alloc_skb(length + 2);
if (!skb)
    return -ENOMEM;

skb_reserve(skb, 2);
memcpy(skb_put(skb, length), packet, length);
skb->dev = dev;
skb->protocol = eth_type_trans(skb, dev);
netif_rx(skb);

Calling skb_put() extends the valid data at the tail and returns the start of the added area. Writing beyond that area corrupts memory, so the allocation and length must agree. eth_type_trans() interprets the Ethernet header, sets the protocol, and adjusts the skb for the network layer.

Ownership changes at netif_rx(). After handing the skb to the network stack, the driver must not free or inspect it as though it still belongs to the receive routine. Error paths before that handoff must free what they allocated.

Transmit has the opposite pressure. The driver’s hard_start_xmit receives an skb and must obey the device queue rules. If hardware resources are exhausted, stopping the queue and waking it when descriptors become available is better than quietly dropping ownership conventions. Interrupt and bottom-half paths also require locking suited to their contexts; sleeping locks are not available there.

I prefer using skb helpers over pointer arithmetic, even when the arithmetic looks obvious. The helpers document whether code is adding, removing, or reserving bytes and retain their checks in debugging builds. The caveat is that not every skb is necessarily laid out as one convenient linear region at every layer, so code must respect the guarantees of its exact call site. Packets are simple on diagrams. The diagrams do not have cache lines.

I changed a file-opening routine from local paths to URLs and immediately made the window feel stuck. The transfer was not especially slow. My code was waiting synchronously in the user-interface path, so even a short delay became visible.

KIO is built around jobs. A call such as KIO::get() starts work and returns a KIO::TransferJob; data and completion arrive through signals while the event loop continues processing input and painting.

KIO::TransferJob *job = KIO::get(url, false, false);
connect(job, SIGNAL(data(KIO::Job *, const QByteArray &)),
        this, SLOT(slotData(KIO::Job *, const QByteArray &)));
connect(job, SIGNAL(result(KIO::Job *)),
        this, SLOT(slotResult(KIO::Job *)));

The job chooses an appropriate KIO slave from the URL’s protocol. The slave runs separately, speaks the protocol, and reports data, progress, redirections, and errors back through KIO. My application consumes a common job interface instead of containing FTP, HTTP, and file code.

There are two details I now handle every time. First, data may arrive in several chunks. The data signal is not a promise that one byte array equals one document. I append or parse incrementally. Second, I treat result as the final authority. An empty final chunk does not mean success, and a non-empty first chunk does not mean the transfer will finish.

For a tiny operation, KIO::NetAccess can provide a convenient synchronous wrapper. I still avoid it from slots reached directly through buttons and menus. Nested event loops and blocked windows are an expensive price for saving two slots.

Cancellation also matters. I keep the job pointer, clear it when the job finishes, and kill the job when the owning view goes away. Because jobs are QObjects, sensible parentage helps, but explicit user cancellation should still be reflected in the interface.

Redirection is another reason not to reduce a transfer to one blocking read. The final URL may differ from the requested one, and policy about accepting it belongs in the job flow. Authentication and certificate questions can also require interaction supplied by KDE. A hand-written socket loop tends to rediscover these cases individually, generally when somebody is trying to use the program rather than when I am prepared to debug it.

For uploads I apply the same rule in reverse: feed data when the job requests it and finish according to the job’s protocol. I do not assume that writing a local temporary file and copying it afterward has identical overwrite and error behavior.

I prefer KIO jobs even for local URLs when the operation already accepts a KURL. The mechanism remains uniform and remote files stop being an awkward special case. The caveat is that asynchronous code forces the state to be honest. That is inconvenient for about an hour and useful for the rest of the program’s life.