Threads on Roberto Selbach

One Event Loop, One Owner

Fri, 02 Mar 2007 20:16:00 +0000

A worker emitted a progress signal today, the label updated twice, and then the application hung while closing. The visible update tempted me to declare the cross-thread code correct. The shutdown was less charitable.

Qt gives each QObject thread affinity. Queued events for that object are processed by the event loop in its owning thread. A signal connected across threads normally becomes a queued call, so the receiver’s slot runs where the receiver lives. This permits a worker to report results without directly touching GUI objects.

The safe shape is simple:

connect(worker, SIGNAL(progress(int)),
 dialog, SLOT(setProgress(int)),
 Qt::QueuedConnection);

The explicit connection type documents the boundary. The worker emits an integer; the dialog owns the label and updates it on the GUI thread.

My hang came from destruction. The dialog waited for the worker while handling its close event, and the worker was waiting for a queued reply from the dialog’s event loop. The GUI event loop could not deliver the reply because it was blocked waiting. Both sides were admirably patient.

I removed the synchronous callback. Closing now requests cancellation, disables the interface, and lets normal event processing continue until the worker announces completion. Only then is it destroyed. Data passed in signals is copied as needed, so custom value types must be registered if they cross a queued connection.

I prefer message-like queued signals between worker and GUI code. They make ownership visible and avoid shared widget state. The caveat is that queued delivery is asynchronous: the sender cannot assume the slot has run, event loops must remain active, and ordering guarantees do not replace a proper state machine.

Nested event loops are another temptation. Opening a modal operation or manually calling event processing can keep paints moving, but it also permits unrelated events to re-enter code whose invariants are half changed. I use that technique only with a very clear boundary, never as a general cure for long work.

The practical test now closes the window at several stages of the job. Starting and finishing are easy; cancellation and teardown reveal who actually owns what. If a design works only while the user behaves, the user is not the unreliable component.

Const Does Not Mean Concurrent

Thu, 21 Sep 2006 19:26:00 +0000

A cache lookup crashed under load today even though every worker called a const method. That sentence sounds reassuring only if const is mistaken for a locking primitive.

The method updated a mutable cache:

QImage ImageStore::image(const QString &key) const
{
 if (!m_cache.contains(key))
 m_cache.insert(key, loadImage(key));
 return m_cache.value(key);
}

const prevents ordinary modification through the public type. It says nothing about simultaneous calls, and mutable explicitly allows internal state to change. Two processors can both inspect and modify the cache, corrupting its internal bookkeeping.

I first added a mutex around the whole operation. That was correct but serialised disk loading, so unrelated images waited behind one slow file. The improved version checks under lock, loads outside it, then locks again to publish:

QImage ImageStore::image(const QString &key) const
{
 m_mutex.lock();
 if (m_cache.contains(key)) {
 QImage image = m_cache.value(key);
 m_mutex.unlock();
 return image;
 }
 m_mutex.unlock();

 QImage image = loadImage(key);

 m_mutex.lock();
 if (!m_cache.contains(key))
 m_cache.insert(key, image);
 else
 image = m_cache.value(key);
 m_mutex.unlock();
 return image;
}

Duplicate loading is possible, but publication is safe and the common cached path is short. For expensive loads I might track in-progress keys with a wait condition; that complexity needs measurements first.

Implicitly shared Qt values make returning an image cheap in the usual case, but they do not make shared storage safe for concurrent mutation. Copy-on-write protects value semantics, not arbitrary access patterns.

The lock itself belongs beside the state it protects. Passing the cache to helper code without the mutex invites a second access path that quietly ignores the rule. I am documenting the invariant as “all cache lookup and publication occurs while holding m_mutex,” then checking each method against that sentence.

I prefer immutable inputs and per-job results when using several cores. Shared caches should be small, explicitly locked, and treated as an optimisation rather than the foundation of correctness. The caveat is memory and copying: some data is too large to duplicate, so careful shared ownership remains necessary.

The failure appeared only on the dual-core machine and only after several minutes. That does not make it a rare bug. It makes the single-core test machine an excellent accomplice.

ThreadWeaver and the Size of a Job

Wed, 14 Jun 2006 18:54:00 +0000

I fed 12,000 thumbnail jobs into a ThreadWeaver experiment today. The queue accepted them with admirable composure. The application then spent more time managing tiny jobs than producing thumbnails.

The mechanism is straightforward: jobs enter a queue, worker threads take runnable jobs, and completion is reported back. This is much better than creating one thread per thumbnail. Threads have stacks, scheduling cost, and an unfortunate tendency to outlive the code that launched them.

My mistake was choosing a job that was too small. Each item performed a little path work, checked a cache, and often returned without decoding anything. Queue locking and signal delivery became a noticeable fraction of total work.

Batching nearby items helped:

class ThumbnailBatch : public ThreadWeaver::Job
{
public:
 ThumbnailBatch(const QStringList &paths)
 : m_paths(paths)
 {
 }

protected:
 void run()
 {
 foreach (const QString &path, m_paths)
 createThumbnail(path);
 }

private:
 QStringList m_paths;
};

The batch is large enough to amortise scheduling but small enough that several workers remain busy. There is no universal number. Disk latency, decoding cost, and cache hit rate all change the useful grain size, so I am measuring rather than engraving 32 on a tablet.

Cancellation also becomes a design issue. Removing queued jobs is easy; stopping one already decoding is cooperative. Long operations need checkpoints, and the job must own or safely reference everything it touches. A window closing while a worker holds its raw pointer is not cancellation. It is delayed excitement.

I prefer a shared job queue such as ThreadWeaver for independent background work because it bounds concurrency and centralises scheduling. The caveat is that it does not make unsafe code safe. Shared caches still need locking, GUI objects still belong to the GUI thread, and dependencies between jobs still need explicit representation.

Priority needs restraint as well. Marking every visible thumbnail urgent merely recreates a first-in queue with more bookkeeping. I reserve priority for work whose delay is observable now, and still ensure ordinary jobs eventually run.

The current prototype sends immutable input into a job and returns a completed image through a queued signal. That copy may cost something, but the ownership story fits in one sentence. For now, clarity beats shaving a copy whose cost I have not measured.

Threads in Userspace and the Kernel

Thu, 18 Mar 2004 17:50:00 +0000

A program I was debugging had six POSIX threads, ps showed one process, and /proc showed several task directories. This produced the inevitable question: are Linux threads implemented in userspace or in the kernel? The unhelpful but accurate answer is both.

The pthread library provides the interface the application sees: pthread_create, mutexes, condition variables, thread-local state, and POSIX rules. With NPTL, creating a thread eventually uses the kernel’s clone() facility to create another schedulable task that shares selected resources with its siblings.

The kernel does not need a completely different object named “thread.” It schedules tasks. Flags passed to clone() determine whether tasks share an address space, file descriptor table, signal handlers, and other state. Tasks in the same thread group form what userspace presents as one process.

This is visible under procfs:

ls /proc/$(pidof myprogram)/task

Each entry is a task ID. The main thread’s task ID is also the process ID normally shown by tools. Updated ps can display either the process-oriented view or individual threads, depending on its options. Neither view is lying; they answer different questions.

Synchronization also crosses the boundary selectively. An uncontended NPTL mutex is normally handled with atomic operations in userspace. Entering the kernel for every lock and unlock would be expensive. When a thread cannot acquire the lock, a futex operation lets it sleep in the kernel; an unlock can wake a waiter when necessary.

This “fast userspace, kernel on contention” pattern explains why futex means fast userspace mutex. It does not mean the kernel knows nothing about waiting threads. The kernel supplies the queueing and scheduling that userspace cannot safely improvise.

Signal handling is where abstractions become less tidy. Some signals target the process or thread group, while others target a particular task. The library and kernel cooperate to provide POSIX behaviour. Code should use pthread signal operations instead of guessing that numeric task IDs are interchangeable with old Linux process IDs.

The same warning applies to debugging. If one thread blocks, inspect the task view; if the question concerns resource ownership, remember that file descriptors and memory may be shared. Killing a random task because it looks like a child process is not thread debugging. It is experimental program termination.

I find the model easier once I stop asking whether a thread is “really” a process. In Linux it is a schedulable task with a particular set of shared resources, grouped so userspace can provide process and pthread semantics. That is a mechanism, not a philosophical position, and it matches what the tools are showing.

NPTL and the Disappearing Processes

Tue, 14 Oct 2003 20:05:00 +0000

I upgraded a test machine to a recent glibc with NPTL and immediately thought several processes had disappeared. An old LinuxThreads program used to make ps look as if every thread had invited itself to the process list. With NPTL and updated tools, the display is much closer to what the programmer meant: one process containing several threads.

The old implementation was clever, but its compromises leaked. LinuxThreads created tasks with clone() and used a manager thread for coordination. Thread IDs and process IDs did not line up neatly with POSIX expectations, signal delivery had awkward corners, and tools often presented threads as peculiar sibling processes.

NPTL also builds threads on kernel tasks, but newer kernel facilities let it implement POSIX semantics much more directly. Thread groups identify tasks belonging to one process. Futexes provide a fast synchronization primitive: uncontended locking can happen in userspace, while the kernel becomes involved when a thread must sleep or wake another waiter. That avoids a system call for every successful mutex operation.

The practical lesson is to upgrade the tools with the library. A stale ps, debugger, or monitoring script can report technically available data in a thoroughly unhelpful way. On this machine I check both the process view and the individual tasks under /proc/PID/task when debugging:

ls /proc/1234/task

Each directory there is a kernel task ID. The process has a thread-group leader, and the other tasks share resources according to the flags used when they were created. “Thread” is not a completely separate kernel creature. It is a task sharing an address space and other state with its group.

Compatibility deserves some attention too. Programs should use the POSIX thread API and not depend on LinuxThreads accidents such as each thread appearing to have an unrelated process identity. Code that sends signals to guessed numeric IDs is especially suspicious. Correct code generally needs no source change, but incorrect assumptions have enjoyed years to become tradition.

NPTL is faster in useful places, particularly thread creation and synchronization, but I like the semantic cleanup more. Better benchmarks are welcome; fewer explanations beginning with “well, on Linux a thread looks like…” are better. The implementation is finally making the common abstraction less dishonest.