Concurrency on Roberto Selbach

When a system call takes a thread

Thu, 24 Oct 2013 16:39:00 +0000

I assumed GOMAXPROCS(1) meant my program would use one operating-system thread. A blocking system call corrected me without displaying much tact.

In Go 1.1’s scheduler, GOMAXPROCS controls Ps, the runtime resources required to execute Go code in parallel. It does not impose a hard limit on Ms, the operating-system threads. The runtime may need more threads because one can become stuck in a system call while runnable goroutines remain.

Suppose G1 is running on M1 with P1 and enters a blocking call the network poller cannot handle asynchronously. The runtime marks the transition and detaches P1. Another thread, M2, can acquire P1 and execute G2. There is still only one P executing Go code at a time, but two operating-system threads exist because M1 is inside the kernel.

When M1 returns, its goroutine must rejoin runnable work. If no P is immediately available, the runtime can queue the goroutine and park or reuse the thread. The M does not permanently own the P, which is precisely why useful execution can continue during the call.

Runtime-managed network operations often take a cheaper path. Non-blocking sockets are registered with the network poller, and the goroutine is parked while its M continues with other work. Ordinary blocking system calls and calls into C do not necessarily have that integration, so the thread itself may be occupied.

This matters when a program makes many concurrent calls that block unpredictably. The scheduler can preserve progress by creating threads, but threads are not free. They consume kernel resources and stack address space, and excessive creation adds scheduling overhead. Wrapping a slow C function in ten thousand goroutines does not convert the function into scalable asynchronous I/O. It converts optimism into threads.

A bounded worker group is appropriate when calling a blocking interface with limited capacity:

jobs := make(chan Job)
for i := 0; i < workers; i++ {
	go func() {
		for job := range jobs {
			runBlocking(job)
		}
	}()
}

The useful bound should come from measured service capacity, not a decorative constant selected because eight feels parallel. Backpressure at the job channel also makes overload visible instead of allowing an unbounded pile of blocked goroutines and threads.

The distinction among G, M, and P is practical here. Counting goroutines does not reveal thread use. Counting threads does not reveal Go parallelism. GOMAXPROCS says how many Ps may run Go code; it does not promise a process with exactly that many threads.

I now treat blocking foreign calls and unusual system calls as resources to measure and usually bound. The scheduler is very good at preventing one blocked thread from freezing unrelated Go work. It is not obliged to make an unlimited number of blocked threads inexpensive.

How the race detector knows

Fri, 13 Sep 2013 18:21:00 +0000

After the race detector found a real bug, I wanted to know whether it was merely running the program many times and hoping for an unfortunate schedule. It is doing something much more useful.

A data race exists when two goroutines access the same memory concurrently, at least one access writes, and synchronization does not establish the required ordering. The detector instruments loads and stores so they can be recorded in shadow state associated with application memory. It also observes synchronization operations such as channel communication and mutex locking.

The key idea is causality rather than wall-clock time. Each goroutine’s operations have an order. Synchronization joins those orders: a channel send and corresponding receive, for example, establish a happens-before edge. The detector maintains enough logical-clock information to determine whether conflicting memory accesses are ordered by such edges.

Consider this broken counter:

var n int

func increment(done chan bool) {
	n++
	done <- true
}

Two goroutines executing n++ each perform a read and a write. Sending completion values afterward does not order the increments with respect to each other. The detector sees conflicting accesses without a happens-before path and reports both stacks.

Putting a mutex around the increment changes more than timing:

mu.Lock()
n++
mu.Unlock()

Unlock and a later lock establish ordering, so accesses in the critical sections no longer race. Sleeping between increments might make overlap less likely, but it adds no synchronization relation and therefore is not a fix.

Instrumentation explains the overhead of -race. Every relevant access carries bookkeeping, and shadow memory plus goroutine histories consume substantial memory. This is the cost of seeing relationships that ordinary execution discards. I run race-enabled tests separately rather than using their timings as application benchmarks.

The detector is dynamic. If a branch is never executed, its accesses are never observed. If a test uses one worker while production uses twenty, the most interesting paths may remain untouched. Good concurrent tests still matter: vary inputs, create contention, repeat operations, and run realistic integration workloads when possible.

There can also be benign-looking races that do not crash today, such as an approximate statistics counter. They remain races under the memory model. The compiler is not required to preserve the comforting behaviour I happened to observe. Use synchronization or documented atomic operations when approximation is acceptable but racing is not.

Understanding the mechanism changed how I read reports. I no longer ask only whether the two lines could execute at the exact same nanosecond. I ask what synchronization orders them. If I cannot point to that edge, the detector has probably found a hole rather than developed an artistic difference of opinion.

The race detector found my clever cache

Tue, 04 Jun 2013 14:09:00 +0000

Go 1.1’s race detector found a bug in a cache I had described as “effectively read-only.” Adverbs remain an important source of software defects.

The cache was reduced to this:

var cached map[string]string

func lookup(key string) string {
	if cached == nil {
		cached = loadAll()
	}
	return cached[key]
}

Two handlers could observe cached == nil and assign it concurrently. Worse, one could read the map while another was publishing it. Running tests with go test -race reported the conflicting accesses and the goroutine stacks involved.

The detector instruments memory accesses in the compiled program and tracks synchronization events. When two goroutines access the same location concurrently, at least one access is a write, and no synchronization orders them, it can report a data race. The stack traces are the useful part: they connect an abstract memory-model violation to the two paths I actually wrote.

I fixed the cache with a mutex and a clear initialization path. A channel-based owner goroutine would also have worked, but building a miniature cache server around one assignment would have been enthusiasm rather than design.

The detector only sees executions that happen. A test that never drives two requests through initialization cannot reveal this race. I added a test that starts several goroutines together and exercises the path repeatedly. Even then, a clean run is evidence, not proof that every possible schedule is safe.

Race-enabled binaries are slower and use more memory because instrumentation and bookkeeping are not free. I would use them in tests and representative staging workloads rather than deciding the overhead is a mysterious production regression. The point is to make races observable, not to benchmark the detector against uninstrumented code and appear shocked.

A race report is also not permission to sprinkle locks around the named lines. The conflicting accesses may reveal confused ownership, publication without synchronization, or a larger invariant spanning several fields. Protecting one variable while leaving the invariant split can silence one report and preserve the bug.

Nor should the test ignore a report because the final value happened to be correct. The detected execution violated the memory model. A passing assertion after that is luck arriving late and asking to be mistaken for synchronization.

The race detector does not find deadlocks, logical races built from properly synchronized but wrongly ordered operations, or races in code it never executes. It finds data races, and that is already an excellent service. Mine was real, small, and previously certified by the prestigious It Seems Fine Institute.

Go 1.1 and the G-M-P scheduler

Thu, 16 May 2013 18:44:00 +0000

Go 1.1 was released this week, so naturally I ignored the pleasant surface additions and went looking for the scheduler changes. This is normal behaviour in some households.

The new scheduler is commonly described with three letters: G, M, and P. A G is a goroutine, including its stack and scheduling state. An M is an operating-system thread, the machine on which Go code eventually runs. A P is a processor-like runtime resource required to execute Go code. GOMAXPROCS controls the number of Ps and therefore how much Go code can execute simultaneously.

This extra P may seem unnecessary. Why not put runnable goroutines directly on threads? The answer becomes clearer when a thread blocks.

The naive picture

Imagine each M owns a queue of goroutines:

M0: G1 G2 G3
M1: G4 G5

If G1 enters a blocking system call, M0 blocks with it. The runtime must somehow preserve access to G2 and G3, arrange another thread, and keep global scheduling state coherent. If queues and caches belong permanently to Ms, every blocked call complicates their ownership.

In the G-M-P design, runnable work and resources are associated with P. An M must acquire a P to execute Go code. If its goroutine enters a blocking system call, the M can detach from its P. Another M acquires that P and continues running other goroutines. When the system call returns, its G becomes eligible to run again, possibly on a different M.

The identities are separate because their lifetimes and reasons for existing are separate. G represents work. M represents a kernel execution context. P represents permission and resources to execute Go work.

Local queues

Each P has a local run queue. Creating a runnable goroutine can usually place it on the current P’s queue instead of contending on one global scheduler lock. The current M draws work from its P locally, which improves locality and reduces synchronization.

Local queues create an obvious question: what happens when one P has no work while another has a long queue? Idle Ps steal runnable goroutines from busy Ps. Work stealing balances execution without forcing every enqueue and dequeue through a single shared structure.

There is still a global queue for cases that need it, and the scheduler periodically considers it so global work is not ignored. “Local queues” does not mean each P becomes an isolated principality with its own foreign policy.

This organization also gives runtime caches a natural home. Per-P allocation state, for example, can be used by whichever M currently runs that P. If such state belonged to M, blocking and replacing threads would either strand it or require more coordination.

A small scheduler experiment

I used a deliberately crude program:

func burn(done chan bool) {
	x := uint64(1)
	for i := 0; i < 200000000; i++ {
		x = x*1664525 + 1013904223
	}
	fmt.Println(x)
	done <- true
}

func main() {
	runtime.GOMAXPROCS(2)
	done := make(chan bool)
	go burn(done)
	go burn(done)
	<-done
	<-done
}

With two Ps and enough operating-system support, the two CPU-bound goroutines can execute in parallel. With one P they execute one at a time and, because these loops contain no scheduling point, one may finish before the other starts running. The distinction matters. Goroutines make concurrency cheap; GOMAXPROCS and available processors determine parallel Go execution.

This benchmark is useful for observing CPU use and almost useless for choosing an application’s GOMAXPROCS. Real programs block, allocate, communicate, and touch memory. A tight arithmetic loop mostly proves that arithmetic loops enjoy processors.

Blocking, parking, and spinning

Not all waiting has the same consequence. A goroutine waiting on a channel or runtime-managed network operation can be parked, allowing the M and P to run another G. A blocking system call may hold the M, so the P is handed off. A call into C may similarly occupy a thread the scheduler cannot use for Go work until the call returns.

The scheduler also needs to avoid excessive thread creation. It tracks spinning workers searching for work and parked workers waiting to be awakened. Waking every idle M whenever one G becomes runnable would produce a thundering herd. Waking none would leave parallel capacity unused. The details are subtle because scheduling is largely the art of replacing one kind of waste with a smaller kind.

What this means in programs

The design makes several common Go patterns scale better, but it does not absolve application code.

A goroutine that blocks while holding a mutex can prevent unrelated goroutines from making useful progress even though the scheduler itself is healthy. A producer can create runnable goroutines faster than Ps can execute them. CPU-bound goroutines can compete with latency-sensitive work. The scheduler distributes runnable work; it cannot determine which business request deserves sympathy.

The safest optimization remains architectural: block goroutines on channels or network operations when they have no work, bound queues where overload is possible, and measure latency under realistic contention. Reaching into scheduler behaviour to arrange exact execution order is not a design. It is a hostage negotiation with an implementation detail.

I like the G-M-P model because it separates concerns that were tangled in earlier schedulers. It gives runnable work local queues, lets processor resources survive blocked threads, and creates better places for per-execution caches. Go 1.1 should spend less time fighting a global scheduler lock and more time running the program.

Scheduler improvements cannot make sequential code parallel, remove lock contention, or turn infinite work into finite work. They do make the straightforward goroutine model a better foundation, which beats an application-level scheduler I would have to debug myself.

The Go memory model and my broken flag

Tue, 19 Feb 2013 19:03:00 +0000

I wrote the concurrency equivalent of checking whether the refrigerator light is off by repeatedly opening the door:

var ready bool
var value string

func load() {
	value = "finished"
	ready = true
}

func main() {
	go load()
	for !ready {
	}
	fmt.Println(value)
}

It worked on my machine, which is information about my machine rather than evidence that the program is correct.

Two goroutines access ready, one writes it, and there is no synchronization. The same is true for value: seeing ready does not create a language guarantee that the write to value is visible. Compilers and processors may reorder operations where a single goroutine cannot observe the difference. Caches and registers add further opportunities for my intuition to be wrong.

The Go memory model describes which operations establish a “happens before” relationship. If one event happens before another, the effects required by that relationship are visible in the defined order. Goroutine creation, channel communication, mutex operations, and package initialization provide useful ordering guarantees. A hopeful loop around a boolean does not.

The channel version says what I meant:

func load(done chan bool) {
	value = "finished"
	done <- true
}

func main() {
	done := make(chan bool)
	go load(done)
	<-done
	fmt.Println(value)
}

The send happens before the corresponding receive completes. Since value is written before the send, reading it after the receive is properly ordered. Better still, the channel documents the handoff instead of hiding it behind a global flag.

A mutex would also work. Unlocking a mutex and subsequently locking it establishes the required ordering around protected data. The choice is not that channels are holy and mutexes are regrettable. Channels fit ownership transfer and signalling; mutexes fit shared state with a clear invariant. Using the mechanism that expresses the relationship makes the program easier to audit.

This is not merely a question of whether reading and writing a machine-sized boolean is individually atomic. Even if a load cannot tear, atomicity of that one load does not order the separate write to value. “But the flag is only one byte” answers the wrong question with admirable confidence.

The safe rule is pleasantly boring: when multiple goroutines access data and at least one modifies it, arrange synchronization unless the operations use a specifically documented atomic mechanism. Do not infer ordering from sleep calls, observed scheduling, processor brand, or the phase of the moon.

I like Go’s concurrency syntax because it makes the correct version short. That does not make every short concurrent program correct. Goroutines remove ceremony from concurrency, not causality.

Measuring a hot stack split

Mon, 03 Dec 2012 17:11:00 +0000

One benchmark had a strange cliff: adding an innocent-looking local array made a short recursive walk much slower. I suspected cache effects first. The CPU profile pointed at the runtime’s stack-growth path instead.

A goroutine begins with a small stack. Before a function uses its frame, generated code checks whether enough stack remains. If not, the runtime grows the stack by adding another segment and resumes the call. A special stack-splitting path does the unpleasant bookkeeping so ordinary Go code can continue pretending stacks are infinite, at least until memory becomes considerably less theoretical.

The benchmark was more useful as a depth sweep than as one headline number. I ran the same walk at depths 8, 16, 32, 64, and 128, then repeated it with a slightly larger local frame. Most points scaled smoothly. One narrow range jumped when calls repeatedly reached the end of a segment.

The segmented arrangement makes the initial cost of a goroutine small, but it is not magic. Each goroutine still has scheduler state and a stack segment. Every additional segment consumes memory. Deep recursion can grow a stack dramatically, and keeping the goroutine alive keeps its stack alive as well.

That narrow range is the interesting bit. If a function arrives with almost, but not quite, enough room, it grows the stack; returning can release the segment, and the next call grows it again. This “hot split” makes a particular call depth look much worse than its neighbours. A single benchmark depth can therefore blame the algorithm for standing on an implementation seam.

I checked the profile at the slow depth, then moved the benchmark one call shallower and deeper. The runtime samples faded at both neighbours. That was better evidence than merely noticing that recursion was involved. Changing the benchmark’s local array size moved the cliff too.

I did not contort the application to dodge one segment boundary. The runtime implementation will change, and ordinary code should not depend on today’s segment size. I did keep the depth sweep: it tells me whether a future optimization improves the work or merely moves the cliff.

Small stacks still make many goroutines practical. They also leave fingerprints in microbenchmarks. When one input size falls off a ledge, profile it and test both sides before rewriting the loop.

A look at the old Go scheduler

Wed, 05 Oct 2011 18:55:00 +0000

I wrote a small fan-out test and expected twice as many goroutines to mean twice as much parallel work. The result barely moved. That sent me away from the programme and into the runtime scheduler.

The test looked innocent:

for i := 0; i < 4; i++ {
	go crunch(work[i])
}

Goroutines are cheap concurrent activities, but concurrency does not itself promise simultaneous execution on every processor. In the current runtime, scheduling is still comparatively simple, and processor use depends on runtime settings, operating-system threads, blocking calls, and where the scheduler gets a chance to run.

The surprise was how many layers my little loop depended upon. A goroutine begins with a small stack and runtime bookkeeping. Runnable goroutines wait for the scheduler. The scheduler places work onto operating-system threads, and the kernel finally decides when those threads run. Linux 3.0 may schedule the threads beautifully, but it cannot create parallel Go execution the runtime never offered it.

This scheduler predates the more ambitious designs being discussed for later Go. It should not be described using machinery that does not exist yet. There is no reason to draw a modern diagram over an old runtime and then congratulate the diagram for explaining it.

Blocking is the interesting case. If a goroutine enters a system call, the runtime may need another thread so runnable Go code can continue. Channel operations and synchronization also move goroutines between runnable and waiting states. The scheduler therefore coordinates cheap user-level goroutines with more expensive kernel threads rather than mapping one permanently to the other.

I prefer to write synchronization for correctness first and treat parallel speedup as a measured property. A channel can make ownership clear even on one processor. If CPU parallelism matters, I set the runtime’s processor allowance explicitly, run a long enough workload, and compare against a serial version.

The caveat is that scheduler behaviour is an implementation detail in active development. A trick that coaxes today’s scheduler may become useless or harmful after a snapshot. Explicit communication and coarse useful work are safer investments than depending on an accidental scheduling order.

My test eventually did improve, but only after I made each unit of work large enough to outweigh coordination and enabled the processors I intended to use. Four goroutines were not four tiny employees waiting inside the computer. This was disappointing for payroll, but helpful for the benchmark.

Making the Scheduler Show Its Work

Fri, 27 May 2011 20:48:00 +0000

Scheduler discussions become folklore quickly, so I built a few experiments that force observable events instead of inferring everything from total runtime.

The first launches workers that announce when they start, wait on a gate, perform fixed CPU work, and announce completion. A channel carries each announcement to one logger goroutine. That gives me a trace without concurrent writes making the output resemble ransom typography.

func worker(id int, gate <-chan bool, events chan<- Event) {
 events <- Event{id, "ready"}
 <-gate
 burnCPU()
 events <- Event{id, "done"}
}

I run the same executable with different GOMAXPROCS values. With one processor thread allowed, CPU-bound completion is serialized even though all workers are runnable. With two, completions overlap in wall time on my dual-core machine. More than two adds runnable work but no additional cores, and eventually loses to scheduling overhead.

The second experiment replaces CPU work with pipe reads. Many goroutines can wait for I/O while the runtime arranges for other work to proceed. Increasing the processor allowance barely changes throughput because the external producer is the limit. This is why one benchmark cannot characterize “goroutine performance.”

The third experiment is intentionally rude: one goroutine runs a tight arithmetic loop with no communication while another tries to report periodically. On this early runtime, the reporter can be delayed noticeably. Scheduling behavior is under active development, and I should not assume the preemption properties of a mature operating-system scheduler.

Adding explicit communication points makes the test cooperative and reflects real pipeline code better. I do not sprinkle meaningless calls into production loops merely to influence today’s runtime; I break large work into units with natural channel operations or function calls, then measure again.

The event channel affects the experiment, of course. Observing a scheduler changes timing by adding synchronization. I keep messages outside the inner loop and compare against an uninstrumented wall-clock run. The trace explains ordering; it does not provide cycle-accurate truth.

These results belong to the May 2011 snapshot and my machine. They are not a specification of future Go scheduling, and I am deliberately avoiding an internal diagram that will expire before the ink dries. The practical findings are modest: concurrency is not parallelism, blocking and CPU work stress different paths, and processor settings should follow measurements. Schedulers are quite willing to show their work, but only if the experiment asks a specific question.

One Thread, Two Threads

Tue, 14 Dec 2010 17:58:00 +0000

I ran two independent CPU loops in goroutines and expected my dual-core machine to halve the time. It did not. Expectations remain the cheapest profiler available and the least accurate.

This runtime defaults to limited processor use. Setting GOMAXPROCS=2 for the experiment allowed Go work to execute on two operating-system threads:

$ time GOMAXPROCS=1 ./spin
real 0m3.84s

$ time GOMAXPROCS=2 ./spin
real 0m2.17s

The result is not exactly half. Both loops touch memory, the scheduler has work, and the rest of the machine has declined to disappear for my benchmark.

Goroutines and parallel execution are separate ideas. Thousands of goroutines can make a concurrent design clear while one thread runs them. Increasing the thread allowance can provide parallelism when there is independent CPU work and hardware to execute it.

The early scheduler also has rough edges. A tight loop with no calls may delay other goroutines more than I expect from mature thread schedulers. I add real synchronization rather than relying on hopeful fairness, and I test the weekly runtime I intend to use.

For I/O-heavy programs, two processor threads may change little. For CPU loops, it can help until memory bandwidth or coordination dominates. The variable is a knob, not a turbo button. Turbo buttons at least had a satisfying light.

Channel Pipelines With Back Pressure

Thu, 29 Apr 2010 22:11:00 +0000

I rewrote a small Python log processor as three stages: read lines, parse records, and aggregate counters. The Python version used queues and worker threads. The Go version uses channels, mostly because I wanted to discover where they stop being cute.

The parser looks like this in the weekly snapshot I have installed:

func parse(in <-chan string, out chan<- Record) {
 for line := range in {
 r, ok := parseLine(line)
 if ok {
 out <- r
 }
 }
 close(out)
}

Directional channel types document who sends and who receives. They also let the compiler reject an accidental send in a consumer. The reader closes its output, the parser drains that input and closes the next channel, and the aggregator stops when its input is closed.

An unbuffered channel forces a rendezvous. If aggregation falls behind, parsing blocks, then reading blocks. That is back pressure without a separate queue-length policy. For this tool it is desirable because there is no value in reading a gigabyte ahead merely to admire it in memory.

A buffered channel changes the timing, not the ownership:

records := make(chan Record, 64)

Now the parser can get 64 records ahead. I tried capacities from zero to 4096. Small buffers smoothed disk and parser timing; large ones mostly increased memory use. The best number depended on record size and machine, so naturally 64 is now a sacred constant until the next measurement.

Closing requires discipline. The sender should close a channel when no more values will arrive. A receiver closing it is guessing about other senders and can provoke a panic. Multiple producers therefore need one coordinator to wait for them and close the shared output.

The current implementation is experimental, as are some channel syntax and library details around it. This is not a claim that channels beat every queue or lock. A shared read-mostly table may be clearer behind a mutex. But for a pipeline, channels put flow control, synchronization, and the handoff of values in one visible operation. I removed two condition variables and, more importantly, stopped having to prove that I signaled both of them on every path.

Goroutines Are Not Pthreads

Thu, 18 Mar 2010 21:25:00 +0000

My reflex when seeing go f() was to translate it into “start a thread.” That translation is convenient and wrong enough to cause bad designs.

A goroutine is a language-level activity scheduled by the Go runtime. Its initial stack is small and can grow, so creating thousands is practical in cases where creating thousands of pthreads would be a conversation with both the kernel and the person carrying the pager.

I tested this with a deliberately dull program:

package main

import "fmt"

func wait(done chan bool) {
 done <- true
}

func main() {
 done := make(chan bool)
 for i := 0; i < 10000; i++ {
 go wait(done)
 }
 for i := 0; i < 10000; i++ {
 <-done
 }
 fmt.Println("finished")
}

The channel is doing two jobs: carrying a value and synchronizing sender with receiver. With an unbuffered channel, the send waits for a receive. The program does not need a mutex around a shared completion counter.

That does not make races impossible. If all those goroutines update the same map, I still have a race. “Do not communicate by sharing memory” is good direction, not a force field.

The early runtime is another caveat. On the snapshot I used, scheduling and stack management are active work. Timing results vary, and a tight goroutine that never calls into the runtime can be an unfriendly neighbor. I would not build a latency promise from this experiment.

The cheap creation cost also does not excuse leaving goroutines blocked forever. Their stacks are small, not imaginary, and their work still needs a defined end.

What I do like is the change in scale. In C, a thread is sufficiently costly and awkward that I first ask how to avoid one. In Go, I can model each independent operation directly, then decide where synchronization belongs. The runtime may multiplex many goroutines over fewer operating-system threads; that implementation is precisely why “goroutine equals pthread” is a poor mental model.

This is not magic parallelism either. More runnable work does not create more processors, and today’s defaults may use only one. It does make concurrency cheap enough to express before I optimize it. For server and pipeline code, that changes the first sketch substantially.

Backpressure Is Not an Error Message

Fri, 16 Oct 2009 20:09:00 +0000

My event-driven server stayed responsive under load, but its memory use climbed steadily. I had removed blocking and accidentally built an extremely efficient machine for accepting work it could not finish.

Each connection had an output buffer. Producers appended responses faster than the network drained them, so those buffers became an unbounded queue distributed across clients. Nothing was technically stuck. The server was merely saving enough unfinished work to become stuck later.

Backpressure means carrying limited downstream capacity toward the producer. When a connection’s output crosses a high-water mark, I stop reading more requests from it or stop scheduling new work. Once buffered output falls below a lower mark, reading resumes. Separate thresholds avoid switching state on every small write.

The same rule applies between the event loop and worker pool. A bounded job queue makes overload visible. If it is full, the loop must defer input, reject work or shed a connection according to policy. Adding another unbounded queue does not increase capacity; it increases the delay before admitting there is none.

I prefer bounded queues and explicit overload behavior, even when rejection feels impolite. The exact limits require measurement, and short bursts deserve some room. But a service that says “not now” can recover. One that accepts everything may respond to nobody, which is very accommodating in principle and less so in practice.

Event Loops Before More Threads

Thu, 19 Feb 2009 22:25:00 +0000

I wrote a small TCP service using one thread per connection. It was the obvious design: accept a socket, start a thread, perform blocking reads and write the response. Each connection looked like a normal sequential program, and the first version was finished before lunch. This is an excellent property in prototypes and an alarming one in architecture.

The service was fine with a few clients. With many mostly idle connections, however, it accumulated threads whose principal activity was owning stacks and waiting. Shutdown required waking them all, shared statistics needed locks, and one slow operation could hold a mutex needed by several innocent connections. I had converted network concurrency into scheduler concurrency whether the work needed it or not.

An event loop starts from a different observation: most connections are not executing. They are waiting for a socket to become readable or writable, a timer to expire, or another operation to complete. One thread can wait for many such events, dispatch a small handler for each ready source, then wait again.

State instead of stacks

The thread-per-connection version stores progress in the thread’s call stack. A function reads a header, calls another function to read a body, and eventually writes a reply. When a read blocks, the stack quietly remembers where execution should continue.

With an event loop, handlers must not block. The connection’s progress becomes explicit state: reading headers, reading a body, preparing a response, or writing remaining bytes. A read handler consumes whatever is available, updates the state and returns. If the message is incomplete, the loop will call it again after the socket becomes readable.

This looks more complicated because it is more explicit. Partial reads and writes were always possible; blocking calls merely kept the unfinished operation on a private stack. The state machine makes those boundaries visible. That visibility is annoying for the first implementation and useful for every timeout, cancellation and protocol error added later.

A connection object in my revised service holds the socket, input and output buffers, parser state and deadlines. The loop owns the objects and invokes them serially. Because one loop thread mutates this state, most of it needs no locking. Ownership becomes a rule I can state rather than a hope expressed through mutexes.

Readiness is not completion

The operating system reports that an operation is likely to make progress. A readable socket may provide only part of a request. A writable socket may accept only part of a response. Handlers therefore loop until the operation would block, preserve what remains, and return control.

Nonblocking mode is essential. If one handler performs a blocking disk lookup or name resolution, every connection assigned to that loop waits behind it. Event-driven code does not eliminate blocking; it makes accidental blocking more expensive and easier to notice when the entire service pauses at once. Very democratic, if not especially helpful.

Timers belong in the same model. The loop computes the next deadline, limits its wait accordingly, then expires idle connections or retries work. I originally used one timer thread that sent messages to connection threads. Replacing that arrangement with a deadline queue removed both threads and a surprising amount of code.

Where threads still fit

An event loop is not an argument for one thread in the entire program. CPU-heavy work can be sent to a bounded worker pool. Blocking interfaces that cannot be integrated with readiness may need workers too. The important constraint is that results return to the owning loop as events, rather than allowing workers to mutate connection state directly.

Multiple loops can run on multiple processors, each owning a set of connections. This keeps the single-owner rule while using more than one core. Distribution and load balancing then become concerns, but they are explicit and usually less tangled than allowing every connection thread to touch global state.

Threads remain attractive when there are few concurrent operations, the blocking flow closely matches the problem, or the platform’s asynchronous facilities are poor. A thread can also be a good unit of isolation for independent subsystems. I do not want to replace clear blocking code with callbacks merely to win a concurrency argument on the Internet.

The event-driven version has its own failure modes. A handler that performs too much work increases latency for everything behind it. Unbounded input or output buffers merely move resource exhaustion from stacks to heaps. Reentrant callbacks can produce transitions the state machine did not expect. Fairness must be designed: a permanently busy socket should not monopolize one pass through the loop.

My current preference

For a service with many mostly waiting connections, I now strongly prefer an event loop with explicit ownership and a small worker pool for unavoidable blocking or CPU work. It uses threads according to available parallelism, not according to the number of clients. It also gives timeouts, cancellation and backpressure natural places in the design.

That preference is qualified. The state machine must be kept small, protocol parsing should be separated from readiness plumbing, and every handler must have a bounded amount of work. If those rules cannot be maintained, a modest number of threads may be safer than one ingenious loop nobody can debug.

My first threaded service was easier to write. The event-loop version is easier to reason about under load, which is a different and more valuable form of easy. It took me several pages of notes to discover that sleeping threads are still resources. The kernel, rather rudely, knew this already.