Internals

I assumed GOMAXPROCS(1) meant my program would use one operating-system thread. A blocking system call corrected me without displaying much tact.

In Go 1.1’s scheduler, GOMAXPROCS controls Ps, the runtime resources required to execute Go code in parallel. It does not impose a hard limit on Ms, the operating-system threads. The runtime may need more threads because one can become stuck in a system call while runnable goroutines remain.

Suppose G1 is running on M1 with P1 and enters a blocking call the network poller cannot handle asynchronously. The runtime marks the transition and detaches P1. Another thread, M2, can acquire P1 and execute G2. There is still only one P executing Go code at a time, but two operating-system threads exist because M1 is inside the kernel.

When M1 returns, its goroutine must rejoin runnable work. If no P is immediately available, the runtime can queue the goroutine and park or reuse the thread. The M does not permanently own the P, which is precisely why useful execution can continue during the call.

Runtime-managed network operations often take a cheaper path. Non-blocking sockets are registered with the network poller, and the goroutine is parked while its M continues with other work. Ordinary blocking system calls and calls into C do not necessarily have that integration, so the thread itself may be occupied.

This matters when a program makes many concurrent calls that block unpredictably. The scheduler can preserve progress by creating threads, but threads are not free. They consume kernel resources and stack address space, and excessive creation adds scheduling overhead. Wrapping a slow C function in ten thousand goroutines does not convert the function into scalable asynchronous I/O. It converts optimism into threads.

A bounded worker group is appropriate when calling a blocking interface with limited capacity:

jobs := make(chan Job)
for i := 0; i < workers; i++ {
	go func() {
		for job := range jobs {
			runBlocking(job)
		}
	}()
}

The useful bound should come from measured service capacity, not a decorative constant selected because eight feels parallel. Backpressure at the job channel also makes overload visible instead of allowing an unbounded pile of blocked goroutines and threads.

The distinction among G, M, and P is practical here. Counting goroutines does not reveal thread use. Counting threads does not reveal Go parallelism. GOMAXPROCS says how many Ps may run Go code; it does not promise a process with exactly that many threads.

I now treat blocking foreign calls and unusual system calls as resources to measure and usually bound. The scheduler is very good at preventing one blocked thread from freezing unrelated Go work. It is not obliged to make an unlimited number of blocked threads inexpensive.

After the race detector found a real bug, I wanted to know whether it was merely running the program many times and hoping for an unfortunate schedule. It is doing something much more useful.

A data race exists when two goroutines access the same memory concurrently, at least one access writes, and synchronization does not establish the required ordering. The detector instruments loads and stores so they can be recorded in shadow state associated with application memory. It also observes synchronization operations such as channel communication and mutex locking.

The key idea is causality rather than wall-clock time. Each goroutine’s operations have an order. Synchronization joins those orders: a channel send and corresponding receive, for example, establish a happens-before edge. The detector maintains enough logical-clock information to determine whether conflicting memory accesses are ordered by such edges.

Consider this broken counter:

var n int

func increment(done chan bool) {
	n++
	done <- true
}

Two goroutines executing n++ each perform a read and a write. Sending completion values afterward does not order the increments with respect to each other. The detector sees conflicting accesses without a happens-before path and reports both stacks.

Putting a mutex around the increment changes more than timing:

mu.Lock()
n++
mu.Unlock()

Unlock and a later lock establish ordering, so accesses in the critical sections no longer race. Sleeping between increments might make overlap less likely, but it adds no synchronization relation and therefore is not a fix.

Instrumentation explains the overhead of -race. Every relevant access carries bookkeeping, and shadow memory plus goroutine histories consume substantial memory. This is the cost of seeing relationships that ordinary execution discards. I run race-enabled tests separately rather than using their timings as application benchmarks.

The detector is dynamic. If a branch is never executed, its accesses are never observed. If a test uses one worker while production uses twenty, the most interesting paths may remain untouched. Good concurrent tests still matter: vary inputs, create contention, repeat operations, and run realistic integration workloads when possible.

There can also be benign-looking races that do not crash today, such as an approximate statistics counter. They remain races under the memory model. The compiler is not required to preserve the comforting behaviour I happened to observe. Use synchronization or documented atomic operations when approximation is acceptable but racing is not.

Understanding the mechanism changed how I read reports. I no longer ask only whether the two lines could execute at the exact same nanosecond. I ask what synchronization orders them. If I cannot point to that edge, the detector has probably found a hole rather than developed an artistic difference of opinion.

Go 1.1 was released this week, so naturally I ignored the pleasant surface additions and went looking for the scheduler changes. This is normal behaviour in some households.

The new scheduler is commonly described with three letters: G, M, and P. A G is a goroutine, including its stack and scheduling state. An M is an operating-system thread, the machine on which Go code eventually runs. A P is a processor-like runtime resource required to execute Go code. GOMAXPROCS controls the number of Ps and therefore how much Go code can execute simultaneously.

This extra P may seem unnecessary. Why not put runnable goroutines directly on threads? The answer becomes clearer when a thread blocks.

The naive picture

Imagine each M owns a queue of goroutines:

M0: G1 G2 G3
M1: G4 G5

If G1 enters a blocking system call, M0 blocks with it. The runtime must somehow preserve access to G2 and G3, arrange another thread, and keep global scheduling state coherent. If queues and caches belong permanently to Ms, every blocked call complicates their ownership.

In the G-M-P design, runnable work and resources are associated with P. An M must acquire a P to execute Go code. If its goroutine enters a blocking system call, the M can detach from its P. Another M acquires that P and continues running other goroutines. When the system call returns, its G becomes eligible to run again, possibly on a different M.

The identities are separate because their lifetimes and reasons for existing are separate. G represents work. M represents a kernel execution context. P represents permission and resources to execute Go work.

Local queues

Each P has a local run queue. Creating a runnable goroutine can usually place it on the current P’s queue instead of contending on one global scheduler lock. The current M draws work from its P locally, which improves locality and reduces synchronization.

Local queues create an obvious question: what happens when one P has no work while another has a long queue? Idle Ps steal runnable goroutines from busy Ps. Work stealing balances execution without forcing every enqueue and dequeue through a single shared structure.

There is still a global queue for cases that need it, and the scheduler periodically considers it so global work is not ignored. “Local queues” does not mean each P becomes an isolated principality with its own foreign policy.

This organization also gives runtime caches a natural home. Per-P allocation state, for example, can be used by whichever M currently runs that P. If such state belonged to M, blocking and replacing threads would either strand it or require more coordination.

A small scheduler experiment

I used a deliberately crude program:

func burn(done chan bool) {
	x := uint64(1)
	for i := 0; i < 200000000; i++ {
		x = x*1664525 + 1013904223
	}
	fmt.Println(x)
	done <- true
}

func main() {
	runtime.GOMAXPROCS(2)
	done := make(chan bool)
	go burn(done)
	go burn(done)
	<-done
	<-done
}

With two Ps and enough operating-system support, the two CPU-bound goroutines can execute in parallel. With one P they execute one at a time and, because these loops contain no scheduling point, one may finish before the other starts running. The distinction matters. Goroutines make concurrency cheap; GOMAXPROCS and available processors determine parallel Go execution.

This benchmark is useful for observing CPU use and almost useless for choosing an application’s GOMAXPROCS. Real programs block, allocate, communicate, and touch memory. A tight arithmetic loop mostly proves that arithmetic loops enjoy processors.

Blocking, parking, and spinning

Not all waiting has the same consequence. A goroutine waiting on a channel or runtime-managed network operation can be parked, allowing the M and P to run another G. A blocking system call may hold the M, so the P is handed off. A call into C may similarly occupy a thread the scheduler cannot use for Go work until the call returns.

The scheduler also needs to avoid excessive thread creation. It tracks spinning workers searching for work and parked workers waiting to be awakened. Waking every idle M whenever one G becomes runnable would produce a thundering herd. Waking none would leave parallel capacity unused. The details are subtle because scheduling is largely the art of replacing one kind of waste with a smaller kind.

What this means in programs

The design makes several common Go patterns scale better, but it does not absolve application code.

A goroutine that blocks while holding a mutex can prevent unrelated goroutines from making useful progress even though the scheduler itself is healthy. A producer can create runnable goroutines faster than Ps can execute them. CPU-bound goroutines can compete with latency-sensitive work. The scheduler distributes runnable work; it cannot determine which business request deserves sympathy.

The safest optimization remains architectural: block goroutines on channels or network operations when they have no work, bound queues where overload is possible, and measure latency under realistic contention. Reaching into scheduler behaviour to arrange exact execution order is not a design. It is a hostage negotiation with an implementation detail.

I like the G-M-P model because it separates concerns that were tangled in earlier schedulers. It gives runnable work local queues, lets processor resources survive blocked threads, and creates better places for per-execution caches. Go 1.1 should spend less time fighting a global scheduler lock and more time running the program.

Scheduler improvements cannot make sequential code parallel, remove lock contention, or turn infinite work into finite work. They do make the straightforward goroutine model a better foundation, which beats an application-level scheduler I would have to debug myself.

I wrote an echo server with one goroutine per connection and then wondered where all the operating-system threads were. I had accepted the pleasant API before understanding the trick beneath it.

The simple server is almost suspiciously simple:

func serve(c net.Conn) {
	defer c.Close()
	buf := make([]byte, 4096)
	for {
		n, err := c.Read(buf)
		if err != nil {
			return
		}
		if _, err := c.Write(buf[:n]); err != nil {
			return
		}
	}
}

func main() {
	l, err := net.Listen("tcp", ":9000")
	if err != nil {
		panic(err)
	}
	for {
		c, err := l.Accept()
		if err != nil {
			return
		}
		go serve(c)
	}
}

Read appears to block the goroutine until bytes arrive. If every blocked goroutine required a blocked kernel thread, this design would approach the old thread-per-connection model with nicer syntax but similar scaling limits. The runtime instead integrates network descriptors with an operating-system readiness notification facility.

The toy server exits on an Accept error. A real daemon may retry temporary errors with a delay, but blindly continuing on every error turns a closed listener or exhausted descriptor table into a very efficient busy loop.

On Linux that facility is epoll; other systems provide equivalents such as kqueue. The exact backend is platform-specific, but the shape is similar. The socket is placed in non-blocking mode. A read attempts the system call. If the kernel says the operation would block, the runtime records that the goroutine is waiting for readiness on that descriptor and parks it. The underlying thread is then free to run another goroutine.

Parking is not polling

Parking a goroutine removes it from runnable work. It does not spin around asking the socket whether it feels conversational yet. The runtime’s network poller waits for the kernel to report descriptors that may now make progress. When an event arrives, the corresponding goroutine is made runnable and eventually tries the operation again.

Readiness is not completion. A descriptor reported readable may yield some bytes, an end-of-file condition, or another transient result by the time code runs. The runtime and net package retain the usual loop around non-blocking operations. The poller says “worth trying,” not “your complete application message has arrived on a silver tray.”

This distinction also explains partial reads. TCP is a byte stream. One Write at the sender does not imply one equally sized Read at the receiver. The poller reports available progress; framing remains the protocol’s responsibility. My echo loop can return each chunk because its protocol is deliberately trivial. A real message parser must accumulate and delimit data.

Deadlines join the machinery

Network code also needs a way to stop waiting. SetReadDeadline and SetWriteDeadline associate times with operations on a connection. The runtime combines descriptor waiting with timers so a parked goroutine can become runnable because the socket is ready or because its deadline expired.

Without deadlines, a slow or vanished peer can retain a goroutine and everything reachable from its stack indefinitely. Small goroutine stacks make this cheaper than a thread leak, not desirable. A server that accepts untrusted connections should make an explicit decision about idle time rather than interpreting silence as a lifelong subscription.

Closing a connection is another wake-up path. Code blocked in an operation must be released with an error rather than left attached to a descriptor that no longer has meaning. This is one reason descriptor state belongs in runtime-coordinated structures instead of being only an integer passed casually among goroutines.

What still blocks a thread

The network poller helps operations the runtime knows how to put into non-blocking mode and register. It does not transform every possible system call into asynchronous work. A call into C, an ordinary blocking file operation, or an unfamiliar kernel interface may occupy an operating-system thread. The scheduler can create or use another thread so runnable goroutines continue, but blocked threads still have a cost.

Regular disk files are particularly different from sockets. Readiness interfaces commonly report them ready even when the operation may wait on storage. Treating every Read method as equivalent because it satisfies the same interface would be tidy and wrong.

There is a nice Linux comparison here. An application written in C can build an event loop directly around epoll, maintain a state machine per connection, and achieve excellent performance. It also has to preserve that state across every partial operation and error path. Go’s runtime uses the same broad kernel capability, then presents the waiting state as an ordinary goroutine stack. The state machine has not vanished; the runtime and compiler help store it in a form humans generally prefer reading.

A practical test

I connected many idle clients to the echo server and watched thread count and memory rather than merely measuring requests per second. The number of goroutines rose with connections. Threads did not rise one-for-one. Memory grew, because every connection and goroutine is real, but not as though each connection had received a traditional large thread stack.

This does not prove every Go server scales automatically. File descriptors have process limits. Buffers consume memory. A goroutine can retain a large object graph. Handlers can block on locks, allocate furiously, or perform CPU work that has nothing to do with the poller. The mechanism solves efficient waiting, not careless architecture.

Still, it changes how I write network services. I can begin with one goroutine per connection because it is clear and maps directly to the protocol conversation. Then I measure descriptor use, memory, scheduler behaviour, and latency. I do not begin by translating the protocol into callback soup merely because the C implementation had to.

The net package’s blocking appearance is therefore honest from the goroutine’s perspective. That goroutine really does stop until it can proceed. The useful deception is that the operating-system thread need not stop with it.

I needed to print arbitrary structs for a debugging tool. After writing the third type switch I reached for reflect, with the same cautious expression normally reserved for an unfamiliar electrical panel.

Reflection starts from interfaces. reflect.TypeOf(x) reports the dynamic type stored in the interface, while reflect.ValueOf(x) represents its dynamic value. This distinction is the same type/value pair that explains why an interface containing a nil pointer is not itself nil.

The first trap is that a reflect.Value has its own validity and kind. A nil interface produces an invalid Value. A typed nil pointer produces a valid Value whose kind is Ptr and for which IsNil is true. Calling methods that do not apply to a Value’s kind panics. Reflection replaces compile-time checks with a detailed list of ways to be wrong at runtime.

A modest struct printer looks like this:

func fields(x interface{}) {
	v := reflect.ValueOf(x)
	if v.Kind() == reflect.Ptr {
		v = v.Elem()
	}
	if !v.IsValid() || v.Kind() != reflect.Struct {
		return
	}

	t := v.Type()
	for i := 0; i < v.NumField(); i++ {
		fmt.Println(t.Field(i).Name, v.Field(i).Interface())
	}
}

Even this needs qualifications. Elem on a nil pointer yields an invalid Value. Access restrictions apply to unexported fields, and asking for an interface representation where it is not permitted can panic. Production code must decide whether to skip, report, or reject these cases rather than treating every struct as an unlocked cupboard.

Settable values are another common surprise. reflect.ValueOf(n) contains a copy and is not settable. To modify n, pass &n, call Elem, verify CanSet, and then use a setter appropriate to its kind. The pointer is not ritual; it gives reflection access to the original storage.

Reflection is invaluable at boundaries where types genuinely vary: encoders, decoders, formatters, and tools that inspect user-defined structures. It is much less convincing as a way to avoid writing an interface for ordinary application behaviour. A reflected method call loses static checking, is harder to read, and performs more runtime work than a direct call.

Calling reflection “unsafe” is unfair in one important sense. The package checks its rules and panics instead of letting arbitrary memory corruption proceed. It is polite. It knocks before ruining the evening.

For my debugging printer, reflection was exactly right. I kept it at the edge, converted the inspected data into a simpler representation, and let the rest of the program remain statically typed. That boundary is where I find reflection most useful: a small dynamic vestibule leading into an otherwise ordinary Go house.