Internals

One request in a test server occasionally took much longer than its neighbours. Rather than convict the collector from one latency graph, I recorded request times and the runtime’s pause samples on the same workload.

The current collector stops application work while it traces the reachable heap. runtime.ReadMemStats exposes NumGC, PauseTotalNs, and a ring of recent PauseNs values. I sampled before and after a fixed five-minute run, then compared only collections added during that window. Mixing in old pauses made short tests look wonderfully stable or mysteriously awful depending on what had run first.

That pause is easy to describe and harder to judge. A short command-line tool may never care. A busy service with latency requirements can care very much, even if its average throughput looks fine. Averages are talented at concealing the request somebody was actually waiting for.

My first table kept four numbers for each run: requests per second, the 50th and 99th percentile request latency, collection count, and the largest observed GC pause. Average latency alone hid the event I cared about.

The naive handler allocated a buffer for every item:

func checksum(items [][]byte) uint32 {
	var sum uint32
	for _, item := range items {
		buf := make([]byte, len(item))
		copy(buf, item)
		for _, b := range buf {
			sum += uint32(b)
		}
	}
	return sum
}

The copy served no purpose. Removing it reduced collections and improved the long tail. This was not clever tuning; I had simply stopped manufacturing garbage.

The collector’s job depends on both allocated volume and live data. Producing temporary objects quickly causes collections to happen more often. Retaining a large graph gives each marking pass more reachable memory to inspect. These are different problems: one concerns allocation rate, the other heap size and object reachability.

After that fix I tried GOGC=50, the default 100, and 200, restarting the process for each run. Lowering it collected more often with a smaller heap. Raising it collected less often but used more memory. On this service, 200 helped the tail a little and increased memory enough that I kept the default. That is a result, not a disappointment.

Manual runtime.GC() calls were useful to prove that the latency recorder could see a pause. They did not belong in the handler. Forcing collection chooses when to pay and can easily turn an occasional pause into a dependable stream of them.

The useful knob came last. First came a repeatable load, pause deltas from the same time window, and removal of an accidental allocation. Otherwise GOGC is just a dial with excellent opportunities for superstition.

One benchmark had a strange cliff: adding an innocent-looking local array made a short recursive walk much slower. I suspected cache effects first. The CPU profile pointed at the runtime’s stack-growth path instead.

A goroutine begins with a small stack. Before a function uses its frame, generated code checks whether enough stack remains. If not, the runtime grows the stack by adding another segment and resumes the call. A special stack-splitting path does the unpleasant bookkeeping so ordinary Go code can continue pretending stacks are infinite, at least until memory becomes considerably less theoretical.

The benchmark was more useful as a depth sweep than as one headline number. I ran the same walk at depths 8, 16, 32, 64, and 128, then repeated it with a slightly larger local frame. Most points scaled smoothly. One narrow range jumped when calls repeatedly reached the end of a segment.

The segmented arrangement makes the initial cost of a goroutine small, but it is not magic. Each goroutine still has scheduler state and a stack segment. Every additional segment consumes memory. Deep recursion can grow a stack dramatically, and keeping the goroutine alive keeps its stack alive as well.

That narrow range is the interesting bit. If a function arrives with almost, but not quite, enough room, it grows the stack; returning can release the segment, and the next call grows it again. This “hot split” makes a particular call depth look much worse than its neighbours. A single benchmark depth can therefore blame the algorithm for standing on an implementation seam.

I checked the profile at the slow depth, then moved the benchmark one call shallower and deeper. The runtime samples faded at both neighbours. That was better evidence than merely noticing that recursion was involved. Changing the benchmark’s local array size moved the cliff too.

I did not contort the application to dodge one segment boundary. The runtime implementation will change, and ordinary code should not depend on today’s segment size. I did keep the depth sweep: it tells me whether a future optimization improves the work or merely moves the cliff.

Small stacks still make many goroutines practical. They also leave fingerprints in microbenchmarks. When one input size falls off a ledge, profile it and test both sides before rewriting the loop.

I changed a small parser to reuse a helper and made it slower. This was irritating because the new version looked cleaner, performed the same amount of useful work, and did not contain the traditional performance feature known as “a very silly loop.”

The reduced example looked like this:

type point struct {
	x, y int
}

func origin() *point {
	p := point{}
	return &p
}

Anyone arriving from C may feel a brief alarm here. p is local, yet the function returns its address. Go makes this safe by deciding where the value must live. The important distinction is not whether the source declares a value inside a function. It is whether the compiler can prove that the value stops being reachable when that function returns.

If it cannot, the value escapes and is allocated somewhere with a lifetime long enough for its uses, normally the heap. If it can prove the value remains local, it can use the goroutine’s stack. This analysis is why Go can permit returning a pointer to a local without turning every local variable into a heap allocation.

A small experiment

I wrote two deliberately uninteresting benchmarks:

var sink *point

func local() int {
	p := point{x: 20, y: 22}
	return p.x + p.y
}

func escaping() int {
	p := point{x: 20, y: 22}
	sink = &p
	return p.x + p.y
}

The arithmetic is identical. The assignment to the package variable is not. After escaping returns, another function can read sink, so p must survive. The compiler cannot put it in a dead stack frame and hope nobody notices.

The compiler can print its decisions. Building with the compiler’s escape-analysis diagnostics enabled is considerably more useful than staring at new(point) and guessing. The exact wording and flags are compiler details and may change, but reports identifying values moved to the heap make a good starting point.

This matters because source syntax alone is a poor allocation detector. new(T) does not decree that T lives on the heap. Returning a value does not decree that a copy must be heap allocated. Conversely, innocent-looking interface conversions, closures, or calls through code the compiler cannot fully inspect may cause values to escape.

Interfaces and hidden addresses

Suppose I replace a direct call with this:

func printAny(v interface{}) {
	fmt.Println(v)
}

func report() {
	p := point{x: 20, y: 22}
	printAny(p)
}

Putting p into an interface constructs an interface value containing dynamic type information and the value’s data. Depending on what the compiler knows about the call and how that data is represented, it may need storage beyond the current frame. The right answer is not “interfaces allocate.” The right answer is to inspect this particular program with this compiler, then measure it. Blanket rules age badly.

Closures offer a more obvious case:

func counter() func() int {
	n := 0
	return func() int {
		n++
		return n
	}
}

The returned function still uses n, so n cannot disappear when counter returns. The compiler arranges storage for the captured variable. This is not a leak. It is the semantics I asked for, delivered with a bill attached.

Stack is not a moral achievement

It is tempting to read an escape report as a list of compiler failures. That is not helpful. Heap allocation is necessary whenever data outlives a call, and Go’s garbage collector exists precisely because many useful objects do. The report says where the compiler could prove a lifetime, not whether the code is virtuous.

Still, allocations have costs. Allocating requires runtime work. More live heap data gives the garbage collector more to consider. A short-lived object may be cheap, but a hot loop producing millions of them deserves attention. I use benchmarks to establish that the path matters, allocation diagnostics to locate candidates, and another benchmark to verify the change. Reversing this order produces beautifully optimized code nobody needed.

Sometimes a small API change keeps a value local. A function can fill a caller-owned value instead of returning a pointer:

func setOrigin(p *point) {
	p.x = 0
	p.y = 0
}

func distance() int {
	var p point
	setOrigin(&p)
	return p.x + p.y
}

Passing an address does not automatically force escape. If the compiler can see that setOrigin does not retain it, p may remain on the caller’s stack. Again, the analysis follows lifetime, not punctuation.

I fixed my parser by removing one retained pointer from a frequently called helper. The result allocated less and recovered the lost time. I did not redesign every function to avoid pointers, because that would exchange a measured problem for a readability problem.

Escape analysis is best treated as an explanation facility. It tells me why the runtime is allocating when a profile says allocation matters. That is much better than the old C technique of declaring stack allocation “fast,” then accidentally returning its address and discovering a more exciting category of performance bug.

I was timing a small word counter and noticed that making the map with a rough size helped more than I expected.

func count(words []string) map[string]int {
	m := make(map[string]int, len(words))
	for _, word := range words {
		m[word]++
	}
	return m
}

Of course len(words) is an overestimate when words repeat. It was still cheaper than repeatedly growing a map created with make(map[string]int).

A Go map is a hash table arranged as buckets. A hash of the key chooses a bucket, then the runtime examines entries in that bucket for a matching key. The bucket stores several keys and values rather than allocating a separate node for each item. That is a useful contrast with the traditional chained hash table I first learned, where every collision grew a little linked list and the allocator got invited to dinner.

When too many entries accumulate, the table grows. Growth is not free: keys and values must end up in buckets appropriate for the larger table. Supplying a reasonable hint lets the runtime start closer to the required size. It is a hint, not a promise about exact allocation, and writing make(map[K]V, oneMillion) because perhaps one day there may be a million entries is not optimization. It is interior decorating for an empty warehouse.

The representation explains a few visible rules. Map elements are not addressable. If the table grows and moves entries among buckets, a pointer to an element would become troublesome. This is illegal:

p := &m["answer"]

For a struct value, retrieve it, change the copy, and assign it back; or store pointers as the map values when that ownership model makes sense.

Iteration order is another thing not to build upon. The language does not specify it. Bucket layout and table state are implementation details, so code that happens to print keys in a pleasant order today has merely received a temporary kindness. Sort the keys when order matters.

There is also no general protection for concurrent access. A map operation changes more than the apparent key/value pair when it triggers growth or updates bucket metadata. I put a mutex around shared maps rather than trying to reason that two writers probably hash to different buckets. “Probably” is a peculiar synchronization primitive.

My practical rule is now simple: give make an honest size estimate when one is already available, never depend on iteration order, and make ownership of a map boringly clear. Knowing about buckets is useful. Trying to outsmart them from application code generally is not.

Two rows in a parser changed at once. I had stored each row by appending to a scratch slice, and enough spare capacity let the next row reuse the first row’s backing array.

scratch := make([]byte, 0, 64)
first := append(scratch, "Troy"...)
scratch = scratch[:0]
second := append(scratch, "Abed"...)

fmt.Printf("%q %q\n", first, second)

Both printed "Abed". No assignment to first appeared in the debugger, which sent me looking in entirely the wrong place.

The quickest diagnostic was printing lengths, capacities, and the first-element addresses at the point each slice was saved:

fmt.Printf("first  len=%d cap=%d p=%p\n", len(first), cap(first), &first[0])
fmt.Printf("second len=%d cap=%d p=%p\n", len(second), cap(second), &second[0])

Matching addresses exposed the alias. Capacity was the clue: resetting the length to zero did not discard the storage. The next append was allowed to write there.

The saved row needed ownership, so I copied it:

row := make([]byte, len(scratch))
copy(row, scratch)
rows = append(rows, row)

For scratch data that is consumed before reuse, sharing capacity is exactly the optimization I want. For data stored beyond that reuse, it is an ownership bug. I now inspect cap and backing addresses before staring at the append line as if it had betrayed me.

One caution: taking &s[0] requires a non-empty slice. In this parser the rows are non-empty; a general diagnostic needs to check len(s) first. That tiny check is cheaper than another afternoon blaming the debugger.