Garbage-Collection on Roberto Selbach

Go 1.4 moves the runtime toward Go

Tue, 16 Dec 2014 17:40:00 +0000

I installed Go 1.4 to test a service and found the most interesting changes below my source code. Parts of the runtime and tools are being translated from C into Go. That work is not a promise that applications suddenly run faster; it is groundwork for making the implementation easier to evolve with one type system and toolchain.

The collector is now fully precise. It identifies pointers in stacks and the heap rather than retaining objects because non-pointer data happens to resemble an address. Precision also requires cooperation from writes: write barriers record pointer updates that the collector must not miss while maintaining its view of the object graph. This is infrastructure for collector work, not the concurrent collector people keep casually attributing to this release. Go 1.4 still has stop-the-world collection behavior worth measuring.

Goroutine stacks now start at 2 KiB, down from the larger starting size in previous releases, and grow by copying when needed. This materially reduces initial memory for programs with many mostly shallow goroutines. It does not make deep stacks free; growth allocates a larger contiguous stack and copies live contents using precise pointer information.

Two source-level tools are immediately practical. go generate lets a package record commands that generate source or other build inputs:

//go:generate stringer -type=State

Running go generate is explicit; go build does not run generators automatically. I like that separation because builds should not unexpectedly require every generator or rewrite the tree. Generated files still need a clear policy in the repository, and the directive is a command for developers, not a dependency manager.

The new internal directory convention has an important 1.4-sized footnote. In this release, the restriction is enforced for packages in the main Go source repository, keeping implementation packages inside the standard library and toolchain. The intended rule is that code under a/b/internal/x may be imported only by packages rooted within a/b, but the go command does not yet enforce that boundary for general repositories. Broader enforcement is future work.

example.com/project/internal/wire
example.com/project/server

I can use the same layout in my own repository to advertise intent, but in Go 1.4 that part is still convention rather than a boundary enforced by the tool. It is worth adopting without pretending the lock is already on the door.

After the upgrade I reran tests, allocation benchmarks, and latency measurements rather than assuming runtime changes were universally beneficial. The service used less memory with thousands of idle goroutines, while its GC tail latency remained visible. That is a believable improvement: smaller starting stacks and more precise tracing reduce waste, but write barriers and collection still have costs.

Go 1.4 mostly removes future constraints. More runtime code in Go, complete pointer precision, and barriers prepare the implementation for deeper collector changes. Meanwhile, go generate is useful now, and internal shows where package boundaries are headed.

What Go 1.3 did to GC pauses

Tue, 15 Jul 2014 13:20:00 +0000

I upgraded a memory-heavy worker to Go 1.3 expecting the garbage collector to become invisible. It did not. What changed was more specific and more useful than my expectation.

Go 1.3 performs stack scanning precisely. The collector knows which stack slots contain pointers instead of treating pointer-shaped integers as possible references. This avoids keeping some dead heap objects alive accidentally and enables the runtime to copy growing stacks while correcting pointers.

The release also adds concurrent sweeping. After marking determines which heap objects remain reachable, reclaiming unmarked spans can overlap with application execution rather than keeping all sweep work inside one stop-the-world interval. Allocation may assist or encounter sweep work later, so the cost has moved and changed shape rather than vanished.

I compared GC traces from the same batch input:

Go 1.2: collections less regular, several pauses near 20ms
Go 1.3: shorter observed pauses, sweep work visible during execution

Those are observations from one program, not a portable benchmark. Heap layout, processor count, allocation pattern, and runtime revisions all matter. The collector still stops the world for portions of collection. Go 1.3 is not the future fully concurrent collector some discussions seem to have installed by enthusiasm.

One accidental retention did disappear after the upgrade because a pointer-looking value on a stack no longer kept an object alive. I did not count that as permission to ignore memory profiles. The worker also retained a 16 MB byte slice through a 40-byte subslice, and precise scanning quite correctly kept the whole backing array reachable.

The fix for that case remained a copy:

key := append([]byte(nil), largeBuffer[start:end]...)

Copying 40 bytes allowed the large buffer to die. Runtime precision can distinguish pointers from integers; it cannot infer that I only care about a tiny section of a live array.

Marking begins from roots such as globals and goroutine stacks and follows pointers through reachable heap objects. Precise stack maps improve the root set, while concurrent sweeping deals with memory already proven dead. They improve different phases; calling both simply “the new GC” hides why a workload changes.

I keep pause percentiles and total CPU in view. Moving sweep work out of a pause can reduce tail latency while leaving processor work to be paid during ordinary execution. That is usually a good trade for my service, but a batch program may value throughput differently.

My practical conclusion is to upgrade for the collector improvements, then measure again. Lower pauses do not make allocation free, concurrent sweep does not eliminate all latency, and a precise collector will faithfully preserve every object my program still references, including the embarrassingly large ones.

Chasing a Go latency spike

Thu, 12 Jun 2014 19:05:00 +0000

An HTTP service of mine usually answered in a few milliseconds but occasionally took more than 100. Average latency looked healthy, which is one of the average’s less charming habits. I wanted to know whether requests were waiting on the network, the scheduler, or garbage collection.

My first attempt added timers around every handler. That established that a slow request was slow, a breakthrough unlikely to threaten the scientific establishment. I needed observations below the handler.

I started with GODEBUG=gctrace=1. A run under load printed lines resembling:

gc1(1): 0+6+1 ms, 18 -> 22 MB 12033 -> 8411 (123004-114593) objects
gc2(1): 0+9+1 ms, 25 -> 29 MB 15020 -> 9170 (171220-162050) objects

The exact format is runtime-version dependent, but the useful correlation was clear: several latency spikes lined up with collections. The Go collector at this point includes stop-the-world work. During those pauses, goroutines do not make application progress. More processors do not turn a global pause into concurrent work.

I then reduced allocation rate rather than immediately tuning the collector. A JSON path decoded first into interface{} and then into a struct. Removing that intermediate object graph reduced collection frequency. Reusing a scratch byte slice within a request also helped, provided I kept ownership local and did not retain a huge backing array for a tiny value.

Heap size and pause time are related but not identical. Setting GOGC lower collects more often with a smaller heap; setting it higher collects less often while retaining more memory. Neither setting fixes accidental allocation. I leave GOGC at its default until measurements show a reason to trade memory for collection frequency.

The scheduler was the next suspect. Go multiplexes goroutines onto operating-system threads. Runnable goroutines wait in scheduler queues; goroutines blocked in syscalls or channel operations allow others to run. GOMAXPROCS controls how many processors may execute Go code simultaneously, not how many sockets the program may hold and not a magic goroutine limit.

I had one CPU-heavy goroutine parsing a large report. Function calls provide scheduling opportunities in the current runtime, but preemption is only partial. A long computation with few safe points can delay other runnable work, particularly with a low GOMAXPROCS. Breaking the report into bounded jobs improved fairness and made cancellation by closing a work channel practical. Adding runtime.Gosched inside the innermost loop helped one test and made the algorithm uglier, so I did not keep it.

Network waiting has a different path. On supported systems, the runtime’s network poller integrates non-blocking descriptors with mechanisms such as kqueue or epoll. A goroutine waiting for socket readiness can park without consuming an operating-system thread for the duration. When the descriptor becomes ready, the poller makes the goroutine runnable again.

This is why one goroutine per connection can be practical. It does not mean each connection has a dedicated thread. It also does not mean network code is immune to thread exhaustion: file operations and syscalls that the poller cannot manage may block threads, DNS behavior varies by platform and resolver path, and cgo calls have their own costs.

I made the wait visible by taking a goroutine profile from net/http/pprof while the service was loaded. Most goroutines were parked in network reads, which was normal. The interesting stack was a growing set waiting on one mutex around a response cache. My “thread-safe map” protected JSON encoding under the lock, turning a short critical section into serialized CPU work.

The repair copied the cached value’s reference while holding the lock, unlocked, and encoded afterward. That shortened scheduler queues during bursts. I verified the data was immutable before doing this; moving mutable state outside a lock would have exchanged latency for corruption, a poor bargain even by my standards.

One test run then looked roughly like this:

before: p50=4ms p95=37ms p99=128ms
after: p50=3ms p95=11ms p99=42ms

The remaining tail still correlated with collections. This was not a proof that all pauses were GC pauses: kernel scheduling, packet loss, disk activity, and clients contribute too. I kept wall-clock request histograms beside allocation and GC observations rather than assigning every gap to the runtime.

The mechanism suggests a practical diagnostic order. First, establish whether time is spent runnable, blocked on synchronization, parked on network I/O, or stopped globally. Goroutine profiles reveal blocking locations, mutex and channel structure reveal serialization, GC traces reveal collection timing, and CPU profiles reveal actual execution. Then change the source of delay.

I also capped the work queue. Before that change, a burst created more live request state than workers could consume, raising heap size and GC cost after traffic subsided. Back pressure made overload visible to callers instead of translating it into a delayed collector problem. A goroutine waiting in a queue is still retained work.

My direct opinion is that “goroutines are cheap” is useful sales material and terrible capacity planning. Their stacks consume memory, their work must be scheduled, their objects burden the collector, and their shared locks serialize. The runtime makes concurrency remarkably approachable, but it cannot make an unbounded queue or a giant critical section reasonable. Mine was a runtime mystery only until I measured my own code.

Measuring and tuning Go GC pauses

Fri, 28 Dec 2012 14:36:00 +0000

One request in a test server occasionally took much longer than its neighbours. Rather than convict the collector from one latency graph, I recorded request times and the runtime’s pause samples on the same workload.

The current collector stops application work while it traces the reachable heap. runtime.ReadMemStats exposes NumGC, PauseTotalNs, and a ring of recent PauseNs values. I sampled before and after a fixed five-minute run, then compared only collections added during that window. Mixing in old pauses made short tests look wonderfully stable or mysteriously awful depending on what had run first.

That pause is easy to describe and harder to judge. A short command-line tool may never care. A busy service with latency requirements can care very much, even if its average throughput looks fine. Averages are talented at concealing the request somebody was actually waiting for.

My first table kept four numbers for each run: requests per second, the 50th and 99th percentile request latency, collection count, and the largest observed GC pause. Average latency alone hid the event I cared about.

The naive handler allocated a buffer for every item:

func checksum(items [][]byte) uint32 {
	var sum uint32
	for _, item := range items {
		buf := make([]byte, len(item))
		copy(buf, item)
		for _, b := range buf {
			sum += uint32(b)
		}
	}
	return sum
}

The copy served no purpose. Removing it reduced collections and improved the long tail. This was not clever tuning; I had simply stopped manufacturing garbage.

The collector’s job depends on both allocated volume and live data. Producing temporary objects quickly causes collections to happen more often. Retaining a large graph gives each marking pass more reachable memory to inspect. These are different problems: one concerns allocation rate, the other heap size and object reachability.

After that fix I tried GOGC=50, the default 100, and 200, restarting the process for each run. Lowering it collected more often with a smaller heap. Raising it collected less often but used more memory. On this service, 200 helped the tail a little and increased memory enough that I kept the default. That is a result, not a disappointment.

Manual runtime.GC() calls were useful to prove that the latency recorder could see a pause. They did not belong in the handler. Forcing collection chooses when to pay and can easily turn an occasional pause into a dependable stream of them.

The useful knob came last. First came a repeatable load, pause deltas from the same time window, and removal of an accidental allocation. Otherwise GOGC is just a dial with excellent opportunities for superstition.

Go 1 stop-the-world garbage collection

Wed, 01 Aug 2012 20:20:00 +0000

I graphed request times for a Go 1 service and found narrow spikes hiding behind a good average. CPU was not saturated and the network was quiet. Allocation tracing pointed toward the garbage collector.

The reduced workload simply creates and drops temporary objects:

for _, line := range lines {
	fields := bytes.Split(line, []byte(","))
	consume(fields)
}

In Go 1, garbage collection stops ordinary Go execution while the collector identifies reachable heap objects and reclaims unreachable storage. The runtime coordinates goroutines at safe points, considers roots including stacks and global data, traces pointers through live objects, and sweeps storage that is no longer reachable. Those pauses are a direct property of the collector in today’s release.

The surprise was the difference between throughput and latency. The total collection work can be a tolerable fraction of a ten-second run while one request arriving during a stop sees the pause directly. An average conceals where time is concentrated. Percentiles and a timeline expose it.

Live heap size and allocation rate both matter, but differently. More live objects give the marker more reachable data to examine. A high rate of short-lived allocation fills available heap space quickly and brings the next collection nearer. Retaining a tiny pointer into a large structure can keep the structure reachable even when the programme considers most of it finished.

The collector does not compact the heap into one tidy block. That avoids rewriting every pointer but leaves allocation and free-space management to the runtime. Details of size classes, metadata, and precisely how particular regions are scanned are implementation-specific. The safe source-level rule is reachability, not a guessed address or object movement policy.

I prefer reducing obvious temporary allocation at a clear boundary. In this parser, reusing a scratch slice and avoiding repeated separator construction improved the latency graph without turning the package into an object pool. A profile before and after keeps that change honest.

The caveat is ownership. Reusing a buffer while another goroutine still reads it trades collection pauses for data races and corrupted requests, an impressively bad bargain. Pools can also retain far more memory than expected. Simpler allocation remains the right default until measurement says otherwise.

Go’s collector will certainly evolve, but application explanations should not arrive from the future. For Go 1 today, stop-the-world pauses are part of the runtime’s cost model. My service became steadier after allocating less, and the graph became considerably less interesting. Production graphs are at their best when they are poor conversationalists.

Watching the Go garbage collector stop

Wed, 16 Nov 2011 19:40:00 +0000

I noticed a small server pausing at intervals under an allocation-heavy test. The average request time looked respectable, but individual requests occasionally took a visible detour. That pattern made the old garbage collector worth examining.

I reduced the trigger to a loop that replaces short-lived buffers:

for i := 0; i < n; i++ {
	b := make([]byte, 32*1024)
	consume(b)
}

The first surprise was that allocation speed and collection latency are separate observations. Allocation can be cheap while creating enough work for periodic collections. An average over the entire run smooths those pauses into a harmless-looking number.

The collector in this runtime is a stop-the-world collector. When a collection begins, Go execution is brought to a safe point and paused while the runtime determines which heap objects are reachable. It starts from roots such as stacks and global data, follows pointers through reachable objects, and eventually reclaims storage that was not found. Application goroutines do not continue doing ordinary work during that collection.

This explains why adding processors does not simply make the pause disappear. It may improve the application between collections, but the collector’s coordination and tracing remain a shared event. It also explains why retaining one pointer to a large structure matters: reachability, not whether I personally remember using the object, determines whether it survives.

I prefer reducing needless allocation only after a profile or latency measurement identifies it. Reusing a buffer at a clear ownership boundary can help. Converting every local variable into a global cache can help the heap grow mysterious hair. Smaller live data and fewer transient objects are good outcomes; cleverness measured only by line count is not.

The caveat is that stack and heap placement are compiler decisions, and the compiler is changing. I avoid writing code that depends on a guessed escape decision. I also avoid claims about a future collector when explaining this one. Today’s pauses are real even if the runtime will improve.

For the server, pooling two large scratch buffers around a serialized operation removed most of the allocation burst. I kept the simpler allocating version in a benchmark so the trade remains measurable. The garbage collector was doing exactly what it had been asked to do; I had simply been sending rather enthusiastic invitations.