Performance on Roberto Selbach

sync.Pool is not a cache

Thu, 18 Sep 2014 16:25:00 +0000

I used Go 1.3’s new sync.Pool to reuse temporary buffers in an HTTP formatter. Then I almost used it to cache parsed templates. The first idea reduced garbage; the second would have randomly forgotten application data.

A pool holds temporary values that may be removed at any time. Garbage collection is explicitly allowed to clear it. That makes it suitable for amortizing allocation of scratch objects, not for storing anything whose presence affects correctness.

My buffer pool looks like this:

var buffers = sync.Pool{
	New: func() interface{} {
		return new(bytes.Buffer)
	},
}

func render(v value) []byte {
	b := buffers.Get().(*bytes.Buffer)
	b.Reset()

	writeValue(b, v)
	out := append([]byte(nil), b.Bytes()...)
	if b.Len() < 64<<10 {
		b.Reset()
		buffers.Put(b)
	}
	return out
}

The copy before Put is essential. b.Bytes() aliases the buffer’s storage. Returning that slice and putting the buffer back would allow another goroutine to overwrite the caller’s result. Pool safety does not transfer ownership safety to pooled objects.

The pool synchronizes Get and Put, but an object obtained from it belongs to the caller until returned. Two goroutines must not use the same buffer concurrently. Go 1.3’s bytes.Buffer does not expose its capacity, so I use the rendered length as a simple retention policy: a response that grows a buffer past 64 KiB does not return that buffer to the pool. It is only a proxy for capacity, but it keeps a freshly oversized allocation out of the pool.

In a benchmark of the formatter, pooling reduced allocation volume:

BenchmarkRender-4 500000 4100 ns/op 2304 B/op 12 allocs/op
BenchmarkRenderPool-4 500000 3000 ns/op 768 B/op 5 allocs/op

The output copy accounts for necessary ownership. If the function wrote directly to an io.Writer, even that copy could disappear, probably a better API than ever more pooling.

New should return the expected type consistently. A pool may omit New, in which case Get can return nil and every caller must handle that. I define it when all callers need the same scratch type; this keeps allocation policy together without pretending values are permanent.

Copying a sync.Pool after first use is invalid because its synchronization state and local storage are not value-like application data. I keep pools as package variables or pointer-owned fields initialized once.

I would not add sync.Pool without allocation profiles and a benchmark. It obscures object lifetime and may provide little benefit when allocations are already short and cheap. A cache promises retrieval; a pool merely offers to rummage in a box the collector is free to empty.

Asking the compiler what escapes

Thu, 07 Aug 2014 18:35:00 +0000

I was reviewing a parser and found a function returning a pointer to a local variable:

func parseCount(s string) (*int, error) {
	n, err := strconv.Atoi(s)
	if err != nil {
		return nil, err
	}
	return &n, nil
}

My C reflex announced that this must be invalid. Go quite deliberately permits it. The compiler sees that n must outlive the call and arranges for it to live on the heap. The pointer remains valid; the cost is allocation and later garbage-collection work.

The useful question was not “is taking a local address legal?” but “where did this value end up, and why?” The compiler can explain with optimization diagnostics. With the current toolchain I use a command along these lines:

$ go build -gcflags=-m
./count.go:8: moved to heap: n
./count.go:8: &n escapes to heap

The wording changes as the compiler evolves, but the report is far more reliable than guessing from syntax. Taking an address does not always force a heap allocation. If the compiler proves the pointer does not outlive its frame, the value can remain on the stack.

For example:

func addOne(p *int) { *p++ }

func work() int {
	n := 4
	addOne(&n)
	return n
}

If addOne is visible and its parameter does not escape, n need not move to the heap. Conversely, code without an obvious new can allocate because a value flows somewhere the analysis cannot bound.

Interfaces are a common source of surprises. I had a logging helper:

func debug(v interface{}) {
	fmt.Fprintln(os.Stderr, v)
}

Passing a local struct through interface{} and into formatting made it harder for the compiler to prove the lifetime. Formatting itself uses reflection and retains enough generality that values may escape. Replacing the call in a hot parser with a concrete, disabled-by-default trace path removed allocations. I did not replace every fmt call in the program; human time is also a resource, occasionally.

Closures can extend lifetimes too:

func counter() func() int {
	n := 0
	return func() int {
		n++
		return n
	}
}

The returned closure needs n after counter returns, so the captured state cannot remain in the departed frame. That is the semantics I asked for. Rewriting it merely to satisfy an allocation count would be silly unless profiles identify this path as important.

I built a benchmark around two count parsers, one returning *int and one returning int plus error:

func BenchmarkPointer(b *testing.B) {
	for i := 0; i < b.N; i++ {
		p, _ := parseCount("42")
		sink = *p
	}
}

func BenchmarkValue(b *testing.B) {
	for i := 0; i < b.N; i++ {
		n, _ := parseCountValue("42")
		sink = n
	}
}

The reduced result looked like:

BenchmarkPointer-4 20000000 92.1 ns/op 8 B/op 1 allocs/op
BenchmarkValue-4 30000000 55.7 ns/op 0 B/op 0 allocs/op

I included a package-level sink so the compiler could not discard the result. I also checked the generated diagnostics after changing code; benchmarks without understanding can reward removal of the work being measured.

Returning the value was the better API anyway. The pointer had been introduced to signal absence, but error already represented failure and every successful parse had a number. One unnecessary state created one unnecessary allocation.

Large values complicate the slogan “prefer values.” Copying a large struct can cost more than indirect access, and a pointer may express shared mutable identity. Escape analysis may also keep a pointer target on the stack when its lifetime is bounded. API semantics come first; diagnostics and benchmarks settle performance questions.

Stack allocation is attractive because reclaiming it is nearly free when a function returns. Heap allocation requires allocator work and gives the garbage collector another object to trace. Yet avoiding every heap object is neither possible nor desirable in a program with goroutines, closures, shared state, and dynamic data structures.

Goroutine arguments deserve the same attention. A closure launched with go can outlive the current function, so captured locals commonly escape. Passing a value as an explicit parameter clarifies what is copied, though it does not guarantee stack allocation when the new goroutine itself needs durable storage. Lifetime, not spelling, drives the decision.

Compiler boundaries matter too. Analysis is strongest when function bodies and call behavior are visible. Calls through interfaces, reflection, or assembly can force conservative decisions because the compiler cannot prove retention does not occur. A future compiler may prove more, another reason not to encode today’s diagnostics as an API religion.

I keep the compiler version beside benchmark results. An escape decision is an implementation result, not a language guarantee, and a toolchain upgrade can legitimately move the allocation again.

My current method is deliberately unromantic. I profile to find an allocation-heavy path, reduce it to a benchmark, ask -gcflags=-m what escapes, and change ownership or API shape if the result remains clear. I do not hunt ampersands by sight. The compiler has a data-flow analysis; I have prejudices from another language. One of those scales better.

What json.Unmarshal really does

Tue, 25 Mar 2014 18:25:00 +0000

I blamed encoding/json for an allocation spike in an API client. That accusation was directionally correct but not particularly useful. I was asking it to decode into interface{}, inspect maps, convert numbers, and then copy everything into structs. The package was faithfully performing the work I had requested twice.

Here was the convenient version:

var raw interface{}
if err := json.Unmarshal(body, &raw); err != nil {
	return err
}

m := raw.(map[string]interface{})
id := int64(m["id"].(float64))
name := m["name"].(string)

Without a concrete destination, JSON objects become map[string]interface{}, arrays become []interface{}, strings become strings, booleans become bools, null becomes nil, and numbers become float64. Every assertion is another opportunity to panic. Large integer identifiers can also lose precision when represented as floating point.

A concrete struct gives the decoder much better instructions:

type record struct {
	ID int64 `json:"id"`
	Name string `json:"name"`
}

var r record
if err := json.Unmarshal(body, &r); err != nil {
	return err
}

The decoder scans the input, matches object keys to exported struct fields, allocates strings and nested values as needed, and uses reflection to assign converted values. Struct field metadata is cached internally, so it is not rediscovering every field from first principles on every object. Reflection still has a cost, but my intermediate tree was the larger mistake.

I measured a reduced case with a 40-element array. The exact numbers depend on machine and revision, but the shape was stable:

BenchmarkInterface-4 5000 356000 ns/op 60400 B/op 1012 allocs/op
BenchmarkStruct-4 10000 181000 ns/op 14500 B/op 126 allocs/op

The benchmark reset the destination each iteration and used testing.B. It was not a production trace, but it proved that decoding into a generic tree and translating it was expensive enough to stop doing.

There are a few details behind apparently simple field matching. A json tag overrides the field name. A tag value of - ignores the field. Embedded fields participate in selection, with conflicts resolved according to the package’s rules. Only exported fields are candidates. Case-insensitive matches are accepted, though I prefer exact wire names because ambiguity is not a feature I need.

Missing and null are another source of false confidence. If a field is absent, unmarshaling leaves the existing Go value alone. If I reuse a struct, stale data can survive:

r := record{ID: 99, Name: "old"}
json.Unmarshal([]byte(`{"name":"new"}`), &r)
fmt.Printf("%+v\n", r)

The output still contains ID:99. I decode each independent document into a fresh value unless merging is intentional. For optional scalar fields, a pointer can distinguish missing or null from a present zero only if that distinction is actually part of the application contract.

For streams, json.Decoder avoids reading the whole input before decoding:

dec := json.NewDecoder(resp.Body)
for {
	var r record
	if err := dec.Decode(&r); err == io.EOF {
		break
	} else if err != nil {
		return err
	}
	consume(r)
}

It buffers beyond the current value, so code must not assume the underlying reader sits exactly at a JSON boundary afterward. Decoder.UseNumber is useful when decoding unknown shapes and preserving number text; json.Number can then be parsed deliberately rather than silently becoming float64.

Custom UnmarshalJSON methods are the escape hatch for special wire forms. I use them sparingly. They hide work behind ordinary decoding and are easy to make recursive by calling json.Unmarshal into the same type. An alias type avoids that recursion, but a separate wire struct is often clearer.

Syntax errors carry an offset, which I include in diagnostics without returning the entire untrusted document. Type errors differ: valid JSON may contain a string where the destination expects an integer. Other fields may already have been assigned when decoding fails, so I treat any error as invalidating the whole destination rather than using a partially filled struct.

Unknown object fields are ignored by default. That helps forwards-compatible clients but hurts configuration, where a typo should fail. In this Go version I make strict configuration explicit by decoding an object into map[string]json.RawMessage, deleting recognized keys as I decode them, and rejecting leftovers. RawMessage also helps when one discriminator selects the concrete shape of a payload. I decode the small envelope first, then decode only its raw payload into the chosen type.

Decoder convenience does not impose resource limits. Before unmarshaling an HTTP body I bound what I read, and for a stream I bound the number and size of accepted records at the protocol level. A syntactically valid JSON array can still be an excellent memory exhaustion device.

The direct opinion I took from this exercise is simple: interface{} is not a schema. If I know the shape of a response, I write it down as Go types. This improves errors, numeric behavior, allocations, and the next reader’s chances. encoding/json was not slow because reflection is cursed. It was slow because I asked for a completely generic representation and then complained that it was completely generic.

Counting allocations in Go 1.1

Fri, 09 Aug 2013 15:46:00 +0000

I had two versions of a formatter with nearly identical benchmark times. One looked cleaner, so naturally I distrusted it.

Go 1.1’s benchmark allocation reporting made the difference visible. Benchmarks can call b.ReportAllocs(), or the test command can request memory statistics for benchmarks. The output includes allocations per operation and bytes allocated per operation alongside timing.

func BenchmarkLabel(b *testing.B) {
	b.ReportAllocs()
	for i := 0; i < b.N; i++ {
		_ = label(42)
	}
}

My first formatter repeatedly concatenated strings:

func join(parts []string) string {
	var out string
	for _, part := range parts {
		out += part
	}
	return out
}

As out grows, concatenation creates new strings and copies prior contents. A bytes.Buffer version reduced allocation for larger inputs. For two tiny pieces, however, the direct expression remained clearer and entirely adequate. Allocation counts are measurements, not commandments delivered from a heap-shaped mountain.

The numbers need careful interpretation. “Bytes per operation” is an average over repeated benchmark iterations. Setup performed inside the timed loop counts against the operation, including test data accidentally rebuilt every time. Use b.ResetTimer() after setup when the setup is not part of what should be measured.

The benchmark result also depends on escape analysis and compiler decisions. A value that remains on the stack does not appear as a heap allocation merely because source code takes its address. Conversely, converting values to interfaces or retaining pointers can move data to the heap in ways that are not obvious from syntax. Compiler diagnostics can explain candidates after the benchmark shows there is a problem.

Allocation count and allocated bytes tell different stories. Many tiny objects increase allocator and collector work. One large object can dominate memory traffic while counting as a single allocation. Live heap is different again: an object retained for minutes affects collection even if it was allocated only once.

I now include allocation reports for benchmarks on hot parsing, formatting, and request paths. I first verify the output cannot be optimized away, keep setup honest, and compare behaviour as well as speed. Reducing allocations often improves performance, but an obscure zero-allocation function can still be a bad bargain.

I keep the benchmark input representative as well. A formatter tested only with an empty string can truthfully report zero allocations while answering a question no caller was likely to ask.

In this case the cleaner formatter also allocated less after a small adjustment. I was forced to accept the pleasant result. Software occasionally lacks respect for a good suspicion.

Two small Go 1.1 speedups

Tue, 25 Jun 2013 09:57:00 +0000

I rebuilt a little indexing program with Go 1.1 and it became faster without receiving any of my expert optimization, which is generally the safest kind.

The program mostly fills maps and then produces enough temporary objects to keep the garbage collector socially engaged:

for _, word := range words {
	counts[word]++
}

Go 1.1 includes a new map implementation and substantial runtime improvements. Map operations are faster, particularly for common key types, and the garbage collector does less work in several important paths. Programs heavy on either can improve simply by recompiling.

This is not a universal percentage discount. Key sizes, hit rates, map growth, allocation rate, live heap, processor count, and workload shape all matter. My toy index is evidence about my toy index.

The right comparison is to build the same source with both releases, run multiple times on an otherwise quiet machine, and look at distributions rather than choosing the friendliest result. If the program is a service, latency and pauses may matter more than total runtime.

Runtime speedups are welcome because they improve ordinary code without making it strange. I will still provide map size hints when I know them and avoid pointless allocation. A faster collector is not a licence to create garbage professionally.

Where did that Go value escape to?

Wed, 14 Nov 2012 15:27:00 +0000

I changed a small parser to reuse a helper and made it slower. This was irritating because the new version looked cleaner, performed the same amount of useful work, and did not contain the traditional performance feature known as “a very silly loop.”

The reduced example looked like this:

type point struct {
	x, y int
}

func origin() *point {
	p := point{}
	return &p
}

Anyone arriving from C may feel a brief alarm here. p is local, yet the function returns its address. Go makes this safe by deciding where the value must live. The important distinction is not whether the source declares a value inside a function. It is whether the compiler can prove that the value stops being reachable when that function returns.

If it cannot, the value escapes and is allocated somewhere with a lifetime long enough for its uses, normally the heap. If it can prove the value remains local, it can use the goroutine’s stack. This analysis is why Go can permit returning a pointer to a local without turning every local variable into a heap allocation.

A small experiment

I wrote two deliberately uninteresting benchmarks:

var sink *point

func local() int {
	p := point{x: 20, y: 22}
	return p.x + p.y
}

func escaping() int {
	p := point{x: 20, y: 22}
	sink = &p
	return p.x + p.y
}

The arithmetic is identical. The assignment to the package variable is not. After escaping returns, another function can read sink, so p must survive. The compiler cannot put it in a dead stack frame and hope nobody notices.

The compiler can print its decisions. Building with the compiler’s escape-analysis diagnostics enabled is considerably more useful than staring at new(point) and guessing. The exact wording and flags are compiler details and may change, but reports identifying values moved to the heap make a good starting point.

This matters because source syntax alone is a poor allocation detector. new(T) does not decree that T lives on the heap. Returning a value does not decree that a copy must be heap allocated. Conversely, innocent-looking interface conversions, closures, or calls through code the compiler cannot fully inspect may cause values to escape.

Interfaces and hidden addresses

Suppose I replace a direct call with this:

func printAny(v interface{}) {
	fmt.Println(v)
}

func report() {
	p := point{x: 20, y: 22}
	printAny(p)
}

Putting p into an interface constructs an interface value containing dynamic type information and the value’s data. Depending on what the compiler knows about the call and how that data is represented, it may need storage beyond the current frame. The right answer is not “interfaces allocate.” The right answer is to inspect this particular program with this compiler, then measure it. Blanket rules age badly.

Closures offer a more obvious case:

func counter() func() int {
	n := 0
	return func() int {
		n++
		return n
	}
}

The returned function still uses n, so n cannot disappear when counter returns. The compiler arranges storage for the captured variable. This is not a leak. It is the semantics I asked for, delivered with a bill attached.

Stack is not a moral achievement

It is tempting to read an escape report as a list of compiler failures. That is not helpful. Heap allocation is necessary whenever data outlives a call, and Go’s garbage collector exists precisely because many useful objects do. The report says where the compiler could prove a lifetime, not whether the code is virtuous.

Still, allocations have costs. Allocating requires runtime work. More live heap data gives the garbage collector more to consider. A short-lived object may be cheap, but a hot loop producing millions of them deserves attention. I use benchmarks to establish that the path matters, allocation diagnostics to locate candidates, and another benchmark to verify the change. Reversing this order produces beautifully optimized code nobody needed.

Sometimes a small API change keeps a value local. A function can fill a caller-owned value instead of returning a pointer:

func setOrigin(p *point) {
	p.x = 0
	p.y = 0
}

func distance() int {
	var p point
	setOrigin(&p)
	return p.x + p.y
}

Passing an address does not automatically force escape. If the compiler can see that setOrigin does not retain it, p may remain on the caller’s stack. Again, the analysis follows lifetime, not punctuation.

I fixed my parser by removing one retained pointer from a frequently called helper. The result allocated less and recovered the lost time. I did not redesign every function to avoid pointers, because that would exchange a measured problem for a readability problem.

Escape analysis is best treated as an explanation facility. It tells me why the runtime is allocating when a profile says allocation matters. That is much better than the old C technique of declaring stack allocation “fast,” then accidentally returning its address and discovering a more exciting category of performance bug.

Making the Scheduler Show Its Work

Fri, 27 May 2011 20:48:00 +0000

Scheduler discussions become folklore quickly, so I built a few experiments that force observable events instead of inferring everything from total runtime.

The first launches workers that announce when they start, wait on a gate, perform fixed CPU work, and announce completion. A channel carries each announcement to one logger goroutine. That gives me a trace without concurrent writes making the output resemble ransom typography.

func worker(id int, gate <-chan bool, events chan<- Event) {
 events <- Event{id, "ready"}
 <-gate
 burnCPU()
 events <- Event{id, "done"}
}

I run the same executable with different GOMAXPROCS values. With one processor thread allowed, CPU-bound completion is serialized even though all workers are runnable. With two, completions overlap in wall time on my dual-core machine. More than two adds runnable work but no additional cores, and eventually loses to scheduling overhead.

The second experiment replaces CPU work with pipe reads. Many goroutines can wait for I/O while the runtime arranges for other work to proceed. Increasing the processor allowance barely changes throughput because the external producer is the limit. This is why one benchmark cannot characterize “goroutine performance.”

The third experiment is intentionally rude: one goroutine runs a tight arithmetic loop with no communication while another tries to report periodically. On this early runtime, the reporter can be delayed noticeably. Scheduling behavior is under active development, and I should not assume the preemption properties of a mature operating-system scheduler.

Adding explicit communication points makes the test cooperative and reflects real pipeline code better. I do not sprinkle meaningless calls into production loops merely to influence today’s runtime; I break large work into units with natural channel operations or function calls, then measure again.

The event channel affects the experiment, of course. Observing a scheduler changes timing by adding synchronization. I keep messages outside the inner loop and compare against an uninstrumented wall-clock run. The trace explains ordering; it does not provide cycle-accurate truth.

These results belong to the May 2011 snapshot and my machine. They are not a specification of future Go scheduling, and I am deliberately avoiding an internal diagram that will expire before the ink dries. The practical findings are modest: concurrency is not parallelism, blocking and CPU work stress different paths, and processor settings should follow measurements. Schedulers are quite willing to show their work, but only if the experiment asks a specific question.

Cache Lines Beat Clever Loops

Thu, 03 Mar 2011 21:31:00 +0000

I translated a matrix filter from C to Go and then spent an afternoon optimizing arithmetic that was not the main cost. The profile pointed at memory movement, so I reduced the problem to traversal order.

The matrix is stored in one slice in row-major order. These loops calculate the same sum:

for y := 0; y < height; y++ {
 for x := 0; x < width; x++ {
 sum += pixels[y*width+x]
 }
}

for x := 0; x < width; x++ {
 for y := 0; y < height; y++ {
 sum += pixels[y*width+x]
 }
}

The first walks adjacent elements. The second jumps by an entire row. On a large image the row-wise loop is much faster because each fetched cache line supplies several values used immediately. The column-wise loop asks the processor to fetch many lines and uses a small part of each before moving on.

Bounds checks and multiplication were my original suspects. Hoisting the row offset made the code clearer and helped a little:

for y := 0; y < height; y++ {
 row := pixels[y*width:(y+1)*width]
 for x := 0; x < width; x++ {
 sum += row[x]
 }
}

But traversal order remained the large effect. A more sophisticated arithmetic trick could not compensate for unfriendly access.

I benchmark with enough data to exceed the small caches, repeat the operation, and verify the sum so the compiler cannot discard the work. These are 2011 compiler snapshots, so absolute timings and optimization behavior are temporary facts. The memory hierarchy is less temporary.

I alternate test order as well. Running one version first every time can give it consistently colder data and turn benchmark order into a hidden input.

This also affects data structure choice. A slice of small structures may have better locality than a slice of pointers to separately allocated structures. The latter adds pointer chasing and gives the allocator freedom to scatter objects. It may still be right when identity and independent lifetimes matter, but it is not free.

My practical conclusion is embarrassingly physical: visit data in the order it lies in memory. Before unrolling loops or translating a bit trick from C, inspect the access pattern. The processor is very fast when fed and becomes an expensive space heater when asked to wait.

One Thread, Two Threads

Tue, 14 Dec 2010 17:58:00 +0000

I ran two independent CPU loops in goroutines and expected my dual-core machine to halve the time. It did not. Expectations remain the cheapest profiler available and the least accurate.

This runtime defaults to limited processor use. Setting GOMAXPROCS=2 for the experiment allowed Go work to execute on two operating-system threads:

$ time GOMAXPROCS=1 ./spin
real 0m3.84s

$ time GOMAXPROCS=2 ./spin
real 0m2.17s

The result is not exactly half. Both loops touch memory, the scheduler has work, and the rest of the machine has declined to disappear for my benchmark.

Goroutines and parallel execution are separate ideas. Thousands of goroutines can make a concurrent design clear while one thread runs them. Increasing the thread allowance can provide parallelism when there is independent CPU work and hardware to execute it.

The early scheduler also has rough edges. A tight loop with no calls may delay other goroutines more than I expect from mature thread schedulers. I add real synchronization rather than relying on hopeful fairness, and I test the weekly runtime I intend to use.

For I/O-heavy programs, two processor threads may change little. For CPU loops, it can help until memory bandwidth or coordination dominates. The variable is a knob, not a turbo button. Turbo buttons at least had a satisfying light.

Counting Bits Like the Kernel

Tue, 09 Nov 2010 20:39:00 +0000

Kernel code contains algorithms that look unnecessarily peculiar until the constraints are visible. Population count, the number of set bits in a machine word, is a good example.

The obvious C version examines every bit:

unsigned int count_bits(unsigned long word)
{
 unsigned int n = 0;

 while (word) {
 n += word & 1;
 word >>= 1;
 }
 return n;
}

This performs one iteration for every bit position up to the highest set bit. A sparse bitmap still pays for all the zeros in between.

Brian Kernighan’s familiar operation removes the lowest set bit at each iteration:

unsigned int count_bits(unsigned long word)
{
 unsigned int n = 0;

 while (word) {
 word &= word - 1;
 n++;
 }
 return n;
}

Subtracting one flips the lowest set bit to zero and turns lower zeros into ones. The & keeps everything above that bit and clears the changed suffix. The loop therefore runs once per set bit.

That is excellent for sparse words and less impressive for dense ones. Kernels also use table lookups and architecture-specific instructions where available. A byte table has predictable work but adds memory access; a processor instruction can win decisively but needs dispatch or compile-time selection. There is no universally fastest source fragment independent of data and machine.

Word width matters too. Kernel types and helpers make that choice explicit, while a casual userspace benchmark can accidentally compare different widths and announce a victory produced by less work.

I benchmarked the two loops over generated sparse and dense arrays, keeping generation outside the timed section. As expected, the clear-lowest-bit version dominated sparse input. Dense input narrowed the gap considerably. Repeating one constant word produced lovely numbers and measured the compiler more than the routine, so I stopped doing that.

The broader lesson from kernel work is to understand the representation before admiring the trick. Bitmaps pack state compactly and permit word-at-a-time operations, but contention, cache placement, and scan direction can matter more than shaving an instruction from population count.

I like this algorithm because its mechanism fits in one sentence and its performance caveat requires another. Most optimization stories should have both sentences. The ones with only the first usually end in a benchmark that accidentally proves the author’s laptop exists.

The Slice Is Not the Array

Thu, 04 Nov 2010 21:04:00 +0000

Coming from C, I first read a Go slice as a pointer with nicer indexing. That model works until append allocates a new backing array or a tiny subslice keeps a large allocation alive. The missing piece is that a slice is a small descriptor, not the array itself.

Conceptually, the descriptor contains three items:

+---------+--------+----------+
| pointer | length | capacity |
+---------+--------+----------+
 |
 v
+----+----+----+----+----+
| 10 | 20 | 30 | 40 | 50 |
+----+----+----+----+----+

The pointer identifies an element in a backing array. Length controls the elements currently accessible through indexing. Capacity describes how far the slice may grow from that starting point before another array is needed. This is a conceptual layout, not a promise that user code should depend on runtime internals.

Given an array and a slice:

a := [5]int{10, 20, 30, 40, 50}
s := a[1:3]

fmt.Println(len(s), cap(s))

my snapshot prints:

2 4

The slice sees 20 and 30, and it has capacity through the end of a. Assigning s[0] = 99 changes a[1] because both views refer to the same storage. Passing s to a function copies the descriptor, but both descriptors still point at that storage.

This explains a function that appears not to resize its caller:

func grow(s []int) {
 s = append(s, 60)
}

The local descriptor changes. The caller’s descriptor does not. If the append fits within capacity, the backing array may contain the new element, but the caller’s length still excludes it. If it does not fit, append obtains another backing array and the local slice points there. The useful pattern is to return the result:

func grow(s []int) []int {
 return append(s, 60)
}

s = grow(s)

The same mechanism makes append performance less mysterious. Growing one element at a time does not normally allocate an array on every call. Capacity grows in larger steps and existing elements are copied when storage changes. The exact growth policy belongs to this experimental runtime and should not become an application assumption.

I verified allocations indirectly by timing construction with and without an initial capacity:

a := make([]int, 0, 100000)
for i := 0; i < 100000; i++ {
 a = append(a, i)
}

When the final size is known, reserving capacity avoids repeated growth and copying. When it is not known, append’s policy is generally better than inventing my own allocation scheme. Capacity is a performance hint with semantic consequences, not a target to maximize.

Subslice lifetime is the less obvious consequence. I wrote a reader that loaded a large file, found one short token, and returned a slice containing that token. The token was only a few bytes, but its descriptor still pointed into the large backing array. As long as the small slice remained reachable, the collector could not reclaim the array.

The fix was to copy the useful bytes into a new, correctly sized slice:

token := make([]byte, end-start)
copy(token, data[start:end])
return token

Copying sounds wasteful until the alternative is retaining several megabytes for a twelve-byte identifier. Profiling the whole process settled that argument quickly.

Overlapping slices deserve similar care. copy is the operation intended for moving slice contents, including overlap supported by the implementation. Hand-written forward loops can overwrite input before reading it. This is one of those cases where code that looks closer to C is merely closer to a C bug.

Slices also make zero values useful. A nil slice has length and capacity zero, can be ranged over, and can be appended to. For many producers there is no need to allocate an empty slice eagerly. An empty non-nil slice may still matter at an encoding boundary, but inside the program I prefer the simpler zero value unless behavior says otherwise.

Arrays remain values with their length in the type. Assigning an array copies its elements, and [4]int differs from [5]int. Slices provide the flexible view normally wanted by functions. Remembering that distinction prevents a great deal of accidental copying and aliasing.

All of this describes the November 2010 toolchain I am using. Syntax, built-ins, and growth details may change before a stable release. The useful mental model is simpler: a slice is a copied descriptor over shared storage. Ask which descriptor changes, which array it references, and who keeps that array alive. That usually finds the surprise.

Profiling the Program I Wrote

Thu, 19 Aug 2010 19:19:00 +0000

I was certain my parser spent its time converting decimal numbers. Certainty is a useful indication that I should profile before touching anything.

The tools include a real-time sampling profiler named 6prof. Despite the name, it also understands the other supported architectures. The interface is still moving, so this is a note about the weekly release on my desk rather than scripture.

I let the profiler start the real program:

$ 6prof ./logsum
 46.8% bytes.(*Buffer).Write
 21.1% runtime.memmove
 8.7% parseNumber

On Linux/amd64 I can ask it to write pprof data, provided the program was linked with 6l -e:

$ 6prof -P cpu.prof ./logsum

The exact output format varies, but the result was unambiguous. My “efficient” reporting path repeatedly grew a byte buffer. Number parsing was visible and not remotely the first problem.

Preallocating a reasonable output buffer removed most growth and copies. Runtime dropped by roughly a third on the same input. Replacing the decimal parser with a clever version afterward produced noise-sized improvement, so I reverted it. Clever code that cannot beat measurement is just decorative risk.

Sampling has limits. A short run may not gather enough samples, and compiler optimization can make line attribution odd. This profiler samples threads while they are running, asleep, or waiting for I/O, so a hot entry may be telling me about waiting rather than arithmetic. I feed several seconds of representative data and repeat runs. I also keep wall-clock timing around the complete operation because users do not experience percentages.

Input matters just as much. A profile from the tiny sample used by unit tests mostly measures startup and can confidently direct optimization toward irrelevant code. I use a captured workload large enough to reach steady behavior.

The young runtime itself appears in profiles. Garbage collection, allocation, copying, and scheduler work are part of the program’s cost, even if I did not type those function names. They should not automatically be dismissed as profiler clutter. Often they point back to an allocation pattern I control.

Now I reproduce, time, profile, change one thing, and time again. This lacks the emotional satisfaction of immediately rewriting the function I dislike. Annoyingly, it makes the program faster.

cgo Is a Border, Not a Bridge

Fri, 11 Jun 2010 20:52:00 +0000

I have too much useful C code to pretend a new language arrives on an empty disk. My first serious Go experiment therefore needed an old image-processing library. That led directly to cgo, then to several hours learning that a foreign-function interface is a border crossing, not a transparent bridge.

The smallest call is almost suspiciously simple. In the snapshot I used, a preamble before import "C" declares the C side:

package scale

/*
#include <stdint.h>

static uint32_t clamp(uint32_t v, uint32_t limit) {
 return v < limit ? v : limit;
}
*/
import "C"

func clamp(v, limit uint32) uint32 {
 return uint32(C.clamp(C.uint32_t(v), C.uint32_t(limit)))
}

The exact generated names and build steps are tied to this weekly release. cgo is changing quickly, so this is a field note rather than a durable manual. I run cgo as part of the package build, compile its generated C, then let the normal architecture-specific tools finish the package.

Scalar conversion is the easy part. Buffers force the real questions: who owns the memory, how long does an address remain valid, and may either runtime retain it after the call? I settled on a strict rule for this experiment: Go calls C synchronously, C consumes the supplied bytes before returning, and C stores no pointer into Go-managed memory.

For output allocated by the C library, I expose a C release function and call it with defer immediately after checking the returned pointer. That keeps allocation and release in the same ownership domain. Wrapping a C allocation in a Go slice may avoid a copy, but it also makes lifetime assumptions much easier to hide. I copy unless profiling proves the copy matters.

Strings are another ownership test disguised as convenience. A Go string is not a NUL-terminated C string. Conversion may allocate. Embedded zero bytes have C semantics whether I find them philosophically agreeable or not. Passing length explicitly is better whenever the C API permits it.

Then I measured the crossing. I called a trivial C function in a loop and compared it with the same arithmetic written in Go. The absolute figures are not worth publishing because both cgo and the compiler are moving, but the shape was obvious: crossing for every pixel was disastrous; crossing once per image was lost in the actual work.

That changed my wrapper. The first version exposed the C library nearly function for function:

Go pixel loop -> cgo -> C operation -> return

The revised version sends a whole row or image:

Go request -> cgo -> C loop over image -> return

The second interface is less “complete” and much more useful. It minimizes transitions and expresses the operation at the level where ownership is clear.

Callbacks in the other direction are harder. They combine scheduler state, foreign threads, and object lifetime. I avoided them by collecting results in C and returning a batch. That is not always possible, but it is a better starting point than assuming a C callback can enter arbitrary Go code safely. The runtime is young and its constraints deserve respect, not exploratory production outages.

Failures need translation too. The C library reports integer status codes and sometimes leaves detail in a structure. My wrapper converts those into a small Go value implementing the snapshot’s os.Error convention. I do not leak C constants throughout callers. If the library returns a partial result, the wrapper states whether that result remains valid and who frees it.

The build is the least elegant part. Header search paths, linker flags, and platform differences have not vanished; they have merely moved to the package boundary. This is still better than spreading them through the program, but cgo does not make a C dependency portable by blessing it with a comment.

I keep one integration test that allocates, calls, copies, releases, and repeats. It catches ownership regressions more reliably than staring harder at the wrapper.

My conclusion is deliberately conservative. cgo is excellent for preserving a tested, substantial C implementation. It is poor as a shortcut for tiny functions that cross the boundary constantly. Design a coarse interface, keep ownership explicit, translate failures once, and benchmark the complete operation. A border can support trade. It should not run through the middle of the kitchen.

perf Turns Counters Into Questions

Tue, 08 Dec 2009 18:53:00 +0000

I had two versions of a parser. One finished faster, so I declared victory and nearly deleted the slower one. Before doing that, I tried the new perf tools included with recent kernels and discovered that my explanation for the improvement was wrong.

The kernel’s performance-counter infrastructure provides a common way to measure hardware and software events. The perf tool can count events for a command, sample execution and report where those samples landed. Instead of beginning with a profiler tied to one processor model, I can ask through one kernel interface and use the events available on this machine.

perf stat is a useful first question. It reports elapsed time along with counts such as cycles, instructions, context switches and page faults. Ratios matter more than isolated totals. Instructions per cycle can suggest whether the processor is retiring useful work efficiently, while cache-related events may explain why an apparently smaller algorithm still stalls.

My faster parser did not execute dramatically fewer instructions. It incurred fewer cache misses because its data was laid out more compactly. I had credited a clever branch change that happened nearby. The benchmark result was real; my story about it was fan fiction.

Sampling answers a different question. perf record periodically captures the current instruction pointer, and perf report aggregates samples by symbol. With suitable symbols, hot functions become visible without instrumenting every call. Sampling has overhead and statistical uncertainty, but it is often much less disruptive than logging entry and exit around a hot path.

Counters are constrained resources. The processor can measure only a limited number simultaneously, so events may be multiplexed. Some events are model-specific, and virtualized or restricted environments may expose less. A cache-miss count without knowing which cache or how the event is defined is an attractive number with an uncertain biography.

I also avoid optimizing from one profile. Workload, input size, compiler options and machine state all matter. I run repeated measurements, preserve the input and compare complete behavior. A ten-percent gain in a microbenchmark is not useful if the changed layout doubles memory for the real service.

I strongly prefer starting performance work with perf stat, then sampling when the totals suggest a question. ftrace remains better for many scheduling and kernel-flow investigations; perf is especially convenient for connecting program hotspots to processor and kernel counters. Neither tool replaces understanding, but both are considerably more reliable than staring at source until one loop begins to look guilty.

The Allocator Is Part of the Program

Thu, 21 May 2009 21:08:00 +0000

A multithreaded parser scaled nicely to two workers and then barely improved. I inspected locks in my code and found no obvious villain. Eventually a profile pointed at allocation and deallocation, which felt unfair because malloc() had not appeared on my list of interesting algorithms.

An allocator manages more than a pointer advancing through memory. It finds suitably sized free regions, records metadata, returns memory to reusable pools and sometimes asks the operating system for more pages. In multiple threads it must also protect shared structures. A program creating millions of short-lived objects can therefore spend substantial time contending inside code it never explicitly called.

The system allocator already uses techniques to reduce this cost. Implementations may maintain size classes, bins and multiple arenas so not every allocation takes one global lock. Alternative allocators such as tcmalloc and jemalloc make different tradeoffs involving per-thread caches, fragmentation and concurrency. Swapping one in can be an excellent experiment, but it is not a substitute for understanding the allocation pattern.

My parser allocated a small object for every token, then freed the whole tree after processing a request. The lifetimes were almost identical. A simple region allocator fit better: reserve larger blocks, hand out aligned pieces by moving a pointer, and release the region at once when the request completes. Individual deallocation disappears because the ownership model says nothing outlives the request.

This improved both speed and clarity. The gain did not come from a magical allocator; it came from expressing lifetime in the data structure. It also introduced a strict rule: pointers into the region cannot escape. Violating that rule turns cleanup into a mass dangling-pointer production line, which is efficient in the wrong direction.

Object pools are useful when objects have uniform shape and repeated lifetimes, but I use them cautiously. A pool retains memory, complicates construction and may conceal an unbounded queue. Per-thread caches can reduce contention while increasing total memory consumption. Peak resident size matters as much as allocations per second on a shared machine.

Measurement must include realistic concurrency and duration. A short benchmark can reward an allocator for keeping every page. A long-running service eventually reveals fragmentation and cache growth. I watch CPU profiles, allocation counts and resident memory rather than treating one throughput number as a complete biography.

I prefer changing ownership and allocation frequency before replacing the general allocator. When patterns are genuinely general and contention remains measurable, testing another mature allocator is reasonable. In this case, the region was the stronger result because it made the program’s lifetime visible. I had gone looking for a faster malloc() and found that asking for less was cheaper.

Zero Copy Is Mostly Fewer Copies

Fri, 24 Apr 2009 18:37:00 +0000

I wrote a file server in the most direct way possible: read a block into a user-space buffer, then write that block to the socket. The code was simple and, for small files, entirely satisfactory. With large transfers, the process spent an impressive amount of CPU moving bytes it never inspected.

The ordinary path copies file data from the kernel’s page cache into my buffer and then back into kernel-managed socket buffers. It also crosses the system-call boundary repeatedly. The data needs to travel from disk or cache to the network device, but my temporary ownership adds work without adding information.

Linux offers sendfile() for this case. It transfers data between a file descriptor and a socket while keeping the payload out of the application’s address space. The kernel can connect the file and networking paths more directly, reducing copies and context transitions. “Zero copy” is convenient shorthand, not a sworn statement that no hardware or kernel component ever copies a byte.

The loop still has to handle partial progress. sendfile() can transfer fewer bytes than requested, be interrupted, or return EAGAIN for a nonblocking socket. I keep an offset and remaining length in the connection state, request another writable notification when necessary, and continue later. Replacing read() and write() with one call does not replace flow control.

Headers complicate matters slightly. A protocol header may be generated in memory while the body comes from a file. I can send the small header normally and use sendfile() for the payload. Trying to eliminate the final tiny copy at all costs generally produces more cleverness than performance.

splice() is a more general Linux mechanism for moving data between descriptors through a pipe, and can connect paths that sendfile() does not cover. It also adds another set of constraints and bookkeeping. For serving regular files, sendfile() expresses the intention clearly, so that is where I start.

There are cases where a user-space buffer belongs in the path. Compression, encryption in user space, content transformation and checksums may require examining the bytes. Small responses may show no measurable benefit. Portability also matters; this is an operating-system facility, not ISO C acquiring telepathy.

I strongly prefer the kernel-assisted path when the application is merely forwarding an unchanged large file. The preference comes after profiling, not before. In my test it lowered CPU use while maintaining throughput, which leaves the processor available for actual work. The original loop remains easier to explain, but explaining why a server copies every byte twice is not especially satisfying.

Idle Host, Busy Guest

Wed, 25 Jun 2008 20:45:00 +0000

My laptop stayed warm with an apparently idle KVM guest running. The guest reported nearly no processor use, and the host showed only small bursts, so I first blamed inaccurate temperature sensors. Sensors are convenient suspects because they cannot write rebuttals.

I stopped services in the guest one by one. A periodic timer user was waking it frequently, but even a minimal guest produced more activity than expected. Virtual timer interrupts, emulated devices and host-side handling all require exits from guest execution. An idle guest is therefore not necessarily an idle host.

Tickless operation helps when the guest kernel can avoid unnecessary periodic ticks and when KVM presents timers accurately enough for longer idle periods. It cannot eliminate device emulation or a program requesting frequent wakeups. The host must also be tickless while idle; saving ticks in one layer and generating them in the other is elaborate bookkeeping, not power management.

After using a tickless guest kernel and disabling a polling monitor, host wakeups dropped and the machine cooled. Timekeeping remained the thing to watch: aggressive experiments are worthless if the guest clock drifts or timers fire late.

My conclusion is to shut down unused guests on battery and to measure host wakeups, not merely guest CPU percentage. For a guest that must remain available, remove polling and use appropriate virtual devices where supported. Virtualization can make a machine appear to be doing nothing in two operating systems simultaneously while the processor is, with admirable professionalism, doing work for both.

CFS After the First Week

Sun, 28 Oct 2007 17:35:00 +0000

I upgraded my workstation to Linux 2.6.23 mainly to try the Completely Fair Scheduler. The machine compiles code, plays music, runs a browser with an unreasonable number of pages, and occasionally hosts a virtual machine. Under a large parallel build, the old setup could make switching desktops feel as though the keyboard had been sent by post.

My first test was not clever. I started a build, dragged windows around and declared the new scheduler smoother. Then I rebooted into the older kernel and it also seemed smooth. Human perception is a wonderful measurement instrument if the desired unit is confidence.

For a better comparison, I repeated the same build with the same compiler jobs, timed it, and introduced an interactive task that periodically needed a small amount of processor time. Total build time changed little. The interesting difference was the delay before the interactive task ran when all cores were busy. Large stalls became less frequent with 2.6.23, though disk activity could still make the desktop miserable for entirely different reasons.

CFS approaches scheduling without the old collection of active and expired priority arrays. It tracks each runnable task’s virtual runtime, an adjusted measure of how much processor service that task has received. The task with the smallest virtual runtime is the one most entitled to run next. Runnable tasks are kept in a red-black tree, making that leftmost choice efficient as tasks arrive, block and wake.

The word “fair” needs care. Equal nice levels should receive roughly equal shares over time; it does not mean every task receives an identical slice at every instant. Nice values weight the virtual runtime, so a task with lower priority accumulates entitlement differently and receives a smaller share. Sleeper behavior emerges because a task that blocks stops consuming processor time. When it wakes, it may be behind its competitors and therefore run promptly, within limits intended to prevent absurd advantages.

My naive mental model was that CFS simply gives tiny slices to interactive programs and large slices to compilers. That is not how it identifies interactivity. There is no reliable label saying “this is the music player.” Instead, the accounting rewards tasks that use little processor time and wake to perform brief work. A music player, terminal or window manager often behaves that way, but so can a less noble process.

The scheduler also has to balance two competing costs. Switching too often improves apparent responsiveness but loses time to context switches and cache disruption. Waiting too long lets a runnable task monopolise a processor. CFS uses a target scheduling latency and derives slices from the number and weights of runnable tasks, subject to a minimum granularity. As the run queue grows, it cannot promise that every task runs within the same tiny interval without turning the processor into a context-switching demonstration.

This explains one of my results. With a modest build load, interactive delays stayed short. With a deliberately ridiculous number of runnable jobs, latency grew again. Fair scheduling cannot manufacture processor time. It can distribute scarcity coherently, which is less exciting but more useful.

Another misleading result came from I/O. During linking, the desktop paused even though CFS was selecting processor tasks correctly. The disk queue was saturated and applications blocked waiting for files. Changing the CPU scheduler did not cure that. Likewise, swapping overwhelmed every subtle scheduler difference. Before blaming CFS, I now check whether the delayed task is actually runnable rather than sleeping on storage.

Nice levels behaved more predictably in my tests than before, but group fairness remains a practical concern. If one user starts one processor-bound task and another starts twenty, scheduling every task independently can grant the second user most of the machine. There is work around grouping tasks, but the default desktop case still mostly exposes per-task fairness. The name should not be read as a constitutional guarantee.

I also tried changing scheduler tunables because numbers invite interference. Reducing latency made window feedback slightly sharper under synthetic load and increased context switches. Increasing it helped throughput by an amount too small to distinguish confidently while making the terminal occasionally sticky. The defaults survived my experiments, an outcome that wounded my pride but spared my configuration files.

After a week, I am keeping 2.6.23. The improvement is not that compilations finish magically sooner. They do not. The improvement is that short, waking tasks more often get service without elaborate interactivity heuristics, and nice values map onto a cleaner accounting model. That makes a loaded workstation feel less capricious.

The practical conclusion is to test CFS with repeatable mixed workloads, not window wiggling alone. Separate CPU delay from disk delay, compare distributions of latency rather than one lucky run, and resist tuning until a real workload demonstrates a problem. CFS is a substantial scheduler change, not a performance patch that makes every number smaller. On my machine it makes contention more orderly. Given what computers usually do under pressure, orderly is a fairly ambitious achievement.

Tickless Is Not Sleepless

Mon, 10 Sep 2007 18:45:00 +0000

I built a recent kernel with tickless idle enabled because the laptop fan had begun running while the machine did apparently nothing. I expected the new kernel to become silent by virtue of one configuration option. It did not. The fan is apparently not impressed by release notes.

My naive test was to leave a terminal open, wait a minute, and compare the battery estimate. That number wandered by nearly an hour between readings. I then watched processor usage, which stayed close to zero while the fan continued. Neither test told me what was waking the processor.

The periodic timer tick normally interrupts each processor at a fixed frequency so the kernel can account for time and perform scheduling work. During idle periods, those interrupts wake an otherwise sleeping processor for very little purpose. Tickless idle programs the next timer event instead, allowing a longer uninterrupted sleep. It cannot help if an application or driver requests a timer every few milliseconds.

On my machine, an application polling for status updates was doing exactly that. Closing it reduced wakeups dramatically and the fan eventually slowed. A network driver also produced regular activity when the interface was unused. Disabling the interface for a test separated that problem from the timer configuration.

Measuring several idle intervals also mattered. One short sample could be dominated by mail checking, disk flushing or my own impatient mouse movement.

There are limits to the name. Tickless does not mean the kernel never uses timer interrupts, and the current work mainly avoids ticks while a processor is idle. A busy processor still needs scheduling and accounting. Hardware timer support matters too; unreliable timers can turn a power improvement into missed timeouts or clocks that drift.

The practical lesson is that tickless operation removes one systematic source of wakeups. It does not forgive programs that poll constantly, nor drivers that chatter with hardware for entertainment. I am keeping it enabled because idle residency improved and battery measurements over several complete runs are better. But the useful debugging question is now “who wakes the processor?” rather than “is CPU usage low?” Zero percent can be composed of thousands of tiny interruptions, which is a very computer-like definition of resting.