Blog

After the race detector found a real bug, I wanted to know whether it was merely running the program many times and hoping for an unfortunate schedule. It is doing something much more useful.

A data race exists when two goroutines access the same memory concurrently, at least one access writes, and synchronization does not establish the required ordering. The detector instruments loads and stores so they can be recorded in shadow state associated with application memory. It also observes synchronization operations such as channel communication and mutex locking.

The key idea is causality rather than wall-clock time. Each goroutine’s operations have an order. Synchronization joins those orders: a channel send and corresponding receive, for example, establish a happens-before edge. The detector maintains enough logical-clock information to determine whether conflicting memory accesses are ordered by such edges.

Consider this broken counter:

var n int

func increment(done chan bool) {
	n++
	done <- true
}

Two goroutines executing n++ each perform a read and a write. Sending completion values afterward does not order the increments with respect to each other. The detector sees conflicting accesses without a happens-before path and reports both stacks.

Putting a mutex around the increment changes more than timing:

mu.Lock()
n++
mu.Unlock()

Unlock and a later lock establish ordering, so accesses in the critical sections no longer race. Sleeping between increments might make overlap less likely, but it adds no synchronization relation and therefore is not a fix.

Instrumentation explains the overhead of -race. Every relevant access carries bookkeeping, and shadow memory plus goroutine histories consume substantial memory. This is the cost of seeing relationships that ordinary execution discards. I run race-enabled tests separately rather than using their timings as application benchmarks.

The detector is dynamic. If a branch is never executed, its accesses are never observed. If a test uses one worker while production uses twenty, the most interesting paths may remain untouched. Good concurrent tests still matter: vary inputs, create contention, repeat operations, and run realistic integration workloads when possible.

There can also be benign-looking races that do not crash today, such as an approximate statistics counter. They remain races under the memory model. The compiler is not required to preserve the comforting behaviour I happened to observe. Use synchronization or documented atomic operations when approximation is acceptable but racing is not.

Understanding the mechanism changed how I read reports. I no longer ask only whether the two lines could execute at the exact same nanosecond. I ask what synchronization orders them. If I cannot point to that edge, the detector has probably found a hole rather than developed an artistic difference of opinion.

Go 1.1 permits a useful expression I kept trying to write before it existed: a method value.

type counter struct{ n int }

func (c *counter) add(n int) { c.n += n }

c := &counter{}
add := c.add
add(3)
add(4)
fmt.Println(c.n) // 7

The expression c.add produces a function value with c bound as its receiver. Its type is func(int). This is handy when an API wants a callback and the work naturally belongs to an object.

It differs from a method expression:

add := (*counter).add
add(c, 3)

Here the receiver is explicit, so the function type is func(*counter, int). The distinction is captured receiver versus receiver as the first argument.

The receiver is evaluated when the method value is created. For a pointer receiver, the saved pointer continues to identify the same object. For a value receiver, the receiver value is copied, just as it is for an ordinary value-receiver call. This can matter for large values and for anyone expecting later replacement of a variable to retarget an already-created callback.

Method values may require storage for the bound receiver, so I would measure before putting their creation in a very hot loop. Their main virtue is not speed. It is that callback code can say server.handle instead of wrapping the same call in an otherwise pointless closure.

I had two versions of a formatter with nearly identical benchmark times. One looked cleaner, so naturally I distrusted it.

Go 1.1’s benchmark allocation reporting made the difference visible. Benchmarks can call b.ReportAllocs(), or the test command can request memory statistics for benchmarks. The output includes allocations per operation and bytes allocated per operation alongside timing.

func BenchmarkLabel(b *testing.B) {
	b.ReportAllocs()
	for i := 0; i < b.N; i++ {
		_ = label(42)
	}
}

My first formatter repeatedly concatenated strings:

func join(parts []string) string {
	var out string
	for _, part := range parts {
		out += part
	}
	return out
}

As out grows, concatenation creates new strings and copies prior contents. A bytes.Buffer version reduced allocation for larger inputs. For two tiny pieces, however, the direct expression remained clearer and entirely adequate. Allocation counts are measurements, not commandments delivered from a heap-shaped mountain.

The numbers need careful interpretation. “Bytes per operation” is an average over repeated benchmark iterations. Setup performed inside the timed loop counts against the operation, including test data accidentally rebuilt every time. Use b.ResetTimer() after setup when the setup is not part of what should be measured.

The benchmark result also depends on escape analysis and compiler decisions. A value that remains on the stack does not appear as a heap allocation merely because source code takes its address. Conversely, converting values to interfaces or retaining pointers can move data to the heap in ways that are not obvious from syntax. Compiler diagnostics can explain candidates after the benchmark shows there is a problem.

Allocation count and allocated bytes tell different stories. Many tiny objects increase allocator and collector work. One large object can dominate memory traffic while counting as a single allocation. Live heap is different again: an object retained for minutes affects collection even if it was allocated only once.

I now include allocation reports for benchmarks on hot parsing, formatting, and request paths. I first verify the output cannot be optimized away, keep setup honest, and compare behaviour as well as speed. Reducing allocations often improves performance, but an obscure zero-allocation function can still be a bad bargain.

I keep the benchmark input representative as well. A formatter tested only with an empty string can truthfully report zero allocations while answering a question no caller was likely to ask.

In this case the cleaner formatter also allocated less after a small adjustment. I was forced to accept the pleasant result. Software occasionally lacks respect for a good suspicion.

Um amigo publicou um texto no Facebook hoje que me lembrou do MBA. Vou deixar o texto aqui, como exemplo do tipo de texto comum em escolas de negócio e deixar o julgamento ao leitor:

ASSALTO MUITO INTERESSANTE.

Durante um assalto, em Guangzhou, China, o assaltante de bancos gritou para todos no banco: “Não se mova O dinheiro pertence ao Estado Sua vida pertence a você…”

Todo mundo no banco deitou-se calmamente no chão. Isso é chamado de “Mudando o conceito mental” Mudar a forma convencional de pensar.

Quando uma senhora apresentou-se sobre a mesa provocativamente, em que o ladrão gritou para ela: “Por favor, seja civilizada isto é um assalto e não um estupro!”

Isso é chamado de “ser profissional” Concentre-se apenas no que você está treinado para fazer!

Quando os assaltantes voltaram para casa, o ladrão mais jovem (MBA treinee), disse ao ladrão mais velho (que só completou seis anos na escola primária): “Big brother, vamos contar o quanto nós temos.”

O assaltante mais velho rebateu e disse: “Você é muito estúpido. Há tanto dinheiro que vai nos levar muito tempo para contar. Hoje à noite, o noticiário da TV vai nos dizer o quanto nós roubamos do banco…!”

Isso é chamado de “experiência”. Hoje em dia, a experiência é mais importante do que as qualificações do papel!

Depois que os ladrões saíram, o gerente do banco disse ao supervisor bancário para chamar a polícia rapidamente. Mas o supervisor lhe disse: “Espere, vamos retirar U$ 10 milhões do banco para nós mesmos e adicioná-lo aos 70 milhões dólares que já foram desviados do banco”.

Isso é chamado de “nadar à favor da maré.” Convertendo uma situação desfavorável para a sua vantagem!

O supervisor diz: “Vai ser bom para nós se houver um assalto a cada mês.”

Isso é chamado de “morte do tédio”. Felicidade pessoal é mais importante do que o seu trabalho.

No dia seguinte, o noticiário da TV informou que U$ 100 milhões, foram retirados do banco. Os ladrões contaram e contaram e contaram, mas eles só podiam contar o montante de U$20 milhões. Os ladrões estavam muito irritados e reclamaram: “Nós arriscamos nossas vidas e só levamos 20 milhões de dólares. O gerente do banco levou 80 milhões de dólares com apenas um estalar de seus dedos. Parece que é melhor ser educado do que ser um ladrão..!”

Isso é chamado de “Conhecimento que vale tanto quanto ouro!”

O gerente do banco estava sorrindo feliz porque suas perdas no mercado de ações foram agora cobertas por este roubo.

Isso é chamado de “Aproveitando a oportunidade.” Ousadia para assumir riscos!

Então, quem são os ladrões reais aqui?

Conde de Kakflour

Every small text-processing program I write begins with the same lie: “I only need to read a file line by line.” Five minutes later I am handling buffering, end-of-file, and the final line without a newline.

Go 1.1 added bufio.Scanner, which packages that loop neatly:

scanner := bufio.NewScanner(r)
for scanner.Scan() {
	line := scanner.Text()
	fmt.Println(line)
}
if err := scanner.Err(); err != nil {
	return err
}

By default Scanner splits input into lines and removes the line ending. Scan advances to the next token. Text returns that token as a string; Bytes returns the token as bytes. After the loop, Err distinguishes a clean end-of-input from a read or tokenization error.

That final check is easy to omit because the loop looks complete without it. It is not. An I/O error halfway through a configuration file should not quietly turn the remaining configuration into an unusually convincing empty section.

Scanner is more general than lines. Split accepts a split function, and the package supplies functions for words, bytes, and runes. A split function receives available data and whether the underlying reader has reached its end. It reports how far to advance, which token to return, and any error. This supports simple lexical work without writing the buffering loop again.

There is an important limit: Scanner uses a maximum token size. It is intended for reasonably sized tokens, not arbitrary enormous records. A log format that permits a single multi-megabyte line may exceed the limit and stop scanning with an error. For such input, bufio.Reader and an explicit loop provide more control. Choosing Scanner means accepting its token-oriented contract, not merely admiring its shorter spelling.

Bytes also deserves care. The returned slice may refer to storage Scanner reuses on the next call to Scan. Process it before advancing, or copy it if it must survive. Text gives a string representation that is safer to retain, with the corresponding conversion and allocation considerations.

I also avoid mixing direct reads from the underlying reader with Scanner. Scanner buffers ahead, so the reader’s apparent position need not correspond to the end of the last token I processed. One owner for the stream is a much less surprising arrangement.

I replaced my hand-written line reader with Scanner and deleted more edge-case code than I added. That is a good trade when its token limits fit the data. When they do not, the lower-level reader remains available. Go’s I/O packages are at their best when the simple case is genuinely simple and the escape hatch is still visible.