Parallel benchmarks and wandering maps

Thu, 09 Oct 2014 13:10:00 +0000

I tested a locked lookup table with Go 1.3’s parallel benchmark support:

func BenchmarkLookupParallel(b *testing.B) {
	b.RunParallel(func(pb *testing.PB) {
		for pb.Next() {
			_ = table.Lookup("troy")
		}
	})
}

RunParallel creates goroutines and distributes benchmark iterations among them. This exposed lock contention that a serial benchmark politely concealed. It is still synthetic: all workers hammering one key may be less realistic than production, so I use representative key distributions too.

The same upgrade broke a test that compared map traversal output to a fixed string. Go 1.3 randomizes iteration order for small maps, making an assumption that was always invalid fail more reliably.

for k := range m {
	keys = append(keys, k)
}
sort.Strings(keys)

Sorting is correct when output order is part of the result. For equality, I compare contents instead of rendering iteration order.

Random order is not cryptographic randomness, nor should a program depend on getting a different order every pass. It is an implementation defense against accidental ordering assumptions, not a shuffling API.

I approve of the randomization. A runtime that accidentally rewards a bad assumption lets the assumption become an API. My test was not flaky; it was finally honest.

Maps on Roberto Selbach

Parallel benchmarks and wandering maps