Algorithms on Roberto Selbach

Cache Lines Beat Clever Loops

Thu, 03 Mar 2011 21:31:00 +0000

I translated a matrix filter from C to Go and then spent an afternoon optimizing arithmetic that was not the main cost. The profile pointed at memory movement, so I reduced the problem to traversal order.

The matrix is stored in one slice in row-major order. These loops calculate the same sum:

for y := 0; y < height; y++ {
 for x := 0; x < width; x++ {
 sum += pixels[y*width+x]
 }
}

for x := 0; x < width; x++ {
 for y := 0; y < height; y++ {
 sum += pixels[y*width+x]
 }
}

The first walks adjacent elements. The second jumps by an entire row. On a large image the row-wise loop is much faster because each fetched cache line supplies several values used immediately. The column-wise loop asks the processor to fetch many lines and uses a small part of each before moving on.

Bounds checks and multiplication were my original suspects. Hoisting the row offset made the code clearer and helped a little:

for y := 0; y < height; y++ {
 row := pixels[y*width:(y+1)*width]
 for x := 0; x < width; x++ {
 sum += row[x]
 }
}

But traversal order remained the large effect. A more sophisticated arithmetic trick could not compensate for unfriendly access.

I benchmark with enough data to exceed the small caches, repeat the operation, and verify the sum so the compiler cannot discard the work. These are 2011 compiler snapshots, so absolute timings and optimization behavior are temporary facts. The memory hierarchy is less temporary.

I alternate test order as well. Running one version first every time can give it consistently colder data and turn benchmark order into a hidden input.

This also affects data structure choice. A slice of small structures may have better locality than a slice of pointers to separately allocated structures. The latter adds pointer chasing and gives the allocator freedom to scatter objects. It may still be right when identity and independent lifetimes matter, but it is not free.

My practical conclusion is embarrassingly physical: visit data in the order it lies in memory. Before unrolling loops or translating a bit trick from C, inspect the access pattern. The processor is very fast when fed and becomes an expensive space heater when asked to wait.

Merging Sorted Streams With Channels

Thu, 13 Jan 2011 20:16:00 +0000

I had several sorted result files and wanted one sorted stream without loading everything into memory. The classic algorithm is a k-way merge: keep the next item from each input, emit the smallest, then advance only that input.

My first Go version gave every file reader a goroutine and a channel. The merger holds one current value per channel:

type item struct {
 value int
 open bool
}

func source(values []int) <-chan int {
 out := make(chan int)
 go func() {
 for _, value := range values {
 out <- value
 }
 close(out)
 }()
 return out
}

The merge loop initializes one item from every source, scans those current values for the smallest, emits it, and receives the replacement from the chosen source. When that receive reports the channel closed, the source leaves consideration. Memory remains proportional to the number of sources, apart from buffering in the file readers.

A linear scan makes each output operation $O(k)$ , where $k$ is the number of inputs. That is fine for four files and silly for four thousand. Replacing the scan with a binary heap makes selection $O(\log k)$ . I implemented both and found the simple scan faster for my actual case of eight inputs. Algorithms still have constants, despite textbooks’ efforts to protect us from them.

Channels help isolate blocking input, but they do not improve the comparison algorithm. That distinction matters. Concurrency lets readers overlap waiting on disks or pipes; the heap changes CPU work. Combining both without measuring can add scheduling overhead to a problem that was already fast.

There is also a shutdown question. If the consumer stops early, source goroutines may remain blocked trying to send. My real version includes a separate stop channel selected by each producer, and the caller closes it when abandoning the merge. A finite example that always drains output can omit this, but reusable code cannot.

This code targets the January 2011 snapshot. Channel receive conventions and library calls should be checked against that compiler. The useful design survives those changes: one lookahead value per sorted source, explicit ownership of advancement, and a selection structure chosen for the actual number of streams.

Counting Bits Like the Kernel

Tue, 09 Nov 2010 20:39:00 +0000

Kernel code contains algorithms that look unnecessarily peculiar until the constraints are visible. Population count, the number of set bits in a machine word, is a good example.

The obvious C version examines every bit:

unsigned int count_bits(unsigned long word)
{
 unsigned int n = 0;

 while (word) {
 n += word & 1;
 word >>= 1;
 }
 return n;
}

This performs one iteration for every bit position up to the highest set bit. A sparse bitmap still pays for all the zeros in between.

Brian Kernighan’s familiar operation removes the lowest set bit at each iteration:

unsigned int count_bits(unsigned long word)
{
 unsigned int n = 0;

 while (word) {
 word &= word - 1;
 n++;
 }
 return n;
}

Subtracting one flips the lowest set bit to zero and turns lower zeros into ones. The & keeps everything above that bit and clears the changed suffix. The loop therefore runs once per set bit.

That is excellent for sparse words and less impressive for dense ones. Kernels also use table lookups and architecture-specific instructions where available. A byte table has predictable work but adds memory access; a processor instruction can win decisively but needs dispatch or compile-time selection. There is no universally fastest source fragment independent of data and machine.

Word width matters too. Kernel types and helpers make that choice explicit, while a casual userspace benchmark can accidentally compare different widths and announce a victory produced by less work.

I benchmarked the two loops over generated sparse and dense arrays, keeping generation outside the timed section. As expected, the clear-lowest-bit version dominated sparse input. Dense input narrowed the gap considerably. Repeating one constant word produced lovely numbers and measured the compiler more than the routine, so I stopped doing that.

The broader lesson from kernel work is to understand the representation before admiring the trick. Bitmaps pack state compactly and permit word-at-a-time operations, but contention, cache placement, and scan direction can matter more than shaving an instruction from population count.

I like this algorithm because its mechanism fits in one sentence and its performance caveat requires another. Most optimization stories should have both sentences. The ones with only the first usually end in a benchmark that accidentally proves the author’s laptop exists.