Merging Sorted Streams With Channels

Thu, 13 Jan 2011 20:16:00 +0000

I had several sorted result files and wanted one sorted stream without loading everything into memory. The classic algorithm is a k-way merge: keep the next item from each input, emit the smallest, then advance only that input.

My first Go version gave every file reader a goroutine and a channel. The merger holds one current value per channel:

type item struct {
 value int
 open bool
}

func source(values []int) <-chan int {
 out := make(chan int)
 go func() {
 for _, value := range values {
 out <- value
 }
 close(out)
 }()
 return out
}

The merge loop initializes one item from every source, scans those current values for the smallest, emits it, and receives the replacement from the chosen source. When that receive reports the channel closed, the source leaves consideration. Memory remains proportional to the number of sources, apart from buffering in the file readers.

A linear scan makes each output operation $O(k)$ , where $k$ is the number of inputs. That is fine for four files and silly for four thousand. Replacing the scan with a binary heap makes selection $O(\log k)$ . I implemented both and found the simple scan faster for my actual case of eight inputs. Algorithms still have constants, despite textbooks’ efforts to protect us from them.

Channels help isolate blocking input, but they do not improve the comparison algorithm. That distinction matters. Concurrency lets readers overlap waiting on disks or pipes; the heap changes CPU work. Combining both without measuring can add scheduling overhead to a problem that was already fast.

There is also a shutdown question. If the consumer stops early, source goroutines may remain blocked trying to send. My real version includes a separate stop channel selected by each producer, and the caller closes it when abandoning the merge. A finite example that always drains output can omit this, but reusable code cannot.

This code targets the January 2011 snapshot. Channel receive conventions and library calls should be checked against that compiler. The useful design survives those changes: one lookahead value per sorted source, explicit ownership of advancement, and a selection structure chosen for the actual number of streams.

Channel Pipelines With Back Pressure

Thu, 29 Apr 2010 22:11:00 +0000

I rewrote a small Python log processor as three stages: read lines, parse records, and aggregate counters. The Python version used queues and worker threads. The Go version uses channels, mostly because I wanted to discover where they stop being cute.

The parser looks like this in the weekly snapshot I have installed:

func parse(in <-chan string, out chan<- Record) {
 for line := range in {
 r, ok := parseLine(line)
 if ok {
 out <- r
 }
 }
 close(out)
}

Directional channel types document who sends and who receives. They also let the compiler reject an accidental send in a consumer. The reader closes its output, the parser drains that input and closes the next channel, and the aggregator stops when its input is closed.

An unbuffered channel forces a rendezvous. If aggregation falls behind, parsing blocks, then reading blocks. That is back pressure without a separate queue-length policy. For this tool it is desirable because there is no value in reading a gigabyte ahead merely to admire it in memory.

A buffered channel changes the timing, not the ownership:

records := make(chan Record, 64)

Now the parser can get 64 records ahead. I tried capacities from zero to 4096. Small buffers smoothed disk and parser timing; large ones mostly increased memory use. The best number depended on record size and machine, so naturally 64 is now a sacred constant until the next measurement.

Closing requires discipline. The sender should close a channel when no more values will arrive. A receiver closing it is guessing about other senders and can provoke a panic. Multiple producers therefore need one coordinator to wait for them and close the shared output.

The current implementation is experimental, as are some channel syntax and library details around it. This is not a claim that channels beat every queue or lock. A shared read-mostly table may be clearer behind a mutex. But for a pipeline, channels put flow control, synchronization, and the handoff of values in one visible operation. I removed two condition variables and, more importantly, stopped having to prove that I signaled both of them on every path.

Channels on Roberto Selbach

Merging Sorted Streams With Channels

Channel Pipelines With Back Pressure