Merging Sorted Streams With Channels

I had several sorted result files and wanted one sorted stream without loading everything into memory. The classic algorithm is a k-way merge: keep the next item from each input, emit the smallest, then advance only that input.

My first Go version gave every file reader a goroutine and a channel. The merger holds one current value per channel:

type item struct {
    value int
    open  bool
}

func source(values []int) <-chan int {
    out := make(chan int)
    go func() {
        for _, value := range values {
            out <- value
        }
        close(out)
    }()
    return out
}

The merge loop initializes one item from every source, scans those current values for the smallest, emits it, and receives the replacement from the chosen source. When that receive reports the channel closed, the source leaves consideration. Memory remains proportional to the number of sources, apart from buffering in the file readers.

A linear scan makes each output operation $O(k)$, where $k$ is the number of inputs. That is fine for four files and silly for four thousand. Replacing the scan with a binary heap makes selection $O(\log k)$. I implemented both and found the simple scan faster for my actual case of eight inputs. Algorithms still have constants, despite textbooks’ efforts to protect us from them.

Channels help isolate blocking input, but they do not improve the comparison algorithm. That distinction matters. Concurrency lets readers overlap waiting on disks or pipes; the heap changes CPU work. Combining both without measuring can add scheduling overhead to a problem that was already fast.

There is also a shutdown question. If the consumer stops early, source goroutines may remain blocked trying to send. My real version includes a separate stop channel selected by each producer, and the caller closes it when abandoning the merge. A finite example that always drains output can omit this, but reusable code cannot.

This code targets the January 2011 snapshot. Channel receive conventions and library calls should be checked against that compiler. The useful design survives those changes: one lookahead value per sorted source, explicit ownership of advancement, and a selection structure chosen for the actual number of streams.