Programming

I rewrote a small Python log processor as three stages: read lines, parse records, and aggregate counters. The Python version used queues and worker threads. The Go version uses channels, mostly because I wanted to discover where they stop being cute.

The parser looks like this in the weekly snapshot I have installed:

func parse(in <-chan string, out chan<- Record) {
    for line := range in {
        r, ok := parseLine(line)
        if ok {
            out <- r
        }
    }
    close(out)
}

Directional channel types document who sends and who receives. They also let the compiler reject an accidental send in a consumer. The reader closes its output, the parser drains that input and closes the next channel, and the aggregator stops when its input is closed.

An unbuffered channel forces a rendezvous. If aggregation falls behind, parsing blocks, then reading blocks. That is back pressure without a separate queue-length policy. For this tool it is desirable because there is no value in reading a gigabyte ahead merely to admire it in memory.

A buffered channel changes the timing, not the ownership:

records := make(chan Record, 64)

Now the parser can get 64 records ahead. I tried capacities from zero to 4096. Small buffers smoothed disk and parser timing; large ones mostly increased memory use. The best number depended on record size and machine, so naturally 64 is now a sacred constant until the next measurement.

Closing requires discipline. The sender should close a channel when no more values will arrive. A receiver closing it is guessing about other senders and can provoke a panic. Multiple producers therefore need one coordinator to wait for them and close the shared output.

The current implementation is experimental, as are some channel syntax and library details around it. This is not a claim that channels beat every queue or lock. A shared read-mostly table may be clearer behind a mutex. But for a pipeline, channels put flow control, synchronization, and the handoff of values in one visible operation. I removed two condition variables and, more importantly, stopped having to prove that I signaled both of them on every path.

I usually notice a missing cleanup path during review, which is a dignified way of saying after I wrote it.

Go’s defer lets me put cleanup beside acquisition:

f, err := os.Open(name, os.O_RDONLY, 0)
if err != nil {
    return err
}
defer f.Close()

The call runs when the surrounding function returns, including an early return. Several deferred calls run in last-in, first-out order. That order fits nested resources rather nicely.

Arguments are evaluated when the defer statement executes, not later. This matters in loops and when variables change:

for i := 0; i < 3; i++ {
    defer fmt.Println(i)
}

The output is 2, 1, 0. It is a stack of already-prepared calls, not three promises to inspect i at the end.

I would not put defer in a hot inner loop without measuring it. This compiler and runtime are young, and convenience has a cost even when the syntax hides the invoice. A helper function can give each iteration its own cleanup boundary.

The larger win is boring correctness. Files close on every return, locks can be paired with unlocks, and the cleanup is visible before the function accumulates three branches. C’s goto out remains perfectly respectable, but defer makes the common case harder to omit. This note matches my current weekly snapshot; the surrounding library calls are still moving.

My reflex when seeing go f() was to translate it into “start a thread.” That translation is convenient and wrong enough to cause bad designs.

A goroutine is a language-level activity scheduled by the Go runtime. Its initial stack is small and can grow, so creating thousands is practical in cases where creating thousands of pthreads would be a conversation with both the kernel and the person carrying the pager.

I tested this with a deliberately dull program:

package main

import "fmt"

func wait(done chan bool) {
    done <- true
}

func main() {
    done := make(chan bool)
    for i := 0; i < 10000; i++ {
        go wait(done)
    }
    for i := 0; i < 10000; i++ {
        <-done
    }
    fmt.Println("finished")
}

The channel is doing two jobs: carrying a value and synchronizing sender with receiver. With an unbuffered channel, the send waits for a receive. The program does not need a mutex around a shared completion counter.

That does not make races impossible. If all those goroutines update the same map, I still have a race. “Do not communicate by sharing memory” is good direction, not a force field.

The early runtime is another caveat. On the snapshot I used, scheduling and stack management are active work. Timing results vary, and a tight goroutine that never calls into the runtime can be an unfriendly neighbor. I would not build a latency promise from this experiment.

The cheap creation cost also does not excuse leaving goroutines blocked forever. Their stacks are small, not imaginary, and their work still needs a defined end.

What I do like is the change in scale. In C, a thread is sufficiently costly and awkward that I first ask how to avoid one. In Go, I can model each independent operation directly, then decide where synchronization belongs. The runtime may multiplex many goroutines over fewer operating-system threads; that implementation is precisely why “goroutine equals pthread” is a poor mental model.

This is not magic parallelism either. More runnable work does not create more processors, and today’s defaults may use only one. It does make concurrency cheap enough to express before I optimize it. For server and pipeline code, that changes the first sketch substantially.

Qt taught me to appreciate explicit contracts. It also taught me that an object system can acquire enough machinery to qualify for municipal government.

Go’s interfaces are much smaller. A type satisfies an interface by having the required methods; it does not announce the relationship in its declaration. This took a moment to trust because I am used to inheritance lists serving as signposts.

Here is the complete contract for something that can write bytes in my current snapshot:

type Writer interface {
    Write(p []byte) (n int, err os.Error)
}

A file can satisfy it. So can a network connection or a tiny test buffer. The producer only needs to accept a Writer:

func emit(w Writer, p []byte) os.Error {
    _, err := w.Write(p)
    return err
}

There is no implements Writer marker on those concrete types. The advantage is that I can define an interface around behavior already provided by a package. The package author did not need to predict my abstraction.

The cost is discoverability. Looking at a type alone does not list every interface it happens to satisfy, and a method signature changed by one type is a different method. Compiler diagnostics are doing more of the navigation work than they do in my C++ code.

I tried the idea on a log collector. The original Python version accepted anything with a write method because Python accepts almost anything until it does not. The Go version gets similar flexibility, but verifies the method set when values are connected. That is a useful middle ground.

Interfaces also contain a pair of things internally: a dynamic type and a dynamic value. An interface value can therefore be non-empty even when the pointer stored inside it is nil. I managed to manufacture that surprise within an hour, which must be some kind of efficiency record.

These details reflect a weekly build, not a stable release. Method-set and library conventions may still shift. Even so, structural satisfaction feels important. It encourages narrow contracts at the point of use rather than grand interface families designed before the program has done anything useful.

Go declarations looked backward to me for about ten minutes:

var retries int
var name string = "worker"
count := 12

Then I noticed how little punctuation I was reading. The name comes first, the type follows when it adds information, and := handles the common local case. My C-trained eyes complained briefly and then found something else to complain about.

The useful distinction is scope, not cosmetic shorthand. := declares at least one new variable in the current block. Assignment remains plain =:

line, err := readLine()
line, err = readLine()

Multiple results make this especially tidy. In C I would pass pointers or return a status and fill a buffer. Here the success value and os.Error can travel together without inventing a result structure.

There is also a practical trap. A short declaration inside an if can create a new variable and leave the outer one untouched. The compiler catches many unused-variable accidents, but it cannot infer my intention. Small scopes help.

This is from the current weekly snapshot, so details may move. The principle already feels sound: write the type when the reader needs it, let the initializer provide it otherwise, and keep declarations close to use. It is not revolutionary. Neither is a sharp screwdriver, and I still prefer one.