Blog

My reflex when seeing go f() was to translate it into “start a thread.” That translation is convenient and wrong enough to cause bad designs.

A goroutine is a language-level activity scheduled by the Go runtime. Its initial stack is small and can grow, so creating thousands is practical in cases where creating thousands of pthreads would be a conversation with both the kernel and the person carrying the pager.

I tested this with a deliberately dull program:

package main

import "fmt"

func wait(done chan bool) {
    done <- true
}

func main() {
    done := make(chan bool)
    for i := 0; i < 10000; i++ {
        go wait(done)
    }
    for i := 0; i < 10000; i++ {
        <-done
    }
    fmt.Println("finished")
}

The channel is doing two jobs: carrying a value and synchronizing sender with receiver. With an unbuffered channel, the send waits for a receive. The program does not need a mutex around a shared completion counter.

That does not make races impossible. If all those goroutines update the same map, I still have a race. “Do not communicate by sharing memory” is good direction, not a force field.

The early runtime is another caveat. On the snapshot I used, scheduling and stack management are active work. Timing results vary, and a tight goroutine that never calls into the runtime can be an unfriendly neighbor. I would not build a latency promise from this experiment.

The cheap creation cost also does not excuse leaving goroutines blocked forever. Their stacks are small, not imaginary, and their work still needs a defined end.

What I do like is the change in scale. In C, a thread is sufficiently costly and awkward that I first ask how to avoid one. In Go, I can model each independent operation directly, then decide where synchronization belongs. The runtime may multiplex many goroutines over fewer operating-system threads; that implementation is precisely why “goroutine equals pthread” is a poor mental model.

This is not magic parallelism either. More runnable work does not create more processors, and today’s defaults may use only one. It does make concurrency cheap enough to express before I optimize it. For server and pipeline code, that changes the first sketch substantially.

Qt taught me to appreciate explicit contracts. It also taught me that an object system can acquire enough machinery to qualify for municipal government.

Go’s interfaces are much smaller. A type satisfies an interface by having the required methods; it does not announce the relationship in its declaration. This took a moment to trust because I am used to inheritance lists serving as signposts.

Here is the complete contract for something that can write bytes in my current snapshot:

type Writer interface {
    Write(p []byte) (n int, err os.Error)
}

A file can satisfy it. So can a network connection or a tiny test buffer. The producer only needs to accept a Writer:

func emit(w Writer, p []byte) os.Error {
    _, err := w.Write(p)
    return err
}

There is no implements Writer marker on those concrete types. The advantage is that I can define an interface around behavior already provided by a package. The package author did not need to predict my abstraction.

The cost is discoverability. Looking at a type alone does not list every interface it happens to satisfy, and a method signature changed by one type is a different method. Compiler diagnostics are doing more of the navigation work than they do in my C++ code.

I tried the idea on a log collector. The original Python version accepted anything with a write method because Python accepts almost anything until it does not. The Go version gets similar flexibility, but verifies the method set when values are connected. That is a useful middle ground.

Interfaces also contain a pair of things internally: a dynamic type and a dynamic value. An interface value can therefore be non-empty even when the pointer stored inside it is nil. I managed to manufacture that surprise within an hour, which must be some kind of efficiency record.

These details reflect a weekly build, not a stable release. Method-set and library conventions may still shift. Even so, structural satisfaction feels important. It encourages narrow contracts at the point of use rather than grand interface families designed before the program has done anything useful.

Go declarations looked backward to me for about ten minutes:

var retries int
var name string = "worker"
count := 12

Then I noticed how little punctuation I was reading. The name comes first, the type follows when it adds information, and := handles the common local case. My C-trained eyes complained briefly and then found something else to complain about.

The useful distinction is scope, not cosmetic shorthand. := declares at least one new variable in the current block. Assignment remains plain =:

line, err := readLine()
line, err = readLine()

Multiple results make this especially tidy. In C I would pass pointers or return a status and fill a buffer. Here the success value and os.Error can travel together without inventing a result structure.

There is also a practical trap. A short declaration inside an if can create a new variable and leave the outer one untouched. The compiler catches many unused-variable accidents, but it cannot infer my intention. Small scopes help.

This is from the current weekly snapshot, so details may move. The principle already feels sound: write the type when the reader needs it, let the initializer provide it otherwise, and keep declarations close to use. It is not revolutionary. Neither is a sharp screwdriver, and I still prefer one.

Since I cannot give any details on Suquia Creek, what could I possibly talk about? A lot, it turns out. I can talk about things I’m learning while working on the project.

Ah boy, am I learning!

It turns out Suquia Creek is going to be the longest, most complex project I’ve worked on. Until SC came along, the longest project of my life had been Lava Peak, which ran for about two years. But that included post-launch activities. We managed to go from ideation to shipping in just under a quarter (I even won an award because of that back in 2007.) Suquia Creek, on the other hand, is scheduled to ship in 2014! That’s nearly half a decade of work.

As well, for earlier projects I only had to worry about software. Suquia Creek, however, is also hardware. And on the software side, off the top of my head, it involves —

• Processor µcode

• Chipset code

• BIOS extensions

• Firmware code

• Drivers

• An SDK

• User-level apps

• And a few other things I can’t say without revealing more than I should.

Bottom line: it’s huge as far as I’m concerned! And we should support multiple versions of Windows and one distro release of Linux. It also involves several cross-functional, geographically-dispersed teams based mainly in Argentina and the US, but also with some smaller efforts coming out of China, India, and Israel. That amounts to six different timezones, for a current maximum time difference of 17 hours.

And then we come back to schedule. I have no idea what I’ll be doing at home during this weekend, but I have to have a rough idea of what we’ll be delivering on, say, week 41 of 2013! That assumes the world will not end in 2012, of course.

This week we had our first engineering meeting to plan on a tentative schedule. Late next month I’ll be flying around between Silicon Valley and Silicon Forest to work out the (semi-)hard schedule. By then, we’ll actually have one internal alpha release in place already. After that, we’ll have another alpha before our first “release”, an internal proof of concept, which will then be used by customer as part of a (quasi-confidential) pilot. The customer? One of the world’s largest… well, can’t say what is their industry yet. It’s huge though.

After that pilot I expect to be able to open up a bit on what the project actually does. In the meantime, I’m going to be sharing my learning experience.

It’s going to be an exciting half-decade for me :–)

Most of my systems work still begins in C. If the job needs a user interface, it usually grows a Qt shell. If it needs to be finished before lunch, a Python script appears beside it and quietly becomes permanent.

That arrangement works, but every boundary charges rent. The C tool has ownership rules, the Qt side has objects and signals, and the Python helper has a different deployment story. I have spent an unreasonable amount of time explaining to machines where their libraries live.

I have been trying the experimental Go compiler from the weekly snapshots. It is plainly unfinished, and the language may change under this post, but it occupies an interesting spot: compiled programs, garbage collection, a small syntax, and concurrency that does not begin with a page of pthread ceremony.

My first useful experiment was the sort of filter I normally write in Python:

package main

import (
    "bufio"
    "fmt"
    "os"
)

func main() {
    in := bufio.NewReader(os.Stdin)
    for {
        line, err := in.ReadString('\n')
        if len(line) != 0 {
            fmt.Printf("%d %s", len(line), line)
        }
        if err == os.EOF {
            break
        }
        if err != nil {
            fmt.Fprintf(os.Stderr, "read: %s\n", err.String())
            os.Exit(1)
        }
    }
}

ReadString may return bytes together with os.EOF, so the line is handled before the error. That keeps a final line without a newline. EOF ends input normally; another os.Error gets reported instead of masquerading as a clean finish.

On my snapshot, I build that directly with 6g and 6l. A later snapshot may spell an import or an I/O call differently; that is currently part of the admission price.

The pleasant surprise is not that this is shorter than C. Python already wins that contest while barely getting out of bed. The surprise is that the result is a small native executable and the source still exposes the mechanics I care about. There is no class hierarchy between the input and me.

I do miss deterministic destruction from C++, especially around Qt resources. Garbage collection is not a substitute for closing a file. Go’s defer looks like the intended answer for many such cases, and I plan to abuse it scientifically.

I am not replacing working C or Qt code. That would be migration as a hobby, a particularly expensive hobby. I am moving the little glue programs first and watching where the new language becomes awkward. So far, it feels less like Python made fast and more like C with several recurring arguments removed.