The Go proposal committee has declined the try proposal.

I am disappointed. I also think this is open source working exactly as it should.

The proposal was not dropped because one person with commit access disliked it. A concrete design was published, tools were written to try it on existing code, hundreds of examples and objections were discussed, and the people behind it kept answering questions long after I would have quietly disconnected my router and moved to the woods.

Most importantly, the Proposal Review Committee made its decision in response to that discussion. The closing note mentions problems the original design had missed, including debugging prints and code coverage. It also admits something more fundamental: many Go programmers do not agree that the verbosity of error forwarding is a problem worth changing the language to solve.

That is a good reason to stop. A language belongs to the people who have to read it, and a change to control flow needs more than a clever specification. The community worked through the tradeoffs together and the committee made the call. I may disagree with the result, but I trust the process far more after seeing it work.

Still, I think this will eventually look like a missed opportunity.

The argument against try was strongest when it was used inside nested expressions:

return write(encode(try(read(name))))

I do not want to debug that either. But bad nesting was not the only possible future. The ordinary form was much less alarming:

b := try(ioutil.ReadFile(name))
try(json.Unmarshal(b, &cfg))

One operation per line, errors still in the function signature, no exceptions, and an explicit if whenever local handling mattered. I think that style would have become normal very quickly. It looks unusual today mostly because it is new.

There is a difficult bias in evaluating syntax: familiar boilerplate does not feel like syntax anymore. We see if err != nil and read “return the error.” We see try and inspect every unusual corner because it is new. That scrutiny is necessary before changing a language, but it is not a neutral comparison.

Rust went through a remarkably similar argument around try! and later ?. People worried about hidden control flow and two ways to write the same thing. Once programmers had lived with it, many came to consider it one of the language’s better features. Go is not Rust and should not become Rust, but people do learn new control-flow notation. We already did it with defer and go.

There is another group absent from a discussion among current Go programmers: people who did not choose Go because they found its error handling tedious. It is almost impossible to measure them from inside the community. The people most comfortable with the status quo are, unsurprisingly, the people who stayed.

None of that means the committee should have ignored the reaction and forced try through. That would have damaged the language and the community more than a little error boilerplate ever could. Declining it was the right decision now, given the lack of agreement about both the problem and the solution.

But I suspect we solved the social problem by leaving the technical one untouched. Years from now we may add something remarkably similar under another name, after enough other languages have made the idea feel ordinary. Or we may simply keep writing the same four lines and insist they are valuable because we have become very fast at not seeing them.

Either way, try was worth trying. The proposal improved our understanding of the problem, and its rejection proved that the Go community can say no even after a great deal of work has been invested.

That is not failure. I just hope it is not the last word on improving error handling.

It has been six days since the try proposal was posted and the issue already has hundreds of comments.

Robert Griesemer has posted a useful summary of the discussion. After reading far too much of the thread, I think the arguments are clearer now. They are also better than “I hate typing if err != nil” versus “real programmers enjoy typing it,” which is where these discussions usually go to die.

The strongest case for try is ordinary sequential code:

func readConfig(name string) (*Config, error) {
	b := try(ioutil.ReadFile(name))
	var cfg Config
	try(json.Unmarshal(b, &cfg))
	return &cfg, nil
}

There is very little error handling in the expanded version. Both errors are simply returned. The extra if statements interrupt the description of what the function does without adding a decision.

This is not merely speculation. Some people have tried converting code from the standard library. One conclusion was that try made most of those examples easier to read. Another person looked at much the same evidence and reached almost exactly the opposite conclusion.

The proposal has other good properties. It does not introduce exceptions. Errors remain ordinary return values. Existing code keeps working. try handles the dull case while an if remains available whenever the error needs local attention. I find that separation appealing.

The invisible return

The best argument against it is simple: try looks like a function call but changes the control flow of its caller.

Consider this:

req.Header = Header(try(tp.ReadMIMEHeader()))

That assignment can return from the enclosing function. A struct literal containing try can return. A nested expression can contain several possible returns on the same source line. Today my eyes know that constructing a value does not quietly leave the function. With try, they would have to search inside every expression.

Some commenters therefore prefer a keyword or statement form:

try f := os.Open(name)

It is harder to miss, but it is also a larger language change and it gives up the proposal’s useful ability to fit where an expression fits. Restricting try may make it safer while removing much of what makes it attractive. Language design is fun like that.

There are practical problems too. If I add a debugging print to one particular failure today, there is an obvious block in which to put it. If several operations use try and one deferred handler wraps all of their errors, identifying the exact operation may require changing the code back to an if.

Coverage has a similar problem. The explicit error block is visible to coverage tools. A hidden return inside try creates a branch that source-level coverage needs to represent somehow. Debuggers and stack traces also need enough information to distinguish multiple try calls on one line. These are practical objections, not just a matter of taste.

The context problem

I am less convinced by the proposed use of defer for error context than I am by try itself.

A deferred handler can add function-level context once:

defer fmt.HandleErrorf(&err, "reading configuration %s", name)

That is lovely when every failure in the function deserves the same context. It is less useful when opening the file, parsing it, and validating it each need different information. The handler also runs on every return and encourages named error results. None of this is fatal, but it means the proposal is strongest exactly where error forwarding is least interesting.

So I still like try, with qualifications large enough to require their own parking space. I would avoid nesting it and use one operation per line. I would keep if err != nil wherever an error needs local context, cleanup, logging, or translation. In that style the hidden control flow is bounded and the useful code becomes much easier to see.

But Go cannot add a feature on the assumption that everybody will use my preferred style. If nested try is legal, nested try will exist. If a return can hide in a composite literal, people will write code that does exactly that.

I thought the original proposal was likely to fail because it felt unlike Go. The discussion has produced better reasons than feelings. I still think we may be rejecting an idea that would become ordinary and useful with time, but it is no longer difficult to understand why so many people are wary of it.

Yesterday Robert Griesemer posted a proposal for a new try built-in in Go. I have read the design document twice now and I think I like it.

I am not entirely comfortable saying that.

Two years ago I wrote that explicit error handling was exactly one of the things I loved about Go. I still do. The interesting question here is whether returning an error unchanged really counts as handling it.

The proposal turns this:

f, err := os.Open(filename)
if err != nil {
	return nil, err
}

into this:

f := try(os.Open(filename))

If os.Open returns an error, try places it in the enclosing function’s error result and returns from that function. Otherwise it gives us the other result. Errors remain values, functions still say that they return errors, and there are no exceptions jumping across some unknowable number of stack frames.

That is the part I like. A surprising amount of Go code is not really handling an error at all. It is checking the error and immediately passing it to the caller. I love explicit error handling, but writing four lines does not make that decision four times more explicit.

And yet try feels wrong.

It looks like a function but it can return from the function that called it. That is not something ordinary Go functions can do. The control flow is hidden inside an expression, and Go has spent the last ten years making control flow painfully, wonderfully obvious. There is no return on the line. It just returns.

The proposed way to add context also takes some getting used to. Instead of wrapping each error where it happens, a function can use a deferred handler to decorate the returned error. Using the helper sketched in the design, it would look something like this:

func readConfig(name string) (cfg Config, err error) {
	defer fmt.HandleErrorf(&err, "read configuration %s", name)

	data := try(ioutil.ReadFile(name))
	try(json.Unmarshal(data, &cfg))
	return cfg, nil
}

I am not sure I love the named result or the indirection in the handler. But I do like being able to see the actual operation without each line being separated from the next by the same little error-return ceremony.

My guess is that this proposal will not pass. It is a small addition in terms of specification and implementation, but a radical change in how Go code reads. Those are often the hardest changes: technically tiny and culturally enormous.

I also suspect that, if it does not pass, we may look back several years from now and wonder why it seemed so frightening. New syntax always has the disadvantage of looking unfamiliar in every example. The existing syntax has had a decade to become invisible.

Maybe I am wrong and try will turn real programs into a collection of secret exits. But the idea behind it is sound: keep explicit errors, while admitting that forwarding an error is not the interesting part of error handling.

That seems worth trying.

I’m a big fan of the way Go does zero values, meaning it initializes every variable to a default value. This is in contrast with the way other languages such as, say, C behave. For instance, the printed result of the following C program is unpredictable.

#include <stdio.h>

int main(void) {
    int i;
    printf("%d\n", i);
    return 0;

}

The value of i will be whatever happens to be at the position in memory where the compiler happened to allocate the variable. Contrast this with the equivalent program in Go —

package main

import "fmt"

func main() {
    var i int
    fmt.Println(i)
}

This will always print `` because i is initialized by the compiler to the default value of an int, which happens to be 0.

This happens for every variable of any type, including our own custom types. What’s even cooler is that this is done recursively, so if you the fields inside a struct will also be themselves initialized.

I strive to make all my zero values useful, but it’s not always that simple. Sometimes you need to use different default values for your fields, or maybe you need to initialize one of those fields. This is especially important when we remember that the zero value of a pointer is nil.

Imagine the following type —

type Foobar struct {
    db *DB
}

func NewFoobar() *Foobar {
    return &Foobar{db: DB.New()}
}

func (f *Foobar) Get(key string) (*Foo, error) {
    foo, err := db.Get(key)
    if err != nil {
        return nil, err
    }
    return foo, nil
}

In the example above, our zero value is no longer useful: we’d cause a runtime error because db will be nil inside the Get() function. We’re forced to call NewFoobar() before using our functions.

But there’s a simple trick to make the Foobar zero value useful again. As it turns out, being lazy sometimes pays off. Our technique is called lazy initialization —

type Foobar struct {
    dbOnce sync.Once
    db *DB
}

// lazy initialize db
func (f *Foobar) lazyInit() {
    f.dbOnce.Do(func() {
        f.db = DB.New()
    })
}

We added a sync.Once to our type. From the Go docs:

Once an object that will perform exactly one action.

The function we pass to sync.Once.Do() is guaranteed to run once and only once, so it is perfect for initializations. Now we can call lazyInit() at the top of our exported function and it will ensure db is initialized —

func (f *Foobar) Get(key string) (*Foo, error) {
    f.lazyInit()

    foo, err := db.Get(key)
    if err != nil {
        return nil, err
    }
    return foo, nil
}

...

var f Foobar
foo, err := f.Get("baz")

We are now free to use our zero value with no additional initialization. I love it.

Of course, it is not always possible to use zero values. For example, our Foobar assumes a magical object DB that can be initialized by itself, but in real life we probably need to connect to an external database, authenticate, etc and then pass the created DB to our Foobar.

Still, using lazy initialization allows us to make a lot of objects’ zero values useful that would otherwise not be.

(This article has been graciously translated to Russian here. Huge thanks to Akhmad Karimov.)

I wrote a brief introduction to Go modules and in it I talked briefly about Go modules proxies and now that Go 1.11 is out, I thought I’d play a bit these proxies to figure our how they’re supposed to work.

Why

One of the goals of Go modules is to provide reproducible builds and it does a very good job by fetching the correct and expected files from a repository.

But what if the servers are offline? What if the repository simply vanishes?

One way teams deal with these risks is by vendoring the dependencies, which is fine. But Go modules offers another way: the use of a module proxy.

The Download Protocol

When Go modules support is enabled and the go command determines that it needs a module, it first looks at the local cache (under $GOPATH/pkg/mods). If it can’t find the right files there, it then goes ahead and fetches the files from the network (i.e. from a remote repo hosted on Github, Gitlab, etc.)

If we want to control what files go can download, we need to tell it to go through our proxy by setting the GOPROXY environment variable to point to our proxy’s URL. For instance:

export GOPROXY=http://gproxy.mycompany.local:8080

The proxy is nothing but a web server that responds to the module download protocol, which is a very simple API to query and fetch modules. The web server may even serve static files.

A typical scenario would be the go command trying to fetch github.com/pkg/errors:

The first thing go will do is ask the proxy for a list of available versions. It does this by making a GET request to /{module name}/@v/list. The server then responds with a simple list of versions it has available:

v0.8.0
v0.7.1

The go will determine which version it wants to download — the latest unless explicitly told otherwise¹. It will then request information about that given version by issuing a GET request to /{module name}/@v/{module revision} to which the server will reply with a JSON representation of the struct:

type RevInfo struct {
    Version string    // version string
    Name    string    // complete ID in underlying repository
    Short   string    // shortened ID, for use in pseudo-version
    Time    time.Time // commit time
}

So for instance, we might get something like this:

{
    "Version": "v0.8.0",
    "Name": "v0.8.0",
    "Short": "v0.8.0",
    "Time": "2018-08-27T08:54:46.436183-04:00"
}

The go command will then request the module’s go.mod file by making a GET request to /{module name}/@v/{module revision}.mod. The server will simply respond with the contents of the go.mod file (e.g. module github.com/pkg/errors.) This file may list additional dependencies and the cycle restarts for each one.

Finally, the go command will request the actual module by getting /{module name}/@v/{module revision}.zip. The server should respond with a byte blob (application/zip) containing a zip archive with the module files where each file must be prefixed by the full module path and version (e.g. github.com/pkg/errors@v0.8.0/), i.e. the archive should contain:

github.com/pkg/errors@v0.8.0/example_test.go
github.com/pkg/errors@v0.8.0/errors_test.go
github.com/pkg/errors@v0.8.0/LICENSE
...

And not:

errors/example_test.go
errors/errors_test.go
errors/LICENSE
...

This seems like a lot when written like this, but it’s in fact a very simple protocol that simply fetches 3 or 4 files:

1. The list of versions (only if go does not already know which version it wants)
2. The module metadata
3. The go.mod file
4. The module zip itself

Creating a simple local proxy

To try out the proxy support, let’s create a very basic proxy that will serve static files from a directory. First we create a directory where we will store our in-site copies of our dependencies. Here’s what I have in mine:

$ find . -type f
./github.com/robteix/testmod/@v/v1.0.0.mod
./github.com/robteix/testmod/@v/v1.0.1.mod
./github.com/robteix/testmod/@v/v1.0.1.zip
./github.com/robteix/testmod/@v/v1.0.0.zip
./github.com/robteix/testmod/@v/v1.0.0.info
./github.com/robteix/testmod/@v/v1.0.1.info
./github.com/robteix/testmod/@v/list

These are the files our proxy will serve. You can find these files on Github if you’d like to play along. For the examples below, let’s assume we have a devel directory under our home directory; adapt accordingly.

$ cd $HOME/devel
$ git clone https://github.com/robteix/go-proxy-blog.git

Our proxy server is simple (it could be even simpler, but I wanted to log the requests):

package main

import (
    "flag"
    "log"
    "net/http"
)

func main() {
    addr := flag.String("http", ":8080", "address to bind to")
    flag.Parse()

    dir := "."
    if flag.NArg() > 0 {
        dir = flag.Arg(0)
    }

    log.Printf("Serving files from %s on %s\n", dir, *addr)

    h := handler{http.FileServer(http.Dir(dir))}

    panic(http.ListenAndServe(*addr, h))
}

type handler struct {
    h http.Handler
}

func (h handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    log.Println("New request:", r.URL.Path)
    h.h.ServeHTTP(w, r)
}

Now run the code above:

$ go run proxy.go -http :8080 $HOME/devel/go-proxy-blog
2018/08/29 14:14:31 Serving files from /home/robteix/devel/go-proxy-blog on :8080

$ curl http://localhost:8080/github.com/robteix/testmod/@v/list
v1.0.0
v1.0.1

Leave the proxy running and move to a new terminal. Now let’s create a new test program. we create a new directory $HOME/devel/test and create a file named test.go inside it with the following code:

package main

import (
    "github.com/robteix/testmod"
)

func main() {
    testmod.Hi("world")
}

And now, inside this directory, let’s enable Go modules:

$ go mod init test

And we set the GOPROXY variable:

export GOPROXY=http://localhost:8080

Now let’s try building our new program:

$ go build
go: finding github.com/robteix/testmod v1.0.1
go: downloading github.com/robteix/testmod v1.0.1

And if you check the output from our proxy:

2018/08/29 14:56:14 New request: /github.com/robteix/testmod/@v/list
2018/08/29 14:56:14 New request: /github.com/robteix/testmod/@v/v1.0.1.info
2018/08/29 14:56:14 New request: /github.com/robteix/testmod/@v/v1.0.1.mod
2018/08/29 14:56:14 New request: /github.com/robteix/testmod/@v/v1.0.1.zip

So as long as GOPROXY is set, go will only download files from out proxy. If I go ahead and delete the repository from Github, things will continue to work.

Using a local directory

It is interesting to note that we don’t even need our proxy.go at all. We can set GOPROXY to point to a directory in the filesystem and things will still work as expected:

export GOPROXY=file://home/robteix/devel/go-proxy-blog

If we do a go build now², we’ll see exactly the same thing as with the proxy:

$ go build
go: finding github.com/robteix/testmod v1.0.1
go: downloading github.com/robteix/testmod v1.0.1

Of course, in real life, we probably will prefer to have a company/team proxy server where our dependencies are stored, because a local directory is not really much different from the local cache that go already maintains under $GOPATH/pkg/mod, but still, nice to know that it works.

There is a project called Athens that is building a proxy and that aims — if I don’t misunderstand it — to create a central repository of packages à la npm.