Blog

I reproduced a build failure only after deleting my GOPATH. That sentence contains most of the problem.

My normal workspace had this shape:

$GOPATH/
    src/example.com/service
    src/github.com/some/dependency
    pkg/darwin_amd64
    bin

go get fetched the dependency’s current source into the expected import path. My machine had an older checkout that still provided a function the service used. A fresh machine fetched a newer revision where the function had changed. The import path named a repository location, not the exact source revision I had tested.

I briefly considered copying dependencies under a random directory in the service. In 2014 the go command does not define vendoring semantics that make such a copy transparently override the canonical import. Rewriting imports to local copies creates new package identities and a permanent maintenance chore.

For deployable applications, I record the repository revisions and make the build checkout those revisions into an isolated GOPATH. Some teams use scripts; others use emerging third-party dependency tools. None is yet the universal answer. Libraries have an even harder choice because pinning their transitive world can conflict with the application assembling the final program.

I also keep import paths canonical. Relative imports make a package depend on where it happens to sit and behave poorly outside a narrow local experiment.

The decisive experiment was a new empty workspace:

$ mkdir /tmp/service-build
$ GOPATH=/tmp/service-build go get example.com/service/cmd/server

That made every source checkout visible under one disposable root. I recorded git rev-parse HEAD for each dependency and reproduced the working set by checking out those revisions. The script fails if a revision cannot be fetched; silently falling back to a branch tip would restore the original uncertainty.

Import-path compatibility remains social rather than solver-enforced. If upstream makes an incompatible change at the same path, the go command cannot select “the old API” from the import statement. Forking under a new path changes package identity, so values from original and forked packages are distinct even when declarations match.

For every recorded revision I also keep its canonical repository URL. Import paths can redirect discovery, and repeatability requires knowing both what commit was selected and where the build expects to obtain it.

GOPATH solves workspace layout and import discovery well. It does not solve repeatable dependency selection. Saying that plainly avoids a lot of mystical advice about cleaning $GOPATH/pkg. My build was not stale; its input was unspecified.

A few days ago I posted about a new Go package called validator that we initially developed for our own internal use at project7.io but what fun is internal stuff, right? So we opensourced it.

Then Matthew Holt pointed me to an ongoing discussion
on validation happening on martini-contrib’s github. By the way, if you don’t know martini yet, go rectify that right now.

And today I learned about check, which takes a completely different
approach to data validation than the one we took, which by the way is totally fine and you should check it out.

Good to know we were not the only ones with the problem. Although I’m happy with how our package turned out, I kind of wish
I had found these other guys earlier. Could have shared some code or just ideas.

The thing about working in a startup under stealth mode is you can’t often talk about what you’re doing. Thankfully, from time to time, an opportunity appears that lets us at least share something tangential.

A large part of our project at project7.io involves receiving data from a client (generally in JSON) for processing. This involves unmarshaling the JSON into a struct, something that Go does very well. But then comes the boring part: checking that the information sent from the client is correct and complete before doing anything with it.

The boring life of validating stuff

if t.Username != "" && t.Age > 18 && t.Foo != nil && len(t.Bar.Baz) > 8 && ...

We had to do this often and for a large number of different structs and sometimes a struct gained a new field and we had to go back and see that it was being properly validated everywhere. It was so boring that we ended up writing something to make it easier for us. We’ve been using this for a while and now we decided to open source it in the hopes that it might be useful for others.

Package validator implements validation of variables. Initially we had implemented JSONschema but we don’t always deal with JSON, we also get data as XML and sometimes as form-encoded. So we changed our approach and went right to the struct definitions.

A struct definition with some validation rules attached

type T struct {
  A int    `validate:"nonzero"`
  B string `validate:"nonzero,max=10"`
  C struct {
      Ca int    `validate:"min=5,max=10"`
      Cb string `validate:"min=8, regexp:^[a-zA-Z]+"`
  }
}

This allowed us to attach validation rules right to the data structure definitions. Then instead of large, boring list of if statements, we were able to validate in single function call.

Validating an instance of a struct

if valid, _ := validator.Validate(t); !valid {
  // not valid, so return an http.StatusBadRequest of something
}

Multiple rules for different situations

We often use the same struct to deal with different data coming from the client. Sometimes we only care about one or two of the fields in one scenario, so we also supported multiple rules like, say

A struct with two different sets of rules

type T struct {
  A int    `foo:"nonzero" bar:"min=5,max=10"`
  B string `bar:"nonzero"`
}

In scenario foo, we need A to be non-zero, but we don’t care for what is in B. In the bar scenario, however, we need B to be non-zero and the value of A to be between 5 and 10, inclusive. To validate, we then make validator use a different tag name for each case

WithTag() FTW

t := T{A: 3}
validator.WithTag("foo").Validate(t) // valid
validator.WithTag("bar").Validate(t) // invalid

We can also change the tag by using SetTag with will then make the tag name persistent until changed again by another call to SetTag.

Please refer to http://godoc.org/gopkg.in/validator.v1 for a lot more documentation, use cases, and how to access the individual validation errors found for each field.

I split a package this week because its name had become meaningless. util contained HTTP parsing, identifiers, retry logic, and one function that formatted dates. Importing it told a reader only that I had eventually run out of nouns.

My first repair was worse: one package per file. That produced retry, identifier, httputil, and dateutil, each with one exported function and several awkward dependencies. Package boundaries are not decorative folders. They should group a coherent capability and hide its implementation.

The useful split placed request parsing beside the server code that owns its rules. Identifier generation became a small package because several unrelated commands use the same format. Date formatting stayed with the output package that defines that presentation.

I also removed exported names. This API:

type RetryManager struct {
	RetryCount int
	RetryDelay time.Duration
}

func NewRetryManager(count int, delay time.Duration) *RetryManager
func (r *RetryManager) ExecuteRetryOperation(f func() error) error

became:

func Retry(n int, delay time.Duration, f func() error) error

There was no durable manager state and only one operation. The type existed because objects felt respectable. The function is harder to misuse, easier to test, and gives me fewer compatibility promises.

Exported identifiers are the public surface, even before a package reaches version one. Renaming one breaks every importer. Adding a method to an interface also breaks implementations outside the package, so I prefer accepting the smallest interface at the point of use rather than publishing a large “service” interface beside a concrete type.

Package names deserve the same economy. Callers write the package qualifier, so httpclient.Client stutters while client.Client or a more specific package often reads naturally. I avoid common, base, and misc; they attract unrelated code through gravity rather than design.

The test after the reorganization was not just go test ./.... I opened several call sites and read them without the implementation. If id.New() and server.ParseRequest(r) communicated enough, the boundary was doing work. If understanding required knowing which former utility file held the function, I had merely rearranged shelves.

Errors are part of the surface too. Callers should not need to compare prose. When they must distinguish a condition, I expose a stable sentinel error or a small concrete error type and document it. Most internal errors remain application details rather than becoming a taxonomy the package must support forever.

Small packages are useful. Tiny packages are not automatically virtuous. I now split around ownership and stable concepts, not line count. That gives me APIs boring enough that I can return to the actual program.

I blamed encoding/json for an allocation spike in an API client. That accusation was directionally correct but not particularly useful. I was asking it to decode into interface{}, inspect maps, convert numbers, and then copy everything into structs. The package was faithfully performing the work I had requested twice.

Here was the convenient version:

var raw interface{}
if err := json.Unmarshal(body, &raw); err != nil {
	return err
}

m := raw.(map[string]interface{})
id := int64(m["id"].(float64))
name := m["name"].(string)

Without a concrete destination, JSON objects become map[string]interface{}, arrays become []interface{}, strings become strings, booleans become bools, null becomes nil, and numbers become float64. Every assertion is another opportunity to panic. Large integer identifiers can also lose precision when represented as floating point.

A concrete struct gives the decoder much better instructions:

type record struct {
	ID   int64  `json:"id"`
	Name string `json:"name"`
}

var r record
if err := json.Unmarshal(body, &r); err != nil {
	return err
}

The decoder scans the input, matches object keys to exported struct fields, allocates strings and nested values as needed, and uses reflection to assign converted values. Struct field metadata is cached internally, so it is not rediscovering every field from first principles on every object. Reflection still has a cost, but my intermediate tree was the larger mistake.

I measured a reduced case with a 40-element array. The exact numbers depend on machine and revision, but the shape was stable:

BenchmarkInterface-4    5000    356000 ns/op    60400 B/op    1012 allocs/op
BenchmarkStruct-4      10000    181000 ns/op    14500 B/op     126 allocs/op

The benchmark reset the destination each iteration and used testing.B. It was not a production trace, but it proved that decoding into a generic tree and translating it was expensive enough to stop doing.

There are a few details behind apparently simple field matching. A json tag overrides the field name. A tag value of - ignores the field. Embedded fields participate in selection, with conflicts resolved according to the package’s rules. Only exported fields are candidates. Case-insensitive matches are accepted, though I prefer exact wire names because ambiguity is not a feature I need.

Missing and null are another source of false confidence. If a field is absent, unmarshaling leaves the existing Go value alone. If I reuse a struct, stale data can survive:

r := record{ID: 99, Name: "old"}
json.Unmarshal([]byte(`{"name":"new"}`), &r)
fmt.Printf("%+v\n", r)

The output still contains ID:99. I decode each independent document into a fresh value unless merging is intentional. For optional scalar fields, a pointer can distinguish missing or null from a present zero only if that distinction is actually part of the application contract.

For streams, json.Decoder avoids reading the whole input before decoding:

dec := json.NewDecoder(resp.Body)
for {
	var r record
	if err := dec.Decode(&r); err == io.EOF {
		break
	} else if err != nil {
		return err
	}
	consume(r)
}

It buffers beyond the current value, so code must not assume the underlying reader sits exactly at a JSON boundary afterward. Decoder.UseNumber is useful when decoding unknown shapes and preserving number text; json.Number can then be parsed deliberately rather than silently becoming float64.

Custom UnmarshalJSON methods are the escape hatch for special wire forms. I use them sparingly. They hide work behind ordinary decoding and are easy to make recursive by calling json.Unmarshal into the same type. An alias type avoids that recursion, but a separate wire struct is often clearer.

Syntax errors carry an offset, which I include in diagnostics without returning the entire untrusted document. Type errors differ: valid JSON may contain a string where the destination expects an integer. Other fields may already have been assigned when decoding fails, so I treat any error as invalidating the whole destination rather than using a partially filled struct.

Unknown object fields are ignored by default. That helps forwards-compatible clients but hurts configuration, where a typo should fail. In this Go version I make strict configuration explicit by decoding an object into map[string]json.RawMessage, deleting recognized keys as I decode them, and rejecting leftovers. RawMessage also helps when one discriminator selects the concrete shape of a payload. I decode the small envelope first, then decode only its raw payload into the chosen type.

Decoder convenience does not impose resource limits. Before unmarshaling an HTTP body I bound what I read, and for a stream I bound the number and size of accepted records at the protocol level. A syntactically valid JSON array can still be an excellent memory exhaustion device.

The direct opinion I took from this exercise is simple: interface{} is not a schema. If I know the shape of a response, I write it down as Go types. This improves errors, numeric behavior, allocations, and the next reader’s chances. encoding/json was not slow because reflection is cursed. It was slow because I asked for a completely generic representation and then complained that it was completely generic.