Programming

The thing about working in a startup under stealth mode is you can’t often talk about what you’re doing. Thankfully, from time to time, an opportunity appears that lets us at least share something tangential.

A large part of our project at project7.io involves receiving data from a client (generally in JSON) for processing. This involves unmarshaling the JSON into a struct, something that Go does very well. But then comes the boring part: checking that the information sent from the client is correct and complete before doing anything with it.

The boring life of validating stuff

if t.Username != "" && t.Age > 18 && t.Foo != nil && len(t.Bar.Baz) > 8 && ...

We had to do this often and for a large number of different structs and sometimes a struct gained a new field and we had to go back and see that it was being properly validated everywhere. It was so boring that we ended up writing something to make it easier for us. We’ve been using this for a while and now we decided to open source it in the hopes that it might be useful for others.

Package validator implements validation of variables. Initially we had implemented JSONschema but we don’t always deal with JSON, we also get data as XML and sometimes as form-encoded. So we changed our approach and went right to the struct definitions.

A struct definition with some validation rules attached

type T struct {
  A int    `validate:"nonzero"`
  B string `validate:"nonzero,max=10"`
  C struct {
      Ca int    `validate:"min=5,max=10"`
      Cb string `validate:"min=8, regexp:^[a-zA-Z]+"`
  }
}

This allowed us to attach validation rules right to the data structure definitions. Then instead of large, boring list of if statements, we were able to validate in single function call.

Validating an instance of a struct

if valid, _ := validator.Validate(t); !valid {
  // not valid, so return an http.StatusBadRequest of something
}

Multiple rules for different situations

We often use the same struct to deal with different data coming from the client. Sometimes we only care about one or two of the fields in one scenario, so we also supported multiple rules like, say

A struct with two different sets of rules

type T struct {
  A int    `foo:"nonzero" bar:"min=5,max=10"`
  B string `bar:"nonzero"`
}

In scenario foo, we need A to be non-zero, but we don’t care for what is in B. In the bar scenario, however, we need B to be non-zero and the value of A to be between 5 and 10, inclusive. To validate, we then make validator use a different tag name for each case

WithTag() FTW

t := T{A: 3}
validator.WithTag("foo").Validate(t) // valid
validator.WithTag("bar").Validate(t) // invalid

We can also change the tag by using SetTag with will then make the tag name persistent until changed again by another call to SetTag.

Please refer to http://godoc.org/gopkg.in/validator.v1 for a lot more documentation, use cases, and how to access the individual validation errors found for each field.

I upgraded a small service to Go 1.2 and immediately found a use for the new full slice expression. That sounds implausibly efficient for an upgrade, so naturally it followed a bug.

I had a helper that accepted a prefix of a work buffer and appended a marker:

func mark(p []byte) []byte {
	return append(p, '!')
}

buf := []byte("abcdef")
prefix := buf[:3]
marked := mark(prefix)
fmt.Printf("%s %s\n", marked, buf)

I expected a separate abc!. I got:

abc! abc!ef

The ordinary two-index slice preserves the unused capacity of the backing array. prefix had length 3 but capacity 6, so append reused the array and replaced d. The helper’s signature did not advertise permission to alter bytes beyond the visible slice, but capacity quietly granted exactly that permission.

Go 1.2 adds a third index:

prefix := buf[:3:3]

For a[low:high:max], the length is high-low and the capacity is max-low. Now the output is:

abc! abcdef

Because len(prefix) == cap(prefix), append must allocate another backing array. This is not an immutable slice. The helper can still change prefix[0], and anyone sharing the original array will see that change. The third index controls append’s room, nothing more.

It is also useful when dividing one large buffer among independent workers:

left := buf[0:3:3]
right := buf[3:6:6]

Appending to left can no longer wander into right. Before 1.2 I used append([]byte(nil), buf[:3]...), which copies eagerly and obscures why. The full slice expression is both cheaper and more precise when all I need is a capacity fence.

The indices must satisfy 0 <= low <= high <= max <= cap(a). The upper capacity bound is checked at run time just like an ordinary slice bound. I find it useful to print both dimensions when debugging ownership:

fmt.Printf("len=%d cap=%d\n", len(prefix), cap(prefix))

Reslicing later cannot recover the fenced-off capacity. The slice header handed to the helper records only a pointer, length, and reduced capacity; it carries no secret reference to my original wider view.

For an array rather than a slice, the omitted low index still defaults to zero. I write all three values at boundaries anyway; explicit fences are easier to review than clever shorthand.

My caveat is that I would not scatter three-index slices everywhere. Most slices should share capacity normally. I use the extra index at ownership boundaries, where an append by a callee would otherwise mutate storage it does not conceptually own.

I lost an hour today to an error value that was plainly nil and just as plainly refused to compare equal to nil.

The reduced version looked like this:

type problem struct{}

func (*problem) Error() string { return "problem" }

func check() error {
	var p *problem
	return p
}

fmt.Println(check() == nil) // false

My first theory involved compiler spite. The real explanation is less personal: an interface value contains a dynamic type and a dynamic value. check returns an interface containing *problem and a nil pointer. The pointer is nil, but the interface’s type word is not, so the interface itself is not nil.

Printing %#v made the situation clearer:

(*main.problem)(nil)

The fix was not reflection or a clever nil helper. It was to return a literal nil when there is no error:

func check() error {
	var p *problem
	if p == nil {
		return nil
	}
	return p
}

This matters most with error, because callers naturally write if err != nil. I now keep concrete pointer errors inside the function and only convert them to error on an actual failure. Interfaces are two-word descriptions, not magic boxes. Remembering that is cheaper than another hour of accusing the compiler.

I assumed GOMAXPROCS(1) meant my program would use one operating-system thread. A blocking system call corrected me without displaying much tact.

In Go 1.1’s scheduler, GOMAXPROCS controls Ps, the runtime resources required to execute Go code in parallel. It does not impose a hard limit on Ms, the operating-system threads. The runtime may need more threads because one can become stuck in a system call while runnable goroutines remain.

Suppose G1 is running on M1 with P1 and enters a blocking call the network poller cannot handle asynchronously. The runtime marks the transition and detaches P1. Another thread, M2, can acquire P1 and execute G2. There is still only one P executing Go code at a time, but two operating-system threads exist because M1 is inside the kernel.

When M1 returns, its goroutine must rejoin runnable work. If no P is immediately available, the runtime can queue the goroutine and park or reuse the thread. The M does not permanently own the P, which is precisely why useful execution can continue during the call.

Runtime-managed network operations often take a cheaper path. Non-blocking sockets are registered with the network poller, and the goroutine is parked while its M continues with other work. Ordinary blocking system calls and calls into C do not necessarily have that integration, so the thread itself may be occupied.

This matters when a program makes many concurrent calls that block unpredictably. The scheduler can preserve progress by creating threads, but threads are not free. They consume kernel resources and stack address space, and excessive creation adds scheduling overhead. Wrapping a slow C function in ten thousand goroutines does not convert the function into scalable asynchronous I/O. It converts optimism into threads.

A bounded worker group is appropriate when calling a blocking interface with limited capacity:

jobs := make(chan Job)
for i := 0; i < workers; i++ {
	go func() {
		for job := range jobs {
			runBlocking(job)
		}
	}()
}

The useful bound should come from measured service capacity, not a decorative constant selected because eight feels parallel. Backpressure at the job channel also makes overload visible instead of allowing an unbounded pile of blocked goroutines and threads.

The distinction among G, M, and P is practical here. Counting goroutines does not reveal thread use. Counting threads does not reveal Go parallelism. GOMAXPROCS says how many Ps may run Go code; it does not promise a process with exactly that many threads.

I now treat blocking foreign calls and unusual system calls as resources to measure and usually bound. The scheduler is very good at preventing one blocked thread from freezing unrelated Go work. It is not obliged to make an unlimited number of blocked threads inexpensive.

I fixed a database slowdown by adding an index, then briefly credited the Go connection pool. This was generous of me and inaccurate.

*sql.DB manages underlying driver connections and permits concurrent use. Left alone, it can open more connections as concurrent work arrives. That is convenient until the database reaches its own useful concurrency limit.

The development tree for the upcoming Go 1.2 release has added SetMaxOpenConns, which turns the pool into a concurrency bound. Code staying on Go 1.1 still needs to bound database work before it reaches *sql.DB; when 1.2 lands, the limit can live on the handle itself:

db.SetMaxOpenConns(20)

Once 20 connections are in use, another database operation waits for one. That can protect a server from a burst, but it moves queueing into the application. A lower database CPU graph paired with rising request latency may simply mean callers are waiting at the pool.

The limit also makes connection lifetime matter more. An open result set or transaction occupies one slot. Calling back through db while holding a transaction can deadlock when every slot is similarly occupied: each holder waits for a new connection, and none can finish to release one. Code under a transaction should use tx and keep that scope short.

SetMaxIdleConns answers a different question. Idle connections reduce setup cost but consume server resources; the open limit bounds active plus idle connections. Setting both to the same number can keep a warm pool, while a smaller idle limit lets unused connections go. The right values depend on server capacity and traffic, not the number of goroutines I can create.

I watch query latency at the database, total application latency around calls, server connection counts, and request concurrency. In this Go version there is no convenient pool statistics API, so the gap between server query time and time around the call is especially useful evidence of waiting.

A connection limit is backpressure, not acceleration. I set it from measurements of the real database, then test saturation rather than only an idle local server. It still cannot compensate for a missing index; that is asking the pipes to improve the water.