What json.Unmarshal really does

Tue, 25 Mar 2014 18:25:00 +0000

I blamed encoding/json for an allocation spike in an API client. That accusation was directionally correct but not particularly useful. I was asking it to decode into interface{}, inspect maps, convert numbers, and then copy everything into structs. The package was faithfully performing the work I had requested twice.

Here was the convenient version:

var raw interface{}
if err := json.Unmarshal(body, &raw); err != nil {
	return err
}

m := raw.(map[string]interface{})
id := int64(m["id"].(float64))
name := m["name"].(string)

Without a concrete destination, JSON objects become map[string]interface{}, arrays become []interface{}, strings become strings, booleans become bools, null becomes nil, and numbers become float64. Every assertion is another opportunity to panic. Large integer identifiers can also lose precision when represented as floating point.

A concrete struct gives the decoder much better instructions:

type record struct {
	ID int64 `json:"id"`
	Name string `json:"name"`
}

var r record
if err := json.Unmarshal(body, &r); err != nil {
	return err
}

The decoder scans the input, matches object keys to exported struct fields, allocates strings and nested values as needed, and uses reflection to assign converted values. Struct field metadata is cached internally, so it is not rediscovering every field from first principles on every object. Reflection still has a cost, but my intermediate tree was the larger mistake.

I measured a reduced case with a 40-element array. The exact numbers depend on machine and revision, but the shape was stable:

BenchmarkInterface-4 5000 356000 ns/op 60400 B/op 1012 allocs/op
BenchmarkStruct-4 10000 181000 ns/op 14500 B/op 126 allocs/op

The benchmark reset the destination each iteration and used testing.B. It was not a production trace, but it proved that decoding into a generic tree and translating it was expensive enough to stop doing.

There are a few details behind apparently simple field matching. A json tag overrides the field name. A tag value of - ignores the field. Embedded fields participate in selection, with conflicts resolved according to the package’s rules. Only exported fields are candidates. Case-insensitive matches are accepted, though I prefer exact wire names because ambiguity is not a feature I need.

Missing and null are another source of false confidence. If a field is absent, unmarshaling leaves the existing Go value alone. If I reuse a struct, stale data can survive:

r := record{ID: 99, Name: "old"}
json.Unmarshal([]byte(`{"name":"new"}`), &r)
fmt.Printf("%+v\n", r)

The output still contains ID:99. I decode each independent document into a fresh value unless merging is intentional. For optional scalar fields, a pointer can distinguish missing or null from a present zero only if that distinction is actually part of the application contract.

For streams, json.Decoder avoids reading the whole input before decoding:

dec := json.NewDecoder(resp.Body)
for {
	var r record
	if err := dec.Decode(&r); err == io.EOF {
		break
	} else if err != nil {
		return err
	}
	consume(r)
}

It buffers beyond the current value, so code must not assume the underlying reader sits exactly at a JSON boundary afterward. Decoder.UseNumber is useful when decoding unknown shapes and preserving number text; json.Number can then be parsed deliberately rather than silently becoming float64.

Custom UnmarshalJSON methods are the escape hatch for special wire forms. I use them sparingly. They hide work behind ordinary decoding and are easy to make recursive by calling json.Unmarshal into the same type. An alias type avoids that recursion, but a separate wire struct is often clearer.

Syntax errors carry an offset, which I include in diagnostics without returning the entire untrusted document. Type errors differ: valid JSON may contain a string where the destination expects an integer. Other fields may already have been assigned when decoding fails, so I treat any error as invalidating the whole destination rather than using a partially filled struct.

Unknown object fields are ignored by default. That helps forwards-compatible clients but hurts configuration, where a typo should fail. In this Go version I make strict configuration explicit by decoding an object into map[string]json.RawMessage, deleting recognized keys as I decode them, and rejecting leftovers. RawMessage also helps when one discriminator selects the concrete shape of a payload. I decode the small envelope first, then decode only its raw payload into the chosen type.

Decoder convenience does not impose resource limits. Before unmarshaling an HTTP body I bound what I read, and for a stream I bound the number and size of accepted records at the protocol level. A syntactically valid JSON array can still be an excellent memory exhaustion device.

The direct opinion I took from this exercise is simple: interface{} is not a schema. If I know the shape of a response, I write it down as Go types. This improves errors, numeric behavior, allocations, and the next reader’s chances. encoding/json was not slow because reflection is cursed. It was slow because I asked for a completely generic representation and then complained that it was completely generic.

Json on Roberto Selbach

What json.Unmarshal really does