Reflection on Roberto Selbach

What json.Unmarshal really does

Tue, 25 Mar 2014 18:25:00 +0000

I blamed encoding/json for an allocation spike in an API client. That accusation was directionally correct but not particularly useful. I was asking it to decode into interface{}, inspect maps, convert numbers, and then copy everything into structs. The package was faithfully performing the work I had requested twice.

Here was the convenient version:

var raw interface{}
if err := json.Unmarshal(body, &raw); err != nil {
	return err
}

m := raw.(map[string]interface{})
id := int64(m["id"].(float64))
name := m["name"].(string)

Without a concrete destination, JSON objects become map[string]interface{}, arrays become []interface{}, strings become strings, booleans become bools, null becomes nil, and numbers become float64. Every assertion is another opportunity to panic. Large integer identifiers can also lose precision when represented as floating point.

A concrete struct gives the decoder much better instructions:

type record struct {
	ID int64 `json:"id"`
	Name string `json:"name"`
}

var r record
if err := json.Unmarshal(body, &r); err != nil {
	return err
}

The decoder scans the input, matches object keys to exported struct fields, allocates strings and nested values as needed, and uses reflection to assign converted values. Struct field metadata is cached internally, so it is not rediscovering every field from first principles on every object. Reflection still has a cost, but my intermediate tree was the larger mistake.

I measured a reduced case with a 40-element array. The exact numbers depend on machine and revision, but the shape was stable:

BenchmarkInterface-4 5000 356000 ns/op 60400 B/op 1012 allocs/op
BenchmarkStruct-4 10000 181000 ns/op 14500 B/op 126 allocs/op

The benchmark reset the destination each iteration and used testing.B. It was not a production trace, but it proved that decoding into a generic tree and translating it was expensive enough to stop doing.

There are a few details behind apparently simple field matching. A json tag overrides the field name. A tag value of - ignores the field. Embedded fields participate in selection, with conflicts resolved according to the package’s rules. Only exported fields are candidates. Case-insensitive matches are accepted, though I prefer exact wire names because ambiguity is not a feature I need.

Missing and null are another source of false confidence. If a field is absent, unmarshaling leaves the existing Go value alone. If I reuse a struct, stale data can survive:

r := record{ID: 99, Name: "old"}
json.Unmarshal([]byte(`{"name":"new"}`), &r)
fmt.Printf("%+v\n", r)

The output still contains ID:99. I decode each independent document into a fresh value unless merging is intentional. For optional scalar fields, a pointer can distinguish missing or null from a present zero only if that distinction is actually part of the application contract.

For streams, json.Decoder avoids reading the whole input before decoding:

dec := json.NewDecoder(resp.Body)
for {
	var r record
	if err := dec.Decode(&r); err == io.EOF {
		break
	} else if err != nil {
		return err
	}
	consume(r)
}

It buffers beyond the current value, so code must not assume the underlying reader sits exactly at a JSON boundary afterward. Decoder.UseNumber is useful when decoding unknown shapes and preserving number text; json.Number can then be parsed deliberately rather than silently becoming float64.

Custom UnmarshalJSON methods are the escape hatch for special wire forms. I use them sparingly. They hide work behind ordinary decoding and are easy to make recursive by calling json.Unmarshal into the same type. An alias type avoids that recursion, but a separate wire struct is often clearer.

Syntax errors carry an offset, which I include in diagnostics without returning the entire untrusted document. Type errors differ: valid JSON may contain a string where the destination expects an integer. Other fields may already have been assigned when decoding fails, so I treat any error as invalidating the whole destination rather than using a partially filled struct.

Unknown object fields are ignored by default. That helps forwards-compatible clients but hurts configuration, where a typo should fail. In this Go version I make strict configuration explicit by decoding an object into map[string]json.RawMessage, deleting recognized keys as I decode them, and rejecting leftovers. RawMessage also helps when one discriminator selects the concrete shape of a payload. I decode the small envelope first, then decode only its raw payload into the chosen type.

Decoder convenience does not impose resource limits. Before unmarshaling an HTTP body I bound what I read, and for a stream I bound the number and size of accepted records at the protocol level. A syntactically valid JSON array can still be an excellent memory exhaustion device.

The direct opinion I took from this exercise is simple: interface{} is not a schema. If I know the shape of a response, I write it down as Go types. This improves errors, numeric behavior, allocations, and the next reader’s chances. encoding/json was not slow because reflection is cursed. It was slow because I asked for a completely generic representation and then complained that it was completely generic.

Validation without a reflection panic

Thu, 13 Feb 2014 13:50:00 +0000

I wrote a small validator that accepted interface{} and walked fields marked validate:"required". The happy path worked, so I naturally declared victory before trying a pointer, a nil pointer, or anything that was not a struct.

The first version began with this:

v := reflect.ValueOf(x)
for i := 0; i < v.NumField(); i++ {
	// inspect v.Field(i)
}

Passing &User{} produced reflect: call of reflect.Value.NumField on ptr Value. Recovering the panic would hide the real question: what inputs does this validator promise to accept? I chose structs and non-nil pointers to structs, with ordinary errors for everything else.

func validate(x interface{}) error {
	v := reflect.ValueOf(x)
	if !v.IsValid() {
		return errors.New("cannot validate nil")
	}
	if v.Kind() == reflect.Ptr {
		if v.IsNil() {
			return errors.New("cannot validate nil pointer")
		}
		v = v.Elem()
	}
	if v.Kind() != reflect.Struct {
		return fmt.Errorf("cannot validate %s", v.Kind())
	}

	t := v.Type()
	for i := 0; i < v.NumField(); i++ {
		if t.Field(i).Tag.Get("validate") != "required" {
			continue
		}
		if isZero(v.Field(i)) {
			return fmt.Errorf("%s is required", t.Field(i).Name)
		}
	}
	return nil
}

The harder policy is what “required” means. For this validator it rejects empty strings, numeric zero, false, and nil pointers, maps, slices, and interfaces. That makes false invalid, so a required boolean is usually a design smell; a pointer can represent the distinct states absent, false, and true.

I ignore unexported fields. A general validator should not turn another package’s private representation into public input policy.

Nested structs are not automatically validated. A field gets nested validation only when its tag asks for it; otherwise adding a private detail to a child type could unexpectedly change the parent’s contract. Nil optional pointers remain acceptable, while a required nil pointer fails at its own field path.

Errors report the Go field path as well as the failed rule. A bare “required” is compact but useless once validation descends into two nested structs with similarly named fields.

Reflection removes field-by-field plumbing, but the useful part is still the contract: accepted top-level shapes, the meaning of “required,” whether nesting is opt-in, and useful error paths. Once those choices are explicit, the reflective code can stay small and unsurprising.

Reading struct tags with reflection

Tue, 28 Jan 2014 17:05:00 +0000

I wanted one struct to describe both its JSON name and a tiny validation rule. The obvious declaration was:

type User struct {
	Name string `json:"name" validate:"required"`
}

My first parser split the raw tag on spaces and colons. It worked until quoting became interesting, which took approximately seven minutes. reflect.StructTag already understands the convention:

t := reflect.TypeOf(User{})
f, _ := t.FieldByName("Name")
fmt.Println(f.Tag.Get("json"))
fmt.Println(f.Tag.Get("validate"))

Output:

name
required

Tags are metadata attached to fields in the type. Reflection exposes them; the compiler does not enforce what validate:"required" means. Misspelling the key gives an empty string, indistinguishable from an explicitly empty value when using Get.

Only exported fields are useful to packages such as encoding/json, because reflection cannot generally set unexported fields from another package. I also avoid turning tags into a miniature programming language. Once a rule needs conditions, database access, or cross-field state, ordinary Go is clearer.

One more trap: reflect.TypeOf on a pointer yields a pointer type. To inspect its fields I use t.Elem(). Reflection is precise, but it is not especially interested in guessing what I meant.

Reflection is a polite form of unsafe

Tue, 12 Mar 2013 12:24:00 +0000

I needed to print arbitrary structs for a debugging tool. After writing the third type switch I reached for reflect, with the same cautious expression normally reserved for an unfamiliar electrical panel.

Reflection starts from interfaces. reflect.TypeOf(x) reports the dynamic type stored in the interface, while reflect.ValueOf(x) represents its dynamic value. This distinction is the same type/value pair that explains why an interface containing a nil pointer is not itself nil.

The first trap is that a reflect.Value has its own validity and kind. A nil interface produces an invalid Value. A typed nil pointer produces a valid Value whose kind is Ptr and for which IsNil is true. Calling methods that do not apply to a Value’s kind panics. Reflection replaces compile-time checks with a detailed list of ways to be wrong at runtime.

A modest struct printer looks like this:

func fields(x interface{}) {
	v := reflect.ValueOf(x)
	if v.Kind() == reflect.Ptr {
		v = v.Elem()
	}
	if !v.IsValid() || v.Kind() != reflect.Struct {
		return
	}

	t := v.Type()
	for i := 0; i < v.NumField(); i++ {
		fmt.Println(t.Field(i).Name, v.Field(i).Interface())
	}
}

Even this needs qualifications. Elem on a nil pointer yields an invalid Value. Access restrictions apply to unexported fields, and asking for an interface representation where it is not permitted can panic. Production code must decide whether to skip, report, or reject these cases rather than treating every struct as an unlocked cupboard.

Settable values are another common surprise. reflect.ValueOf(n) contains a copy and is not settable. To modify n, pass &n, call Elem, verify CanSet, and then use a setter appropriate to its kind. The pointer is not ritual; it gives reflection access to the original storage.

Reflection is invaluable at boundaries where types genuinely vary: encoders, decoders, formatters, and tools that inspect user-defined structures. It is much less convincing as a way to avoid writing an interface for ordinary application behaviour. A reflected method call loses static checking, is harder to read, and performs more runtime work than a direct call.

Calling reflection “unsafe” is unfair in one important sense. The package checks its rules and panics instead of letting arbitrary memory corruption proceed. It is polite. It knocks before ruining the evening.

For my debugging printer, reflection was exactly right. I kept it at the edge, converted the inspected data into a simpler representation, and let the rest of the program remain statically typed. That boundary is where I find reflection most useful: a small dynamic vestibule leading into an otherwise ordinary Go house.

Reflection needs a reason

Wed, 17 Aug 2011 17:05:00 +0000

I reached for reflection after writing the same diagnostic print twice. The obvious first experiment was to inspect one value:

func describe(x interface{}) {
	v := reflect.ValueOf(x)
	fmt.Println(v.Type(), v.Kind())
}

What surprised me was that type and kind answer different questions. A named integer type has its own type, but its kind is still integer. The type preserves identity; the kind tells reflective code which family of operations is available.

An interface value passed to describe carries a dynamic type and data. reflect.ValueOf exposes a view of that pair. From there, operations are checked at run time. Asking for an integer from a string value is not a clever conversion; it is a panic with good timing.

Settable values were the second lesson. Reflecting on a copied value lets me inspect it, not rewrite the caller’s variable. To change the original I must pass a pointer, obtain the pointed-to value, and ensure it is settable. Reflection follows the same addressability rules as ordinary Go, only with more opportunities to discover mistakes while running.

I prefer ordinary interfaces whenever the required behaviour can be named. They give compile-time checking and explain intent better than a tour through Kind. Reflection earns its place when the types themselves are the input, as in formatting or decoding.

The caveat is that a generic-looking helper can become a private, badly documented type system. My diagnostic printer stayed reflective. The business logic did not. One magician in the programme is plenty.

Reflection and a Tiny Config Decoder

Tue, 10 May 2011 17:22:00 +0000

My programs often begin with three configuration fields and end with a parser that believes it is a database. I tried using reflection to make one small key-value decoder reusable without building another private language.

The input is deliberately boring:

host=buildbox
workers=4
verbose=true

The destination is a structure. The decoder walks exported fields with the early reflect package, matches lowercased field names, converts the text according to each field’s kind, and sets the value. The exact calls have changed across weekly snapshots, so the important part is the control flow rather than names copied from my May 2011 tree:

parse lines
 -> find structure field
 -> inspect field kind
 -> convert text
 -> set addressable field or return os.Error

Passing a pointer is required because the decoder must modify the original structure. It checks that the argument points to a structure before examining fields. A non-pointer, unknown key, unsupported kind, or failed conversion returns an ordinary failure value with the key name.

I initially ignored unknown keys for forward compatibility. That converted a misspelled workres setting into a silent default of zero workers. Strict input is friendlier here: configuration is written by a person who would rather receive a precise complaint now than investigate strange behavior later.

The decoder supports strings, booleans, and signed integers. That is all this program needs. Slices, nested structures, aliases, default tags, and custom conversion hooks can wait until an actual configuration requires them. Reflection makes adding features easy in the same way an empty notebook makes writing a trilogy easy.

There is runtime cost, but startup configuration is not a hot path. The larger cost is weaker static checking inside the decoder, which I address with table tests over valid and invalid input. The application still uses an ordinary typed structure after startup.

I would not use this mechanism for a protocol or large data stream. Gob already solves typed Go-to-Go encoding, and explicit parsing is clearer for a public format. For a dozen local settings, a constrained reflective decoder removes repetitive assignment while keeping failures visible. The constraint is the feature, not an embarrassing first release.

Reflection With the Lights On

Tue, 05 Oct 2010 18:27:00 +0000

I wanted a diagnostic printer for configuration structures and reached for reflection. That is normally the moment a small helper begins applying for framework status, so I kept the experiment deliberately narrow.

An interface value carries a dynamic type and value. The reflect package exposes descriptions of those two parts. In my weekly snapshot the entry points and names differ from examples written against other snapshots, so I will not pretend this code is timeless:

v := reflect.NewValue(x)
t := reflect.Typeof(x)
fmt.Println(t.String(), v.Kind())

From there I can switch on kind, inspect structure fields, and recursively print supported values. The mechanism is not source-code introspection. It is runtime examination of type information retained for a value.

The first rule I learned is to separate validity, kind, and concrete type. A pointer is not its element. An interface can contain a pointer. A nil interface and an interface containing a nil pointer are different states. Reflection faithfully presents these distinctions, even when I would prefer it to edit my mistake.

The second rule is that settable values are special. Inspecting a copy obtained from an interface does not grant permission to modify the original. To change a caller’s value, reflection needs an addressable value, usually reached through a pointer and then its element. This is good friction. Generic mutation should look dangerous because it is.

I ended with a read-only printer supporting strings, integers, slices, and exported structure fields. Unsupported kinds produce an explicit marker. I did not add automatic conversion, tag syntax, or clever cycle handling because the tool does not need them.

Reflection trades compiler knowledge for runtime decisions. Misspelled field assumptions become execution-path problems, and every branch is harder to read than direct code. It is justified when the types truly vary, as in encoders and diagnostics. It is not justified merely to avoid writing three straightforward assignments.

My opinion after using it is positive but cautious. Go’s reflection model follows the interface representation and type system rather than inventing a parallel object universe. Still, if ordinary interfaces can express the job, they are clearer and checked earlier. Reflection is a sharp tool. Keeping the lights on means seeing both the blade and the fingers.