gofmt is an AST program

Tue, 22 Jan 2013 13:48:00 +0000

I wanted to rename a field across a pile of Go files. My first impulse was a regular expression. My second, considerably healthier impulse was to remember that Go ships with a parser.

Source code is text, but edits usually concern syntax. The go/parser package reads source into nodes from go/ast. An identifier is an *ast.Ident; a function declaration is an *ast.FuncDecl; an expression has structure beyond where its parentheses happened to land. Once the source is a tree, a tool can ask precise questions instead of hoping a comment does not look executable.

Here is a tiny program that lists function names:

package main

import (
	"fmt"
	"go/ast"
	"go/parser"
	"go/token"
)

func main() {
	fset := token.NewFileSet()
	f, err := parser.ParseFile(fset, "sample.go", nil, 0)
	if err != nil {
		panic(err)
	}
	ast.Inspect(f, func(n ast.Node) bool {
		if fn, ok := n.(*ast.FuncDecl); ok {
			fmt.Println(fn.Name.Name)
		}
		return true
	})
}

token.FileSet connects tree positions to files, lines, and columns. It looks like administrative furniture until an error must say exactly where it found something. Then it becomes the only chair in the room.

After changing a tree, go/printer can write Go source in the standard layout. This is the important lesson of gofmt: parse syntax, then print it consistently. It is not lining up text with an especially ambitious set of regular expressions.

Comments make transformations subtler. Parsing with parser.ParseComments retains comment groups, but moving nodes while preserving the intended comment association still requires care. Imports are another edge. A rename may change whether an import is used, and blindly printing the tree does not make semantic mistakes disappear.

There is also a limit to what the AST alone knows. The syntax tree can tell me that x.Name is a selector expression. It cannot, by itself, tell me which package or type supplied Name. That needs type information. For a local mechanical transformation with an unambiguous syntactic shape, the AST is often enough. For a semantic rename, it may not be.

My field change ended up being a dozen lines around ast.Inspect, followed by formatting and a review of the diff. That was more code than a one-line substitution and much less code than repairing a one-line substitution after it renamed examples, comments, and an unrelated field with the same spelling.

I used to think gofmt was mainly a successful ceasefire in the whitespace wars. It also shows the value of shipping language tools as ordinary libraries: the compiler is not the only program allowed to understand Go source.

Ast on Roberto Selbach

gofmt is an AST program