Reading lines with bufio.Scanner

Tue, 16 Jul 2013 17:33:00 +0000

Every small text-processing program I write begins with the same lie: “I only need to read a file line by line.” Five minutes later I am handling buffering, end-of-file, and the final line without a newline.

Go 1.1 added bufio.Scanner, which packages that loop neatly:

scanner := bufio.NewScanner(r)
for scanner.Scan() {
	line := scanner.Text()
	fmt.Println(line)
}
if err := scanner.Err(); err != nil {
	return err
}

By default Scanner splits input into lines and removes the line ending. Scan advances to the next token. Text returns that token as a string; Bytes returns the token as bytes. After the loop, Err distinguishes a clean end-of-input from a read or tokenization error.

That final check is easy to omit because the loop looks complete without it. It is not. An I/O error halfway through a configuration file should not quietly turn the remaining configuration into an unusually convincing empty section.

Scanner is more general than lines. Split accepts a split function, and the package supplies functions for words, bytes, and runes. A split function receives available data and whether the underlying reader has reached its end. It reports how far to advance, which token to return, and any error. This supports simple lexical work without writing the buffering loop again.

There is an important limit: Scanner uses a maximum token size. It is intended for reasonably sized tokens, not arbitrary enormous records. A log format that permits a single multi-megabyte line may exceed the limit and stop scanning with an error. For such input, bufio.Reader and an explicit loop provide more control. Choosing Scanner means accepting its token-oriented contract, not merely admiring its shorter spelling.

Bytes also deserves care. The returned slice may refer to storage Scanner reuses on the next call to Scan. Process it before advancing, or copy it if it must survive. Text gives a string representation that is safer to retain, with the corresponding conversion and allocation considerations.

I also avoid mixing direct reads from the underlying reader with Scanner. Scanner buffers ahead, so the reader’s apparent position need not correspond to the end of the last token I processed. One owner for the stream is a much less surprising arrangement.

I replaced my hand-written line reader with Scanner and deleted more edge-case code than I added. That is a good trade when its token limits fit the data. When they do not, the lower-level reader remains available. Go’s I/O packages are at their best when the simple case is genuinely simple and the escape hatch is still visible.

Io on Roberto Selbach

Reading lines with bufio.Scanner