What a SQL pool can and cannot do

Thu, 03 Oct 2013 11:08:00 +0000

I fixed a database slowdown by adding an index, then briefly credited the Go connection pool. This was generous of me and inaccurate.

*sql.DB manages underlying driver connections and permits concurrent use. Left alone, it can open more connections as concurrent work arrives. That is convenient until the database reaches its own useful concurrency limit.

The development tree for the upcoming Go 1.2 release has added SetMaxOpenConns, which turns the pool into a concurrency bound. Code staying on Go 1.1 still needs to bound database work before it reaches *sql.DB; when 1.2 lands, the limit can live on the handle itself:

db.SetMaxOpenConns(20)

Once 20 connections are in use, another database operation waits for one. That can protect a server from a burst, but it moves queueing into the application. A lower database CPU graph paired with rising request latency may simply mean callers are waiting at the pool.

The limit also makes connection lifetime matter more. An open result set or transaction occupies one slot. Calling back through db while holding a transaction can deadlock when every slot is similarly occupied: each holder waits for a new connection, and none can finish to release one. Code under a transaction should use tx and keep that scope short.

SetMaxIdleConns answers a different question. Idle connections reduce setup cost but consume server resources; the open limit bounds active plus idle connections. Setting both to the same number can keep a warm pool, while a smaller idle limit lets unused connections go. The right values depend on server capacity and traffic, not the number of goroutines I can create.

I watch query latency at the database, total application latency around calls, server connection counts, and request concurrency. In this Go version there is no convenient pool statistics API, so the gap between server query time and time around the call is especially useful evidence of waiting.

A connection limit is backpressure, not acceleration. I set it from measurements of the real database, then test saturation rather than only an idle local server. It still cannot compensate for a missing index; that is asking the pipes to improve the water.

database/sql is not a database connection

Fri, 10 May 2013 10:31:00 +0000

I put sql.Open inside an HTTP handler because the function was named Open and I wanted to open the database. The code worked, which delayed the useful realization that *sql.DB is not one database connection.

sql.Open creates a database handle using a registered driver and its data source name. It may not contact the server immediately. The first operation can therefore be where a bad password, missing server, or malformed configuration finally becomes visible. When startup must verify the database, run a trivial query and handle its error.

The handle manages a pool of underlying connections. It is intended to be long-lived and shared by concurrent goroutines. Opening one per request creates needless pool machinery and closing each one prevents useful connection reuse.

The practical shape is:

db, err := sql.Open("driver-name", dataSource)
if err != nil {
	log.Fatal(err)
}
defer db.Close()

var one int
if err := db.QueryRow("select 1").Scan(&one); err != nil {
	log.Fatal(err)
}

The driver import and data source are database-specific. database/sql deliberately supplies common operations while drivers deal with wire protocols and server peculiarities. This boundary is useful, but it does not make SQL dialects identical or transactions portable by divine intervention.

Rows carry a connection resource until they are exhausted or closed. That makes this pattern important:

rows, err := db.Query("select name from people")
if err != nil {
	return err
}
defer rows.Close()

for rows.Next() {
	var name string
	if err := rows.Scan(&name); err != nil {
		return err
	}
}
return rows.Err()

Checking only the initial Query error misses errors encountered while iterating. Forgetting Close can keep the underlying connection unavailable to other work. A leak in a pooled abstraction tends to present itself as an unrelated request mysteriously waiting, which is a thoughtful way for a bug to meet new people.

Transactions bind operations to the transaction’s connection. Once Begin succeeds, calls that belong to the transaction must use tx.Query, tx.Exec, and friends, not the original db handle. Calling db.Exec in the middle may use another connection and therefore sit outside the transaction entirely.

Statements and transactions need closing too. I keep their lifetime in the smallest useful scope and arrange cleanup immediately after successful creation. This is less about tidiness than returning scarce server and connection resources on every error path.

I now create one *sql.DB for a configured database, verify it during startup when appropriate, pass it to code that needs it, close rows promptly, and inspect iteration errors. Boring database plumbing is good database plumbing.

Database on Roberto Selbach

What a SQL pool can and cannot do

database/sql is not a database connection