Where an HTTP timeout actually fires

Thu, 30 Oct 2014 18:50:00 +0000

I added a dial timeout to an HTTP client and assumed I had bounded the entire request. Then a server accepted the connection, sent headers, and dribbled the body slowly enough to occupy a worker for minutes. The timeout had completed its duty before the interesting failure began.

In Go 1.3, the first choice for a simple end-to-end limit is http.Client.Timeout:

client := &http.Client{Timeout: 5 * time.Second}

It covers connection setup, redirects, and reading the response body. If the policy really is “this request gets five seconds,” start there. My narrower transport setting looked like this:

tr := &http.Transport{
	Dial: (&net.Dialer{
		Timeout: 2 * time.Second,
	}).Dial,
}
client := &http.Client{Transport: tr}

The dial timeout covers establishing the network connection. It does not cover waiting for response headers, reading a response body, or the time spent following redirects. DNS resolution may consume part of dialing behavior depending on the platform and resolver path, but the timeout is still not an end-to-end request deadline.

To see the phases, I wrote a deliberately bad server:

func slow(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "text/plain")
	w.WriteHeader(http.StatusOK)
	if f, ok := w.(http.Flusher); ok {
		f.Flush()
	}
	time.Sleep(10 * time.Second)
	fmt.Fprintln(w, "eventually")
}

The client connected immediately and received headers immediately. Its Do call returned a response, then reading resp.Body waited. A connect timeout could not help because the connection was exemplary.

The transport exposes other phase-specific controls. ResponseHeaderTimeout bounds the wait for response headers after the request is written. TLSHandshakeTimeout bounds the TLS handshake. Neither bounds a slowly arriving body. These knobs are useful because failures differ: a slow handshake may indicate different capacity trouble than a handler that never writes headers.

For policies that Client.Timeout cannot express, I can arrange cancellation through the transport’s request cancellation support and a timer. Merely returning from a selecting caller while leaving Do running leaks work: timeout handling must interrupt the network operation, not only stop waiting for its answer.

For direct net.Conn code, deadlines are clearer:

if err := conn.SetDeadline(time.Now().Add(5 * time.Second)); err != nil {
	return err
}

The deadline applies to future reads and writes and causes them to fail after the time. SetReadDeadline and SetWriteDeadline split those directions. A deadline is an absolute time, not an idle timeout that automatically moves after every successful byte. For an idle protocol I refresh it after each accepted unit of progress.

HTTP complicates direct connection deadlines because http.Transport owns and reuses pooled connections. A per-request caller should not reach underneath and set arbitrary deadlines on a connection that may later serve another request. Transport-level controls and cancellation preserve that ownership boundary.

As with the earlier runtime-managed network mechanism, a goroutine can wait without occupying a thread. Cheap waiting is not a timeout policy, though. In-flight requests, response body size, queue length, and elapsed time still need deliberate limits.

Servers have the same issue from the opposite side. A handler can be tied up by a client that sends or receives slowly. Depending on the Go version and server configuration, read and write timeouts establish connection-level deadlines around phases of serving a request. They are blunt instruments, especially for streaming responses, but leaving every deadline at zero means “forever.”

I tested the repaired client with three bad endpoints: one that never completed a TCP connection, one that accepted but withheld headers, and one that streamed a byte periodically. Each exercises a different phase. The output made the distinction visible:

dial stall: timeout awaiting connection
header stall: timeout awaiting response headers
body stall: request deadline reached while reading body

Error strings vary and should not become program logic. I care that each operation terminates, closes its body or cancellation path, and leaves no steadily growing goroutine count.

The temptation is to choose one five-second number and call the system safe. Real policies differ. A metadata request might have a strict total deadline. A large download may allow minutes overall but reject 30 seconds without progress. A streaming endpoint may be intentionally unbounded while still needing heartbeat and cancellation rules.

Timeout errors should remain distinguishable from protocol failures. I check whether a returned network error reports Timeout() rather than matching its text, then include the operation and peer in the higher-level error. Retrying blindly is still wrong: a timed-out request may have reached the server, so non-idempotent operations need application-level rules.

“The HTTP timeout” is not a very useful phrase. There are limits for connecting, handshaking, receiving headers, reading bodies, remaining idle, and completing the whole operation. Naming the phase put the mechanism in the right place; before that, I had a very reliable timeout on the part that was not slow.

Draining HTTP response bodies

Tue, 04 Mar 2014 16:40:00 +0000

I had a polling client that became steadily slower. The server was fine, DNS was fine, and my diagnostic method of staring at it was producing limited returns.

The client checked only the status:

resp, err := http.Get(url)
if err != nil {
	return err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
	return fmt.Errorf("status: %s", resp.Status)
}

Closing the body is necessary, but for persistent HTTP connections it is not always sufficient. If I do not read the response body to EOF, the transport may be unable to reuse that TCP connection. My server returned a small explanatory body for an error status; I ignored it and paid for a new connection on the next poll.

For responses whose contents I deliberately discard, I now do this:

defer resp.Body.Close()
if _, err := io.Copy(ioutil.Discard, resp.Body); err != nil {
	return err
}

The transport owns a pool of persistent connections keyed by destination. Reading through EOF lets its body wrapper observe completion and return the connection to the idle pool. Close still matters because it releases resources on every exit path.

This is not permission to drain an unbounded hostile response. If I only need to preserve reuse for a known-small error body, I put a limit around it and accept that an oversized body may force the connection closed:

r := io.LimitReader(resp.Body, 4<<10)
_, err = io.Copy(ioutil.Discard, r)

The same ownership rule applies on success. The caller that receives *http.Response generally owns Body and must close it. A helper that consumes the body should close it itself and return decoded data instead. Splitting those responsibilities is how bodies remain open.

I verified reuse by wrapping Transport.Dial with a counter. Ten requests to the same server produced ten dials before the fix and one after it. That is a crude instrument, but better than inferring connection behavior from elapsed time. A server can close persistent connections itself, so one dial is not a universal expected value.

There is a related server-side rule: a handler should not assume unread request bytes are harmless. net/http often manages request bodies for handlers, but large or malformed input still needs explicit size limits. Connection reuse is valuable; reading arbitrary quantities merely to obtain it is not.

After draining the tiny responses, repeated requests settled onto the same few connections instead of continually dialing. It was not an exotic kernel problem. It was one unread paragraph of error text, patiently converting itself into sockets.

Http on Roberto Selbach

Where an HTTP timeout actually fires

Draining HTTP response bodies