Draining HTTP response bodies
I had a polling client that became steadily slower. The server was fine, DNS was fine, and my diagnostic method of staring at it was producing limited returns.
The client checked only the status:
resp, err := http.Get(url)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("status: %s", resp.Status)
}
Closing the body is necessary, but for persistent HTTP connections it is not always sufficient. If I do not read the response body to EOF, the transport may be unable to reuse that TCP connection. My server returned a small explanatory body for an error status; I ignored it and paid for a new connection on the next poll.
For responses whose contents I deliberately discard, I now do this:
defer resp.Body.Close()
if _, err := io.Copy(ioutil.Discard, resp.Body); err != nil {
return err
}
The transport owns a pool of persistent connections keyed by destination. Reading through EOF lets its body wrapper observe completion and return the connection to the idle pool. Close still matters because it releases resources on every exit path.
This is not permission to drain an unbounded hostile response. If I only need to preserve reuse for a known-small error body, I put a limit around it and accept that an oversized body may force the connection closed:
r := io.LimitReader(resp.Body, 4<<10)
_, err = io.Copy(ioutil.Discard, r)
The same ownership rule applies on success. The caller that receives *http.Response generally owns Body and must close it. A helper that consumes the body should close it itself and return decoded data instead. Splitting those responsibilities is how bodies remain open.
I verified reuse by wrapping Transport.Dial with a counter. Ten requests to the same server produced ten dials before the fix and one after it. That is a crude instrument, but better than inferring connection behavior from elapsed time. A server can close persistent connections itself, so one dial is not a universal expected value.
There is a related server-side rule: a handler should not assume unread request bytes are harmless. net/http often manages request bodies for handlers, but large or malformed input still needs explicit size limits. Connection reuse is valuable; reading arbitrary quantities merely to obtain it is not.
After draining the tiny responses, repeated requests settled onto the same few connections instead of continually dialing. It was not an exotic kernel problem. It was one unread paragraph of error text, patiently converting itself into sockets.