ASP.NET Tips #53 - Efficiently Streaming Large HTTP Responses With HttpClient

HttpClient in .NET has a default behaviour of buffering the response body when you make the call through GetAsync, PostAsync, PutAsync, etc. This is generally okay if you are dealing with small sized response bodies. However, if you wanted to download a large image and write it to a disk, you might end up consuming too much memory unnecessarily.

The following code, for example, will use up lots of memory if the response body is large:

static async Task HttpGetForLargeFileInWrongWay()
{
  using (HttpClient client = new HttpClient())
  {
    const string url = "https://github.com/tugberkugurlu/ASPNETWebAPISamples/archive/master.zip";
    using (HttpResponseMessage response = await client.GetAsync(url))
    using (Stream streamToReadFrom = await response.Content.ReadAsStreamAsync())
    {
      string fileToWriteTo = Path.GetTempFileName();
      using (Stream streamToWriteTo = File.Open(fileToWriteTo, FileMode.Create))
      {
        await streamToReadFrom.CopyToAsync(streamToWriteTo);
      }
      response.Content = null;
    }
  }
}

By calling GetAsync method directly, every single byte is loaded into memory.

A far better method is to only read the headers of the response and then get a handle for the network stream as below:

static async Task HttpGetForLargeFileInRightWay()
{
  using (HttpClient client = new HttpClient())
  {
    const string url = "https://github.com/tugberkugurlu/ASPNETWebAPISamples/archive/master.zip";
    using (HttpResponseMessage response = await client.GetAsync(url, HttpCompletionOption.ResponseHeadersRead))
    using (Stream streamToReadFrom = await response.Content.ReadAsStreamAsync())
    {
      string fileToWriteTo = Path.GetTempFileName();
      using (Stream streamToWriteTo = File.Open(fileToWriteTo, FileMode.Create))
      {
        await streamToReadFrom.CopyToAsync(streamToWriteTo);
      }
    }
  }
}

The HttpCompletionOption.ResponseHeadersRead enumeration value reads the headers and returns the control back rather than buffering the response. The CopyTo Async method then streams the content rather than downloading it all to memory.

For more information, see http://www.tugberkugurlu.com/archive/efficiently-streaming-large-http-responses-with-httpclient