Home > Articles > Programming > C#

.NET Reference Guide

Hosted by

Toggle Open Guide Table of ContentsGuide Contents

Close Table of ContentsGuide Contents

Close Table of Contents

Reading the Response Stream Asynchronously

Last updated Mar 14, 2003.

The .NET documentation for BeginGetResponse has an elaborate example that shows how to make an asynchronous request and then read the data asynchronously in chunks. The example works fine except for one thing: there is no read timeout. There’s a request timeout, but there is no example of setting a timeout on the read.

The problem is that there is no documented way to timeout an asynchronous stream read. At least, the documentation for Stream.BeginRead doesn’t mention one. I tried implementing a timeout callback using ThreadPool.RegisterWaitForSingleObject, as in the asynchronous Web request, but closing the stream in the timeout callback didn’t have any effect. If there is a way to implement a timeout callback on an asynchronous read, I don’t know what it is.

However, that doesn’t prevent us from performing the read asynchronously. If you recall, the response callback is executing in a thread pool thread, so it’s already asynchronous. As a result, a synchronous read running in that thread will appear asynchronous to the main program. So all we have to do is find a way to make a synchronous read timeout.

As it turns out, there is a way to specify a timeout value for reading the response stream. The HttpWebRequest.ReadWriteTimeout property lets you set the number of milliseconds before a read or write times out. The "read" in this case refers to reading the stream returned by the HttpWebResponse.GetResponseStream method. The default timeout value is 300,000 milliseconds, or five minutes.

I think five minutes is an absurdly long time to wait for the data to come down. How much time you’re willing to wait will depend, of course, on how critical the information is to you. A Web crawler, for example, that is pulling down dozens or hundreds of pages per second, is not going to wait more than 15 or 30 seconds for a page. Even that is a very long time. On the other hand, if you’re just pulling down a few pages periodically, you can afford to wait a bit for a slow server response.

The other thing that determines how much time you’re willing to wait for a response is the size of the data you’re pulling down. A 25-kilobyte Web page will come down in a fraction of a second. But if you’re downloading an ISO image, you’ll need to set the ReadWriteTimeout value much higher. Or find a different way to do things.

The other problem you’ll run into when reading the data is that you don’t know how big the data is. Most Web servers will return the data size in the Content-Length header, but not all do. There are three cases to think about:

  1. The response header contains a Content-Length.
  2. The response header contains a Content-Length value that is too large.
  3. The response header contains no Content-Length value.

The first is easy: simply allocate a buffer of the given size and start the read.

The second case can be difficult. The definition of "too large" will depend on your specific circumstance, but in general it means that you lack either the memory or the time to download that much data.

The third case can be considered an extension of the second, except that the document is of some unknown and possibly infinite size.

You can handle both of those cases (the second and third above) in one of three ways:

  1. Discard any document that is too large.
  2. Download a portion of the document, up to some maximum size.
  3. Read the document in chunks and write it to disk.

In most cases I’ll choose the second option: reading up to a set number of bytes. It’s simple and covers most cases.

It only takes a few slight modifications to make the previous example download the data and make it available to the main program. Those changes are:

  • Add a data member to the RequestState class.
  • Set the ReadWriteTimeout property on the HttpWebRequest object instance.
  • Add code to check the Content-Length header and allocate the data buffer.
  • Read data from the stream.
  • Convert the retrieved data to a String that we can display.

Most of those changes are very simple, the only exception being the read and associated exception handler. Here’s the modified program that includes those changes:

[C#]

using System;
using System.IO;
using System.Net;
using System.Text;
using System.Threading;

namespace nc_cs
{
  class RequestState
  {
    public HttpWebRequest request = null;
    public HttpWebResponse response = null;
    public byte[] data = null;
  }

  class Program
  {
    static ManualResetEvent allDone = new ManualResetEvent(false);

    // Maximum size of document to retrieve
    private const int MaxDocumentSize = 256 * 1024 * 1024;

    // Document read timeout value
    private const int DocReadTimeout = 15000;

    static void TimeoutCallback(object state, bool timedOut)
    {
      if (timedOut)
      {
        HttpWebRequest request = (HttpWebRequest)state;
        if (request != null)
          request.Abort();
      }
    }

    static void Main(string[] args)
    {
      try
      {
        RequestState myRequestState = new RequestState();
        myRequestState.request = (HttpWebRequest)WebRequest.Create("http://www.informit.com/");

        // set read timeout value
        myRequestState.request.ReadWriteTimeout = DocReadTimeout;

        IAsyncResult ar = myRequestState.request.BeginGetResponse(new AsyncCallback(ResponseCallback), myRequestState);
        // Wait for request to complete
        ThreadPool.RegisterWaitForSingleObject(ar.AsyncWaitHandle,
          new WaitOrTimerCallback(TimeoutCallback),
          myRequestState.request, 60000, true);

        Console.WriteLine("Request completed.");
        // Wait for callback to signal
        allDone.WaitOne();
        Console.WriteLine("Response status code = {0}", myRequestState.response.StatusCode);
        myRequestState.response.Close();
        // Convert data to string and display
        if (myRequestState.data != null)
        {
          string s = Encoding.ASCII.GetString(myRequestState.data, 0, myRequestState.data.Length);
          Console.WriteLine(s);
        }
      }
      catch (WebException ex)
      {
        Console.WriteLine("Exception in Main:");
        Console.WriteLine("Message: {0}", ex.Message);
        Console.WriteLine("Status: {0}", ex.Status);
      }
    }

    static void ResponseCallback(IAsyncResult ar)
    {
      try
      {
        RequestState myRequestState = (RequestState)ar.AsyncState;
        myRequestState.response = (HttpWebResponse)myRequestState.request.EndGetResponse(ar);
        // Open the response stream
        Stream responseStream = myRequestState.response.GetResponseStream();

        // Determine how much data to read.
        // If ContentLength is -1, then no Content-Length header was specified
        // and we allocate the full size.
        int dataSize = (int)myRequestState.response.ContentLength;
        if (dataSize == -1 || dataSize > MaxDocumentSize)
          dataSize = MaxDocumentSize;

        // allocate the input buffer
        myRequestState.data = new byte[dataSize];
        try
        {
          // and read data from the stream
          int bytesRead = responseStream.Read(myRequestState.data, 0, dataSize);

          // if the number of bytes read is less than the size of the buffer,
          // then reallocate the buffer.
          if (bytesRead < dataSize)
          {
            Array.Resize(ref myRequestState.data, bytesRead);
          }
        }
        catch (IOException ioex)
        {
          Console.WriteLine("Exception reading data stream:");
          Console.WriteLine("Message: {0}", ioex.Message);
          myRequestState.data = null;
        }
        finally
        {
          responseStream.Close();
        }
      }
      catch (WebException ex)
      {
        Console.WriteLine("Exception in ResponseCallback:");
        Console.WriteLine("Message: {0}", ex.Message);
        Console.WriteLine("Status: {0}", ex.Status);
      }
      allDone.Set();
    }
  }
}

[Visual Basic]

Imports System.IO
Imports System.Net
Imports System.Threading
Imports System.Text

Public Class RequestState
  Public request As HttpWebRequest = Nothing
  Public response As HttpWebResponse = Nothing
  Public data As Byte() = Nothing
End Class

Public Class AsyncTest

  Shared allDone As ManualResetEvent = New ManualResetEvent(False)
  ’ Maximum size of document to retrieve
  Private Const MaxDocumentSize As Integer = 256 * 1024 * 1024

  ’ Document read timeout value
  Private Const DocReadTimeout As Integer = 15000

  Shared Sub TimeoutCallback(ByVal state As Object, ByVal timedOut As Boolean)
    If timedOut Then
      Dim request As HttpWebRequest = CType(state, HttpWebRequest)
      If Not (request Is Nothing) Then
        request.Abort()
      End If
    End If
  End Sub

  Shared Sub Main(ByVal args As String())
    Try
      Dim myRequestState As New RequestState()
      myRequestState.request = CType(WebRequest.Create("http://www.informit.com/"), HttpWebRequest)

      ’ set read timeout value
      myRequestState.request.ReadWriteTimeout = DocReadTimeout

      Dim ar As IAsyncResult = myRequestState.request.BeginGetResponse(New AsyncCallback(AddressOf ResponseCallback), myRequestState)
      ’ Wait for request to complete
      ThreadPool.RegisterWaitForSingleObject(ar.AsyncWaitHandle, _
        New WaitOrTimerCallback(AddressOf TimeoutCallback), myRequestState.request, 60000, True)
      ar.AsyncWaitHandle.WaitOne()
      Console.WriteLine("Request completed.")
      ’ Wait for callback to signal
      allDone.WaitOne()
      Console.WriteLine("Response status code = {0}", myRequestState.response.StatusCode)
      ’ Convert data to string and display
      If Not (myRequestState.data Is Nothing) Then
        Dim s As String = Encoding.ASCII.GetString(myRequestState.data, 0, myRequestState.data.Length)
        Console.WriteLine(s)
      End If

    Catch ex As WebException
      Console.WriteLine("Exception in Main:")
      Console.WriteLine("Message: {0}", ex.Message)
      Console.WriteLine("Status: {0}", ex.Status)
    End Try
  End Sub

  Shared Sub ResponseCallback(ByVal ar As IAsyncResult)
    Try
      Dim myRequestState As RequestState = CType(ar.AsyncState, RequestState)
      myRequestState.response = CType(myRequestState.request.EndGetResponse(ar), HttpWebResponse)
      ’ Open the response stream
      Dim responseStream As Stream = myRequestState.response.GetResponseStream()

      ’ Determine how much data to read.
      ’ If ContentLength is -1, then no Content-Length header was specified
      ’ and we allocate the full size.
      Dim dataSize As Integer = CType(myRequestState.response.ContentLength, Integer)
      If dataSize = -1 Or dataSize > MaxDocumentSize Then
        dataSize = MaxDocumentSize
      End If

      ’ allocate the input buffer
      ReDim myRequestState.data(dataSize - 1)
      Try
        ’ and read data from the stream
        Dim bytesRead As Integer = responseStream.Read(myRequestState.data, 0, dataSize)

        ’ if the number of bytes read is less than the size of the buffer,
        ’ then reallocate the buffer.
        If bytesRead < dataSize Then
          Array.Resize(myRequestState.data, bytesRead)
        End If
      Catch ioex As IOException
        Console.WriteLine("Exception reading data stream:")
        Console.WriteLine("Message: {0}", ioex.Message)
        myRequestState.data = Nothing
      Finally
        responseStream.Close()
      End Try
    Catch ex As WebException
      Console.WriteLine("Exception in Main:")
      Console.WriteLine("Message: {0}", ex.Message)
      Console.WriteLine("Status: {0}", ex.Status)
    End Try
      allDone.Set()
  End Sub
End Class

With this code, you can ensure that reading information from a Web site doesn’t take too much time, and you can also prevent your program from running out of memory if you run into an extremely large file.

If you do want to download very large files (hundreds of megabytes or more), you have to create a read loop that pulls the file down in chunks and writes each chunk to disk. Each read operation then has its own timeout value, and you’ll need to handle the timeout exception by aborting the read and potentially (depending on your application) deleting the partial data that you downloaded. Even in this case, though, you’ll likely want to set some maximum document size. Otherwise a streaming media file or a malicious Web server could have you downloading an essentially infinite-length document.