Adjusting Httpwebrequest Connection Timeout in C#

Adjusting HttpWebRequest Connection Timeout in C#

I believe that the problem is that the WebRequest measures the time only after the request is actually made. If you submit multiple requests to the same address then the ServicePointManager will throttle your requests and only actually submit as many concurrent connections as the value of the corresponding ServicePoint.ConnectionLimit which by default gets the value from ServicePointManager.DefaultConnectionLimit. Application CLR host sets this to 2, ASP host to 10. So if you have a multithreaded application that submits multiple requests to the same host only two are actually placed on the wire, the rest are queued up.

I have not researched this to a conclusive evidence whether this is what really happens, but on a similar project I had things were horrible until I removed the ServicePoint limitation.

Another factor to consider is the DNS lookup time. Again, is my belief not backed by hard evidence, but I think the WebRequest does not count the DNS lookup time against the request timeout. DNS lookup time can show up as very big time factor on some deployments.

And yes, you must code your app around the WebRequest.BeginGetRequestStream (for POSTs with content) and WebRequest.BeginGetResponse (for GETs and POSTSs). Synchronous calls will not scale (I won't enter into details why, but that I do have hard evidence for). Anyway, the ServicePoint issue is orthogonal to this: the queueing behavior happens with async calls too.

How to set HttpWebRequest.Timeout for a large HTTP request in C#

There are two timeouts that plague us in processing a large file upload. HttpWebRequest.Timeout and HttpWebRequest.ReadWriteTimeout. We'll need to address both.

HttpWebRequest.ReadWriteTimeout

First, let's address HttpWebRequest.ReadWriteTimeout. We need to disable "write stream buffering".

httpRequest.AllowWriteStreamBuffering = false;

With this setting changed, your HttpWebRequest.ReadWriteTimeout values will magically work as you would like them to, you can leave them at the default value (5 minutes) or even decrease them. I use 60 seconds.

This problem comes about because, when uploading a large file, if the data is buffered by the .NET framework, your code will think the upload is finished when it's not, and call HttpWebRequest.GetResponse() too early, which will time out.

HttpWebRequest.Timeout

Now, let's address HttpWebRequest.Timeout.

Our second problem comes about because HttpWebRequest.Timeout applies to the entire upload. The upload process actually consists of three steps (here's a great article that was my primary reference):

1. A request to start the upload.
2. Writing of the bytes.
3. A request to finish the upload and get any response from the server.

If we have one timeout that applies to the whole upload, we need a large number to accomodate uploading large files, but we also face the problem that a legitimate timeout will take a long time to actually time out. This is not a good situation. Instead, we want a short time out (say 30 seconds) to apply to steps #1 and #3. We don't want an overall timeout on #2 at all, but we do want the upload to fail if bytes stop getting written for a period of time. Thankfully, we have already addressed #2 with HttpWebRequest.ReadWriteTimeout, we just need to fix the annoying behaviour of HttpWebRequest.Timeout. It turns out that the async versions of GetRequestStream and GetResponse do exactly what we need.

So you want this code:

public static class AsyncExtensions
{
    public static Task<T> WithTimeout<T>(this Task<T> task, TimeSpan timeout)
    {
        return Task.Factory.StartNew(() =>
        {
            var b = task.Wait((int)timeout.TotalMilliseconds);
            if (b) return task.Result;
            throw new WebException("The operation has timed out", WebExceptionStatus.Timeout);
        });
    }
}

And instead of calling GetRequestStream and GetResponse we'll call the async versions:

var uploadStream = httpRequest.GetRequestStreamAsync().WithTimeout(TimeSpan.FromSeconds(30)).Result;

And similarly for the response:

var response = (HttpWebResponse)httpRequest.GetResponseAsync().WithTimeout(TimeSpan.FromSeconds(30)).Result;

That's all you'll need. Now your uploads will be far more reliable. You might wrap the whole upload in a retry loop for added certainty.

c# HttpWebRequest Timeout setting to zero

Yes, it will timout immediately, you can test it yourself easily:

try
{
    WebRequest myWebRequest = WebRequest.Create("http://stackoverflow.com/questions/38340099/c-sharp-httpwebrequest-timeout-setting-to-zero");
    myWebRequest.Timeout = 0;
    WebResponse myWebResponse = myWebRequest.GetResponse();
}
catch (Exception ex)
{
    Console.WriteLine(ex.Message); // timeout exceeded
}

Why? I've asked myself the same. Maybe for testing purposes or if you need a default value that clearly doesn't work accidentially to ensure that the property will be set in any case.

Interestingly negative values aren't allowed apart from -1, since that is the value of System.Threading.Timeout.Infinite. Here's the source:

public override int Timeout {
    get {
        return _Timeout;
    }
    set {
        if (value<0 && value!=System.Threading.Timeout.Infinite) {
            throw new ArgumentOutOfRangeException("value", SR.GetString(SR.net_io_timeout_use_ge_zero));
        }
        if (_Timeout != value)
        {
            _Timeout = value;
            _TimerQueue = null;
        }
    }
}

How can I specify a connection-only timeout when executing web requests?

You can use a Timer to abort the request if the connection take too much time. Add an event when the time is elapsed. You can use something like this:

static WebRequest request;
private static void sendAndReceive()
{
    // The request with a big timeout for receiving large amout of data
    request = HttpWebRequest.Create("http://localhost:8081/index/");
    request.Timeout = 100000;

    // The connection timeout
    var ConnectionTimeoutTime = 100;
    Timer timer = new Timer(ConnectionTimeoutTime);
    timer.Elapsed += connectionTimeout;
    timer.Enabled = true;

    Console.WriteLine("Connecting...");
    try
    {
        using (var stream = request.GetRequestStream())
        {
            Console.WriteLine("Connection success !");
            timer.Enabled = false;

            /*
             *  Sending data ...
             */
            System.Threading.Thread.Sleep(1000000);
        }

        using (var response = (HttpWebResponse)request.GetResponse())
        {
            /*
             *  Receiving datas...
             */
        }
    }
    catch (WebException e)
    {
        if(e.Status==WebExceptionStatus.RequestCanceled) 
            Console.WriteLine("Connection canceled (timeout)");
        else if(e.Status==WebExceptionStatus.ConnectFailure)
            Console.WriteLine("Can't connect to server");
        else if(e.Status==WebExceptionStatus.Timeout)
            Console.WriteLine("Timeout");
        else
            Console.WriteLine("Error");
    }
}

static void connectionTimeout(object sender, System.Timers.ElapsedEventArgs e)
{
    Console.WriteLine("Connection failed...");
    Timer timer = (Timer)sender;
    timer.Enabled = false;

    request.Abort();
}

Times here are just for example, you have to adjust them to your needs.

Can't set HttpWebRequest timeout higher than 100 seconds when doing a POST?

I figured it out. This is an example of DRY coding coming back and biting me in the butt. The code above is a paraphrase of my real code, and as such the code above will work fine.

The issue was I was setting the Timeout value after I had already called proxyRequest.GetRequestStream() to add the POST data. Because I was setting both the Timeout and ReadWriteTimeout properties, the shortest timeout was winning. In the case of a POST request, even though the framework let me set the Timeout value AFTER a call to GetRequestStream, it ignored whatever value was set (and instead used the default 100 seconds even though inspecting the Timeout property after setting it showed it was set to what I expected). I wish setting the Timeout property worked the same as setting the ReadWriteTimeout property: If you attempt to set the ReadWriteTimeout property after you have called GetRequestStream, it throws an exception. If Timeout did the same thing, that would have saved me a TON of time. I should have caught this sooner, but I'll chalk it up to a learning experience.

So the moral of the story: Set all the timeout properties for your HttpWebRequest right when you create it.

HttpWebRequest timeout handling

You can look at WebException.Status. The WebExceptionStatus enum has a Timeout flag:

try
{
   using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
   {
      WebHeaderCollection headers = response.Headers;    
      using (Stream answer = response.GetResponseStream())
      {
          // Do stuff
      }
   }
}
catch (WebException e)
{
   if (e.Status == WebExceptionStatus.Timeout)
   {
      // Handle timeout exception
   }
   else throw;
}

Using C# 6 exception filters can come in handy here:

try
{
    var request = WebRequest.Create("http://www.google.com");
    using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
    {
        WebHeaderCollection headers = response.Headers;
        using (Stream answer = response.GetResponseStream())
        {
            // Do stuff
        }
    }
}
catch (WebException e) when (e.Status == WebExceptionStatus.Timeout)
{
    // If we got here, it was a timeout exception.
}

---

Related Topics