Httpwebrequest Is Extremely Slow!

HttpWebRequest is extremely slow!

What I have found to be the main culprit with slow web requests is the proxy property. If you set this property to null before you call the GetResponse method the query will skip the proxy autodetect step:

request.Proxy = null;
using (var response = (HttpWebResponse)request.GetResponse())
{
}

The proxy autodetect was taking up to 7 seconds to query before returning the response. It is a little annoying that this property is set on by default for the HttpWebRequest object.

HttpWebRequest is slow with chunked data

This looks like a typical case of the Nagle algorithm clashing with TCP Delayed Acknowledgement. In your case you are sending a small Http Request (~170 bytes according to your numbers). This is likely less than the MSS (Maximum Segment Size) meaning that the Nagle Algorithm will kick in. The server is probably delaying the ACK resulting in a delay of up to 500 ms. See links for details.

You can disable Nagle via ServicePointManager.UseNagleAlgorithm = false (before issuing the first request), see MSDN.

Also see Nagle’s Algorithm is Not Friendly towards Small Requests for a detailed discussion including a Wireshark analysis.

Note: In your answer you are running into the same situation when you do write-write-read. When you switch to write-read you overcome this problem. However I do not believe you can instruct the HttpWebRequest (or HttpClient for that matter) to send small requests as a single TCP write operation. That would probably be a good optimization in some cases. Althought it may lead to some additional array copying, affecting performance negatively.

Why is this WebRequest code slow?

Probably Firefox issues multiple requests at once whereas your code does them one by one. Perhaps adding threads will speed up your program.

Super slow HttpWebRequest

I just ran the following code and ignoring the first initial compile I average about 3.3 seconds for GetResponse() and 0.2 more seconds for Load(). Are you on a fast connection? Are you sure this is where the bottleneck is?

Option Explicit On
Option Strict On

Imports System.Net

Public Class Form1

    Private Const REQUESTURL As String = "http://www.MCServerList.net/?page="
    Private chunkId As Int32 = 1

    Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load
        Dim ST As New System.Diagnostics.Stopwatch()
        ST.Start()
        Dim req = WebRequest.Create(REQUESTURL & chunkId)

        Dim res = req.GetResponse()
        Trace.WriteLine(String.Format("GetResponse() : {0}", ST.Elapsed))
        Using Stream As System.IO.Stream = res.GetResponseStream()
            Dim X As New HtmlAgilityPack.HtmlDocument()
            X.Load(Stream)
        End Using
        Trace.WriteLine(String.Format("Load() : {0}", ST.Elapsed))
        ST.Stop()
    End Sub
End Class

---

Related Topics