Mass-Threaded HTTPWebRequests drains CPU

Vote count:

0

i want to run about 10.000 concurrent requests using .net HttpWebRequest, not all of them going to the same host and some of them go through a pool of proxies.

Im currently using Threads which works fine up to 1000 concurrent requests(equals around 3% in task manager), but when I scale up to 2000 or even 5000 concurrent requests, I get many Exceptions(see below) and 100% cpu load.

When I noticed that I thought its too much for the server, but running for example 2 instances with 1000 concurrent works also, so Im guessing its something with the connection management.

First of all: sample code of request(guessing its all fine, works fine until big scale)

public string HttpGet(string url)
{
    try {
        HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
        request.Timeout = 20000;
        request.CookieContainer = Cookie;
        request.AutomaticDecompression = DecompressionMethods.GZip;
        request.KeepAlive = true;
        request.Method = "GET";
        HttpWebResponse response = (HttpWebResponse)request.GetResponse();

        Stream dataStream = response.GetResponseStream();
        StreamReader reader = new StreamReader(dataStream);
        string tmp = reader.ReadToEnd();
        reader.Close();
        dataStream.Close();
        response.Close();

        return tmp;
    } catch (Exception ex) {
        return "";
    }
}

And Yeah of course I (think I ) also set ServicePointManager right:

ServicePointManager.DefaultConnectionLimit = 20 * 1000;
ServicePointManager.MaxServicePointIdleTime = 1000 * 60 * 20;//maybe this is an issue?
ServicePointManager.UseNagleAlgorithm = false;
ServicePointManager.Expect100Continue = false;

The strange thing is that Im getting Timeouts(no real timeout, its an timeout like it occurs when you have a low DefaultConnectionLimit and trying to make requests parallel) even when the application even doesnt use the half of the limit of connections.

So I decided to try using the ConnectionGroupName to give each thread an unique connection:

request.ConnectionGroupName = randomStringPerThreadBasis;

That increased at least the opened TCP connections, so I not only extended the .NET DefaultConnectionLimit, I also increased the available dynamic TCP ports under windows like this http://ift.tt/1AsP94F and this netsh int ipv4 set dynamicport tcp start=1025 num=50000

The strange thing again is that the app opens now so many connections, which arent needed, but still has a high cpu footage and still times out rarely(but much better than before, so its maybe something with the connection management?).

For completeness: Im also getting these exceptions somtimes(but not so often):

System.Net.WebException: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server. ---> System.IO.IOException:Unable to read data from the transport connection : An existing connection was forcibly closed by the remote host. ---> A connection attempt failed because the connected party did not properly
respond after a period of time, or established connection failed because 
connected host has failed to respond
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
   at System.Net.Security._SslStream.StartFrameHeader(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security._SslStream.StartReading(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security._SslStream.ProcessRead(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.TlsStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.Net.PooledStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.Net.Connection.SyncRead(HttpWebRequest request, Boolean userRetrievedStream, Boolean probeRead)
   at System.Net.HttpWebRequest.GetResponse()

So my question is how can I solve this? Did I sth wrong? Are there workarounds? Can I manage the connections on my own? Is there a library/language which just works for this case? etc.

asked 34 secs ago

Tearsdontfalls

358

Mass-Threaded HTTPWebRequests drains CPU