.net,performance,stream,streamreader,httpwebresponse , C# - StreamReader.ReadToEnd() is extremely slow


C# - StreamReader.ReadToEnd() is extremely slow

Question:

Tag: .net,performance,stream,streamreader,httpwebresponse

I'm making a Web Crawler and I just found out that one of my methods, GetHTML, is very slow because it uses a StreamReader to get a string of the HTML out of the HttpWebResponse object.

Here is the method:

static string GetHTML(string URL)
      {
           HttpWebRequest Request = (HttpWebRequest)WebRequest.Create(URL);
           Request.Proxy = null;
           HttpWebResponse Response = ((HttpWebResponse)Request.GetResponse());
           Stream RespStream = Response.GetResponseStream();
           return new StreamReader(RespStream).ReadToEnd(); // Very slow
      }

I made a test with Stopwatch and used this method on YouTube.

Time it takes to get an HTTP response: 500 MS

Time it takes to convert the HttpWebResponse object to a string: 550 MS

So the HTTP request is fine, it's just the ReadToEnd() that is so slow.

Is there any alternative to the ReadToEnd() method to get an HTML string from the response object? I tried using WebClient.DownloadString() method, but it's just a wrapper around HttpWebRequest that uses streams too.

EDIT: Tried it with Sockets and it's much faster:

static string SocketHTML(string URL)
      {
           string IP = Dns.GetHostAddresses(URL)[0].ToString();
           Socket s = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
           s.Connect(new IPEndPoint(IPAddress.Parse(IP), 80));
           s.Send(Encoding.ASCII.GetBytes("GET / HTTP/1.1\r\n\r\n"));
           List<byte> HTML = new List<byte>();
           int Bytes = 1;
           while (Bytes > 0)
           {
                byte[] Data = new byte[1024];
                Bytes = s.Receive(Data);
                foreach (byte b in Data) HTML.Add(b);
           }
           s.Close();
           return Encoding.ASCII.GetString(HTML.ToArray());
      }

The problem with using it with Sockets, though, is that it most of the time returns errors such as "Moved Permanently" or "Your browser sent a request that the server could not understand".


Answer:

When I call this method but return String.Empty instead of the ReadToEnd, the method takes about 500 MS.

All that says is that starting to get the response takes 500ms. Calling GetResponseStream doesn't consume all the data.

ReadToEnd will also be doing conversion from the binary data to text, but I doubt that's significant - I strongly suspect it's just waiting for the data to arrive over the network. To verify that, you should add logging to every aspect of your code and run Wireshark - you should then be able to see packet-by-packet when the data arrives, and correlate it with the logging.

As a side issue, you should definitely have a using statement for the response:

using (var response = ((HttpWebResponse)Request.GetResponse())
{
    // The stream will be disposed when the response is.
    return new StreamReader(response.GetResponseStream())
        .ReadToEnd();
}

If you don't dispose of the response, you'll tie up connections until the garbage collector finalizes them. That can lead to timeouts.


Related:


Web API - Set each thread with the HttpRequestMessage id?


c#,.net,multithreading,task-parallel-library,web-api
I have a web api coded in c#. The web api uses functionality which is shared with other in-house components. it depends on single threaded flows and uses thread local storage to store objects, and session information. Please don't say if it's good or bad, that's what I have to...

Fastest way to draw sprites in opengles 2.0 on android


android,performance,opengl-es,opengl-es-2.0
So Im trying to figure out how to draw a single textured quad many times. My issue is that since these are create and deleted and every one of them has a unique position and rotation. Im not sure a vbo is the best solution as I've heard modifying buffers...

Microsoft Band and WPF


.net,wpf,dll,microsoft-band,.net-core
I have downloaded the sample code for Band SDK for Windows. Then I added a new WPF project and grabbed the latest SDK through Nuget (Version 1.3.10518.1). Now under packages\Microsoft.Band.1.3.10518\lib, the following folders are contained: netcore451 portable-net45+win+wpa81 portable-win81+wpa81 wpa81 Now as I need .NET 4.5 support I have referenced the...

Should I pass all the state in every change from store to component?


javascript,performance,reactjs,reactjs-flux,flux
I am new on React and Flux and I am making an app that will have an input (like the name or the title), and below a list of n things. How should I handle the event listeners? I saw in the TODO example of flux (https://github.com/facebook/flux/blob/master/examples/flux-todomvc/) that they use...

Why when trying to upload a video file to youtube i'm getting InvalidCredentialsException?


c#,.net,youtube,youtube-api,google-client-login
This is my code in form1: using System; using System.Collections.Generic; using System.ComponentModel; using System.Data; using System.Drawing; using System.Linq; using System.Text; using System.Windows.Forms; using System.IO; using Google.GData.Client; using Google.GData.Extensions; using Google.GData.YouTube; using Google.GData.Extensions.MediaRss; using Google.YouTube; namespace YouTube_Manager { public partial class Form1 : Form { YouTubeRequestSettings settings; YouTubeRequest request; string username...

How to count the number of zero decimals in javascript?


javascript,performance
How do I get the number of zero decimals behind the comma (but not the total)? So to illustrate an example: 0.00001 > 4 0.000015 > 4 0.0000105 > 4 0.001 > 2 I am looking for methods that are efficient (meaning that they optimize the calculation time)....

How does the Take() method work in LINQ


c#,.net,linq,entity-framework
I have a question regarding LINQ's Take() Method. I have a somewhat large table I'm querying in my web app, and I only want to return say N number of rows from the table. I've read through the MSDN documentation, but I can't see where it states if Take() first...

QUERY IN C# ,where statement


c#,.net,select
How can I write this query on c# : SELECT DEPT_ID FROM PERSONNEL_TEMP.DEPARTMENT WHERE DEPARTMENT_NAME=combobox1.text; ...

Consuming and exposing webservices in one project (.NET)


.net,web-services,rest,soap
What is best practice concerning consuming and exposing webservices in one project? (.net) I need to create a rest webservice to expose data. The rest webservice would need to consume this data from another (SOAP) webservice from a third party. (The data needs to be merged with data present in...

performance of executing openjpa query


java,performance,jpa,openjpa
I have a rest web app with cxf 3.0.1 framework and openjpa 2.3.0. The web.xml: <web-app xmlns="http://java.sun.com/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"> <display-name>Mobile app rest services</display-name> <context-param> <param-name>contextConfigLocation</param-name> <param-value>WEB-INF/cxf-servlet.xml</param-value>...

SpecFlow Ambiguity in bindings


c#,.net,bdd,specflow
I'm working with Spec-flow for quite some days. I am facing "Multiple matching found.Navigating to first match", while Debugging this can be solved, but when I'm running the entire solution Its failing because of Ambiguity in bindings. I'm running around 4 C sharp class files in a Single project Feature:...

how can I add a column to IQueryable object and modify its values


c#,.net,linq,grid,devexpress
var packs = from r in new XPQuery<Roll>(session) select new { Number = r.number Selection = new bool() }; gcPack.DataSource = packs; I want to add another column to my grid control with: Selection = new bool(). It will be added to the grid but I can't change its...

Move windows form from a picturebox in C#


c#,.net,winforms,picturebox
I have a windows form without any border. So i added a picture box and i want the whole form to be moved when that picture box is clicked. public const int WM_NCLBUTTONDOWN = 0xA1; public const int HT_CAPTION = 0x2; [DllImportAttribute("user32.dll")] public static extern int SendMessage(IntPtr hWnd, int Msg,...

Formatting large numbers in C#


c#,.net,unity3d,formatting
I am using Unity to make an "Incremental Game" also known as an "Idle Game" and I am trying to format large numbers. For example, when gold gets to say 1000 or more, it will display as Gold: 1k instead of Gold: 1000. using UnityEngine; using System.Collections; public class Click...

How to send Ctrl+S through SendKeys.Send() method to save a file(save as dialog)


c#,.net,windows,sendkeys
I need to save a file which is in an External application using SendKeys.Send() method. The keys needed to be sent are Ctrl+S. I wrote the below code, but its not working: SendKeys.SendWait("^%s?"); // to get the Save As dialog Thread.Sleep(5000); SetForegroundWindow(FindWindow(null, "Save As")); Thread.Sleep(5000); SendKeys.SendWait("xyz"); // Sending FileName ...

Where to store an mp4 file in my project?


.net,vb.net,visual-studio,mp4
I have been studying vb.net for a couple of months now and I want to try something new. I have this sample project to try. It is a web browser and I want that if a button will be clicked, a video will play immediately. This is what my application...

External Datepicker not working


javascript,jquery,.net
I'm working with an external jss library that will let me use a datepicker that doesn't cut off the previous and next months dates. I'm getting an error when going through the coding. code $.fn.calendar = function (options) { var _this = this; var opts = $.extend({}, $.fn.calendar.defaults, options); var...

ASP.NET MVC posting list from view to controller


c#,.net,asp.net-mvc,razor
I have a view model in my ASP.NET MVC application: public class FiltersViewModel { public IEnumerable<SelectListItem> AvailableFilters { get; set; } // fills a drop down menu public IList<TechnologyFilter> TechnologyFilters { get; set; } public IList<ContractTypeFilter> ContractTypeFilters { get; set; } public FiltersViewModel() { this.TechnologyFilters = new List<TechnologyFilter>(); this.ContractTypeFilters =...

String Format: How to add any number of zeros before string


c#,.net
I'd like to have strings that have zeros before them. I use this code: string a = string.Format("{0:00}",7); The above code gives a two digits string, but what if I want a n digit string? Let's say n = 5, how can I use string.format to get something like 0004?...

Listing directories by content size using C# [closed]


c#,.net,windows,linq
I'm trying to list all folders of my c drive excluding the document folder which i do not seem to have access to. This first seemed rather simple to me but i found myself still struggling with it despite the seemingly rich .net library. I can't post any code as...

Related entity not loaded


c#,.net,entity-framework
I'm starting a new entity framework project and I'm defining relations among entities. I defined this simple entities: [Table("db_owner.Students")] public partial class User { [Key] public int Id { get; set; } [Required] [StringLength(50)] public string Name { get; set; } public int? SchoolClassId { get; set; } [ForeignKey("SchoolClassId")] public...

Synchronize local SQLite database with central database


android,performance,sqlite,compare
We are developing an Android App based on a product that already has a website. As a result, when launching the Android app it may happen that a lot of things differ between the local SQLite database and our central database. What is the fastest way to synchronize the Android...

Universal function for getting all unique pairs, trebles etc from an array in javascript


javascript,jquery,arrays,performance,underscore.js
I am looking to create a function in javascript, which would allow me to pass a long array, together with one argument. what I'm looking for is something like this: var ar = [1,2,3,4]; var pairs = superAwesomeFunction(ar,2) //=> [[1,2],[1,3],[1,4],[2,3],[2,4],[3,4]]; var trebles = superAwesomeFunction(ar,3) //=> [[1,2,3],[1,2,4],[1,3,4],[2,3,4] ideally, the function would...

Unconstrained type parameters casting


c#,.net,types,casting
My situation: interface ISomeInterface { void DoSmth<T>(T other); } class Base : ISomeInterface { public virtual void DoSmth<T>(T other){ // for example do nothing } } class Derived<T2> : Base { Action<T2> MyAction {get;set;} public override void DoSmth<T>(T other){ if(typeof(T2).IsAssignableFrom(typeof(T))) MyAction((T2) other); } } This gives me an error: Cannot...

Feature-specific compatibility of a .NET 4.5 application on a system with .NET 4.0


c#,.net,zipfile
I have a project which runs fine on 4.0, however in the interests of utilizing the new ZipFile class I have decided to target 4.5. The vast majority of my users will have 4.5+ installed, but its not impossible someone with only 4.0 may come along and try to use...

Dependency Injection for concrete .Net classes


c#,.net,dependency-injection,ninject
What is the preferred way for injecting/isolating classes that are sealed in dlls and do not implement interfaces? We use Ninject. Let's say that we have a class "Server" and we want to inject/isolate the class TcpServer which "Server" uses. Don't want to be too specific because I want to...

Custom drawing using System.Windows.Forms.BorderStyle?


c#,.net,vb.net,winforms,custom-controls
I want to mimick drawing of default border based on value of property BorderStyle. Instead of single border around the control, my control is visualised as four adjacent custom-drawn boxes (2×2), each having standard border drawn individually. So for example, if Control.Border is set to FixedSingle value I want to...

How does convert.ToString(C0) behave?


c#,.net
I have different scenarios. I need output where the value must return comma separated values in ₹ format which it does in my system where I have the ₹ rupee symbol. Whereas in the user system C0 returns $ value with comma separated values I do not know if he...

Catch concurrency exception in EF6 to change message to be more user friendly


c#,asp.net,.net,entity-framework,entity-framework-6
I am using EF6.1 and i would like to change the message to a more system specific message when the below exception is thrown. Store update, insert, or delete statement affected an unexpected number of rows (0) Now, my problem is i cannot seem to catch the exception? I have...

How can I determine if an object of anonymous type is empty?


c#,.net
I am sure the answer to this is quite simple but I am trying to write an if statement (C# 5.0) to determine whether or not an anonymous type is empty or not. Here is a simplified version of my code: public void DoSomething(object attributes) { // This is the...

Is Serialization the same as GetBytes?


.net
Java provides a mechanism, called object serialization where an object can be represented as a sequence of bytes. Found this on google. I use .Net to send packets using a socket connection. I use Encoding.Default.GetBytes(). Serialization is confusing, what is the difference of it to GetBytes?...

OpenXml.WordProcessing.Justification always comes as OpenXmlUnknownElement


.net,ms-office,openxml,openxml-sdk
I'm building a table using OpenXml and setting the Justification property for every cell. It doesn't really matter where I assign this property, either in the Run element or as a TableCellProperty, when the document is open using the Productivity Tool for MS Office the Justification element always comes as...

How to print iterations per second?


python,performance,loops,cmd,progress
I have a small Python script which sends POST requests to a server and gets their response. It iterates 10000 times, and I managed to print the current progress in command prompt using: code=current_requestnumber print('{0}/{1}'.format(str(code),"10000"),end="\r") at the end of each loop. Because this involves interaction with a webserver, I would...

Dynamic creation of objects vs storing them as fields


java,performance,object
I have this loop which runs about 1,000,000 times. while ((line = br.readLine()) != null) { if(!(line.isEmpty())){ data = new JSONObject(line); name = (String)data.get("name"); surname = (String)data.get("surname"); ar.put(name, surname); //an array } } What are the advantages (if any) in terms of efficiency of having it like that were data,name...

Automapper AfterMap function initialising classes


.net,vb.net,automapper
Can anyone please explain how does the AfterMap function initalises the class objects passed in the lambda expression? When i run this bit of code as myself, it populates the dest object with my details. i can't figure out how is it doing that? but if someone can tell me...

inconsistent timing from .net StopWatch


c#,.net,timing,stopwatch,cudafy.net
I have a C# & .NET application that uses a GPU (NVIDA GTX980) to do image processing. There are 4 stages and I synch the CPU to the GPU (no overlap in time) to do timing. But the numbers do not add up. Launch() will do a async launch of...

Capturing group recursively inside non-capturing group?


.net,regex
So I've got the following input: 1,6-10,10000,2,10-11 And the following regex: ^\d+(?:,(\d+|\d+-\d+))*$ in the .NET Regex I want to match each of the groups between the commas but I also want the capturing groups to end up like the following: Group 1: 1 Group 2: 6-10 Group 3: 10000 Group...

I want to upload video on youtube using client side login. without open web page permission


c#,.net,google-api,youtube-api,google-api-dotnet-client
When I upload video to YouTube using client side login. The first time redirect to UI for permissions. I want upload without redirect to web page. I need to execute this code on server. Here is my code. private async Task<string> Run(string title, string description, string filepath) { var videoID...

Regex to remove `.` from a sub-string enclosed in square brackets


c#,.net,regex,string,replace
I have this regex in C#: \[.+?\] This regex extracts the sub-strings enclosed between square brackets. But before doing that I want to remove . inside these sub-strings. For example, the string hello,[how are yo.u?]There are [300.2] billion stars in [Milkyw.?ay]. should become hello,[how are you?]There are [3002] billion stars...

DialogBox with value verifications


c#,.net,winforms
For my application which deals with graphics, I've made a little DialogBox to set: Max; Min; Major Step (of the scale); Minor Step. Here's a screen capture: I want to validate a few things before allowing the user to click Ok: Max >= Min MaxScale >= MinScale. But it's not...

Angular ng-repeat cache (avoid re-rendering on state change)


javascript,angularjs,performance,caching,angularjs-ng-repeat
We have huge rendering spikes with ng-repeat in Angular application. Main page shows a huge list of cover images ("::" and "track by" are in place). On first load it works acceptable. But if user changes the state (we use UI-Router) and goes back to the home page afterwards then...

Unhandled exceptions and background workers


.net,exception-handling,backgroundworker
In our application we use several background workers which are executed using SERVICENAME__bgw.RunWorkerAsync() where SERVICENAME is one of several different background worker processes executed by different timers. We recently saw this error message that was shown after an unhandled error bubbled up to the top of the call stack: My...

XElement.Value is stripping XML tags from content


c#,.net,xml,xml-parsing,xelement
I have the following XML: <Message> <Identification>c387e36a-0d79-405a-745c-7fc3e1aa8160</Identification> <SerializedContent> {"Identification":"81d090ca-b913-4f15-854d-059055cc49ff","LogType":0,"LogContent":"{\"EntitiesChanges\":\" <audit> <username>acfc</username> <date>2015-06-04T15:15:34.7979485-03:00</date> <entities> <entity> <properties> <property> <name>DepId</name> <current>2</current> </property>...

Does MongoDB successful insert guarantee populated ID's?


c#,.net,mongodb-csharp,mongodb-csharp-2.0
We are using the MongoDB C# driver to insert a collection of records using InsertManyAsync. We currently have code that iterates the entire collection after InsertManyAsync returns, confirming that the ID's are populated. I would expect that the driver takes care of this already, but I have not found documentation...

Visual Studio Assembly force-installs Target Framework


c#,.net,visual-studio-2013,.net-framework-version
I have this Assembly targeted at .NET 3.5. The code will work on later versions as well, but I like this to work on Windows XP. I mean, .NET is backwards compatible, right? I can run apps for .NET 3.5 on Windows 8.1. However, when I run my own assembly,...

performance issues executing list of stored procedures


c#,multithreading,performance,loops
I'm having some performance issues when starting my windows service, the first round my lstSps is long (about 130 stored procedures). Is there anyway to speed this up (except for speeding the stored procedures up)? When the foreach is over and goes over to the second round it goes faster,...

Application is missing required files


c#,.net,windows,winforms,sharpdevelop
Pic of Error: http://s23.postimg.org/7uj6qcxtn/9708083373e57a9ec91e4296e302f88e.png Cannot Download the Application. The Application is missing required Files. Contact Application Vendor For Assistance. So I'm building a windows form application using SharpDevelop 5.2 and I'm trying to make a standalone/version someone else would be able to use on another machine. In sharpdevelop (and visual...

Foreign key not updating in entity frame work


c#,.net,entity-framework,foreign-keys
I have a class like public class Document { [Key] public int Id { get; set; } public User ModifiedBy { get; set; } public DateTime ModifiedDate { get; set; } } Here “User” is a class with some properties (all are primitive types ) Here is my mapping of...

MySQL - How can I know my query is tuned?


mysql,performance,explain
I'm creating a DB from scratch and I'm trying to create queries with performance in mind. For test purpose I filled my tables with test data. My query needs to join 3 tables: 2 with millions of rows and the third with hundred thousands rows. How can I know my...

Best approach to upgrade MVC3 web app to MVC5?


c#,.net,asp.net-mvc,asp.net-mvc-5
I have a web application which runs on Azure which is currently running MVC3/C#, EF6.1, .NET4.5. I would like to upgrade it to MVC5 to be: a) Current b) Get benefit of new features c) Get Performance gains. This is a part of a performance project, so hopefully there will...