C# Httpwebrequest and JavaScript

C# httpwebrequest and javascript

If I correctly interpret your question, there is no simple solution for your problem.

You are scraping the HTML from a server and since your C# code is not a real web browser, it doesn't execute client scripts.

This way you can't access information which the HTML you fetch doesn't contain.

Edit: I don't know how complex these AJAX calls from the original web site are, but you could use Firebug or Fiddler for IE to see how the requests are made in order to call these AJAX calls in your C# application too. So you could add the pieces of information you'll need. But it's only a theoretical solution.

How to deal with JavaScript when fetching http Response in C# using HttpWebRequest?

I finally got the easy solution to my need. following is the link that I followed:
Link to tutorial

Following is the code that'll get the results:

First you'll need to import following:

using System.Drawing;
using OpenQA.Selenium;
using OpenQA.Selenium.PhantomJS;
using System.Text.RegularExpressions;
using System.IO;
using HtmlAgilityPack;

Now the code:

        var options = new PhantomJSOptions();
        var driver = new PhantomJSDriver(options);
        driver.Manage().Window.Size = new Size(1360, 728);
        var size = driver.Manage().Window.Size;

        driver.Navigate().GoToUrl("https://example.com/");
        string url = driver.Url;
        //the driver can now provide you with what you need (it will execute the script)
        //get the source of the page
        var source = driver.PageSource;
        //fully navigate the dom
        var pathElement1 = driver.FindElementByName("username");
        var pathElement2 = driver.FindElementByName("password");
        var pathElement3 = driver.FindElementByXPath("//button[@class='SubmitButton']");

        pathElement1.Clear();
        pathElement1.SendKeys("username");
        pathElement2.Clear();
        pathElement2.SendKeys("password");
        pathElement3.Click();

        //Now get the response after login button click
        source = driver.PageSource;

httpwebrequest and javascript

An HTTP Webrequest cannot execute client-end scripts. Javascript needs that the script be executed, page be loaded and the DOM be constructed for various operations to be performed.

The webrequests & responses just download the resource, and do not process or execute it in anyway.
You are better off using a WebBrowserControl in this case.

C# Pull a webpage with HTTPWebRequest and execute the javascript from the site

You could do this with the C++ http system which gives you a lot of control over the pieces of data that are receievd, but I guess that's not really an answer so...

Why not try putting webkit into your app, and running off the events from it. It has several events that can notify you when downloads are started and finished.

C# Basic web HttpWebRequest, does not support Javascript

HttpWebRequest just implements GET, you need full browser to execute JavaScript (and possibly need CSS files to as scripts may depend on them).

The built in approach is to use WebBrowser control to render pages, that grab innerHTML after you find that JavaScript rendering is done.

How can i get Equivalent method of HttpwebRequest in javascript

A cross domain example by using yql,

var url = 'xyz.com'; // website you want to scrape
var yql = 'http://query.yahooapis.com/v1/public/yql?q=' + encodeURIComponent('select * from html where url="' + url + '"') + '&format=json&callback=?';  
$.getJSON(yql,function(data){
    if (data.results[0]){  
        console.log(data = data.results[0].replace(/<script[^>]*>[\s\S]*?<\/script>/gi, ''));  // The scraped data (the whole webpage)
    }
});

C# httpwebrequest on javascript

I'm assuming you have a program that wants to manipulate the server "back end" for a web page by making the server think that someone pushed a button that POSTs, and sending the data that the web page would include with its POST.

The first tool you need is Microsoft Network Monitor 3.3, or another network packet tracing tool. Use this to look at the POST from the real web page. NetMon (at least) decomposes the packet into the HTTP pieces and headers, so you can easily see what's going on.

Now you will know what data the real POST is sending, and the URL to which it is sending the data (with any possible "query string" - which is unusual for a POST).

Next you need to write C# to create the same sort of POST to the same URL. It seems that you already know about HttpWebRequest/HttpWebResponse so I won't explain them in detail. You may have noticed in your NetMon trace that the Content-Type header was application/x-www-form-urlencoded. This is most often data from an HTML form which is URL-Encoded (like the name), so you need to URL-Encode your data before POSTing it, and you need to know the size of the encoded data for the Content-Length. HttpUtility.UrlEncode() is one method to use for this encoding.

Once you think you have it, try it and use NetMon to inspect your POST request and the response from the server. Keep going until you have duplicated what the mystery web page is doing.

C# Httpwebrequest detection

    private const string Url = "http://services.runescape.com/m=hiscore_oldschool/overall.ws";

    private static HttpWebRequest BuildWebRequest()
    {
        var request = WebRequest.Create(Url) as HttpWebRequest;

        request.Method = "POST";
        request.ContentType = "application/x-www-form-urlencoded";
        request.Timeout = 40000;
        request.ServicePoint.Expect100Continue = true;

        string body = "user1=Zezima&submit=Search";

        byte[] bytes = Encoding.Default.GetBytes(body);

        using (var requestStream = request.GetRequestStream())
        {
            requestStream.Write(bytes, 0, bytes.Length);
        }

        return request;

    }

    static void Main(string[] args)
    {
            try
            {
                HttpWebRequest request = BuildWebRequest();

                var response = request.GetResponse() as HttpWebResponse;
                var responseContent = new StreamReader(response.GetResponseStream()).ReadToEnd();
                Console.Write("Success - " + response.StatusCode);
            }
            catch (Exception e)
            {
                 Console.Write(e);
            }
    }

I can take the response from the website. It is not empty.

---

Related Topics