How to Transfer Authentication from Webbrowser to Webrequest

How to share cookies between WebBrowser control and CookieAwareWebClient?

After much hack and google, I'm going to conclude it's a myth you can make WebClient "cookie aware." I never could make it work, and almost all the threads about it that I read concluded with no solution. And anyway, WebClient is apparently deprecated.

To recap, the mission was to automate the login and download of files from a low-security website that uses forms authentication. The WebBrowser control would have worked fine, except that it uses IE, and IE refuses to download PDF files silently. It insists on prompting whether to open, save, or discard.

I started playing around with HTTPWebRequest, HTTPRequest, WebRequest, HTTPClient, and a bunch of variations, and got nowhere. Then it occurred to me to look for a Chrome-based WebBrowser control, and I stumbled across Selenium. That proved to be the solution for me.

Selenium's principal use appears to be to test software, but it also lets you manipulate web pages. You can easily install it within Visual Studio through NuGet. You also need to install a browser-specific driver. There are drivers for every major browser, but using the IE driver would be pointless because I would still have the problem of being prompted at every file. I instead downloaded the Chrome and Firefox drivers. They allow users here to choose between the two, and it's about 50/50.

Here's how simple the code was in the end.

Dim Options = New FirefoxOptions
Options.SetPreference("browser.download.folderList", 2)
'Options.SetPreference("browser.download.dir", "C:\\Windows\\temp")
Options.SetPreference("browser.download.useDownloadDir", True)
Options.SetPreference("browser.helperApps.neverAsk.saveToDisk", "application/octet-stream")
Options.SetPreference("pdfjs.disabled", True)
Dim driverService = FirefoxDriverService.CreateDefaultService()
driverService.HideCommandPromptWindow = True
Dim browser = New FirefoxDriver(driverService, Options)
browser.Url = "https://[the website]"
browser.Navigate()
Dim elm = browser.FindElementById("username")
elm.SendKeys([the username])
elm = browser.FindElementById("password")
elm.SendKeys([the password])
elm = browser.FindElementById("loginSubmit")
elm.Click()
While InStr(browser.Url, "token") = 0
    Application.DoEvents()
End While
Dim links As IList(Of IWebElement) = browser.FindElementsByPartialLinkText(".")
For Each link As IWebElement In links
    link.Click()
Next

I ran into a problem with the neverAsk.saveToDisk part. It just wasn't working. It turned out that I had the wrong mime type. I got the solution to that from this comment - Set Firefox profile to download files automatically using Selenium and Java

Using WebRequest cookie in WebBrowser

use cookie collection to grab cookies, I've write something similar this month and I can share you some sample code:

    static string GetFromServer(string URL, ref CookieCollection oCookie)
    {
        //first rquest
        // Create a request for the URL. 
        HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(URL);
        request.UserAgent = "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)";
        request.AllowAutoRedirect = false;
        // If required by the server, set the credentials.
        //request.Credentials = CredentialCache.DefaultCredentials;
        request.CookieContainer = new CookieContainer();
        if (oCookie != null)
        {
            foreach (Cookie cook in oCookie)
            {
                request.CookieContainer.Add(cook);
            }
        }

        // Get the response.
        HttpWebResponse response = (HttpWebResponse)request.GetResponse();
        foreach (Cookie cook in response.Cookies)
        {
            oCookie.Add(cook);
        }
        // Display the status.
        while (response.StatusCode == HttpStatusCode.Found)
        {
            response.Close();
            request = (HttpWebRequest)HttpWebRequest.Create(response.Headers["Location"]);
            request.UserAgent = "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)";
            request.AllowAutoRedirect = false;
            request.CookieContainer = new CookieContainer();
            if (oCookie != null)
            {
                foreach (Cookie cook in oCookie)
                {
                    request.CookieContainer.Add(cook);
                }
            }
            response = (HttpWebResponse)request.GetResponse();
            foreach (Cookie cook in response.Cookies)
            {
                oCookie.Add(cook);
            }
        }
        Console.WriteLine(((HttpWebResponse)response).StatusDescription);
        // Get the stream containing content returned by the server.
        Stream dataStream = response.GetResponseStream();
        // Open the stream using a StreamReader for easy access.
        StreamReader reader = new StreamReader(dataStream);
        // Read the content.
        string responseFromServer = reader.ReadToEnd();
        // Display the content.
        Console.WriteLine(responseFromServer);
        // Clean up the streams and the response.
        reader.Close();
        response.Close();
        return responseFromServer;
    }

Now you got cookies, and you just need to set it to the webBrowser control, import this method:

    [DllImport("wininet.dll", CharSet = CharSet.Auto, SetLastError = true)]
    public static extern bool InternetSetCookie(string lpszUrlName, string lbszCookieName, string lpszCookieData);

and call this after you got cookies:

    string cookie_string = string.Empty;
        foreach (Cookie cook in cookieCon)
        {
            cookie_string += cook.ToString() + ";";
            InternetSetCookie(url, cook.Name, cook.Value);
        }
        webBrowser1.Navigate(url, "", null, "Cookie: " + cookie_string + Environment.NewLine);

Please be aware of that this is just my test code and mainly was copied from msdn so it may buggy and you may need more exception handling

Load web browser with web response

 HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create("http://example.com");
 webRequest.Proxy = new WebProxy(host, port);

 HttpWebResponse response = (HttpWebResponse)webRequest.GetResponse();
 Stream receiveStream = response.GetResponseStream();

 WebBrowser webBrowser = new WebBrowser();
 webBrowser.DocumentStream = receiveStream;     

Login session not transferring to new webpage using WebRequest/Response?

I was doing some sort of cookie transfer for a website written with PHP.
Clearly you are passing the cookie, but maybe is like in that situation

var phpsessid = response.Headers["Set-Cookie"].Replace("; path=/", String.Empty);

The Set-Cookie header contains other related info about the cookie and possible other instructions for other cookies. I had one cookie with its info (Path), the session id which I needed to sent back to the server so the server would know that I am the same client which did the GET request.

The new request had to include this cookie

request.Headers["Cookie"] = phpsessid;

You already do this, but make sure the cookies you receive, you sent back to the server.

Considering session and authentication, there are two cookies, one for session, one for authentication and some servers/application might not allow to have an authentication without a valid session. What I want to say is that you might need to pass the session cookie too. So the steps would be:

• Do first a GET request to obtain the session cookie.
• Next do the POST request to authenticate and get the auth cookie.
• Use both cookies to navigate to the protected pages.

Also check this question, it doesn't show the entire class, but the idea is to keep the CookieContainer in the same class, add the new cookies from POST/GET requests and assign them to the each new request, like @Krizz answered.

WebRequest class to post data to login form

Take a look at my aspx sessions scraper on bitbucket. It does exactly what you are asking for, including some aspx webforms specific extensions, like sending postbacks etc.

HttpWebRequest and forms authentication in C#

It depends on how the website is authenticating users. If they are using basic authentication or Windows authentication, then you can set the Credentials property of the HttpWebRequest class to the username/password/domain information and it should work.

However, it sounds like you have to enter the username/password on the site, which means you are going to have to login to the site first. Looking at the main page, this is what I find in the <form> element that handles login:

<form name="formlogin" method="post" action="./defalogin.php" >
  <input name="emtext" type="text" id="emtext" size="12">
  <input name="pstext" type="password" id="pstext" size="12">
  <input type="submit" name="Submit" value="Logn in" 
    onClick="return logincheck()" >
</form>

I've included only the relevant portions.

Given this, you have to go to the ./defalogin.php page first with the HttpWebRequest and POST the emtext and pstext values. Also, make sure you set the CookieContainer property to an instance of CookieContainer. When that call to POST returns, it's more than likely going to be populated with a cookie which you will have to send back to the site. Just keep setting the CookieContainer property on any subsequent HttpWebRequest instances to that CookieContainer to make sure the cookies are passed around.

Then you would go to the page indicated in the link.

Of concern is also the logincheck javascript function, but looking at the script sources, it does nothing of note.

---

Related Topics