Jsoup Posting and Cookie

jsoup posting and cookie

When you login to the site, it is probably setting an authorised session cookie that needs to be sent on subsequent requests to maintain the session.

You can get the cookie like this:

Connection.Response res = Jsoup.connect("http://www.example.com/login.php")
    .data("username", "myUsername", "password", "myPassword")
    .method(Method.POST)
    .execute();

Document doc = res.parse();
String sessionId = res.cookie("SESSIONID"); // you will need to check what the right cookie name is

And then send it on the next request like:

Document doc2 = Jsoup.connect("http://www.example.com/otherPage")
    .cookie("SESSIONID", sessionId)
    .get();

Jsoup login problem (URL/cookie/session/POST/data)

Thanks for Frederic's comment.

I figured out the answer myself.

The point I was missing for this case is the persisting cookies.

In the below solution I

1. Create a map (call it session) to store this data
2. Update it every time I've sent a request, using putAll method
   
   session.putAll(resp.cookies());
3. Include this map in the next request
   
   .cookies(session)

Note that 2 and 3 must be repeated in every request thereafter.

Basically that's all!

public class JsoupTest {
    public static void main(String args[]) throws URISyntaxException {
        try {

            String urlLogIn = "https://invest.firstrade.com/cgi-bin/login";

            // Put the url that you see when you have logged in.
            String urlUnderTest = "https://invest.firstrade.com/cgi-bin/main#/cgi-bin/acctpositions"; 

            // Create a "session" map here to persists cookies across all requests
            Map<String, String> session = new HashMap<String, String>();

            // get initial login form
            Response loginForm = Jsoup.connect(urlLogIn)
                    .userAgent(agent)
                    .method(Method.GET)
                    .execute();

            // initialize session
            session = loginForm.cookies();

            print("started");

            // data map containing all the parameters and its values found in the form
            Map<String, String> mapParams = new HashMap<String, String>();
            mapParams.put("redirect", "");
            mapParams.put("ft_locale", "en-us");
            mapParams.put("login.x", "Log In");
            mapParams.put("username", "MY_USERNAME");
            mapParams.put("password", "MY_PASSWORD");
            mapParams.put("destination_page", "acctpositions");

            // With this you login and a session is created
            Connection.Response res = Jsoup.connect(urlLogIn)
                    .userAgent(agent)
                    .data(mapParams)
                    .cookies(session)
                    .method(Method.POST)
                    .execute();

            // update session after login
            session.putAll(loginResp.cookies());

            print("done");

            // The snippet from "started" to "done" is a full cycle
            // From here on every request is basically as the snippet above
            // For example:
            mapParams = new HashMap<String, String>();
            mapParams.put("other required form data", "DATA");

            resp = Jsoup.connect(urlUnderTest)
                    .userAgent(agent)
                    .data(mapParams)
                    .referrer("https://invest.firstrade.com/cgi-bin/main") // sometimes referrer is necessary
                    .cookies(session)
                    .method(Method.POST)
                    .execute();
            session.putAll(resp.cookies());

            // Then you start your crawler stuff
            Document doc = resp.parse();
            System.out.println(doc.title());
            print(doc.toString());

        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    private static void print(String msg, Object... args) {
        System.out.println(String.format(msg, args));
    }
}

Collect cookies from a website using jsoup

When you call Jsoup.connect("...") you get an object implementing the Connection interface (see API)

The cookies method you are calling on the Connection instance is meant to set the cookies for the HTTP request. In order to retrieve the cookies from the HTTP response, you need to get an instance of Connection.Response, which you can obtain by calling connection.execute().

To sum up, your code should be something like:

try {
    Map<String, String> cookies = Jsoup.connect("https://www.google.com").execute().cookies();
    System.out.println(cookies);
} catch (IOException e) {
    // error handling
}

As an additional consideration, keep in mind that in Android you're not allowed to perform network calls from the main thread, otherwise you'll get an Exception.

Jsoup can't login to establish cookies

Your code gives me

HTTP error fetching URL. Status=404, URL=https://www.rbauction.com/home/auth&

If you change only:

.data("_58_redirect", "%2Fhome%2Fauth&") for .data("_58_redirect", "%2Fhome%2Fauth") (without the trailing ampersand)

It works fine!

If it doesn't, check your user/pass.

JSoup Login and Cookie

I was missing the Login parameter in the form. After including that I was able to get a good cookie back.

---

Related Topics