403 Forbidden With Java But Not Web Browser

403 Forbidden with Java but not web browser?

You just need to set user agent header for it to work:

URLConnection connection = new URL("https://www.google.com/search?q=" + query).openConnection();
connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11");
connection.connect();

BufferedReader r  = new BufferedReader(new InputStreamReader(connection.getInputStream(), Charset.forName("UTF-8")));

StringBuilder sb = new StringBuilder();
String line;
while ((line = r.readLine()) != null) {
    sb.append(line);
}
System.out.println(sb.toString());

The SSL was transparently handled for you as could be seen from your exception stacktrace.

Getting the result amount is not really this simple though, after this you have to fake that you're a browser by fetching the cookie and parsing the redirect token link.

String cookie = connection.getHeaderField( "Set-Cookie").split(";")[0];
Pattern pattern = Pattern.compile("content=\\\"0;url=(.*?)\\\"");
Matcher m = pattern.matcher(response);
if( m.find() ) {
    String url = m.group(1);
    connection = new URL(url).openConnection();
    connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11");
    connection.setRequestProperty("Cookie", cookie );
    connection.connect();
    r  = new BufferedReader(new InputStreamReader(connection.getInputStream(), Charset.forName("UTF-8")));
    sb = new StringBuilder();
    while ((line = r.readLine()) != null) {
        sb.append(line);
    }
    response = sb.toString();
    pattern = Pattern.compile("<div id=\"resultStats\">About ([0-9,]+) results</div>");
    m = pattern.matcher(response);
    if( m.find() ) {
        long amount = Long.parseLong(m.group(1).replaceAll(",", ""));
        return amount;
    }

}

Running the full code I get 2930000000L as a result.

Unexpected HTTP 403 error in Java

Without you showing us the code you're using to access the supplied URL (http://checkdnd.com/api/check_dnd_no_api.php?mobiles=9999999999) it makes it a wee bit difficult to determine exactly where your problem lies but my first guess would be that the link you provided is only accessible through a Secure Socket Layer (SSL). In other words, the link should start with https:// instead of http://

To validate this simply make the change to your url string: https://checkdnd.com/api/check_dnd_no_api.php?mobiles=9999999999 and try again.

You're not going to have an issue with a browser for the simple reason that generally browsers will always try both protocols to make a connection. It is also up to the Website what protocol is acceptable, lots allow for both and some just don't.

To check if a url string is utilizing a valid protocol you can use this little method I quickly whipped up:

/**
 * This method will take the supplied URL String regardless of the protocol (http or https)
 * specified at the beginning of the string, and will return whether or not it is an actual
 * "http" (no SSL) or "https" (is SSL) protocol. A connection to the URL is attempted first 
 * with the http protocol and if successful (by way of data acquisition) will then return 
 * that protocol. If not however, then the https protocol is attempted and if successful then 
 * that protocol is returned. If neither protocols were successful then Null is returned.<br><br>
 * 
 * Returns null if the supplied URL String is invalid, a protocol does not
 * exist, or a valid connection to the URL can not be established.<br><br>
 * 
 * @param webLink (String) The full link path.<br>
 * 
 * @return (String) Either "http" for Non SLL link, "https" for a SSL link. 
 * Null is returned if the supplied URL String is invalid, a protocol does 
 * not exist, or a valid connection to the URL can not be established.
 */
public static String isHttpOrHttps(String webLink) {
    URL url;
    try {
        url = new URL(webLink);
    } catch (MalformedURLException ex) { return null; }

    String protocol = url.getProtocol();
    if (protocol.equals("")) { return null; }

        URLConnection yc;
        try {
            yc = url.openConnection();
            BufferedReader in = new BufferedReader(new InputStreamReader(yc.getInputStream()));
            in.close();
            return "http";
        } catch (IOException e) {
            // Do nothing....check for https instead.
        }
        try {
            yc = new URL(webLink).openConnection();
            //send request for page data...
            yc.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11");
            yc.connect();
            BufferedReader in = new BufferedReader(new InputStreamReader(yc.getInputStream()));
            in.close();
            return "https";
        } catch (IOException e) {
            // Do Nothing....allow for Null to be returned.
        }
    return null;
}

To use this method:

// Note that the http protocol is supplied within the url string:
String protocol = isHttpOrHttps("http://checkdnd.com/api/check_dnd_no_api.php?mobiles=9999999999");
System.out.println(protocol);

The output to console will be: https. The isHttpOrHttps() method has determined that the https protocol is the successful protocol to use in order to acquire data (or whatever) even though http was supplied.

To pull the page source from the web page you can perhaps use a method like this:

/**
 * Returns a List ArrayList containing the page source for the supplied web 
 * page link.<br><br>
 * 
 * @param link (String) The URL address of the web page to process.<br>
 * 
 * @return (List ArrayList) A List ArrayList containing the page source for 
 * the supplied web page link.
 */
public static List<String> getWebPageSource(String link) {
    if (link.equals("")) { return null; }
    try {
        URL url = new URL(link);

        URLConnection yc = null;
        //If url is a SSL Endpoint (using a Secure Socket Layer such as https)...
        if (link.startsWith("https:")) {
            yc = new URL(link).openConnection();
            //send request for page data...
            yc.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11");
            yc.connect();
        }
        //and if not a SLL Endpoint (just http)...
        else { yc = url.openConnection(); }
        BufferedReader in = new BufferedReader(new InputStreamReader(yc.getInputStream()));
        String inputLine;
        List<String> sourceText = new ArrayList<>();
        while ((inputLine = in.readLine()) != null) { 
            sourceText.add(inputLine);
        }
        in.close();
        return sourceText;
    } 
    catch (MalformedURLException ex) { 
        // Do whatever you want with exception.
        ex.printStackTrace();
    } 
    catch (IOException ex) { 
        // Do whatever you want with exception.
        ex.printStackTrace();
    }
    return null;
}

In order to utilize both the methods supplied here you can try something like this:

String netLink = "http://checkdnd.com/api/check_dnd_no_api.php?mobiles=9999999999";
String protocol = isHttpOrHttps(netLink);
String netLinkProtocol = netLink.substring(0, netLink.indexOf(":"));
if (!netLinkProtocol.equals(protocol)) {
    netLink = protocol + netLink.substring(netLink.indexOf(":"));
}
List<String> list = getWebPageSource(netLink);
for (int i = 0; i < list.size(); i++) {
    System.out.println(list.get(i));
}

And the console output will display:

{"msg":"success","msg_text":{"9999999999":"N"}}

403 error in Java Application but not in browser or Postman

I had the wrong password. Once I updated it with the correct password I was able to perform the GET.

---

Related Topics