How to Use Java.Net.Urlconnection to Fire and Handle Http Requests

  • Home
  • Languages
  • Java
  • How to Use Java.Net.Urlconnection to Fire and Handle Http Requests

How to use java.net.URLConnection to fire and handle HTTP requests

First a disclaimer beforehand: the posted code snippets are all basic examples. You'll need to handle trivial IOExceptions and RuntimeExceptions like NullPointerException, ArrayIndexOutOfBoundsException and consorts yourself.

In case you're developing for Android instead of Java, note also that since introduction of API level 28, cleartext HTTP requests are disabled by default. You are encouraged to use HttpsURLConnection, but if it is really necessary, cleartext can be enabled in the Application Manifest.


Preparing

We first need to know at least the URL and the charset. The parameters are optional and depend on the functional requirements.

String url = "http://example.com";
String charset = "UTF-8";  // Or in Java 7 and later, use the constant: java.nio.charset.StandardCharsets.UTF_8.name()
String param1 = "value1";
String param2 = "value2";
// ...

String query = String.format("param1=%s¶m2=%s",
    URLEncoder.encode(param1, charset),
    URLEncoder.encode(param2, charset));

The query parameters must be in name=value format and be concatenated by &. You would normally also URL-encode the query parameters with the specified charset using URLEncoder#encode().

The String#format() is just for convenience. I prefer it when I would need the String concatenation operator + more than twice.


Firing an HTTP GET request with (optionally) query parameters

It's a trivial task. It's the default request method.

URLConnection connection = new URL(url + "?" + query).openConnection();
connection.setRequestProperty("Accept-Charset", charset);
InputStream response = connection.getInputStream();
// ...

Any query string should be concatenated to the URL using ?. The Accept-Charset header may hint the server what encoding the parameters are in. If you don't send any query string, then you can leave the Accept-Charset header away. If you don't need to set any headers, then you can even use the URL#openStream() shortcut method.

InputStream response = new URL(url).openStream();
// ...

Either way, if the other side is an HttpServlet, then its doGet() method will be called and the parameters will be available by HttpServletRequest#getParameter().

For testing purposes, you can print the response body to standard output as below:

try (Scanner scanner = new Scanner(response)) {
    String responseBody = scanner.useDelimiter("\\A").next();
    System.out.println(responseBody);
}

Firing an HTTP POST request with query parameters

Setting the URLConnection#setDoOutput() to true implicitly sets the request method to POST. The standard HTTP POST as web forms do is of type application/x-www-form-urlencoded wherein the query string is written to the request body.

URLConnection connection = new URL(url).openConnection();
connection.setDoOutput(true); // Triggers POST.
connection.setRequestProperty("Accept-Charset", charset);
connection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded;charset=" + charset);

try (OutputStream output = connection.getOutputStream()) {
    output.write(query.getBytes(charset));
}

InputStream response = connection.getInputStream();
// ...

Note: whenever you'd like to submit a HTML form programmatically, don't forget to take the name=value pairs of any <input type="hidden"> elements into the query string and of course also the name=value pair of the <input type="submit"> element which you'd like to "press" programmatically (because that's usually been used in the server side to distinguish if a button was pressed and if so, which one).

You can also cast the obtained URLConnection to HttpURLConnection and use its HttpURLConnection#setRequestMethod() instead. But if you're trying to use the connection for output you still need to set URLConnection#setDoOutput() to true.

HttpURLConnection httpConnection = (HttpURLConnection) new URL(url).openConnection();
httpConnection.setRequestMethod("POST");
// ...

Either way, if the other side is an HttpServlet, then its doPost() method will be called and the parameters will be available by HttpServletRequest#getParameter().


Actually firing the HTTP request

You can fire the HTTP request explicitly with URLConnection#connect(), but the request will automatically be fired on demand when you want to get any information about the HTTP response, such as the response body using URLConnection#getInputStream() and so on. The above examples does exactly that, so the connect() call is in fact superfluous.


Gathering HTTP response information

  1. HTTP response status:

You need an HttpURLConnection here. Cast it first if necessary.

    int status = httpConnection.getResponseCode();

  1. HTTP response headers:

     for (Entry<String, List<String>> header : connection.getHeaderFields().entrySet()) {
         System.out.println(header.getKey() + "=" + header.getValue());
     }

  2. HTTP response encoding:

When the Content-Type contains a charset parameter, then the response body is likely text based and we'd like to process the response body with the server-side specified character encoding then.

    String contentType = connection.getHeaderField("Content-Type");
    String charset = null;

    for (String param : contentType.replace(" ", "").split(";")) {
        if (param.startsWith("charset=")) {
            charset = param.split("=", 2)[1];
            break;
        }
    }

    if (charset != null) {
        try (BufferedReader reader = new BufferedReader(new InputStreamReader(response, charset))) {
            for (String line; (line = reader.readLine()) != null;) {
                // ... System.out.println(line)?
            }
        }
    } else {
        // It's likely binary content, use InputStream/OutputStream.
    }

Maintaining the session

The server side session is usually backed by a cookie. Some web forms require that you're logged in and/or are tracked by a session. You can use the CookieHandler API to maintain cookies. You need to prepare a CookieManager with a CookiePolicy of ACCEPT_ALL before sending all HTTP requests.

// First set the default cookie manager.
CookieHandler.setDefault(new CookieManager(null, CookiePolicy.ACCEPT_ALL));

// All the following subsequent URLConnections will use the same cookie manager.
URLConnection connection = new URL(url).openConnection();
// ...

connection = new URL(url).openConnection();
// ...

connection = new URL(url).openConnection();
// ...

Note that this is known to not always work properly in all circumstances. If it fails for you, then best is to manually gather and set the cookie headers. You basically need to grab all Set-Cookie headers from the response of the login or the first GET request and then pass this through the subsequent requests.

// Gather all cookies on the first request.
URLConnection connection = new URL(url).openConnection();
List<String> cookies = connection.getHeaderFields().get("Set-Cookie");
// ...

// Then use the same cookies on all subsequent requests.
connection = new URL(url).openConnection();
for (String cookie : cookies) {
    connection.addRequestProperty("Cookie", cookie.split(";", 2)[0]);
}
// ...

The split(";", 2)[0] is there to get rid of cookie attributes which are irrelevant for the server side like expires, path, etc. Alternatively, you could also use cookie.substring(0, cookie.indexOf(';')) instead of split().


Streaming mode

The HttpURLConnection will by default buffer the entire request body before actually sending it, regardless of whether you've set a fixed content length yourself using connection.setRequestProperty("Content-Length", contentLength);. This may cause OutOfMemoryExceptions whenever you concurrently send large POST requests (e.g. uploading files). To avoid this, you would like to set the HttpURLConnection#setFixedLengthStreamingMode().

httpConnection.setFixedLengthStreamingMode(contentLength);

But if the content length is really not known beforehand, then you can make use of chunked streaming mode by setting the HttpURLConnection#setChunkedStreamingMode() accordingly. This will set the HTTP Transfer-Encoding header to chunked which will force the request body being sent in chunks. The below example will send the body in chunks of 1 KB.

httpConnection.setChunkedStreamingMode(1024);

User-Agent

It can happen that a request returns an unexpected response, while it works fine with a real web browser. The server side is probably blocking requests based on the User-Agent request header. The URLConnection will by default set it to Java/1.6.0_19 where the last part is obviously the JRE version. You can override this as follows:

connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36"); // Do as if you're using Chrome 41 on Windows 7.

Use the User-Agent string from a recent browser.


Error handling

If the HTTP response code is 4nn (Client Error) or 5nn (Server Error), then you may want to read the HttpURLConnection#getErrorStream() to see if the server has sent any useful error information.

InputStream error = ((HttpURLConnection) connection).getErrorStream();

If the HTTP response code is -1, then something went wrong with connection and response handling. The HttpURLConnection implementation is in older JREs somewhat buggy with keeping connections alive. You may want to turn it off by setting the http.keepAlive system property to false. You can do this programmatically in the beginning of your application by:

System.setProperty("http.keepAlive", "false");

Uploading files

You'd normally use multipart/form-data encoding for mixed POST content (binary and character data). The encoding is in more detail described in RFC2388.

String param = "value";
File textFile = new File("/path/to/file.txt");
File binaryFile = new File("/path/to/file.bin");
String boundary = Long.toHexString(System.currentTimeMillis()); // Just generate some unique random value.
String CRLF = "\r\n"; // Line separator required by multipart/form-data.
URLConnection connection = new URL(url).openConnection();
connection.setDoOutput(true);
connection.setRequestProperty("Content-Type", "multipart/form-data; boundary=" + boundary);

try (
    OutputStream output = connection.getOutputStream();
    PrintWriter writer = new PrintWriter(new OutputStreamWriter(output, charset), true);
) {
    // Send normal param.
    writer.append("--" + boundary).append(CRLF);
    writer.append("Content-Disposition: form-data; name=\"param\"").append(CRLF);
    writer.append("Content-Type: text/plain; charset=" + charset).append(CRLF);
    writer.append(CRLF).append(param).append(CRLF).flush();

    // Send text file.
    writer.append("--" + boundary).append(CRLF);
    writer.append("Content-Disposition: form-data; name=\"textFile\"; filename=\"" + textFile.getName() + "\"").append(CRLF);
    writer.append("Content-Type: text/plain; charset=" + charset).append(CRLF); // Text file itself must be saved in this charset!
    writer.append(CRLF).flush();
    Files.copy(textFile.toPath(), output);
    output.flush(); // Important before continuing with writer!
    writer.append(CRLF).flush(); // CRLF is important! It indicates end of boundary.

    // Send binary file.
    writer.append("--" + boundary).append(CRLF);
    writer.append("Content-Disposition: form-data; name=\"binaryFile\"; filename=\"" + binaryFile.getName() + "\"").append(CRLF);
    writer.append("Content-Type: " + URLConnection.guessContentTypeFromName(binaryFile.getName())).append(CRLF);
    writer.append("Content-Transfer-Encoding: binary").append(CRLF);
    writer.append(CRLF).flush();
    Files.copy(binaryFile.toPath(), output);
    output.flush(); // Important before continuing with writer!
    writer.append(CRLF).flush(); // CRLF is important! It indicates end of boundary.

    // End of multipart/form-data.
    writer.append("--" + boundary + "--").append(CRLF).flush();
}

If the other side is an HttpServlet, then its doPost() method will be called and the parts will be available by HttpServletRequest#getPart() (note, thus not getParameter() and so on!). The getPart() method is however relatively new, it's introduced in Servlet 3.0 (Glassfish 3, Tomcat 7, etc.). Prior to Servlet 3.0, your best choice is using Apache Commons FileUpload to parse a multipart/form-data request. Also see this answer for examples of both the FileUpload and the Servelt 3.0 approaches.


Dealing with untrusted or misconfigured HTTPS sites

In case you're developing for Android instead of Java, be careful: the workaround below may save your day if you don't have correct certificates deployed during development. But you should not use it for production. These days (April 2021) Google will not allow your app be distributed on Play Store if they detect insecure hostname verifier, see https://support.google.com/faqs/answer/7188426.

Sometimes you need to connect an HTTPS URL, perhaps because you're writing a web scraper. In that case, you may likely face a javax.net.ssl.SSLException: Not trusted server certificate on some HTTPS sites who doesn't keep their SSL certificates up to date, or a java.security.cert.CertificateException: No subject alternative DNS name matching [hostname] found or javax.net.ssl.SSLProtocolException: handshake alert: unrecognized_name on some misconfigured HTTPS sites.

The following one-time-run static initializer in your web scraper class should make HttpsURLConnection more lenient as to those HTTPS sites and thus not throw those exceptions anymore.

static {
    TrustManager[] trustAllCertificates = new TrustManager[] {
        new X509TrustManager() {
            @Override
            public X509Certificate[] getAcceptedIssuers() {
                return null; // Not relevant.
            }
            @Override
            public void checkClientTrusted(X509Certificate[] certs, String authType) {
                // Do nothing. Just allow them all.
            }
            @Override
            public void checkServerTrusted(X509Certificate[] certs, String authType) {
                // Do nothing. Just allow them all.
            }
        }
    };

    HostnameVerifier trustAllHostnames = new HostnameVerifier() {
        @Override
        public boolean verify(String hostname, SSLSession session) {
            return true; // Just allow them all.
        }
    };

    try {
        System.setProperty("jsse.enableSNIExtension", "false");
        SSLContext sc = SSLContext.getInstance("SSL");
        sc.init(null, trustAllCertificates, new SecureRandom());
        HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());
        HttpsURLConnection.setDefaultHostnameVerifier(trustAllHostnames);
    }
    catch (GeneralSecurityException e) {
        throw new ExceptionInInitializerError(e);
    }
}

Last words

The Apache HttpComponents HttpClient is much more convenient in this all :)

  • HttpClient Tutorial
  • HttpClient Examples

Parsing and extracting HTML

If all you want is parsing and extracting data from HTML, then better use a HTML parser like Jsoup.

  • What are the pros/cons of leading HTML parsers in Java
  • How to scan and extract a webpage in Java

java.net How to fire an HTTP request

From the documentation, a HttpURLConnection will connect either if you call connect(), or if you call an operation that depends on being connected, like getInputStream() .

Opens a communications link to the resource referenced by this URL, if
such a connection has not already been established. If the connect
method is called when the connection has already been opened
(indicated by the connected field having the value true), the call is
ignored.

URLConnection objects go through two phases: first they are created,
then they are connected. After being created, and before being
connected, various options can be specified (e.g., doInput and
UseCaches). After connecting, it is an error to try to set them.
Operations that depend on being connected, like getContentLength, will
implicitly perform the connection, if necessary.

However, several topics indicate that connect() won't commit the actual request, but getInputStream() (and most likely any method reading the server's response e.g getResponseCode() ), will :

Java URLConnection - When do I need to use the connect() method?

Why does HttpURLConnection not send the HTTP request

How to send PUT, DELETE HTTP request in HttpURLConnection?

Using java.net.URLConnection with a GET, how to get redirect URL?

Given Rishal's link and amobiz's answer was what I was looking for:

URLConnection con = new URL( url ).openConnection();
System.out.println( "orignal url: " + con.getURL() );
con.connect();
System.out.println( "connected url: " + con.getURL() );
InputStream is = con.getInputStream();
System.out.println( "redirected url: " + con.getURL() );
is.close();

HTTPS and HTTP connection in java

In your code:

public static HttpsURLConnection passmyurl(URL url) throws IOException {
    HttpsURLConnection connection = (HttpsURLConnection) url.openConnection();
    return connection;
}

You should change the return type HttpsURLConnection to URLConnection.Because the url.openConnection() result's type is subtype of URLConnection,and the exact type depends on the paramter url's protocol.The implemention document of openConnection() in the URL class:

If for the URL's protocol (such as HTTP or JAR), there exists a public, specialized URLConnection subclass belonging to one of the following packages or one of their subpackages: java.lang, java.io, java.util, java.net, the connection returned will be of that subclass. For example, for HTTP an HttpURLConnection will be returned, and for JAR a JarURLConnection will be returned.

So you can pass the Http url or Https url to your method.

See the following code:

    URLConnection httpConnection = new URL("http://test").openConnection();
    System.out.println(httpConnection.getClass());
    URLConnection httpsConnection = new URL("https://test").openConnection();
    System.out.println(httpsConnection.getClass());
    URLConnection ftpConnection = new URL("ftp://test").openConnection();
    System.out.println(ftpConnection.getClass());`

the print is:

class sun.net.www.protocol.http.HttpURLConnection
class sun.net.www.protocol.https.HttpsURLConnectionImpl
class sun.net.www.protocol.ftp.FtpURLConnection

Plain HTTP Request to UrlConnection (Java)

We can send Request and get Response by using Socket.

We can create a new Socket and connect it with host and port. In the OutputStream we can input the plain request. We can get Response by Socket.getInputStream().

Request : Socket.getOutputStream()
Response : Socket.getInputStream()

But, we'll have to connect to the Server with Socket.connect().

Socket sock = new Socket();
sock.connect(host, port);
DataOutputStream dos = new DataOutputStream(sock.getOutputStream());
dos.write(reqStr.getBytes());
dos.flush();
dos.close();
DataInputStream dis = new DataInputStream(sock.getInputStream());
//Here dis is the response.

javax.net.ssl.HttpsURLConnection when are the request fired

In cases like this you may want to check the source code. Most of JVM classes have included sources, and java.net.HttpURLConnection does.

There is this snip in beginning of method getResponseCode() (as JDK 1.8_71)

/*
 * We're got the response code already
 */
if (responseCode != -1) {
    return responseCode;
}

So its cached. If response is still default value, -1, its executing request to server. But since this behavior is not described in JavaDoc for this method, I would not rely on this and use own integer variable.

JV

Using HttpURLConnection to POST in Java

Why the suspect code in the question has been duplicated all over the internet is something I can't answer. Nor can I answer why it seems to work for some people and not others. I can however answer the other question now, mainly thanks to this link that Luiggi pointed me to.

The key here is understanding the intricacies of the HttpURLConnection class. When first created the class defaults to a "GET" request method, so nothing needs to be changed in this instance. The following is rather unintuitive, but to set the request method to "POST" you should not call setRequestMethod("POST"), but rather setDoOutput(true) which implicitly sets the request method to post. Once you've done that you're good to go.

Below, I believe, is what a post method should look like. This is for posting json, but can obviously be altered for any other content type.

public static String doPostSync(final String urlToRead, final String content) throws IOException {
    final String charset = "UTF-8";
    // Create the connection
    HttpURLConnection connection = (HttpURLConnection) new URL(urlToRead).openConnection();
    // setDoOutput(true) implicitly set's the request type to POST
    connection.setDoOutput(true);
    connection.setRequestProperty("Accept-Charset", charset);
    connection.setRequestProperty("Content-type", "application/json");

    // Write to the connection
    OutputStream output = connection.getOutputStream();
    output.write(content.getBytes(charset));
    output.close();

    // Check the error stream first, if this is null then there have been no issues with the request
    InputStream inputStream = connection.getErrorStream();
    if (inputStream == null)
        inputStream = connection.getInputStream();

    // Read everything from our stream
    BufferedReader responseReader = new BufferedReader(new InputStreamReader(inputStream, charset));

    String inputLine;
    StringBuffer response = new StringBuffer();

    while ((inputLine = responseReader.readLine()) != null) {
        response.append(inputLine);
    }
    responseReader.close();

    return response.toString();
}

Related Topics

Difference Between Jsonobject and Jsonarray

Why Maven Uses Jdk 1.6 But My Java -Version Is 1.7

Why Am I Getting a Noclassdeffounderror in Java

Java Generics Type Erasure: When and What Happens

Why Is Char[] Preferred Over String For Passwords

Preparedstatement in Clause Alternatives

Create Instance of Generic Type in Java

What Is a Daemon Thread in Java

Difference Between Int[] Array and Int Array[]

How to Load a File from Resource Folder

Firebase Android Listview Not Being Displayed

How to Make a .Jar Out from an Android Studio Project

Java_Home and Path Are Set But Java -Version Still Shows the Old One

Java String to Date Conversion

Reading a Plain Text File in Java

How to Deal With "Java.Lang.Outofmemoryerror: Java Heap Space" Error

How to Generate a Random Alpha-Numeric String

How Does Java Handle Integer Underflows and Overflows and How Would You Check For It


Leave a reply



Submit