How to Search Google Programmatically Java API

How can you search Google Programmatically Java API

Some facts:

  1. Google offers a public search webservice API which returns JSON: http://ajax.googleapis.com/ajax/services/search/web. Documentation here

  2. Java offers java.net.URL and java.net.URLConnection to fire and handle HTTP requests.

  3. JSON can in Java be converted to a fullworthy Javabean object using an arbitrary Java JSON API. One of the best is Google Gson.

Now do the math:

public static void main(String[] args) throws Exception {
String google = "http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=";
String search = "stackoverflow";
String charset = "UTF-8";

URL url = new URL(google + URLEncoder.encode(search, charset));
Reader reader = new InputStreamReader(url.openStream(), charset);
GoogleResults results = new Gson().fromJson(reader, GoogleResults.class);

// Show title and URL of 1st result.
System.out.println(results.getResponseData().getResults().get(0).getTitle());
System.out.println(results.getResponseData().getResults().get(0).getUrl());
}

With this Javabean class representing the most important JSON data as returned by Google (it actually returns more data, but it's left up to you as an exercise to expand this Javabean code accordingly):

public class GoogleResults {

private ResponseData responseData;
public ResponseData getResponseData() { return responseData; }
public void setResponseData(ResponseData responseData) { this.responseData = responseData; }
public String toString() { return "ResponseData[" + responseData + "]"; }

static class ResponseData {
private List<Result> results;
public List<Result> getResults() { return results; }
public void setResults(List<Result> results) { this.results = results; }
public String toString() { return "Results[" + results + "]"; }
}

static class Result {
private String url;
private String title;
public String getUrl() { return url; }
public String getTitle() { return title; }
public void setUrl(String url) { this.url = url; }
public void setTitle(String title) { this.title = title; }
public String toString() { return "Result[url:" + url +",title:" + title + "]"; }
}

}

###See also:

  • How to fire and handle HTTP requests using java.net.URLConnection
  • How to convert JSON to Java

Update since November 2010 (2 months after the above answer), the public search webservice has become deprecated (and the last day on which the service was offered was September 29, 2014). Your best bet is now querying http://www.google.com/search directly along with a honest user agent and then parse the result using a HTML parser. If you omit the user agent, then you get a 403 back. If you're lying in the user agent and simulate a web browser (e.g. Chrome or Firefox), then you get a way much larger HTML response back which is a waste of bandwidth and performance.

Here's a kickoff example using Jsoup as HTML parser:

String google = "http://www.google.com/search?q=";
String search = "stackoverflow";
String charset = "UTF-8";
String userAgent = "ExampleBot 1.0 (+http://example.com/bot)"; // Change this to your company's name and bot homepage!

Elements links = Jsoup.connect(google + URLEncoder.encode(search, charset)).userAgent(userAgent).get().select(".g>.r>a");

for (Element link : links) {
String title = link.text();
String url = link.absUrl("href"); // Google returns URLs in format "http://www.google.com/url?q=<url>&sa=U&ei=<someKey>".
url = URLDecoder.decode(url.substring(url.indexOf('=') + 1, url.indexOf('&')), "UTF-8");

if (!url.startsWith("http")) {
continue; // Ads/news/etc.
}

System.out.println("Title: " + title);
System.out.println("URL: " + url);
}

Google Search using Java programmatically

A sample code for google knowledge search.

Generate API using https://console.developers.google.com/apis/dashboard > credentials > Create credentials > API key

import java.util.List;

import org.json.simple.parser.JSONParser;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.scheduling.annotation.Async;
import org.springframework.stereotype.Service;

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.google.api.client.http.GenericUrl;
import com.google.api.client.http.HttpRequest;
import com.google.api.client.http.HttpRequestFactory;
import com.google.api.client.http.HttpResponse;
import com.google.api.client.http.HttpTransport;
import com.google.api.client.http.javanet.NetHttpTransport;

@Service
public class GoogleSearchService {

@Async
public void searchGoogle(List<String> keywords){
try {
ObjectMapper mapper = new ObjectMapper();
HttpTransport httpTransport = new NetHttpTransport();
HttpRequestFactory requestFactory = httpTransport.createRequestFactory();
JSONParser parser = new JSONParser();
GenericUrl url = new GenericUrl("https://kgsearch.googleapis.com/v1/entities:search");
url.put("query", "Kolkata");
url.put("limit", "1");
url.put("indent", "true");
url.put("key", "xxxxxx");
HttpRequest request = requestFactory.buildGetRequest(url);
HttpResponse httpResponse = request.execute();
String responseString = httpResponse.parseAsString();

JsonNode node = mapper.readTree(responseString).get("itemListElement").get(0).get("result");
System.out.println(node.get("name"));
System.out.println(node.get("@type"));
System.out.println(node.get("detailedDescription").get("articleBody"));

} catch (Exception ex) {
ex.printStackTrace();
}
}

}

easiest (legal) way to programmatically get the google search result count?

/**** @author RAJESH Kharche */
//open Netbeans
//Choose Java->prject
//name it GoogleSearchAPP

package googlesearchapp;

import java.io.*;
import java.net.*;
import java.util.*;
import java.util.logging.Level;
import java.util.logging.Logger;

public class GoogleSearchAPP {
public static void main(String[] args) {
try {
// TODO code application logic here

final int Result;

Scanner s1=new Scanner(System.in);
String Str;
System.out.println("Enter Query to search: ");//get the query to search
Str=s1.next();
Result=getResultsCount(Str);

System.out.println("Results:"+ Result);
} catch (IOException ex) {
Logger.getLogger(GoogleSearchAPP.class.getName()).log(Level.SEVERE, null, ex);
}
}

private static int getResultsCount(final String query) throws IOException {
final URL url;
url = new URL("https://www.google.com/search?q=" + URLEncoder.encode(query, "UTF-8"));
final URLConnection connection = url.openConnection();

connection.setConnectTimeout(60000);
connection.setReadTimeout(60000);
connection.addRequestProperty("User-Agent", "Google Chrome/36");//put the browser name/version

final Scanner reader = new Scanner(connection.getInputStream(), "UTF-8"); //scanning a buffer from object returned by http request

while(reader.hasNextLine()){ //for each line in buffer
final String line = reader.nextLine();

if(!line.contains("\"resultStats\">"))//line by line scanning for "resultstats" field because we want to extract number after it
continue;

try{
return Integer.parseInt(line.split("\"resultStats\">")[1].split("<")[0].replaceAll("[^\\d]", ""));//finally extract the number convert from string to integer
}finally{
reader.close();
}
}
reader.close();
return 0;
}
}

Java code for using google custom search API

I have changed the while loop in the code provided by @Zakaria above. It might not be a proper way of working it out but it gives you the result links of google search. You just need to parse the output. See here,

public static void main(String[] args) throws Exception {

String key="YOUR KEY";
String qry="Android";
URL url = new URL(
"https://www.googleapis.com/customsearch/v1?key="+key+ "&cx=013036536707430787589:_pqjad5hr1a&q="+ qry + "&alt=json");
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("GET");
conn.setRequestProperty("Accept", "application/json");
BufferedReader br = new BufferedReader(new InputStreamReader(
(conn.getInputStream())));

String output;
System.out.println("Output from Server .... \n");
while ((output = br.readLine()) != null) {

if(output.contains("\"link\": \"")){
String link=output.substring(output.indexOf("\"link\": \"")+("\"link\": \"").length(), output.indexOf("\","));
System.out.println(link); //Will print the google search links
}
}
conn.disconnect();
}

Hope it works for you too.

How can you search Google Programmatically Java API

Some facts:

  1. Google offers a public search webservice API which returns JSON: http://ajax.googleapis.com/ajax/services/search/web. Documentation here

  2. Java offers java.net.URL and java.net.URLConnection to fire and handle HTTP requests.

  3. JSON can in Java be converted to a fullworthy Javabean object using an arbitrary Java JSON API. One of the best is Google Gson.

Now do the math:

public static void main(String[] args) throws Exception {
String google = "http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=";
String search = "stackoverflow";
String charset = "UTF-8";

URL url = new URL(google + URLEncoder.encode(search, charset));
Reader reader = new InputStreamReader(url.openStream(), charset);
GoogleResults results = new Gson().fromJson(reader, GoogleResults.class);

// Show title and URL of 1st result.
System.out.println(results.getResponseData().getResults().get(0).getTitle());
System.out.println(results.getResponseData().getResults().get(0).getUrl());
}

With this Javabean class representing the most important JSON data as returned by Google (it actually returns more data, but it's left up to you as an exercise to expand this Javabean code accordingly):

public class GoogleResults {

private ResponseData responseData;
public ResponseData getResponseData() { return responseData; }
public void setResponseData(ResponseData responseData) { this.responseData = responseData; }
public String toString() { return "ResponseData[" + responseData + "]"; }

static class ResponseData {
private List<Result> results;
public List<Result> getResults() { return results; }
public void setResults(List<Result> results) { this.results = results; }
public String toString() { return "Results[" + results + "]"; }
}

static class Result {
private String url;
private String title;
public String getUrl() { return url; }
public String getTitle() { return title; }
public void setUrl(String url) { this.url = url; }
public void setTitle(String title) { this.title = title; }
public String toString() { return "Result[url:" + url +",title:" + title + "]"; }
}

}

###See also:

  • How to fire and handle HTTP requests using java.net.URLConnection
  • How to convert JSON to Java

Update since November 2010 (2 months after the above answer), the public search webservice has become deprecated (and the last day on which the service was offered was September 29, 2014). Your best bet is now querying http://www.google.com/search directly along with a honest user agent and then parse the result using a HTML parser. If you omit the user agent, then you get a 403 back. If you're lying in the user agent and simulate a web browser (e.g. Chrome or Firefox), then you get a way much larger HTML response back which is a waste of bandwidth and performance.

Here's a kickoff example using Jsoup as HTML parser:

String google = "http://www.google.com/search?q=";
String search = "stackoverflow";
String charset = "UTF-8";
String userAgent = "ExampleBot 1.0 (+http://example.com/bot)"; // Change this to your company's name and bot homepage!

Elements links = Jsoup.connect(google + URLEncoder.encode(search, charset)).userAgent(userAgent).get().select(".g>.r>a");

for (Element link : links) {
String title = link.text();
String url = link.absUrl("href"); // Google returns URLs in format "http://www.google.com/url?q=<url>&sa=U&ei=<someKey>".
url = URLDecoder.decode(url.substring(url.indexOf('=') + 1, url.indexOf('&')), "UTF-8");

if (!url.startsWith("http")) {
continue; // Ads/news/etc.
}

System.out.println("Title: " + title);
System.out.println("URL: " + url);
}

Getting links of google search results

You can use the Google REST API, as described here: https://developers.google.com/custom-search/v1/using_rest#WorkingResults

The result can be in JSON format, which you can parse to get the links.

This is an example request:

GET https://www.googleapis.com/customsearch/v1?key=INSERT-YOUR-KEY&cx=013036536707430787589:_pqjad5hr1a&q=flowers&alt=json

Now you get a JSON as described. You can parse the JSON either with a JSON library, such as Jackson (recommended!), or you just "grep" through it using a regular expression:

    BufferedReader in = new BufferedReader(new StringReader(resultJson));

Pattern regex = Pattern.compile(".*\"link\": \"(.*)\",");
Collection<String> links = new ArrayList<String>();
String line = null;
while ((line = in.readLine()) != null) {
Matcher matcher = regex.matcher(line);
if (matcher.matches()) {
String link = matcher.group(1);
links.add(link);
}
}


Related Topics



Leave a reply



Submit