Java - Best Approach to Parse Huge (Extra Large) JSON File

JAVA - Best approach to parse huge (extra large) JSON file

You don't need to switch to Jackson. Gson 2.1 introduced a new TypeAdapter interface that permits mixed tree and streaming serialization and deserialization.

The API is efficient and flexible. See Gson's Streaming doc for an example of combining tree and binding modes. This is strictly better than mixed streaming and tree modes; with binding you don't waste memory building an intermediate representation of your values.

Like Jackson, Gson has APIs to recursively skip an unwanted value; Gson calls this skipValue().

Parse large JSON file with JSON Simple (OutOfMemoryError)

If you are open to use other Json parser then you can try Jackson's Streaming API which can be used to parse huge JSON upto even giga bytes of size.It can be used to process huge files without loading them completely in memory.It allows get the data you want and ignore what you don't want also

Read more: https://github.com/FasterXML/jackson-docs/wiki/JacksonStreamingApi

Read a huge json array file of objects

is there a way to read this file using BufferedReader and then to push
object by object ?

Of course, not. Even you can open this file how you can store 40GB as java objects in memory? I think you don't have such amount of memory in you computers (but technically using ObjectMapper you should have about 2 times more operation memory - 40GB for store json + 40GB for store results as java objects = 80 GB).

I think you should use any way from this questions, but store information in databases or files instead of memory. For example, if you have millions rows in json, you should parse and save every rows to database without keeping it all in memory. And then you can get this data from database step by step (for example, not more then 1GB for every time).

Android - Best approach to parse huge (extra large) JSON file from assets

I recommend you to use the GSON lib. It has very good performance.

Just add this line in your gradle file to import the lib .

compile 'com.google.code.gson:gson:2.2.4'

If your JSON starts with "[" (array of User let's say), you can use GSON like this :

public Set<User> getUsers(final Activity activity) {

    Set<User> usersList = new HashSet<>();
    String json = readFromAsset(activity, "myfile_with_array.json");
    Type listType = new TypeToken<HashSet<User>>() {}.getType();
    // convert json into a list of Users
    try {
        usersList = new Gson().fromJson(json, listType);
    }
    catch (Exception e) {
        // we never know :)
        Log.e("error parsing", e.toString());
    }
    return usersList;
}

/**
 * Read file from asset directory
 * @param act current activity
 * @param fileName file to read
 * @return content of the file, string format
 */
private  String readFromAsset(final Activity act, final String fileName)
{
    String text = "";
    try {
        InputStream is = act.getAssets().open(fileName);

        int size = is.available();

        // Read the entire asset into a local byte buffer.
        byte[] buffer = new byte[size];
        is.read(buffer);
        is.close();
        text = new String(buffer, "UTF-8");
    } catch (IOException e) {
        e.printStackTrace();
    }
    return text;
}

This will return you a set of Users.

If your JSON Starts with a "{", so can be mapped to an object (User object let's say), you can use it like that :

public User getUser(final Activity activity) {

        User user = null;
        String json = readFromAsset(activity, "myfile_with_object.json");
        try {
        // convert json in an User object
            user = new Gson.fromJson(json, User.class)
        }
        catch (Exception e) {
            // we never know :)
            Log.e("error parsing", e.toString());
        }
        return user;
    }

Hope this helps !

How to go about parsing a large JSON file for search bar suggestions

Approach 1: A better approach is to request the JSON in background as soon as the app starts up and saving it to a Local storage(SQlite, Room Persistence or just sharedpreference).

Approach 2: The second trick is to show only a large enough number but not to large.
Suppose user there are 1000 results matching the string of user input. Just query the local or remote database to show best matching 100. No sane person will scroll past 100 results without modifying/writing his input text again.

Best Approach: Use a pagination library to load data as the user scrolls down.

---

Related Topics