How to Parse a Json Input Stream

Parsing json directly using input stream

You're using the content stream in a wrong way. Look at the marked lines:

InputStream content = entity.getContent();

// (1)
BufferedReader bReader = new BufferedReader(new InputStreamReader(content));
String line;
while ((line = bReader.readLine()) != null) {
    builder.append(line);
}

// (2)
JsonReader reader = new JsonReader(new InputStreamReader(content));
reader.setLenient(true);
readGson(reader);

The issue here is that you wrap the same stream twice (on marked line).

• After first wrap, you go through the stream collecting the lines and concatenating them in a builder. You must be aware that effectively the bReader is just a wrapper around the content stream. So while you are collecting the lines, the content from the content stream is eaten. So, the condition (line = ...) != null is false when the content stream is at the end of input.
• Then you wrap the content stream again - for parsing JSON contents. But the stream is already at the end of input here, so there is nothing to consume by the JSON reader. And that exactly is the meaning of the line: java.io.EOFException: End of input at line 1 column 1 of your exception.

What you have to do is to read through the content stream only once. So, you have several options here:

• Don't save received contents to a String. You delete the first wrap and the loop which builds the response string using a builder. If that's not an option then I recommend:

• Save the whole response to a String. Note: you can use the EntityUtils class for this since it will do the work for you:

  HttpEntity entity = response.getEntity();
  
  // All the work is done for you here :)
  String jsonContent = EntityUtils.toString(entity);
  
  // Create a Reader from String
  Reader stringReader = new StringReader(jsonContent);
  
  // Pass the string reader to JsonReader constructor
  JsonReader reader = new JsonReader(stringReader);
  reader.setLenient(true);
  readGson(reader);
  
  ...
  // at the end of method return the JSON response
  return jsonContent;
  

How to convert InputStream to JsonArray Object using Java

AFAIU, this is expected, since Your JSON object is only partially valid.

Although it is not a valid JSON array either, it could be parsed into JSONArray after small modifications (mind the starting and closing brackets and a comma between the objects):

[
{"id":4,"productId":9949940,"data":"product data 1","productPrice":"653.90"},
{"id":5,"productId":4940404,"data":"product data 2","productPrice":"94.12"}
]

Or, alternatively, You could split the input into individual JSON objects by hand and parse them one by one.

Streaming large JSON from input stream efficiently in Java

In short,

• your code does not work because it implements a wrong algorithm;
• JsonPath, as it has been suggested, seems to be a good DSL implementation, but it uses a DOM approach collecting entire JSON tree into memory, therefore you'll run into OOM again.

You have two solutions:

• implement a proper algorithm within your current approach (and I agree you were on a right way);
• try implementing something similar to what JsonPath implements breaking down the problem to smaller ones supporting really streaming approach.

I wouldn't document much of my code since it's pretty easy to understand and adapt to other libraries, but you can develop a more advanced thing of the following code using Java 17 (w/ preview features enabled) and javax.json (+ some Lombok for Java boilerplate):

@RequiredArgsConstructor(access = AccessLevel.PRIVATE)
public final class PathJsonParser
        implements JsonParser, Iterator<JsonParser.Event> {

    private static final int DEFAULT_PATH_LENGTH = 32;

    private final JsonParser jsonParser;
    private final AbstractPathElement[] path;
    private int last;

    public static PathJsonParser create(final JsonParser jsonParser) {
        final int maxPathLength = DEFAULT_PATH_LENGTH;
        final PathJsonParser pathJsonParser = new PathJsonParser(jsonParser, new AbstractPathElement[maxPathLength]);
        pathJsonParser.path[0] = AbstractPathElement.Root.instance;
        for ( int i = 1; i < maxPathLength; i++ ) {
            pathJsonParser.path[i] = new AbstractPathElement.Container();
        }
        return pathJsonParser;
    }

    @Override
    public Event next() {
        final Event event = jsonParser.next();
        switch ( event ) {
        case START_ARRAY -> {
            path[last].tryIncreaseIndex();
            path[++last].reset(JsonValue.ValueType.ARRAY);
        }
        case START_OBJECT -> {
            path[last].tryIncreaseIndex();
            path[++last].reset(JsonValue.ValueType.OBJECT);
        }
        case KEY_NAME -> path[last].setKeyName(jsonParser.getString());
        case VALUE_STRING -> path[last].tryIncreaseIndex();
        case VALUE_NUMBER -> path[last].tryIncreaseIndex();
        case VALUE_TRUE -> path[last].tryIncreaseIndex();
        case VALUE_FALSE -> path[last].tryIncreaseIndex();
        case VALUE_NULL -> path[last].tryIncreaseIndex();
        case END_OBJECT -> --last;
        case END_ARRAY -> --last;
        default -> throw new AssertionError(event);
        }
        return event;
    }

    public boolean matchesRoot(final int at) {
        @Nullable
        final AbstractPathElement e = tryElementAt(at);
        return e != null && e.matchesRoot();
    }

    public boolean matchesIndex(final int at, final IntPredicate predicate) {
        @Nullable
        final AbstractPathElement e = tryElementAt(at);
        return e != null && e.matchesIndex(predicate);
    }

    public boolean matchesName(final int at, final Predicate<? super String> predicate) {
        @Nullable
        final AbstractPathElement e = tryElementAt(at);
        return e != null && e.matchesName(predicate);
    }

    // @formatter:off
    @Override public boolean hasNext() { return jsonParser.hasNext(); }
    @Override public String getString() { return jsonParser.getString(); }
    @Override public boolean isIntegralNumber() { return jsonParser.isIntegralNumber(); }
    @Override public int getInt() { return jsonParser.getInt(); }
    @Override public long getLong() { return jsonParser.getLong(); }
    @Override public BigDecimal getBigDecimal() { return jsonParser.getBigDecimal(); }
    @Override public JsonLocation getLocation() { return jsonParser.getLocation(); }
    @Override @SuppressWarnings("MethodDoesntCallSuperMethod") public JsonObject getObject() { return jsonParser.getObject(); }
    @Override @SuppressWarnings("MethodDoesntCallSuperMethod") public JsonValue getValue() { return jsonParser.getValue(); }
    @Override @SuppressWarnings("MethodDoesntCallSuperMethod") public JsonArray getArray() { return jsonParser.getArray(); }
    @Override @SuppressWarnings("MethodDoesntCallSuperMethod") public Stream<JsonValue> getArrayStream() { return jsonParser.getArrayStream(); }
    @Override @SuppressWarnings("MethodDoesntCallSuperMethod") public Stream<Map.Entry<String, JsonValue>> getObjectStream() { return jsonParser.getObjectStream(); }
    @Override @SuppressWarnings("MethodDoesntCallSuperMethod") public Stream<JsonValue> getValueStream() { return jsonParser.getValueStream(); }
    @Override @SuppressWarnings("MethodDoesntCallSuperMethod") public void skipArray() { jsonParser.skipArray(); }
    @Override @SuppressWarnings("MethodDoesntCallSuperMethod") public void skipObject() { jsonParser.skipObject(); }
    @Override public void close() { jsonParser.close(); }
    // @formatter:on

    @Nullable
    private AbstractPathElement tryElementAt(final int at) {
        final int pathAt;
        if ( at >= 0 ) {
            pathAt = at;
        } else {
            pathAt = last + at + 1;
        }
        if ( pathAt < 0 || pathAt > last ) {
            return null;
        }
        return path[pathAt];
    }

    private abstract static sealed class AbstractPathElement
            permits AbstractPathElement.Root, AbstractPathElement.Container {

        abstract void reset(JsonValue.ValueType valueType);

        abstract void setKeyName(String keyName);

        abstract void tryIncreaseIndex();

        abstract boolean matchesRoot();

        abstract boolean matchesIndex(IntPredicate predicate);

        abstract boolean matchesName(Predicate<? super String> predicate);

        @RequiredArgsConstructor(access = AccessLevel.PRIVATE)
        private static final class Root
                extends AbstractPathElement {

            private static final AbstractPathElement instance = new Root();

            @Override
            void reset(final JsonValue.ValueType valueType) {
                throw new UnsupportedOperationException();
            }

            @Override
            void setKeyName(final String keyName) {
                throw new UnsupportedOperationException();
            }

            @Override
            void tryIncreaseIndex() {
                // do nothing
            }

            @Override
            boolean matchesRoot() {
                return true;
            }

            @Override
            boolean matchesIndex(final IntPredicate predicate) {
                return false;
            }

            @Override
            boolean matchesName(final Predicate<? super String> predicate) {
                return false;
            }

        }

        @RequiredArgsConstructor(access = AccessLevel.PACKAGE)
        private static final class Container
                extends AbstractPathElement {

            private static final String NO_KEY_NAME = null;
            private static final int NO_INDEX = -1;

            private JsonValue.ValueType valueType;
            private String keyName = NO_KEY_NAME;
            private int index = NO_INDEX;

            @Override
            void reset(final JsonValue.ValueType valueType) {
                this.valueType = valueType;
                keyName = NO_KEY_NAME;
                index = NO_INDEX;
            }

            @Override
            void setKeyName(final String keyName) {
                this.keyName = keyName;
            }

            @Override
            void tryIncreaseIndex() {
                if ( valueType == JsonValue.ValueType.ARRAY ) {
                    index++;
                }
            }

            @Override
            boolean matchesRoot() {
                return false;
            }

            @Override
            boolean matchesIndex(final IntPredicate predicate) {
                return switch ( valueType ) {
                    case ARRAY -> index != NO_INDEX && predicate.test(index);
                    case OBJECT -> false;
                    case STRING, NUMBER, TRUE, FALSE, NULL -> throw new AssertionError(valueType);
                };
            }

            @Override
            boolean matchesName(final Predicate<? super String> predicate) {
                return switch ( valueType ) {
                    case ARRAY -> false;
                    case OBJECT -> !Objects.equals(keyName, NO_KEY_NAME) && predicate.test(keyName);
                    case STRING, NUMBER, TRUE, FALSE, NULL -> throw new AssertionError(valueType);
                };
            }

        }

    }

}

Example of use:

public final class PathJsonParserTest {

    // $.files.0.content.0.fileContent.subList.0.subList.0.text
    private static boolean matches(final PathJsonParser parser) {
        return parser.matchesName(-1, name -> name.equals("text"))
                && parser.matchesIndex(-2, index -> true)
                && parser.matchesName(-3, name -> name.equals("subList"))
                && parser.matchesIndex(-4, index -> true)
                && parser.matchesName(-5, name -> name.equals("subList"))
                && parser.matchesName(-6, name -> name.equals("fileContent"))
                && parser.matchesIndex(-7, index -> true)
                && parser.matchesName(-8, name -> name.equals("content"))
                && parser.matchesIndex(-9, index -> true)
                && parser.matchesName(-10, name -> name.equals("files"))
                && parser.matchesRoot(-11);
    }

    @Test
    public void test()
            throws IOException {
        try ( final PathJsonParser parser = PathJsonParser.create(JsonParsers.openFromResource(PathJsonParserTest.class, "input.json")) ) {
            for ( ; parser.hasNext(); parser.next() ) {
                if ( matches(parser) ) {
                    parser.next();
                    System.out.println(parser.getValue());
                }
            }
        }
    }

}

Of course, not that cool-looking as JsonPath is, but you can do the following:

• implement a matcher builder API to make it look nicer;
• implement a JSON Path-compliant parser to build matchers;
• wrap the for/if/next() pattern into a generic algorithm (similar to what BufferedReader.readLine() implements or wrap it for Stream API);
• implement some kind of simple JSON-to-objects deserializer.

Or, if possible, find a good code generator that can generate a streamed parser having as small runtime cost as possible (its outcome would be very similar to yours, but working). (Ping me please if you are aware of any.)

Convert InputStream into JSON

Make use of Jackson JSON parser.

Refer - Jackson Home

The only thing you need to do -

ObjectMapper mapper = new ObjectMapper();
Map<String, Object> jsonMap = mapper.readValue(inputStream, Map.class);

Now jsonMap will contain the JSON.

Java read JSON input stream

Wrap it with a BufferedReader and start reading the data from it:

StringBuilder sb = new StringBuilder();
try (BufferedReader br = new BufferedReader(new InputStreamReader(socket.getInputStream()))) {
    String line;
    while ( (line = br.readLine()) != null) {
        sb.append(line).append(System.lineSeparator());
    }
    String content = sb.toString();
    //as example, you can see the content in console output
    System.out.println(content);
}

Once you have it as a String, parse it with a library like Gson or Jackson.

---

Related Topics