How to Decode JSON with Unknown Field Using Gson

How to decode JSON with unknown field using Gson?

(After OP commented that in fact the JSON looks like this, I completely updated the answer.)

Solution for Gson 2.0+

I just learned that with newer Gson versions this is extremely simple:

GsonBuilder builder = new GsonBuilder();
Object o = builder.create().fromJson(json, Object.class);

The created object is a Map (com.google.gson.internal.LinkedTreeMap), and if you print it, it looks like this:

{1145={cities_id=1145, city=Nawanshahr, city_path=nawanshahr, region_id=53, region_district_id=381, country_id=0, million=0, population=null, region_name=Punjab}, 
1148={cities_id=1148, city=Nimbahera, city_path=nimbahera, region_id=54, region_district_id=528, country_id=0, million=0, population=null, region_name=Rajasthan}
...

Solution using a custom deserialiser

(NB: Turns out you don't really a custom deserialiser unless you're stuck with pre-2.0 versions of Gson. But still it is useful to know how to do custom deserialisation (and serialisation) in Gson, and it may often be the best approach, depending on how you want to use the parsed data.)

So we're indeed dealing with random / varying field names. (Of course, this JSON format is not very good; this kind of data should be inside a JSON array, in which case it could be very easily read into a List. Oh well, we can still parse this.)

First, this is how I would model the JSON data in Java objects:

// info for individual city
public class City {
String citiesId;
String city;
String regionName;
// and so on
}

// top-level object, containing info for lots of cities
public class CityList {
List<City> cities;

public CityList(List<City> cities) {
this.cities = cities;
}
}

Then, the parsing. One way to deal with this kind of JSON is to create a custom deserialiser for the top level object (CityList).

Something like this:

public class CityListDeserializer implements JsonDeserializer<CityList> {

@Override
public CityList deserialize(JsonElement element, Type type, JsonDeserializationContext context) throws JsonParseException {
JsonObject jsonObject = element.getAsJsonObject();
List<City> cities = new ArrayList<City>();
for (Map.Entry<String, JsonElement> entry : jsonObject.entrySet()) {
// For individual City objects, we can use default deserialisation:
City city = context.deserialize(entry.getValue(), City.class);
cities.add(city);
}
return new CityList(cities);
}

}

A key point to notice is the call to jsonObject.entrySet() which retuns all the top-level fields (with names like "1145", "1148", etc). This Stack Overflow answer helped me solve this.

Complete parsing code below. Note that you need to use registerTypeAdapter() to register the custom serialiser.

GsonBuilder builder = new GsonBuilder();
builder.registerTypeAdapter(CityList.class, new CityListDeserializer());
Gson gson = builder.setFieldNamingPolicy(LOWER_CASE_WITH_UNDERSCORES).create();
CityList list = gson.fromJson(json, CityList.class);

(Here's a full, executable example that I used for testing. Besides Gson, it uses Guava library.)

Using the Gson library to parse unknown data dynamically

You can simply use java.util.Map that is an associative key/value container where keys and values are arbitrary objects, and can align with JSON dynamic objects using Gson really straight-forward. You just have to define appropriate mappings (I made the field collapsed to save some visual space):

final class Response {
@SerializedName("schools") final HomeSchool school = null;
@SerializedName("school") final Map<Integer, School> schools = null;
}

final class HomeSchool {
@SerializedName("home") final int home = Integer.valueOf(0);
@SerializedName("statelevel") final int stateLevel = Integer.valueOf(0);
}

final class School {
@SerializedName("schoolname") final String name = null;
@SerializedName("nameshort") final String shortName = null;
@SerializedName("students") final Map<Integer, Student> students = null;
}

final class Student {
@SerializedName("name") final String name = null;
@SerializedName("isCR") final boolean isCr = Boolean.valueOf(false);
@SerializedName("maths") final Maths maths = null;
}

final class Maths {
@SerializedName("score") final int score = Integer.valueOf(0);
@SerializedName("lastscore") final int lastScore = Integer.valueOf(0);
}

Now, once you have the mappings, you can easily deserialize your JSON:

private static final Gson gson = new Gson();

public static void main(final String... args) {
final Response response = gson.fromJson(JSON, Response.class);
for ( final Entry<Integer, School> schoolEntry : response.schools.entrySet() ) {
final School school = schoolEntry.getValue();
System.out.println(schoolEntry.getKey() + " " + school.name);
for ( final Entry<Integer, Student> studentEntry : school.students.entrySet() ) {
final Student student = studentEntry.getValue();
System.out.println("\t" + studentEntry.getKey()
+ " " + student.name
+ " CR:" + (student.isCr ? "+" : "-")
+ " (" + student.maths.score + ", " + student.maths.lastScore + ")"
);
}
}
}

1103 Indira School

2201 Ritesh CR:+ (95, 86)

2202 Sanket CR:- (98, 90)

2203 Ajit CR:- (94, 87)

1348 Patil School

3201 Ravi CR:- (95, 86)

3202 Raj CR:+ (98, 90)

3203 Ram CR:- (94, 87)

The type token suggestions are partially correct: they are used to deserialize the objects you cannot or don't have concrete mappings for, like lists of something, or maps of strings to something. In your case Gson simply analyzes the field declarations to resolve the map types (both keys and values).

Java: Parsing Json with Gson for unknown field

If the name of fields are not known in advance then convert it into Map<String, Object> using TypeToken

String str = "{\"initial_obj\": {\"random_name1\": { },\"random_name50\": { },\"random_name9\": { }}}";
Type type = new TypeToken<Map<String, Object>>() {}.getType();
Map<String, Object> data = new Gson().fromJson(str, type);
System.out.println(new GsonBuilder().setPrettyPrinting().create().toJson(data));

output:

{
"initial_obj": {
"random_name1": {},
"random_name50": {},
"random_name9": {}
}
}

Trouble parsing a JSON string with unknown keys using Retrofit and Gson

Try with the following code.

data class NameNW(
val fullname: String,
val nickname: String
)

data class NamesListNWEntity(
val action: String,
val list: Map<String, NameNW>,
val status: String
)

// You can also test with static json string
private fun test() {
val apiResponse = "{\n" +
" \"list\": {\n" +
" \"14\": {\n" +
" \"nickname\": \"Dave\",\n" +
" \"fullname\": \"David Collins\"\n" +
" },\n" +
" \"68\": {\n" +
" \"nickname\": \"Phil\",\n" +
" \"fullname\": \"Phillip Jordan\"\n" +
" }\n" +
" },\n" +
" \"action\": \"load\",\n" +
" \"status\": \"LOADED\"\n" +
"}"

val namesListNWEntity: NamesListNWEntity = Gson().fromJson(apiResponse, NamesListNWEntity::class.java)
}

Java: Deserialize nested JSON of unknown spec to a polymorphic Hashtable

In Gson it's not possible to override the Object type adapter, so there is no a direct way to deserialize strings, lists and maps under the common type. However, you can pretend they three can have a common super type cheating the type system by introducing a marker type, for example an annotation that only marks something to have in common. The fake common type marker will only work for the top-level fromJson call and won't work for really strongly typed fields, however you don't need them as you mentioned in your question.

@Target({})
@Retention(RetentionPolicy.RUNTIME)
public @interface Poly {
}
public final class PolyTypeAdapterFactory
implements TypeAdapterFactory {

private static final TypeAdapterFactory defaultInstance = new PolyTypeAdapterFactory(ArrayList::new, LinkedHashMap::new);

private final Supplier<? extends List<? super Object>> createList;
private final Supplier<? extends Map<String, ? super Object>> createMap;

private PolyTypeAdapterFactory(final Supplier<? extends List<? super Object>> createList, final Supplier<? extends Map<String, ? super Object>> createMap) {
this.createList = createList;
this.createMap = createMap;
}

public static TypeAdapterFactory getDefaultInstance() {
return defaultInstance;
}

@Override
@Nullable
public <T> TypeAdapter<T> create(final Gson gson, final TypeToken<T> typeToken) {
if ( typeToken.getRawType() != Poly.class ) {
return null;
}
final TypeAdapter<Object> polyTypeAdapter = new TypeAdapter<Object>() {
@Override
public void write(final JsonWriter out, final Object o) {
throw new UnsupportedOperationException();
}

@Override
public Object read(final JsonReader in)
throws IOException {
final JsonToken token = in.peek();
switch ( token ) {
case BEGIN_ARRAY:
return readList(in);
case BEGIN_OBJECT:
return readMap(in);
case STRING:
case NUMBER:
case BOOLEAN:
return readString(in);
case NULL:
return readNull(in);
case NAME:
case END_ARRAY:
case END_OBJECT:
throw new AssertionError("Unexpected token: " + token + " at " + in);
case END_DOCUMENT:
throw new UnsupportedOperationException();
default:
throw new AssertionError(token);
}
}

@Nullable
private Object readNull(final JsonReader in)
throws IOException {
in.nextNull();
return null;
}

private String readString(final JsonReader in)
throws IOException {
return in.nextString();
}

private Map<String, Object> readMap(final JsonReader in)
throws IOException {
in.beginObject();
final Map<String, Object> map = createMap.get();
while ( in.hasNext() ) {
final String key = in.nextName();
if ( map.containsKey(key) ) {
throw new IllegalStateException("Duplicate key: " + key);
}
@Nullable
final Object value = read(in);
if ( value != null ) {
map.put(key, value);
}
}
in.endObject();
return map;
}

private List<Object> readList(final JsonReader in)
throws IOException {
in.beginArray();
final List<Object> list = createList.get();
while ( in.hasNext() ) {
final Object element = read(in);
list.add(element);
}
in.endArray();
return list;
}
}
.nullSafe();
@SuppressWarnings("unchecked")
final TypeAdapter<T> typeAdapter = (TypeAdapter<T>) polyTypeAdapter;
return typeAdapter;
}

}

The type adapter above checks if the given type is the fake super type and then deserializes to three types depending on actual JSON tokens:

  • primitive strings for every simple literal (Gson nextString coerces numbers and booleans);
  • maps and lists for containers, recursively.

I slightly modified your JSON to show that numbers will be coerced to strings:

{
"k1": {
"k2": [
"1",
2
],
"k3": {
"k4": "v4",
"k5": "v5"
},
"k6": "v6"
}
}

The following test will pass:

public final class PolyTypeAdapterFactoryTest {

private static final Gson gson = new GsonBuilder()
.disableHtmlEscaping()
.disableInnerClassSerialization()
.registerTypeAdapterFactory(PolyTypeAdapterFactory.getDefaultInstance())
.create();

@Test
public void test()
throws IOException {
try ( final JsonReader jsonReader = new JsonReader(...) ) {
// This is where cheating happens: we tell Gson to apply the Poly handler,
// but it returns a Map. Well, let it be for the top-most level...
final Map<String, ?> actual = gson.fromJson(jsonReader, Poly.class);
final Map<String, ?> expected = ImmutableMap.of(
"k1", ImmutableMap.of(
"k2", ImmutableList.of("1", "2"),
"k3", ImmutableMap.of(
"k4", "v4",
"k5", "v5"
),
"k6", "v6"
)
);
Assertions.assertEquals(expected, actual);
}
}

}

No Gson but org.json solution

As I mentioned in the comments, you can avoid using Gson de/serialization facilities, and deserialize the given JSON fully manually:

public final class OrgJsonPolyReader {

private OrgJsonPolyReader() {
}

@Nullable
public static Object read(final JSONTokener jsonTokener) {
return read(jsonTokener, ArrayList::new, LinkedHashMap::new);
}

@Nullable
public static Object read(final JSONTokener jsonTokener, final Supplier<? extends List<? super Object>> createList,
final Supplier<? extends Map<? super String, ? super Object>> createMap) {
while ( jsonTokener.more() ) {
final char token = jsonTokener.nextClean();
switch ( token ) {
case 'n':
jsonTokener.back();
return readNull(jsonTokener);
case 'f':
case 't':
case '0':
case '1':
case '2':
case '3':
case '4':
case '5':
case '6':
case '7':
case '8':
case '9':
jsonTokener.back();
return readLiteral(jsonTokener);
case '"':
jsonTokener.back();
return readString(jsonTokener);
case '[':
jsonTokener.back();
return readList(jsonTokener, createList, createMap);
case '{':
jsonTokener.back();
return readMap(jsonTokener, createList, createMap);
default:
throw new IllegalStateException("Unexpected token: " + token);
}
}
throw new AssertionError();
}

@Nullable
private static <T> T readNull(final JSONTokener jsonTokener) {
final Object value = jsonTokener.nextValue();
assert value.equals(JSONObject.NULL);
return null;
}

private static String readLiteral(final JSONTokener jsonTokener) {
return jsonTokener.nextValue().toString();
}

private static String readString(final JSONTokener jsonTokener) {
jsonTokener.next('"');
return jsonTokener.nextString('"');
}

private static List<Object> readList(final JSONTokener jsonTokener, final Supplier<? extends List<? super Object>> createList,
final Supplier<? extends Map<? super String, ? super Object>> createMap) {
jsonTokener.next('[');
final List<? super Object> list = createList.get();
for ( ; ; ) {
final char token = jsonTokener.nextClean();
switch ( token ) {
case ']':
return list;
case ',':
break;
default:
jsonTokener.back();
final Object value = read(jsonTokener, createList, createMap);
list.add(value);
break;
}
}
}

private static Map<? super String, Object> readMap(final JSONTokener jsonTokener, final Supplier<? extends List<? super Object>> createList,
final Supplier<? extends Map<? super String, ? super Object>> createMap) {
jsonTokener.next('{');
final Map<? super String, ? super Object> map = createMap.get();
for ( ; ; ) {
final char token = jsonTokener.nextClean();
switch ( token ) {
case '}':
return map;
case ',':
break;
case '"':
final String key = jsonTokener.nextString(token);
if ( map.containsKey(key) ) {
throw new IllegalStateException("Duplicate key: " + key);
}
jsonTokener.next(':');
final Object value = read(jsonTokener, createList, createMap);
map.put(key, value);
break;
default:
throw new IllegalStateException("Unexpected token: " + token);
}
}
}

}

And this is how it can be added to Retrofit instead of the Gson converter factory:

.addConverterFactory(new Converter.Factory() {
@Override
public Converter<ResponseBody, ?> responseBodyConverter(final Type type, final Annotation[] annotations, final Retrofit retrofit) {
return responseBody -> {
try ( final Reader reader = new InputStreamReader(responseBody.byteStream()) ) {
return OrgJsonPolyReader.read(new JSONTokener(reader));
}
};
}
})


Related Topics



Leave a reply



Submit