Line Delimited Json Serializing and De-Serializing

Line delimited json serializing and de-serializing

You can do so by manually parsing your JSON using JsonTextReader and setting the SupportMultipleContent flag to true.

If we look at your first example, and create a POCO called Foo:

public class Foo
{
    [JsonProperty("some")]
    public string Some { get; set; }
}

This is how we parse it:

var json = "{\"some\":\"thing1\"}\r\n{\"some\":\"thing2\"}\r\n{\"some\":\"thing3\"}";
var jsonReader = new JsonTextReader(new StringReader(json))
{
    SupportMultipleContent = true // This is important!
};

var jsonSerializer = new JsonSerializer();
while (jsonReader.Read())
{
    Foo foo = jsonSerializer.Deserialize<Foo>(jsonReader);
}

If you want list of items as result simply add each item to a list inside the while loop to your list.

listOfFoo.Add(jsonSerializer.Deserialize<Foo>(jsonReader));

Note: with Json.Net 10.0.4 and later same code also supports comma separated JSON entries see How to deserialize dodgy JSON (with improperly quoted strings, and missing brackets)?)

Deserialize JSON from a large file

If you cant change the json file do what @dbc is saying and use the line delimited json parsing.

if you can then format it into a proper json array by surrounding it with []. Then change your code to:

var products = serializer.Deserialize(file, type(List<Product>);

Edit: changing to proper json will just save you headaches later

Deserializing JSON from a .json file and adding the members to a List object

It will depend on the format of the file.

You are trying to deserialize a list.
For that you must specify the type to be compatible. Try:

var result = JsonConvert.DeserializeObject<List<Rootobject>>(File.ReadAllText("list_city.json"));

If that doesn't work, and you have all stand alone JSON in each line, you can try:

public IEnumerable<RootObject> ReadAllCities()
{
    var lines = File.ReadAllLines("list_city.json");
    foreach (var line in lines)
    {
       yield return JsonConvert.DeserializeObject<Rootobject>(line);
    }
}

Serialize as NDJSON using Json.NET

As Json.NET does not currently have a built-in method to serialize a collection to NDJSON, the simplest answer would be to write to a single TextWriter using a separate JsonTextWriter for each line, setting CloseOutput = false for each:

public static partial class JsonExtensions
{
    public static void ToNewlineDelimitedJson<T>(Stream stream, IEnumerable<T> items)
    {
        // Let caller dispose the underlying stream 
        using (var textWriter = new StreamWriter(stream, new UTF8Encoding(false, true), 1024, true))
        {
            ToNewlineDelimitedJson(textWriter, items);
        }
    }

    public static void ToNewlineDelimitedJson<T>(TextWriter textWriter, IEnumerable<T> items)
    {
        var serializer = JsonSerializer.CreateDefault();

        foreach (var item in items)
        {
            // Formatting.None is the default; I set it here for clarity.
            using (var writer = new JsonTextWriter(textWriter) { Formatting = Formatting.None, CloseOutput = false })
            {
                serializer.Serialize(writer, item);
            }
            // https://web.archive.org/web/20180513150745/http://specs.okfnlabs.org/ndjson/
            // Each JSON text MUST conform to the [RFC7159] standard and MUST be written to the stream followed by the newline character \n (0x0A). 
            // The newline charater MAY be preceeded by a carriage return \r (0x0D). The JSON texts MUST NOT contain newlines or carriage returns.
            textWriter.Write("\n");
        }
    }
}

Sample fiddle.

Since the individual NDJSON lines are likely to be short but the number of lines might be large, this answer suggests a streaming solution to avoid the necessity of allocating a single string larger than 85kb. As explained in Newtonsoft Json.NET Performance Tips, such large strings end up on the large object heap and may subsequently degrade application performance.

Deserializing json and delimiting newlines NDJ

You can use this code with GSON library:

ObjectMapper mapr = new ObjectMapper();
Map<String,Object> map = mapr.readValue(jsonString, Map.class);

or

public class Units {
  @JsonProperty("user_name")
  public String user_name;

  @JsonProperty("status")
  public List<Attr> attrs;

  ..
}
public class Attr {
  @JsonProperty("hp") 
  public String hp;
  @JsonProperty("karma") 
  public String karma;
  @JsonProperty("mp") 
  public String mp;
}

ObjectMapper mapr = new ObjectMapper();
Units unit = mapr.readValue(jsonString, Units.class);

for both you can use jsonString to define each json unit in your file.

Deserializing newline-delimited JSON from a socket using Serde

Place the TcpStream into a BufReader. This allows you to read until a specific byte (in this case a newline). You can then parse the read bytes with Serde:

use std::io::{BufRead, BufReader};
use std::io::Write;

fn handle_client(mut stream: TcpStream) -> Result<(), Error> {
    let mut data = Vec::new();
    let mut stream = BufReader::new(stream);

    loop {
        data.clear();

        let bytes_read = stream.read_until(b'\n', &mut data)?;
        if bytes_read == 0 {
            return Ok(());
        }

        let input: Point3D = serde_json::from_slice(&data)?;
        let value = input.x.pow(2) + input.y.pow(2) + input.z.pow(2);

        write!(stream.get_mut(), "{}", value)?;
    }
}

I'm being a little fancy by reusing the allocation of data, which means it's very important to reset the buffer at the beginning of each loop. I also avoid allocating memory for the result and just print directly to the output stream.

---

Related Topics