JSONencoder Won't Allow Type Encoded to Primitive Value

JSONEncoder won't allow type encoded to primitive value

There is a bug report for this:

https://bugs.swift.org/browse/SR-6163

SR-6163: JSONDecoder cannot decode RFC 7159 JSON

Basically, since RFC-7159, a value like 123 is valid JSON, but JSONDecoder won't support it. You may follow up on the bug report to see any future fixes on this. [The bug was fixed starting in iOS 13.]

#Where it fails#

It fails in the following line of code, where you can see that if the object is not an array nor dictionary, it will fail:

https://github.com/apple/swift-corelibs-foundation/blob/master/Foundation/JSONSerialization.swift#L120

open class JSONSerialization : NSObject {
        //...

        // top level object must be an Swift.Array or Swift.Dictionary
        guard obj is [Any?] || obj is [String: Any?] else {
            return false
        }

        //...
} 

#Workaround#

You may use JSONSerialization, with the option: .allowFragments:

let jsonText = "123"
let data = Data(jsonText.utf8)

do {
    let myString = try JSONSerialization.jsonObject(with: data, options: .allowFragments)
    print(myString)
}
catch {
    print(error)
}

Encoding to key-value pairs

Finally, you could also have your JSON objects look like this:

{ "integer": 123456 }

or

{ "string": "potatoe" }

For this, you would need to do something like this:

import Foundation 

enum MyValue {
    case integer(Int)
    case string(String)
}

extension MyValue: Codable {
    
    enum CodingError: Error { 
        case decoding(String) 
    }
    
    enum CodableKeys: String, CodingKey { 
        case integer
        case string 
    }

    init(from decoder: Decoder) throws {

        let values = try decoder.container(keyedBy: CodableKeys.self)

        if let integer = try? values.decode(Int.self, forKey: .integer) {
            self = .integer(integer)
            return
        }

        if let string = try? values.decode(String.self, forKey: .string) {
            self = .string(string)
            return
        }

        throw CodingError.decoding("Decoding Failed")
    }

    func encode(to encoder: Encoder) throws {
        var container = encoder.container(keyedBy: CodableKeys.self)

        switch self {
            case let .integer(i):
            try container.encode(i, forKey: .integer)
            case let .string(s):
            try container.encode(s, forKey: .string)
        }
    }

}

let theEncodedValue = try! JSONEncoder().encode(MyValue.integer(123456))
let theEncodedString = String(data: theEncodedValue, encoding: .utf8)
print(theEncodedString!) // { "integer": 123456 }
let theDecodedValue = try! JSONDecoder().decode(MyValue.self, from: theEncodedValue)

Swift 4 decode simple root level json value

It works with good ol' JSONSerialization and the .allowFragments
reading option. From the documentation:

allowFragments

Specifies that the parser should allow top-level objects that are not an instance of NSArray or NSDictionary.

Example:

let json = "22".data(using: .utf8)!

if let value = (try? JSONSerialization.jsonObject(with: json, options: .allowFragments)) as? Int {
    print(value) // 22
}

However, JSONDecoder has no such option and does not accept top-level
objects which are not arrays or dictionaries. One can see in the
source code that the decode() method calls
JSONSerialization.jsonObject() without any option:

open func decode<T : Decodable>(_ type: T.Type, from data: Data) throws -> T {
    let topLevel: Any
    do {
       topLevel = try JSONSerialization.jsonObject(with: data)
    } catch {
        throw DecodingError.dataCorrupted(DecodingError.Context(codingPath: [], debugDescription: "The given data was not valid JSON.", underlyingError: error))
    }

    // ...

    return value
}

How to use Any in Codable Type

Codable needs to know the type to cast to.

Firstly I would try to address the issue of not knowing the type, see if you can fix that and make it simpler.

Otherwise the only way I can think of solving your issue currently is to use generics like below.

struct Person<T> {
    var id: T
    var name: String
}

let person1 = Person<Int>(id: 1, name: "John")
let person2 = Person<String>(id: "two", name: "Steve")

UserDefault property wrapper not saving values iOS versions below iOS 13

Nothing to do with property wrappers! The problem is that in iOS 12 and before, a simple value like a Bool (or String, etc.), though Codable as a property of a Codable struct (for example), cannot itself be JSON encoded. The error (which you are throwing away) is quite clear about this:

Top-level Bool encoded as number JSON fragment.

To see this, just run this code:

    do {
        _ = try JSONEncoder().encode(false)
        print("succeeded")
    } catch {
        print(error)
    }

On iOS 12, we get the error. On iOS 13, we get "succeeded".

But if we wrap our Bool (or String, etc.) in a Codable struct, all is well:

    struct S : Codable { let prop : Bool }
    do {
        _ = try JSONEncoder().encode(S(prop:false))
        print("succeeded")
    } catch {
        print(error)
    }

That works fine on both iOS 12 and iOS 13.

And that fact suggests a solution! Redefine your property wrapper so that it wraps its value in a generic Wrapper struct:

struct UserDefaultWrapper<T: Codable> {

    struct Wrapper<T> : Codable where T : Codable {
        let wrapped : T
    }

    private let key: String
    private let defaultValue: T

    init(key: String, defaultValue: T) {
        self.key = key
        self.defaultValue = defaultValue
    }

    var wrappedValue: T {
        get {
            guard let data = UserDefaults.standard.object(forKey: key) as? Data 
                else { return defaultValue }
            let value = try? JSONDecoder().decode(Wrapper<T>.self, from: data)
            return value?.wrapped ?? defaultValue
        }
        set {
            do {
                let data = try JSONEncoder().encode(Wrapper(wrapped:newValue))
                UserDefaults.standard.set(data, forKey: key)
            } catch {
                print(error)
            }
        }
    }
}

Now it works on iOS 12 and iOS 13.

---

By the way, I actually think you would do better to save as a property list rather than JSON. But that makes no difference to the question generally. You can’t encode a bare Bool as a property list either. You’d still need the Wrapper approach.

How to JSON serialize sets?

JSON notation has only a handful of native datatypes (objects, arrays, strings, numbers, booleans, and null), so anything serialized in JSON needs to be expressed as one of these types.

As shown in the json module docs, this conversion can be done automatically by a JSONEncoder and JSONDecoder, but then you would be giving up some other structure you might need (if you convert sets to a list, then you lose the ability to recover regular lists; if you convert sets to a dictionary using dict.fromkeys(s) then you lose the ability to recover dictionaries).

A more sophisticated solution is to build-out a custom type that can coexist with other native JSON types. This lets you store nested structures that include lists, sets, dicts, decimals, datetime objects, etc.:

from json import dumps, loads, JSONEncoder, JSONDecoder
import pickle

class PythonObjectEncoder(JSONEncoder):
    def default(self, obj):
        try:
            return {'_python_object': pickle.dumps(obj).decode('latin-1')}
        except pickle.PickleError:
            return super().default(obj)

def as_python_object(dct):
    if '_python_object' in dct:
        return pickle.loads(dct['_python_object'].encode('latin-1'))
    return dct

Here is a sample session showing that it can handle lists, dicts, and sets:

>>> data = [1,2,3, set(['knights', 'who', 'say', 'ni']), {'key':'value'}, Decimal('3.14')]

>>> j = dumps(data, cls=PythonObjectEncoder)

>>> loads(j, object_hook=as_python_object)
[1, 2, 3, set(['knights', 'say', 'who', 'ni']), {'key': 'value'}, Decimal('3.14')]

Alternatively, it may be useful to use a more general purpose serialization technique such as YAML, Twisted Jelly, or Python's pickle module. These each support a much greater range of datatypes.

How to make json.dumps in Python ignore a non-serializable field

Keys with a leading _ underscore are not really 'hidden', they are just more strings to JSON. The Construct Container class is just a dictionary with ordering, the _io key is not anything special to that class.

You have two options:

• implement a default hook that just returns a replacement value.
• Filter out the key-value pairs that you know can't work before serialising.

and perhaps a third, but a casual scan of the Construct project pages doesn't tell me if it is available: have Construct output JSON or at least a JSON-compatible dictionary, perhaps by using adapters.

The default hook can't prevent the _io key from being added to the output, but would let you at least avoid the error:

json.dumps(packet, default=lambda o: '<not serializable>')

Filtering can be done recursively; the @functools.singledispatch() decorator can help keep such code clean:

from functools import singledispatch

_cant_serialize = object()

@singledispatch
def json_serializable(object, skip_underscore=False):
    """Filter a Python object to only include serializable object types

    In dictionaries, keys are converted to strings; if skip_underscore is true
    then keys starting with an underscore ("_") are skipped.

    """
    # default handler, called for anything without a specific
    # type registration.
    return _cant_serialize

@json_serializable.register(dict)
def _handle_dict(d, skip_underscore=False):
    converted = ((str(k), json_serializable(v, skip_underscore))
                 for k, v in d.items())
    if skip_underscore:
        converted = ((k, v) for k, v in converted if k[:1] != '_')
    return {k: v for k, v in converted if v is not _cant_serialize}

@json_serializable.register(list)
@json_serializable.register(tuple)
def _handle_sequence(seq, skip_underscore=False):
    converted = (json_serializable(v, skip_underscore) for v in seq)
    return [v for v in converted if v is not _cant_serialize]

@json_serializable.register(int)
@json_serializable.register(float)
@json_serializable.register(str)
@json_serializable.register(bool)  # redudant, supported as int subclass
@json_serializable.register(type(None))
def _handle_default_scalar_types(value, skip_underscore=False):
    return value

I have the above implementation an additional skip_underscore argument too, to explicitly skip keys that have a _ character at the start. This would help skip all additional 'hidden' attributes the Construct library is using.

Since Container is a dict subclass, the above code will automatically handle instances such as packet.

How to make a class JSON serializable

Do you have an idea about the expected output? For example, will this do?

>>> f  = FileItem("/foo/bar")
>>> magic(f)
'{"fname": "/foo/bar"}'

In that case you can merely call json.dumps(f.__dict__).

If you want more customized output then you will have to subclass JSONEncoder and implement your own custom serialization.

For a trivial example, see below.

>>> from json import JSONEncoder
>>> class MyEncoder(JSONEncoder):
        def default(self, o):
            return o.__dict__    

>>> MyEncoder().encode(f)
'{"fname": "/foo/bar"}'

Then you pass this class into the json.dumps() method as cls kwarg:

json.dumps(cls=MyEncoder)

If you also want to decode then you'll have to supply a custom object_hook to the JSONDecoder class. For example:

>>> def from_json(json_object):
        if 'fname' in json_object:
            return FileItem(json_object['fname'])
>>> f = JSONDecoder(object_hook = from_json).decode('{"fname": "/foo/bar"}')
>>> f
<__main__.FileItem object at 0x9337fac>
>>> 

---

Related Topics