Serialization With Qt

Serialization with Qt

QDataStream handles a variety of C++ and Qt data types. The complete list is available at http://doc.qt.io/qt-4.8/datastreamformat.html. We can also add support for our own custom types by overloading the << and >> operators. Here's the definition of a custom data type that can be used with QDataStream:

class Painting
{
public:
    Painting() { myYear = 0; }
    Painting(const QString &title, const QString &artist, int year) {
        myTitle = title;
        myArtist = artist;
        myYear = year;
    }
    void setTitle(const QString &title) { myTitle = title; }
    QString title() const { return myTitle; }
    ...
private:
    QString myTitle;
    QString myArtist;
    int myYear;
};
QDataStream &operator<<(QDataStream &out, const Painting &painting);
QDataStream &operator>>(QDataStream &in, Painting &painting);

Here's how we would implement the << operator:

QDataStream &operator<<(QDataStream &out, const Painting &painting)
{
    out << painting.title() << painting.artist()
        << quint32(painting.year());
    return out;
}

To output a Painting, we simply output two QStrings and a quint32. At the end of the function, we return the stream. This is a common C++ idiom that allows us to use a chain of << operators with an output stream. For example:

out << painting1 << painting2 << painting3;

The implementation of operator>>() is similar to that of operator<<():

QDataStream &operator>>(QDataStream &in, Painting &painting)
{
    QString title;
    QString artist;
    quint32 year;
    in >> title >> artist >> year;
    painting = Painting(title, artist, year);
    return in;
}

This is from: C++ GUI Programming with Qt 4 By Jasmin Blanchette, Mark Summerfield

How to simply serialize complex structures and send them over a network in Qt

You can and should leverage QDataStream for that. Instead from deriving from Serializable, simply implement QDataStream & operator<<(QDataStream &, Type & const) and QDataStream & operator>>(QDataStream &, Type &) for each Type you want to serialize.

You'd then simply dump all data to a QByteArray via a QDataStream. You'd transmit the size of the array, followed by its contents. On the receiving end, you receive the size, then the contents, set a datastream on it, and pull the data out. It should be seamless.

The data block separation is handled automatically for you as long as the individual streaming operators are implemented correctly. The choice of a QHash as a means of serialization is unnecessarily limiting - it may be a good choice for some classes, but not for others.

Instead of serializing into a QHash, you should be serializing into a QDataStream. But the operators for that can be free-standing functions, so you don't need to derive from any special interface class for that. The compiler will remind you of any missing operators.

This is a very simple example, working over UDP.

This is a larger example that shows the details of future-proof versioning, and shows serialization of a rather complex data structure - a QAbstractItemModel.

Generally speaking, the serialization of, say, three objects a,b,c might look as follows:

static const QDataStream::Version kDSVersion = QDataStream::Qt_5_5;

void Foo::send() {
  QByteArray buf;
  QDataStream bds(&buf, QIODevice::WriteOnly));
  bds.setVersion(kDSVersion);
  bds << a << b << c;

  QDataStream ds(socket, QIODevice::WriteOnly);
  ds.setVersion(kDSVersion);
  ds << buf; // buffer size followed by data
}

On the receiving end:

void Foo::readyReadSlot() {
  typedef quint32 QBALength;
  if (socket->bytesAvailable() < sizeof(QBALength)) return;
  auto buf = socket->peek(sizeof(QBALength));
  QDataStream sds(&buf);
  // We use a documented implementation detail:
  // See http://doc.qt.io/qt-5/datastreamformat.html
  // A QByteArray is serialized as a quint32 size followed by raw data.
  QBALength size;
  sds >> size;
  if (size == 0xFFFFFFFF) {
    // null QByteArray, discard
    socket.read(sizeof(QBALength));
    return;
  }
  if (socket->bytesAvailable() < size)  return;
  QByteArray buf;
  QDataStream bds(&socket);
  bds.setVersion(kDSVersion);
  bds >> buf;
  QDataStream ds(&buf);
  ds.setVersion(kDSVersion);
  ds >> a >> b >> c;
}

Boost serialization in Qt: is it a proper way?

QDataStream supports (de)serialization of some popular Qt objects. You can check which ones here.
The "Qt" way would be to use that.

However, there's nothing preventing you from using boost, but you will have to implement the serialization for basic objects such as QList all over again, which can be tiresome.

Note that if you have custom objects, such as your TreeModelItem, you would have to provide an operator<< of your own.

Regarding the serialization of signals/slots: afaik Qt doesn't support this atm, and I believe the Qt team has made it this way intentionally. If you're interested why, maybe this read can be helpful.

Qt data type serialization

QColor can be serialized to QDataStream directly, without any conversions. QDataStream itself, in turn, can write the data to any QIODevice sublass or to a QByteArray.

See Serializing Qt Data Types.

Example:

Serialize color to QByteArray:

QColor color(Qt::red);

QByteArray ba;
QDataStream out(&ba, QIODevice::WriteOnly);
out << color; // serialized to ba

qDebug() << ba.size();

Serialize to TCP socket:

auto socket = new QTcpSocket;
socket->connectToHost(addr, port);
if(socket->waitForConnected())
{
    QDataStream out(socket);
    out << color; // written to socket
}

Qt has universal serialization rules for major core data types. You can serialize QColor directly to a io-device you need, or to QByteArray.

---

You can also present color as a string. But it is not a serialization.

QString colorName = color.name(); // the name of the color in the format "#RRGGBB"; i.e. a "#" character followed by three two-digit hexadecimal numbers
qDebug() << colorName;

---

About changing concrete bits in a QByteArray. See how to convert QByteArray to QBitArray and vice versa: https://wiki.qt.io/Working_with_Raw_Data

Converting Bits to Bytes (and back again)

QByteArray bytes = ...;

// Create a bit array of the appropriate size
QBitArray bits(bytes.count()*8);

// Convert from QByteArray to QBitArray
for(int i=0; i<bytes.count(); ++i) {
    for(int b=0; b<8;b++) {
        bits.setBit( i*8+b, bytes.at(i)&(1<<(7-b)) );
    }
}

...

QBitArray bits = ...;

// Resulting byte array
QByteArray bytes;

// Convert from QBitArray to QByteArray
for(int b=0; b<bits.count();++b) {
    bytes[b/8] = (bytes.at(b/8) | ((bits[b]?1:0)<<(7-(b%8))));
}

---

Do not forget about Byte Order: https://doc.qt.io/qt-5/qdatastream.html#setByteOrder

Serialize a data structure in text format in C++/Qt

What do you do with this data in the database? If you want to use it in another program or language, then your data need to be readable and you can use QXmlStreamWriter/Reader or QJsonDocument (both with QByteArray).

If you don't want to use your data outside your program, you can write them in QByteArray with QDataStream.

It would be something like the code below.

QByteArray data;
QDataStream stream(&data);
stream << yourClasses;
sqlQuery.bindValue(data);

You just need to add stream operators in your classes you want to serialize:

QDataStream &operator<<(QDataStream &, A const&);
QDataStream &operator>>(QDataStream &, A &);

Qt C++ class data json serialization

First add functions to translate your objects into JSON objects.

QJsonObject Person::toJson() const
{
    QJsonObject obj;
    obj["name"] = m_name;
    obj["phone"] = m_phoneNumber;
    if (m_childData)
        obj["child"] = m_childData.toJson();
    return obj;
}

QJsonObject ChildPerson::toJson() const
{
    QJsonObject obj;
    obj["name"] = m_name;
    obj["phone"] = m_phoneNumber;
    return obj;
}

Then create the JSON like that:

QJsonDocument doc;
doc.setObject(myPerson.toJson());
QByteArray data = doc.toJson();

QFile file("save.json");
file.open(QIODevice::WriteOnly);
file.write(data);
file.close();

A few remarks:

• If the child is a person, you should try to have the same class for both to avoid duplicating code.
• Probably you could have several children? If yes, using a vector (QList/QVector) of pointers is better. The children could be added by a function instead of being created in the constructor of the parent.
• The phone number should rather be a string than an int, as an int cannot hold 0's at the beginning, and these 0's matter.

To read a Person back from the file, it could look like this:

Person myPerson;

QFile file("save.json");
if (file.open(QIODevice::ReadOnly))
{
    QByteArray content = file.readAll();
    file.close();

    QJsonDocument doc = QJsonDocument::fromJson(content); // use 2nd argument here to get the parsing error in case the input JSON is malformed
    QJsonObject personJson = doc.object();

    myPerson.fromJson(personJson);
}


void Person::fromJson(const QJsonObject& obj) const
{
    m_name = obj.value("name").toString();
    m_phoneNumber = obj.value("phone").toInt();
    if (obj.contains("child")
    {
        m_childData = new ChildPerson();
        m_childData.fromJson(obj.value("child").toObject());
    }
}

void ChildPerson::fromJson(const QJsonObject& obj) const
{
    m_name = obj.value("name").toString();
    m_phoneNumber = obj.value("phone").toInt(); // or toString()
}

More remarks:

• The fromJson() function should rather return a bool: true if the JSON contained the expected data, false otherwise. You can check whether a field exists in the JSON object with contains(QString key).

Serializing/parsing multiple objects in one file in Qt C++

You can have an array of json object, each of them having an ID so you can parse the relevant ones.

Although you could also parse all of them and add them in a map, as long as you don't have very heavy files it should be fine.

void parseJson(const QString &data)
{
    QJsonDocument doc = QJsonDocument::fromJson(data.toUtf8());

    if (doc.isNull())
    {
        war("invalid json document");
        return;
    }


    QJsonArray jsonArray = doc.array();
    foreach (const QJsonValue & value, jsonArray) {
        QJsonObject obj = value.toObject();
        if (obj.contains("id"))
        {
            if (obj["id"].toInt() == yourId) parseObject(obj);
        }
    }
}


void parseObject(const QJsonObject &obj)
{
    if (obj.contains("valueA")) valueA = obj["valueA"].toDouble();
    if (obj.contains("valueB")) valueB = obj["valueB"].toDouble();
}

This will work just fine if your file is not too big

---

Bigger Files

Now if you have very large file, it might be an issue to load it all in memory and parse it.

Since your structure is always the same and quite simple, JSON might not be the best choice, one more efficient method would be to do your own parser (or use probably some existing ones) that could read the file and process it as a stream.

---

Another method, would be to have one JSON entry per line preceded by an ID with a fixed number of digit. Load this in a QHash lookup and then only read id of interest from the file and only parse a small section.

// This code is not tested and is just to show the principle.
#define IDSIZE 5
QHash<int64, int64> m_lookup; // has to be global var

// For very large file, this might take some time and can be done on a separate thread.
// it needs to be done only once at startup (given the file is not modified externally)
void createLookup(const QString &fileName)
{
    QFile inputFile(fileName);
    if (inputFile.open(QIODevice::ReadOnly))
    {
       QTextStream in(&inputFile);
       while (!in.atEnd())
       {
          int position = in.pos(); // store the position in the file
          QString line = in.readLine();
          int id = line.mid(0,IDSIZE).toInt(); // 5 digit id (like 00001, 00002, etc...
          m_lookup[id] = position + IDSIZE;
       }
       inputFile.close();
    }
}

QString getEntry(const QString &fileName, int64 id)
{
     if (m_lookup.contains(id))
     {
        QFile inputFile(fileName);
        if (inputFile.open(QIODevice::ReadOnly))
        {
            inputFile.seek(m_lookup[id]);
            QString data = inputFile.readLine();
            inputFile.close();
            return data;
        } else {
            return QString(); // or handle error
        }
     } else {
        return QString(); // or handle error
     }
}

// use example
QString data = getEntry(id);
if (data.length() > 0)
{
    QJsonDocument doc = QJsonDocument::fromJson(data.toUtf8());
    if (!doc.isNull())
    {
        // assign your variables
    }
}

and your data file looking like this:

00042{"name":"toto", "id":"42", "description":"tata", "x":"20", "y":"50"}
00044{"name":"toto2", "id":"44", "description":"tata2", "x":"25", "y":"547"}  
00046{"name":"toto3", "id":"46", "description":"tata3", "x":"21", "y":"580"}

The advantage of this method, it will only read the entry of interest, and avoid having to load MB or GB of data in memory just to get a specific entry.

This could further be improved with a lookup table stored at the beginning of the file.

---

Related Topics