Boost Serialization, Deserialization of Raw C Arrays

boost serialization, deserialization of raw C arrays

I heartily recommend you use std::array or std::vector here, because... you messed this up :)

For starters, Monkey doesn't initialize its members. So, loading ends up doing a load_binary to whatever pointer value m.arr happened to have. How would you expect the deserialization to "know" that you needed to allocate memory for that? You need to tell it:

    template<class Archive>
    void serialize(Archive & ar, Monkey& m, const unsigned int version)
    {
        ar & m.num;
        if (Archive::is_loading::value)
        {
            assert(m.arr == nullptr);
            m.arr = new float[m.num];
        }
        ar & make_array<float>(m.arr, m.num);
    }

Now, let's make Monkey a bit less unsafe (by adding initialization and destruction, and, perhaps most importantly, prohibiting copy semantics):

struct Monkey
{
    uint32_t num;
    float* arr;

    Monkey() : num(0u), arr(nullptr) {}

    Monkey(Monkey const&) = delete;
    Monkey& operator=(Monkey const&) = delete;
    ~Monkey() { delete[] arr; }
};

Now, you can see it work:

#include <iostream>
#include <fstream>
#pragma warning(disable: 4244)
#include <boost/serialization/serialization.hpp>
#include <boost/archive/binary_oarchive.hpp>
#include <boost/archive/binary_iarchive.hpp>

struct Monkey
{
    uint32_t num;
    float* arr;

    Monkey() : num(0u), arr(nullptr) {}

    Monkey(Monkey const&) = delete;
    Monkey& operator=(Monkey const&) = delete;
    ~Monkey() { delete[] arr; }
};

namespace boost
{
    namespace serialization
    {
        template<class Archive>
        void serialize(Archive & ar, Monkey& m, const unsigned int version)
        {
            ar & m.num;
            if (Archive::is_loading::value)
            {
                assert(m.arr == nullptr);
                m.arr = new float[m.num];
            }
            ar & make_array<float>(m.arr, m.num);
        }
    }
}

int main(int argc, char* argv[])
{
    const char* name = "monkey.txt";
    {
        Monkey m;
        m.num = 10;
        m.arr = new float[m.num];
        for (uint32_t index = 0; index < m.num; index++)
            m.arr[index] = (float)index;

        std::ofstream outStream(name, std::ios::out | std::ios::binary | std::ios::trunc);
        boost::archive::binary_oarchive oar(outStream);
        oar << (m);
    }

    Monkey m;
    std::ifstream inStream(name, std::ios::in | std::ios::binary);
    boost::archive::binary_iarchive iar(inStream);
    iar >> (m);

    std::copy(m.arr, m.arr + m.num, std::ostream_iterator<float>(std::cout, ";"));
}

Prints

0;1;2;3;4;5;6;7;8;9;

Live on Coliru

How to serialize large malloc'ed float array with Boost allocated outside of constructor?

According to this answer, you can do it like this:

template<class Archive>
void serialize(Archive & ar, Monkey& m, const unsigned int version)
{
    ar & m.num;
    if (Archive::is_loading::value)
    {
        assert(m.arr == nullptr);
        m.arr = new float[m.num];
    }
    ar & make_array<float>(m.arr, m.num);
}

Deserializing array of string with boost and are bigger than expected (after serializing from cv::cuda::GpuMat)

In PrepData, string are initialized without a given length, leading to arbitrary long string (until null '\0' is reached).

The correct code is :

string strX((const char*)matX.col(0).data, matX.rows);
string strY((const char*)matY.col(0).data, matY.rows);

Direct boost serialization to char array

IIUC, you would like to write to a preallocated array of fixed size.

You could use a boost::iostreams::array_sink (wrapped with stream to give it an std::ostream interface) for that.

Unsigned long long serialization in boost

Serializing unsigned long long arrays works for me using gcc 4.7.2 with boost 1.49, gcc 4.2.1 with boost 1.55, and clang 3.4 with boost 1.55:

#include <sstream>
#include <boost/archive/text_iarchive.hpp>
#include <boost/archive/text_oarchive.hpp>
#include <boost/serialization/access.hpp>
#include <boost/version.hpp>

struct Foo {
   unsigned long long bar[3];

   template<class Archive>
   void serialize(Archive& ar, const unsigned int /*version*/) {
      ar & bar;
   }
};

std::ostream& operator<<(std::ostream& os, const Foo& foo) {
   return os << foo.bar[0] << ' ' << foo.bar[1] << ' ' << foo.bar[2];
}

int main() {
   std::cout << "Boost version " << BOOST_LIB_VERSION << '\n';

   Foo before;
   before.bar[0] = 0;
   before.bar[1] = 1;
   before.bar[2] = 2;

   std::cout << "before: " << before << '\n';

   std::ostringstream os;
   {
      boost::archive::text_oarchive oa(os);
      oa << before;
   }

   Foo after;
   {
      std::istringstream is(os.str());
      boost::archive::text_iarchive ia(is);
      ia >> after;
   }

   std::cout << "after: " << after << '\n';

   return 0;
}

Here's gcc 4.8 with boost 1.55 on Coliru, also works.

If you are using a pointer to an allocated array, then I think that is your problem. I don't believe you can serialize a bare pointer to a primitive, and I'm sure you can't serialize a bare pointer to an array of primitives because there is no way for serialization to know how many elements a pointer points to.

I would use a std::vector over an allocated array because there is no speed disadvantage in doing so. However, if you really want to allocate your own array then you can serialize it with the boost::serialization::make_array() wrapper like this:

#include <iostream>
#include <boost/archive/text_oarchive.hpp>
#include <boost/serialization/access.hpp>
#include <boost/serialization/array.hpp>

struct Foo {
   size_t dataSize;
   unsigned long long *data;

   Foo()
      : dataSize(3)
      , data(new unsigned long long[dataSize]) {
   }

   ~Foo() {
      delete[] data;
   }

   // TODO: Production code should disallow default copy constructor
   // and assignment operator.

   template<class Archive>
   void serialize(Archive& ar, const unsigned int /*version*/) {
      ar & dataSize;
      ar & boost::serialization::make_array(data, dataSize);
   }
};

int main() {
   Foo foo;
   foo.data[0] = 0;
   foo.data[1] = 1;
   foo.data[2] = 2;

   boost::archive::text_oarchive oa(std::cout);
   oa << foo;

   return 0;
}

It turns out that this question was not about unsigned long long at all, but is essentially a duplicate of boost serialization, deserialization of raw C arrays.

Boost serialization of class wrapping a pointer

The solution to the riddle is simply that serializing primitive types through a pointer is not supported.

The reason is that object tracking is disabled for primitive types. this is documented here:

Special Considerations / Object Tracking

By default, data types designated primitive by Implementation Level class serialization trait are never tracked. If it is desired to track a shared primitive object through a pointer (e.g. a long used as a reference count), It should be wrapped in a class/struct so that it is an identifiable type. The alternative of changing the implementation level of a long would affect all longs serialized in the whole program - probably not what one would intend.

Here's a minimalist sample that shows the root cause in isolation:

Live On Coliru

#include <boost/archive/xml_iarchive.hpp>
#include <boost/archive/xml_oarchive.hpp>
#include <iostream>
#include <sstream>

int main() {
    // Serialization for double* does not compile
    double* p(new double(2.0));
    std::cout << *p << std::endl;

    std::ostringstream oss;
    boost::archive::xml_oarchive oa(oss);
    oa << BOOST_SERIALIZATION_NVP(p); 
    std::cout << oss.str() << std::endl;
}

You need to rethink your serialization plan. What object identity do you wish/need to track?

You can track the identity of the Ptr<> object, and from the fact that you took the trouble toimplement a custom pointer wrapper type, I get the impression that this is likely all you want/need.

Demo: Live On Coliru

In the unlikely event that you really want bi-level object tracking here (e.g. if you can have two Ptr<T> instances pointing to the same T?) you will need to partially specialize for the case where T is a primitive type.

looking for deserialization example in boost

You were almost there:

int main()
{
    using namespace mydata;
    MyData data { "this is a name", "this is a type", boost::make_shared<MyInfo>("this is info") };

    std::ostringstream oss;
    {
        boost::archive::text_oarchive oa(oss);
        oa << data;
    }

    std::istringstream iss(oss.str());
    {
        boost::archive::text_iarchive ia(iss);
        ia >> data;
    }
}

In fact, you could use std::stringstream for both the input and output, but for purity I showed the symmetric approaches (which does redundant copying).

You'll need

#include <sstream>
#include <boost/archive/text_iarchive.hpp>

and for deserialization your classes need to be defaultconstructible:

MyInfo(std::string info = "") : info(std::move(info)) {}

(unrelated warning: do not use std::string info = {} here because that triggers a compiler bug in MSVC)

Here's a fully working sample that shows that the deserialized object has the same data: Live On Coliru

#include <boost/serialization/access.hpp>
#include <boost/serialization/string.hpp>
#include <boost/serialization/shared_ptr.hpp>
#include <boost/archive/text_oarchive.hpp>
#include <boost/smart_ptr/make_shared.hpp>

#include <sstream>
#include <boost/archive/text_iarchive.hpp>

namespace mydata
{
    struct MyInfo 
    {
        std::string info = "extra info";

        MyInfo(std::string info = "") : info(std::move(info)) {}

        friend class boost::serialization::access;
        template<class Archive>
            void serialize(Archive &ar, const unsigned int /*version*/)
            {
                ar & info;
            }
    };

    struct MyData
    {
        std::string name;
        std::string type;
        boost::shared_ptr<MyInfo> myref;

        private:
        friend class boost::serialization::access;
        template<class Archive>
            void serialize(Archive &ar, const unsigned int /*version*/)
            {
                ar & name;
                ar & type;
                ar & myref;
            }
    };
}

int main()
{
    using namespace mydata;

    std::ostringstream oss;
    {
        MyData data { "this is a name", "this is a type", boost::make_shared<MyInfo>("this is info") };

        boost::archive::text_oarchive oa(oss);
        oa << data;
    }

    MyData cloned;
    std::istringstream iss(oss.str());
    {
        boost::archive::text_iarchive ia(iss);
        ia >> cloned;
    }

    // check equality
    {
        std::ostringstream oss2;

        boost::archive::text_oarchive oa(oss2);
        oa << cloned;

        std::cout << oss.str()  << "\n";
        std::cout << oss2.str() << "\n";
    }
}

Output:

22 serialization::archive 10 0 0 14 this is a name 14 this is a type 0 1 2 1 0
0 12 this is info

22 serialization::archive 10 0 0 14 this is a name 14 this is a type 0 1 2 1 0
0 12 this is info

---

Related Topics