Direct Boost Serialization to Char Array

Direct boost serialization to char array

IIUC, you would like to write to a preallocated array of fixed size.

You could use a boost::iostreams::array_sink (wrapped with stream to give it an std::ostream interface) for that.

Creating custom boost serialization output archive

Don't create your own archive class. Like the commenter said, use the streambuf interface to your advantage. The upside is that things will work for any of the archive implementations, including binary archives, and perhaps more interestingly things like the EOS Portable Archive implementation.

The streambuf interface can be quite flexible. E.g. i've used it to implement hashing/equality operations:

• Generate operator== using Boost Serialization?

In that answer I used Boost Iostreams with its Device concept to make implementing things simpler.

Now if your Buffer type (which you might have shown) has an interface that resembles most buffers (i.e. one or more (void*,size) pairs), you could use existing adapters present in Boost IOstreams. E.g.

• Boost: Re-using/clearing text_iarchive for de-serializing data from Asio:receive()

where I show how to use Serialization with a re-usable fixed buffer. Here's the Proof Of Concept:

#include <boost/archive/text_iarchive.hpp>
#include <boost/archive/text_oarchive.hpp>
#include <boost/serialization/serialization.hpp>

#include <boost/iostreams/device/array.hpp>
#include <boost/iostreams/stream.hpp>
#include <sstream>

namespace bar = boost::archive;
namespace bio = boost::iostreams;

struct Packet {
    int i;
    template <typename Ar> void serialize(Ar& ar, unsigned) { ar & i; }
};

namespace Reader {
    template <typename T>
    Packet deserialize(T const* data, size_t size) {
        static_assert(boost::is_pod<T>::value     , "T must be POD");
        static_assert(boost::is_integral<T>::value, "T must be integral");
        static_assert(sizeof(T) == sizeof(char)   , "T must be byte-sized");

        bio::stream<bio::array_source> stream(bio::array_source(data, size));
        bar::text_iarchive ia(stream);
        Packet result;
        ia >> result;

        return result;
    }

    template <typename T, size_t N>
    Packet deserialize(T (&arr)[N]) {
        return deserialize(arr, N);
    }

    template <typename T>
    Packet deserialize(std::vector<T> const& v) {
        return deserialize(v.data(), v.size());
    }

    template <typename T, size_t N>
    Packet deserialize(boost::array<T, N> const& a) {
        return deserialize(a.data(), a.size());
    }
}

template <typename MutableBuffer>
void serialize(Packet const& data, MutableBuffer& buf)
{
    bio::stream<bio::array_sink> s(buf.data(), buf.size());
    bar::text_oarchive ar(s);

    ar << data;
}

int main() {
    boost::array<char, 1024> arr;

    for (int i = 0; i < 100; ++i) {
        serialize(Packet { i }, arr);

        Packet roundtrip = Reader::deserialize(arr);
        assert(roundtrip.i == i);
    }
    std::cout << "Done\n";
}

Deserializing array of string with boost and are bigger than expected (after serializing from cv::cuda::GpuMat)

In PrepData, string are initialized without a given length, leading to arbitrary long string (until null '\0' is reached).

The correct code is :

string strX((const char*)matX.col(0).data, matX.rows);
string strY((const char*)matY.col(0).data, matY.rows);

Can boost::container::strings be serialized using boost serialization?

Yes. Surprisingly, the necessary support is not baked into Boost. Though if you look inside the string serialization header you will find that it has support as "primitive", and it takes just one line to enable it:

BOOST_CLASS_IMPLEMENTATION(boost::container::string, boost::serialization::primitive_type)

Now it works the same as std::string:

Live On Coliru

#include <boost/archive/text_oarchive.hpp>
#include <boost/archive/text_iarchive.hpp>
#include <boost/container/string.hpp>
#include <iostream>

BOOST_CLASS_IMPLEMENTATION(boost::container::string, boost::serialization::primitive_type)

struct car {
    template<class Ar> void serialize(Ar& ar, unsigned) { ar & make; }
    boost::container::string make;
};

int main() {
    std::stringstream ss;
    {
        boost::archive::text_oarchive oa(ss);
        car my_car{"ford"};
        oa << my_car;
    } // close archive

    std::cout << ss.str() << "\n";
    
    boost::archive::text_iarchive ia(ss);
    car new_car;
    ia >> new_car;    
}

Prints

22 serialization::archive 17 0 0 ford

Serialization into static data using Boost and IOStreams

unsigned char* my_obj = { 10, 48, 48, 30, 20 ... }

It is better to use:

unsigned char my_obj[] = { 10, 48, 48, 30, 20 ... }

So you will have array, which knows it's size, not just pointer to begin. Also, add const if you don't plan to modify it.

---

My question is: is there some easy way to pass byte arrays around as streams? Boost deals with istreams and ostreams for reading and writing the archives. I don't want to use a stringstream or a filestream, but rather what I suppose may be a custom stream that just acts as a giant byte array for whatever is passed to it.

Check Boost's array_source and array_sink.

live demo:

#include <boost/archive/binary_iarchive.hpp>
#include <boost/archive/binary_oarchive.hpp>
#include <boost/iostreams/device/array.hpp>
#include <boost/iostreams/stream.hpp>
#include <iostream>
#include <ostream>
using namespace boost;
using namespace std;

struct Data
{
    double value;

    template<class Archive>
    void serialize(Archive & ar, const unsigned int)
    {
        ar & value;
    }
};

int main()
{
    Data d={3.1415926};
    char buffer[256];
    {
        iostreams::array_sink sink(buffer);
        iostreams::stream<iostreams::array_sink> stream(sink);
        archive::binary_oarchive out_archive(stream);
        out_archive << d;
    }
    Data restored = {0.0};
    {
        iostreams::array_source source(buffer);
        iostreams::stream<iostreams::array_source> stream(source);
        archive::binary_iarchive in_archive(stream);
        in_archive >> restored;
    }
    cout << boolalpha << ( restored.value == d.value ) << endl;
}

Output is:

true

Boost binary serialization - double array of fixed length error

You will want to use

template <class Archive> void serialize(Archive &ar, unsigned) {
    ar & end;
    ar & boost::serialization::make_array(data, end);
}

See http://www.boost.org/doc/libs/1_46_1/libs/serialization/doc/wrappers.html#arrays

Here is the demo, selfcontained:

Live On Coliru

#include <boost/serialization/vector.hpp>
#include <boost/serialization/array.hpp>

struct DataChunk {
  public:
    double data[2048]; // THIS IS THE PROBLEM AREA
    int end;

  private:
    friend class boost::serialization::access;
    template <class Archive> void serialize(Archive &ar, unsigned) {
        ar & end;
        ar & boost::serialization::make_array(data, end);
    }
};

#include <boost/range.hpp>
#include <boost/range/algorithm.hpp>

class RawData {
  private:
    std::vector<DataChunk> chunks;
    friend class boost::serialization::access;
    template <class Archive> void serialize(Archive &ar, unsigned) { ar &chunks; }

  public:
    void add_chunk(DataChunk chunk) { chunks.push_back(chunk); };
    std::vector<DataChunk> get_chunks() { return chunks; };
    static void save(RawData rd, std::string path);
    static bool load(RawData &rd, std::string path);

    void add_raw_data(std::vector<double> raw_data) {
        DataChunk chunk;
        auto const csize = boost::size(chunk.data);

        size_t n    = raw_data.size(), 
               offs = 0ul;

        while (n>0) {
            auto n_ = std::min(n, csize);
            std::copy_n(raw_data.begin() + offs, n_, chunk.data);
            chunk.end = n_;
            chunks.push_back(chunk);
            offs += n_;
            n    -= n_;
        }
    }

    std::vector<double> combine_chunks() {
        std::vector<double> r;
        boost::for_each(chunks, [&r](DataChunk const& c) {std::copy_n(c.data, c.end, back_inserter(r));});
        return r;
    }
};

#include <boost/archive/binary_oarchive.hpp>
#include <boost/archive/binary_iarchive.hpp>
#include <fstream>

void RawData::save(RawData rd, std::string path) {
    std::ofstream file(path);
    if (file.good()) {

        boost::archive::binary_oarchive oa(file, std::ios::binary);
        // boost::archive::text_oarchive oa(file);
        oa << rd;
    }
    file.flush();
    file.close();
}

bool RawData::load(RawData &rd, std::string path) {
    std::ifstream file(path);
    if (file.good()) {

        boost::archive::binary_iarchive ia(file, std::ios::binary);
        // boost::archive::text_iarchive ia(file);
        ia >> rd;
        file.close();
        return true;
    } else
        return false;
}

#include <iostream>

RawData generate() {
    RawData data;
    std::vector<double> raw_data;
    for (int i = 0; i < 5000; i++)
        raw_data.push_back(i * 2048);
    data.add_raw_data(raw_data);
    return data;
}

int main() {
    std::string const path = "test.data";

    {
        // serialize
        RawData const old_data = generate();
        RawData::save(old_data, path);
    }

    {
        // deserialize
        RawData new_data;
        RawData::load(new_data, path);

        // grab the chunks and test the values
        for (auto d : new_data.combine_chunks())
            std::cout << d << ", ";
    }
}

Can't get C++ Boost Pointer Serialization to work

If I must write a char[] one character at a time, or if I must dereference all pointer objects in a container and break them down to their constituent POD parts, writing their sizes in bytes to the stream first, in order to serialize with boost, then there doesn't seem to be much point in using boost.

Yes. But you're trying to serialize a pointer. What should a pointer to char be serialized as? Surely T* should serialize a dynamically allocated T, no? So you would expect to serialize a single char unless name were nullptr?

Or do you expect to get full object tracking for each character serialized?

The whole point here is that you chose a raw pointer to a primitive type. The pointer lacks the necessary information. Hence, the onus is on you to add the information. If you find that tedious, you're free to use C++ types instead.

Note, by the way, that this is not legal in C++03 and up anyways:

a.name = (char *)"this is a test";

A static cast from char const(&)[15] to char * would drop a const qualification. Your compiler should reject this code anyways.

Here's my take on it (use std::string):

#include <boost/serialization/string.hpp>
#include <boost/archive/text_oarchive.hpp>

class A {
  public:
    std::string name;
    template <class Archive> void serialize(Archive &ar, unsigned) {
        ar & name;
    }
};

int main() {
    A a { "this is a test" };
    boost::archive::text_oarchive os(std::cout);
    os << a;
}

Incidentally, the linked duplicate question's title exactly matches what you're trying to do here: Boost c++ serializing a char * and the accepted answer displays exactly the same solution.

If you really insist, you can of course go mud-wrestle some C-style code for no glory:

#include <boost/serialization/serialization.hpp>
#include <boost/serialization/binary_object.hpp>
#include <boost/archive/text_oarchive.hpp>
#include <cstring>
#include <cstdlib>

class A {
  public:
    A(char const* sz = nullptr) : name(sz? strdup(sz) : nullptr) {}
    A(A const&) = delete;
    A& operator=(A const&) = delete;

    ~A() { free(name); }

  private:
    char* name;

    template <class Archive> void save(Archive &ar, unsigned) const {
        bool have_name = name;
        ar & have_name;

        if (have_name)
        {
            size_t len = strlen(name);
            ar & len;
            ar & boost::serialization::make_binary_object(name, len);
        }
    }

    template <class Archive> void load(Archive &ar, unsigned) {
        bool have_name = name;
        ar & have_name;

        if (!have_name)
        {
            free(name);
            name = nullptr;
        } else
        {
            size_t len = 0;
            ar & len;

            name = static_cast<char*>(realloc(name, len));
            ar & boost::serialization::make_binary_object(name, len);
        }
    }

    friend class boost::serialization::access;
    BOOST_SERIALIZATION_SPLIT_MEMBER()
};

int main() {
    A a { "this is a test" };
    boost::archive::text_oarchive os(std::cout);
    os << a;
}

But all this gains you is 6x as much code, crippled value semantics, completely abandoned separation of concerns, a lot of bug potential (did you think of the difference between having name==null and strlen(name)==0? Currently my implementation employs unspecified (implementation-defined) behaviour. I'll leave it as an exercise to you, the reader, to come up with airtight handling here) and... not even readable string output.

Yes, of course make_array would give you 14 116 104 105 115 32 105 115 32 97 32 116 101 115 116 instead of dGhpcyBpcyBhIHRlc3Q=, but that's horrifically storage inefficient and still not human readable.

How does Boost::serialization store user-defined classes in archives?

it seems to just output the first few zeros

What do you mean? You serialized two variables with indeterminate values (you never initialized them). You should not be expecting zeroes. Nor should you be expecting any particular layout (it is determined by archive type, version, library version(s) and platform architecture).

like to know how the archive files are structured, and whether that structure is general

It depends on the archive. It is "general" (for some definition of general). However, most archive formats include minimal meta-data describing its structure (they can not be "visited" or "traversed" in non-sequential fashion, nor is there a "reflectable" schema of some kind). If you need these kinds of serialization features, look at others (protobuf, msgpack, bson, XML/XSD, etc).

The one archive type that will most satisfy your curiosity is the XML archive type. You will find that you are required to supply semantic information for the XML elements, e.g.

Live On Coliru

#include <boost/archive/xml_oarchive.hpp>
#include <iostream>

int main() {
    using boost::serialization::make_nvp;
    {
        boost::archive::xml_oarchive oa(std::cout);

        int a = 42, b = 99, c = a * b;
        oa << make_nvp("a", a) << make_nvp("b", b) << make_nvp("c", c);
    } // closes the archive
}

Which results in

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<!DOCTYPE boost_serialization>
<boost_serialization signature="serialization::archive" version="19">
  <a>42</a>
  <b>99</b>
  <c>4158</c>
</boost_serialization>

This will at least lend some recognizable context to the version (19) and similar information. E.g. relating to Object Tracking and Identity, RTTI mapping and exported class keys etc.

And if so, is it reasonable to read out all of the member variables of Foo from that archive by just reading into the appropriate types

Yes but maybe no. The contract you have to fulfill is to (de)serialize the exact same types of data in the exact same order. Any way you manage that is fine. The typical way is to implement serialize for your type, not to serialize the members separately (Law Of Demeter and simple Encapsulation):

Live On Coliru

#include <boost/archive/xml_oarchive.hpp>

namespace MyLib {
    struct Foo {
        bool        bool1;
        bool        bool2;
        double      Foo_member1;
        std::string Foo_member2;
    };

    template <typename Ar>
    void serialize(Ar& ar, Foo& foo, unsigned /*version*/)
    {
        ar& BOOST_NVP(foo.bool1) & BOOST_NVP(foo.bool2) &
            BOOST_NVP(foo.Foo_member1) & BOOST_NVP(foo.Foo_member2);
    }
} // namespace MyLib

#include <cmath> // for M_PI...
#include <iostream>
int main() {
    using boost::serialization::make_nvp;
    {
        boost::archive::xml_oarchive oa(std::cout);

        MyLib::Foo foo{true, false, M_PI, "Hello world"};
        oa << make_nvp("foo", foo);
    } // closes the archive
}

Prints

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<!DOCTYPE boost_serialization>
<boost_serialization signature="serialization::archive" version="19">
<foo class_id="0" tracking_level="0" version="0">
    <foo.bool1>1</foo.bool1>
    <foo.bool2>0</foo.bool2>
    <foo.Foo_member1>3.14159265358979312e+00</foo.Foo_member1>
    <foo.Foo_member2>Hello world</foo.Foo_member2>
</foo>
</boost_serialization>

CAVEAT

It does sound a little like you're mistaking archives for random-access data streams with some standard format. See for more Archives are not streams

---

Related Topics