Serialize and Send a Data Structure Using Boost

Serialize and send a data structure using Boost?

For such simple structure, boost::serialization is overkill and huge overhead.

Do simpler:

vector<uint16_t> net(3,0);

net[0]=htons(data.m_short1);
net[1]=htons(data.m_short2);
net[2]=htons(data.character);

asio::async_write(socket,buffer((char*)&net.front(),6),callback);

vector<uint16_t> net(3,0);
asio::async_read(socket,buffer((char*)&net.front(),6),callback);

callback:
data.m_short1=ntohs(net[0]);
data.m_short2=ntohs(net[1]);
data.character=ntohs(net[2]);

And Save yourself HUGE overhead that boost::serialization has

And if you private protocol where computers with same order of bytes work (big/little)
that just send structure as is -- POD.

Boost serialization with recursive data structure ends up in stack overflow

Why not simply serialize all elements in order of the matrix and avoid the function call recursion entirely, e.g.:

template<class Archive>
void serialize(Archive& ar, const unsigned int version)
{
    ar& m;
    ar& n;
    for (int i = 0; i < m; ++i)
    {
        Node* node = firstNodesPerRow[i];
        for (int j = 0; j < n; ++j)
        {
            ar & node->gValue();
            node = node->gRight();
        }
    }
}

BTW This works in saving the matrix. I think you need to specialize the serialize function in save and load versions, because for the load version you need to:

load n, m
allocate all nodes and populate the node pointer vectors in matrix
load all values in the same order as during saving

    template<class Archive>
    void save(Archive & ar, const unsigned int version) const
    {
        ...
    }
    template<class Archive>
    void load(Archive & ar, const unsigned int version)
    {
        ...
    }
    BOOST_SERIALIZATION_SPLIT_MEMBER()

In the complicated case where nodes may be missing, the same idea applies. And you still need to split between save/load, adding allocation to load.
But it takes a lot more bookkeeping to correctly save & load again.

E.g., you could first iterate over all nodes as above but creating a map from each Nodes pointer value to an unique increasing ID number. When you save each node's value (as above row by row), also save each direction's node ID. When you load, you first make a empty map: ID -> node pointer. Then restore nodes again row by row, while restoring neighbour pointers as well from map. Of course whenever an ID is first encountered you need to allocate a new node.

Serialize and send objects by TCP using boost

Here you can find a good example on how to use boost::serialization together with boost::asio.

Something like this is the core of what you need:

std::ostringstream archive_stream;
boost::archive::text_oarchive archive(archive_stream);
archive << YOUR_DATA;
outbound_data_ = archive_stream.str();
boost::asio::async_write(socket_, boost::asio::buffer(outbound_data_), handler);

How to pass packed structs as messages in boost::asio? (without serialization)

If you mean that the "packed struct" is actually POD (so standard-layour and trivially constructible/destructible), in short bitwise serializable, then you could say that your struct is a buffer.

Indeed, you may decide that you don't need to copy into another buffer/representation and use the buffer. That's simple, just adapt your object as a buffer:

UserLoginRequest req;
write(socket_or_stream, boost::asio::buffer(&req, sizeof(req)));
read(socket_or_stream, boost::asio::buffer(&req, sizeof(req)));

To avoid doing the math I prefer to use array declaration:

UserLoginRequest req[1];
write(socket_or_stream, boost::asio::buffer(req));
read(socket_or_stream, boost::asio::buffer(req));

Of course on async streams or sockets the async_* variants can be used as well (given that the lifetime of the buffer extends to completion of the operation as always).

Related, you can contiguous storage of POD types as a buffer:

std::vector<UserLoginRequest> massLogin(123); // weird, but just for demo
write(socket_or_stream, boost::asio::buffer(massLogin));
read(socket_or_stream, boost::asio::buffer(massLogin));

Caveat

This kind of bitwise serialization is NOT PORTABLE. I'm assuming that you are well aware of this and don't mind.

Bonus links

TCP Zero copy using boost
How to use Zero-copy sendmsg/Receive in Boost.Asio

How to use boost::serialization with nested structs and minimal code changes?

You can serialize using boost serialization¹:

template <typename Ar> void serialize(Ar& ar, A& a, unsigned) {
    ar & a.Value & a.SomeChar;
}
template <typename Ar> void serialize(Ar& ar, B& b, unsigned) {
    ar & b.data & b.SomeFloat;
}

Using these, you will already have the correct behaviour out of the box with both the C-array and std::vector approaches.

If you want to keep using fixed-size trivially-copyable types², you can use something like Boost Container's static_vector: it will keep track of the current size, but the data is statically allocated right inside the structures.

TRIPLE DEMO

Here's a triple demo program with three implementations depending on the IMPL variable.

As you can see the bulk of the code is kept invariant. However, for "best comparison" I've made sure that all the containers are at half capacity (50/25) before serialization.

The main program also deserializes.

Live On Coliru

#include <boost/iostreams/device/back_inserter.hpp>
#include <boost/iostreams/device/array.hpp>
#include <boost/iostreams/stream.hpp>

#include <boost/archive/binary_oarchive.hpp>
#include <boost/archive/binary_iarchive.hpp>

#include <boost/serialization/access.hpp>
#include <boost/serialization/is_bitwise_serializable.hpp>
#include <boost/serialization/binary_object.hpp>

#include <iostream>

#if (IMPL==0) // C arrays
    struct A {
        int Value[100]; 
        char SomeChar = 'a';
    };

    struct B {
        A data[50];
        float SomeFloat = 0.1f;
    };

    template <typename Ar> void serialize(Ar& ar, A& a, unsigned) {
        ar & a.Value & a.SomeChar;
    }
    template <typename Ar> void serialize(Ar& ar, B& b, unsigned) {
        ar & b.data & b.SomeFloat;
    }

#elif (IMPL==1) // std::vector
    #include <boost/serialization/vector.hpp>
    struct A {
        std::vector<int> Value;
        char SomeChar = 'a';
    };

    struct B {
        std::vector<A> data;
        float SomeFloat = 0.1f;
    };

    template <typename Ar> void serialize(Ar& ar, A& a, unsigned) {
        ar & a.Value & a.SomeChar;
    }
    template <typename Ar> void serialize(Ar& ar, B& b, unsigned) {
        ar & b.data & b.SomeFloat;
    }

#elif (IMPL==2) // static_vector
    #include <boost/serialization/vector.hpp>
    #include <boost/container/static_vector.hpp>
    struct A {
        boost::container::static_vector<int, 100> Value; 
        char SomeChar = 'a';
    };

    struct B {
        boost::container::static_vector<A, 50> data; 
        float SomeFloat = 0.1f;
    };

    template <typename Ar> void serialize(Ar& ar, A& a, unsigned) {
        ar & boost::serialization::make_array(a.Value.data(), a.Value.size()) & a.SomeChar;
    }
    template <typename Ar> void serialize(Ar& ar, B& b, unsigned) {
        ar & boost::serialization::make_array(b.data.data(), b.data.size()) & b.SomeFloat;
    }

#endif

namespace bio = boost::iostreams;
static constexpr auto flags = boost::archive::archive_flags::no_header;
using BinaryData = std::vector</*unsigned*/ char>;

int main() {
    char const* impls[] = {"C style arrays", "std::vector", "static_vector"};
    std::cout << "Using " << impls[IMPL] << " implementation: ";
    BinaryData serialized_data;

    {
        B object = {};
#if IMPL>0
        {
            // makes sure all containers half-full
            A element;
            element.Value.resize(50);
            object.data.assign(25, element);
        }
#endif

        bio::stream<bio::back_insert_device<BinaryData>> os { serialized_data };
        boost::archive::binary_oarchive oa(os, flags);

        oa << object;
    }

    std::cout << "Size: " << serialized_data.size() << "\n";

    {
        bio::array_source as { serialized_data.data(), serialized_data.size() };
        bio::stream<bio::array_source> os { as };
        boost::archive::binary_iarchive ia(os, flags);

        B object;
        ia >> object;
    }
}

Printing

Using C style arrays implementation: Size: 20472
Using std::vector implementation: Size: 5256
Using static_vector implementation: Size: 5039

Final Thoughts

Sending structure over UDP using boost::asio

Sending and receiving structures over network sockets (no mater what type of the socket you are using synchronous or asynchronous TCP, datagram UDP ) logical similar to file read/write. It means - simply dumping the memory approach is not going to work, especially when you structure contains class/structure fields or pointers.
Usually serialization approach used instead, e.g. you can serialize you structure into some binary (ASN1, Google Protocol Buffers etc) or text format (XML,JSON,YAML etc) - send the result by network, receive it and de-serialize back to a structure.

You can use boost serialization for serialization propose.

I.e. something like:

#include <boost/archive/text_oarchive.hpp>
....
struct sample {
  char a;
  char16_t b;
  char c;
  std::string d;
  char e;
};
namespace boost {
namespace serialization {

template<class Archive>
void serialize(Archive & ar,const sample& value, const unsigned int version)
{
    ar & value.a;
    ar & value.b;
    ar & value.c;
    ar & value.d;
    ar & value.e;
}

} // namespace serialization
} // namespace boost
...
sample mystruct;
....
std::ostringstream archive_stream;
boost::archive::text_oarchive archive(archive_stream);
archive << mystruct;
...
sock.send_to( boost::asio::buffer(archive_stream.str()), remote, 0, error);

Then you can de-serialize the structure, when received

#include <boost/archive/text_iarchive.hpp>
....
std::string str;
str.resize(1024);
boost::asio::udp::endpoint sender_endpoint;
std::size_t len = socket.receive_from(
        boost::asio::buffer(str), sender_endpoint);
....
std::istringstream archive_stream(str);
sample mystruct;
boost::archive::text_iarchive archive(archive_stream);
archive >> mystruct;

Reading a serialized struct at the receiver end boost asio

You don't show enough code to know how you declared buf or managed the lifetime.

I'm assuming you used boost::asio::streambuf buf; and it has static storage duration (namespace scope) or is a class member (but you didn't show a class).

Either way, whatever you have you can do "the same" in reverse to receive.

Here's a shortened version (that leaves out the async so we don't have to make guesses about the lifetimes of things like I mentioned above);

Connect

Let's connect to an imaginary server (we can make one below) at port 3001 on localhost:

asio::io_context ioc;
asio::streambuf buf;
tcp::socket s(ioc, tcp::v4());
s.connect({{}, 3001});

Serialize

Basically what you had:

{
    std::ostream os(&buf);
    boost::archive::binary_oarchive oa(os);

    Test req {13,31};
    oa << req;
}

Note the {} scope around the stream/archive make sure the archive is completed before sending.

Send

/*auto bytes_sent =*/ asio::write(s, buf);

Receive

Let's assume our server sends back another Test object serialized in the same way¹.

Reading into the buffer, assuming no framing we'll just "read until the end of the stream":

boost::system::error_code ec;
/*auto bytes_received =*/ asio::read(s, buf, ec);
if (ec && ec != asio::error::eof) {
    std::cout << "Read error: " << ec.message() << "\n";
    return 1;
}

In real life you want timeouts and limits to the amount of data read. Often your protocol will add framing where you know what amount of data to read or what boundary marker to expect.

Deserialize

Test response; // uninitialized
{
    std::istream is(&buf);
    boost::archive::binary_iarchive ia(is);

    ia >> response;
}

Full Demo

Live On Coliru

#include <boost/asio.hpp>
#include <boost/archive/binary_oarchive.hpp>
#include <boost/archive/binary_iarchive.hpp>
#include <boost/serialization/serialization.hpp>
#include <iostream>
namespace asio = boost::asio;
using tcp = boost::asio::ip::tcp;

struct Test {
    int a,b;
    template<typename Ar> void serialize(Ar& ar, unsigned) { ar & a & b; }
};

int main() {
    asio::io_context ioc;
    asio::streambuf buf;
    tcp::socket s(ioc, tcp::v4());
    s.connect({{}, 3001});

    ///////////////////
    // send a "request"
    ///////////////////
    {
        std::ostream os(&buf);
        boost::archive::binary_oarchive oa(os);

        Test req {13,31};
        oa << req;
    }
    /*auto bytes_sent =*/ asio::write(s, buf);

    /////////////////////
    // receive "response"
    /////////////////////

    boost::system::error_code ec;
    /*auto bytes_received =*/ asio::read(s, buf, ec);
    if (ec && ec != asio::error::eof) {
        std::cout << "Read error: " << ec.message() << "\n";
        return 1;
    }

    Test response; // uninitialized
    {
        std::istream is(&buf);
        boost::archive::binary_iarchive ia(is);

        ia >> response;
    }

    std::cout << "Response: {" << response.a << ", " << response.b << "}\n";
}

Using netcat to mock a server with a previously generated response Test{42,99} (base64 encoded here):

base64 -d <<<"FgAAAAAAAABzZXJpYWxpemF0aW9uOjphcmNoaXZlEgAECAQIAQAAAAAAAAAAKgAAAGMAAAA=" | nc -N -l -p 3001

It prints:

Response: {42, 99}

¹ on the same architecture and compiled with the same version of boost, because Boost's binary archives are not portable. The live demo is good demonstration of this

C++ serialization of complex data using Boost

There are many advantages to boost.serialization. For instance, as you say, just including a method with a specified signature, allows the framework to serialize and deserialize your data. Also, boost.serialization includes serializers and readers for all the standard STL containers, so you don't have to bother if all keys have been stored (they will) or how to detect the last entry in the map when deserializing (it will be detected automatically).

There are, however, some considerations to make. For example, if you have a field in your class that it is calculated, or used to speed-up, such as indexes or hash tables, you don't have to store these, but you have to take into account that you have to reconstruct these structures from the data read from the disk.

As for the "file format" you mention, I think some times we try to focus in the format rather than in the data. I mean, the exact format of the file don't matter as long as you are able to retrieve the data seamlessly using (say) boost.serialization. If you want to share the file with other utilities that don't use serialization, that's another thing. But just for the purposes of (de)serialization, you don't have to care about the internal file format.

Serialize and Send a Data Structure Using Boost