boost asio write/read vector
Asio is not a serialization library. It does not serialize random vectors (you could use Boost Serialization for that).
It merely treats your buffer as the data buffer for the IO operation. So, e.g. in the second instance:
std::vector<float> recv_vector;
tcp_socket.async_read_some(boost::asio::buffer(recv_vector), read_handler);
you tell it to receive the amount of data that fits into the POD buffer represented by the vector recv_vector, which is empty. You are therefore telling Asio to receive 0 bytes, which I presume it will do happily for you.
To fix things, either use serialization (putting the responsibility in the hands of another library) or manually send the size of the vector before the actual data.
Full Demo
I have more explanations and a full sample here: How to receive a custom data type from socket read?
Note, as well, that you don't have to do that complicated thing:
boost::asio::write (socket, boost::asio::buffer(&new_buffers->points.front(), nr_points * 3 * sizeof (float)));
Instead, for POD type vectors, just let Asio do the calculations and casts:
boost::asio::write (socket, buffer(new_buffers->points));
Boost ASIO read X bytes synchroniously into a vector
You say that the socket has more than 4 bytes available to read, in which case your code is correctly continuously looping since an eof won't be encountered until all the data is read. It seems to me that you want to use read() rather than read_some() since the latter might return with less than the four bytes you want. From the read_some documentation.
The read_some operation may not read all of the requested number of bytes. Consider using the read function if you need to ensure that the requested amount of data is read before the blocking operation completes.
You need to read the first 4 bytes (using read), process them (you don't need to be in a loop to do this), and then do a looping read and process the data until you get an error condition or eof.
I think you want more like that below:
vector<uint8_t> buf(4);
try
{
size_t len = read(socket, boost::asio::buffer(buf));
assert(len == 4);
// process the 4 bytes in buf
}
catch (boost::system::system_error &err)
{
// handle the error e.g by returning early
}
boost::system::error_code error;
while (!error)
{
size_t len = socket.read_some(boost::asio::buffer(buf), error);
// process len bytes
}
boost::asio and synchrounous read causing compile errors
boost::asio::read takes as second argument buffer which must satisfy Mutable Buffer Sequence requirements, in 1.65 class mutable_buffers_1 does it, not mutable_buffer (it will work since 1.66 version).
So change:
void getData(tcp::socket& socket, boost::asio::mutable_buffers_1& data)
{
boost::asio::read(socket, data);
}
and
boost::asio::mutable_buffers_1 rcv_buf = ...
std::ostringstream os;
//...
snd_buf = boost::asio::buffer(os.str()); // [1]
sendData(client_socket,snd_buf);
The above code cannot work, I mean it compiles fine but it will bring you undefined behaviour. Why?
boost::buffer doesn't make a copy of passed data. It returns just tuple (pointer to data and size of data), underlying data is not copied, buffer just wraps it.
ostringstream::str() returns string by COPY, reference. So buffer takes it, wraps into tuple and at the end of full expression this temporary string is destroyed, hence you have dangling reference in tuple returned by buffer.
Solution?
Create named string:
std::string str = os.str();
snd_buf = boost::asio::buffer(str);
How to create a boost::asio::buffer with uint8_t* [Is it possible?]
On the page, there are all buffer overloads, one of them is
const_buffer buffer(
const void * data,
std::size_t size_in_bytes);
you could use it but you need to define the size of your buffer. Without this, how can async_write know how many bytes must be transmitted ? In case of vector, buffer knows this size by calling vector::size method. In case of raw buffers you have to provide this explicitly.
If GetDataToSend returned pair: pointer to data and its length, you could write:
std::pair<uint8_t*,size_t> GetDataToSend() {
// ...
return result;
}
std::pair<uint8_t*,int> data = GetDataToSend();
boost::asio::async_write(tcp_socket, boost::asio::buffer(data.first, data.second), ...
How to properly store asio connections and reuse them? (non-boost)
It's strange that this line works: asio::write(socket, asio::buffer(message), ignored_error); because you moved a socket before std::shared_ptr<asio::ip::tcp::socket> newconn = std::make_shared<asio::ip::tcp::socket>(std::move(socket)); and later tried to use it. In a range based for you get x which is reference to a shared_ptr<socket>. I think you should use asio::write(*x, asio::buffer(message), ignored_error); if function write uses socket and not shared_ptr<socket>.
Sending raw data over boost::asio
The fundamental part of the question is about serializing and deserializing collections.
Without controlling the compiler and architectures of both the server and client, sending raw structures is typically unsafe as the byte representation may differ between systems. While the compiler and architecture are the same in this specific case, the #pragma pack(1) is irrelevant as WAVEFORM_DATA_STRUCT is not being written as raw memory to the socket. Instead, a multiple buffers of memory are provided for a gathering write operation.
boost::array<boost::asio::mutable_buffer,2> buffer = {{
boost::asio::buffer(&waveformPacket->numWaveforms, ...), // &numWaveforms
boost::asio::buffer(waveformPacket->waveforms) // &waveforms[0]
}};
There are various tools to help with serializing data structures, such as Protocol Buffers.
The code below will demonstrate the basics for serializing a data structure for network communication. To simplify the code and explanation, I have chosen to focus on serialization and deserialization, rather than writing and reading from the socket. Another example located below this section will show more of a raw approach, that assumes the same compiler and architecture.
Starting with a basic foo type:
struct foo
{
char a;
char b;
boost::uint16_t c;
};
It can be determined that the data can be packed into 4 total bytes. Below is one possible wire reprensetation:
0 8 16 24 32
|--------+--------+--------+--------|
| a | b | c |
'--------+--------+--------+--------'
With the wire representation determined, two functions can be used to serialize (save) a foo object to a buffer, and another can be used to deserialize (load) foo from a buffer. As foo.c is larger than a byte, the functions will also need to account for endianness. I opted to use the endian byte swapping functions in the Boost.Asio detail namespace for some platform neutrality.
/// @brief Serialize foo into a network-byte-order buffer.
void serialize(const foo& foo, unsigned char* buffer)
{
buffer[0] = foo.a;
buffer[1] = foo.b;
// Handle endianness.
using ::boost::asio::detail::socket_ops::host_to_network_short;
boost::uint16_t c = host_to_network_short(foo.c);
std::memcpy(&buffer[2], &c, sizeof c);
}
/// @brief Deserialize foo from a network-byte-order buffer.
void deserialize(foo& foo, const unsigned char* buffer)
{
foo.a = buffer[0];
foo.b = buffer[1];
// Handle endianness.
using ::boost::asio::detail::socket_ops::network_to_host_short;
boost::uint16_t c;
std::memcpy(&c, &buffer[2], sizeof c);
foo.c = network_to_host_short(c);
}
With the serialization and deserialization done for foo, the next step is to handle a collection of foo objects. Before writing the code, the wire representation needs to be determine. In this case, I have decided to prefix a sequence of foo elements with a 32-bit count field.
0 8 16 24 32
|--------+--------+--------+--------|
| count of foo elements [n] |
|--------+--------+--------+--------|
| serialized foo [0] |
|--------+--------+--------+--------|
| serialized foo [1] |
|--------+--------+--------+--------|
| ... |
|--------+--------+--------+--------|
| serialized foo [n-1] |
'--------+--------+--------+--------'
Once again, two helper functions can be introduced to serialize and deserialize collection of foo objects, and will also need to account for the byte order of the count field.
/// @brief Serialize a collection of foos into a network-byte-order buffer.
template <typename Foos>
std::vector<unsigned char> serialize(const Foos& foos)
{
boost::uint32_t count = foos.size();
// Allocate a buffer large enough to store:
// - Count of foo elements.
// - Each serialized foo object.
std::vector<unsigned char> buffer(
sizeof count + // count
foo_packed_size * count); // serialize foo objects
// Handle endianness for size.
using ::boost::asio::detail::socket_ops::host_to_network_long;
count = host_to_network_long(count);
// Pack size into buffer.
unsigned char* current = &buffer[0];
std::memcpy(current, &count, sizeof count);
current += sizeof count; // Adjust position.
// Pack each foo into the buffer.
BOOST_FOREACH(const foo& foo, foos)
{
serialize(foo, current);
current += foo_packed_size; // Adjust position.
}
return buffer;
};
/// @brief Deserialize a buffer into a collection of foo objects.
std::vector<foo> deserialize(const std::vector<unsigned char>& buffer)
{
const unsigned char* current = &buffer[0];
// Extract the count of elements from the buffer.
boost::uint32_t count;
std::memcpy(&count, current, sizeof count);
current += sizeof count;
// Handle endianness.
using ::boost::asio::detail::socket_ops::network_to_host_long;
count = network_to_host_long(count);
// With the count extracted, create the appropriate sized collection.
std::vector<foo> foos(count);
// Deserialize each foo from the buffer.
BOOST_FOREACH(foo& foo, foos)
{
deserialize(foo, current);
current += foo_packed_size;
}
return foos;
};
Here is the complete example code:
#include <iostream>
#include <vector>
#include <boost/asio.hpp>
#include <boost/asio/detail/socket_ops.hpp> // endian functions
#include <boost/cstdint.hpp>
#include <boost/foreach.hpp>
#include <boost/tuple/tuple.hpp> // boost::tie
#include <boost/tuple/tuple_comparison.hpp> // operator== for boost::tuple
/// @brief Mockup type.
struct foo
{
char a;
char b;
boost::uint16_t c;
};
/// @brief Equality check for foo objects.
bool operator==(const foo& lhs, const foo& rhs)
{
return boost::tie(lhs.a, lhs.b, lhs.c) ==
boost::tie(rhs.a, rhs.b, rhs.c);
}
/// @brief Calculated byte packed size for foo.
///
/// @note char + char + uint16 = 1 + 1 + 2 = 4
static const std::size_t foo_packed_size = 4;
/// @brief Serialize foo into a network-byte-order buffer.
///
/// @detail Data is packed as follows:
///
/// 0 8 16 24 32
/// |--------+--------+--------+--------|
/// | a | b | c |
/// '--------+--------+--------+--------'
void serialize(const foo& foo, unsigned char* buffer)
{
buffer[0] = foo.a;
buffer[1] = foo.b;
// Handle endianness.
using ::boost::asio::detail::socket_ops::host_to_network_short;
boost::uint16_t c = host_to_network_short(foo.c);
std::memcpy(&buffer[2], &c, sizeof c);
}
/// @brief Deserialize foo from a network-byte-order buffer.
void deserialize(foo& foo, const unsigned char* buffer)
{
foo.a = buffer[0];
foo.b = buffer[1];
// Handle endianness.
using ::boost::asio::detail::socket_ops::network_to_host_short;
boost::uint16_t c;
std::memcpy(&c, &buffer[2], sizeof c);
foo.c = network_to_host_short(c);
}
/// @brief Serialize a collection of foos into a network-byte-order buffer.
///
/// @detail Data is packed as follows:
///
/// 0 8 16 24 32
/// |--------+--------+--------+--------|
/// | count of foo elements [n] |
/// |--------+--------+--------+--------|
/// | serialized foo [0] |
/// |--------+--------+--------+--------|
/// | serialized foo [1] |
/// |--------+--------+--------+--------|
/// | ... |
/// |--------+--------+--------+--------|
/// | serialized foo [n-1] |
/// '--------+--------+--------+--------'
template <typename Foos>
std::vector<unsigned char> serialize(const Foos& foos)
{
boost::uint32_t count = foos.size();
// Allocate a buffer large enough to store:
// - Count of foo elements.
// - Each serialized foo object.
std::vector<unsigned char> buffer(
sizeof count + // count
foo_packed_size * count); // serialize foo objects
// Handle endianness for size.
using ::boost::asio::detail::socket_ops::host_to_network_long;
count = host_to_network_long(count);
// Pack size into buffer.
unsigned char* current = &buffer[0];
std::memcpy(current, &count, sizeof count);
current += sizeof count; // Adjust position.
// Pack each foo into the buffer.
BOOST_FOREACH(const foo& foo, foos)
{
serialize(foo, current);
current += foo_packed_size; // Adjust position.
}
return buffer;
};
/// @brief Deserialize a buffer into a collection of foo objects.
std::vector<foo> deserialize(const std::vector<unsigned char>& buffer)
{
const unsigned char* current = &buffer[0];
// Extract the count of elements from the buffer.
boost::uint32_t count;
std::memcpy(&count, current, sizeof count);
current += sizeof count;
// Handle endianness.
using ::boost::asio::detail::socket_ops::network_to_host_long;
count = network_to_host_long(count);
// With the count extracted, create the appropriate sized collection.
std::vector<foo> foos(count);
// Deserialize each foo from the buffer.
BOOST_FOREACH(foo& foo, foos)
{
deserialize(foo, current);
current += foo_packed_size;
}
return foos;
};
int main()
{
// Create a collection of foo objects with pre populated data.
std::vector<foo> foos_expected(5);
char a = 'a',
b = 'A';
boost::uint16_t c = 100;
// Populate each element.
BOOST_FOREACH(foo& foo, foos_expected)
{
foo.a = a++;
foo.b = b++;
foo.c = c++;
}
// Serialize the collection into a buffer.
std::vector<unsigned char> buffer = serialize(foos_expected);
// Deserialize the buffer back into a collection.
std::vector<foo> foos_actual = deserialize(buffer);
// Compare the two.
std::cout << (foos_expected == foos_actual) << std::endl; // expect 1
// Negative test.
foos_expected[0].c = 0;
std::cout << (foos_expected == foos_actual) << std::endl; // expect 0
}
Which produces the expected results of 1 and 0.
If using the same compiler and architecture, then it may be possible to reinterpret a contiguous sequence of foo objects from a raw buffer as an array of foo objects, and populate std::vector<foo> with copy constructors. For example:
// Create and populate a contiguous sequence of foo objects.
std::vector<foo> foo1;
populate(foo1);
// Get a handle to the contiguous memory block.
const char* buffer = reinterpret_cast<const char*>(&foo1[0]);
// Populate a new vector via iterator constructor.
const foo* begin = reinterpret_cast<const foo*>(buffer);
std::vector<foo> foos2(begin, begin + foos1.size());
In the end, foo1 should be equal to foo2. The foo objects in foo2 will be copy-constructed from the reinterpreted foo objects residing in memory owned by foo1.
#include <iostream>
#include <vector>
#include <boost/cstdint.hpp>
#include <boost/foreach.hpp>
#include <boost/tuple/tuple.hpp> // boost::tie
#include <boost/tuple/tuple_comparison.hpp> // operator== for boost::tuple
/// @brief Mockup type.
struct foo
{
char a;
char b;
boost::uint16_t c;
};
/// @brief Equality check for foo objects.
bool operator==(const foo& lhs, const foo& rhs)
{
return boost::tie(lhs.a, lhs.b, lhs.c) ==
boost::tie(rhs.a, rhs.b, rhs.c);
}
int main()
{
// Create a collection of foo objects with pre populated data.
std::vector<foo> foos_expected(5);
char a = 'a',
b = 'A';
boost::uint16_t c = 100;
// Populate each element.
BOOST_FOREACH(foo& foo, foos_expected)
{
foo.a = a++;
foo.b = b++;
foo.c = c++;
}
// Treat the collection as a raw buffer.
const char* buffer =
reinterpret_cast<const char*>(&foos_expected[0]);
// Populate a new vector.
const foo* begin = reinterpret_cast<const foo*>(buffer);
std::vector<foo> foos_actual(begin, begin + foos_expected.size());
// Compare the two.
std::cout << (foos_expected == foos_actual) << std::endl;
// Negative test.
foos_expected[0].c = 0;
std::cout << (foos_expected == foos_actual) << std::endl;
}
As with the other approach, this produces the expected results of 1 and 0.
Related Topics
Why Isn't "0F" Treated as a Floating Point Literal in C++
What Are the Incompatible Differences Between C(99) and C++(11)
Why Does Rand() Return the Same Value Using Srand(Time(Null)) in This for Loop
Undefined Behavior in C/C++: I++ + ++I VS ++I + I++
How to Check Deallocation of Memory
Emacs C++-Mode Incorrect Indentation
Is 'Volatile' Needed in This Multi-Threaded C++ Code
How to Treat a Specific Warning as an Error
Receiving Only Necessary Data with C++ Socket
How to Set Given Channel of a Cv::Mat to a Given Value Efficiently Without Changing Other Channels
Throwing the Fattest People Off of an Overloaded Airplane
Generate All Sequences of Bits Within Hamming Distance T
How to Compare Two Standard Conversion Sequences Use the Rank of Contained Conversions
Static Member Variable in a Class
C++ Passing an Array Pointer as a Function Argument