Simpler Way to Create a C++ Memorystream from (Char*, Size_T), Without Copying the Data

Simpler way to create a C++ memorystream from (char*, size_t), without copying the data?

I'm assuming that your input data is binary (not text), and that you want to extract chunks of binary data from it. All without making a copy of your input data.

You can combine boost::iostreams::basic_array_source and boost::iostreams::stream_buffer (from Boost.Iostreams) with boost::archive::binary_iarchive (from Boost.Serialization) to be able to use convenient extraction >> operators to read chunks of binary data.

#include <stdint.h>
#include <iostream>
#include <boost/iostreams/device/array.hpp>
#include <boost/iostreams/stream.hpp>
#include <boost/archive/binary_iarchive.hpp>

int main()
{
    uint16_t data[] = {1234, 5678};
    char* dataPtr = (char*)&data;

    typedef boost::iostreams::basic_array_source<char> Device;
    boost::iostreams::stream_buffer<Device> buffer(dataPtr, sizeof(data));
    boost::archive::binary_iarchive archive(buffer, boost::archive::no_header);

    uint16_t word1, word2;
    archive >> word1 >> word2;
    std::cout << word1 << "," << word2 << std::endl;
    return 0;
}

With GCC 4.4.1 on AMD64, it outputs:

1234,5678

Boost.Serialization is very powerful and knows how to serialize all basic types, strings, and even STL containers. You can easily make your types serializable. See the documentation. Hidden somewhere in the Boost.Serialization sources is an example of a portable binary archive that knows how to perform the proper swapping for your machine's endianness. This might be useful to you as well.

If you don't need the fanciness of Boost.Serialization and are happy to read the binary data in an fread()-type fashion, you can use basic_array_source in a simpler way:

#include <stdint.h>
#include <iostream>
#include <boost/iostreams/device/array.hpp>
#include <boost/iostreams/stream.hpp>

int main()
{
    uint16_t data[] = {1234, 5678};
    char* dataPtr = (char*)&data;

    typedef boost::iostreams::basic_array_source<char> Device;
    boost::iostreams::stream<Device> stream(dataPtr, sizeof(data));

    uint16_t word1, word2;
    stream.read((char*)&word1, sizeof(word1));
    stream.read((char*)&word2, sizeof(word2));
    std::cout << word1 << "," << word2 << std::endl;

    return 0;
}

I get the same output with this program.

looking for a MemoryStream in C++

#include <sstream>

std::ostringstream  buffer; // no growth specification necessary
buffer << "a char buffer" << customObject << someOtherObject;

std::string contents = buffer.str();
size_t bufferSize = contents.size();

rawNetworkSocket.Send(contents); // you can take the size in Send

Using this approach you will have to parse the result where you receive it (as the code above just transforms your data into an unstructured string.

Another problem with it is that since C++ doesn't support reflection, you will have to define operator << for your objects. This is the code for a Custom class:

template<typename C, typename T>
std::basic_ostream<C,T>& operator << (
    std::basic_ostream<C,T>& out, const Custom& object)
{
    out << object.member1 << "," << object.member2 /* ... */ << object.memberN;
    return out;
}

If you want structured serialization, have a look at boost::serialization.

initializing a C++ std::istringstream from an in memory buffer?

Look at std::istrstream it has a constructor

 istrstream( char* pch, int nLength );

This class is sort of depreciated or at least you are normally told to use other classes.

The issue with strstream is that it is more complex to manage the memory of the char* buffer so in general you would prefer stringstream as it does the memory management for you. However in this case you are already managing the memory of the char* so the normal benefit is in this case a cost. In fact in this case strstream does exactly what you want with minimal overhead in code or speed. This is similar to the discussion of ostrsteram by Herb Sutter

initializing a C++ std::istringstream from an in memory buffer?

Look at std::istrstream it has a constructor

 istrstream( char* pch, int nLength );

This class is sort of depreciated or at least you are normally told to use other classes.

The issue with strstream is that it is more complex to manage the memory of the char* buffer so in general you would prefer stringstream as it does the memory management for you. However in this case you are already managing the memory of the char* so the normal benefit is in this case a cost. In fact in this case strstream does exactly what you want with minimal overhead in code or speed. This is similar to the discussion of ostrsteram by Herb Sutter

How can I read from memory just like from a file using iostream?

I found a solution that works on VC++ since Nim solution works only on GCC compiler (big thanks, though. Thanks to your answer I found other answers which helped me!).

It seems that other people have similar problem too. I did exactly as here and here.

So to read from a piece of memory just like form a istream you have to do this:

class membuf : public streambuf
{
    public:
        membuf(char* p, size_t n) {
        setg(p, p, p + n);
    }
};

int main()
{
    char buffer[] = "Hello World!\nThis is next line\nThe last line";  
    membuf mb(buffer, sizeof(buffer));

    istream istr(&mb);
    string line;
    while(getline(istr, line))
    {
        cout << "line:[" << line << "]" << endl;
    }
}

EDIT: And if you have '\r\n' new lines do as Nim wrote:

if (*line.rbegin() == '\r') line.erase(line.end() - 1);

---

I'm trying to treat this memory as as wistream. Does anybody know how to do this? I asked separate question for this.

convert from string to byte[] and add the result into a memorystream

You can get a pointer to the string buffer by simply calling std::string.c_str(). That will return a const char* (where char is an int8_t) that you can effectively use as a byte[]. Keep in mind though that the pointer returned is pointing to memory managed by the string object, so if you change anything in the original string class, you will invalidate the pointer. Also since it's a pointer to a const char, you shouldn't change any values in the buffer. So if you need more permanent memory, or need a buffer you can modify, a better way to accomplish your goal would be to-do (using gcc, which shouldn't be a problem since you're on Ubuntu):

std::string my_string;
char string_array[my_string.length() + 1];
strcpy(string_array, my_string.c_str());

Now use the string_array as your memory buffer.

If you need to return the buffer from a function, you're going to have to allocate the buffer on the heap and return a pointer. That also means you're going to have to call delete [] on the pointer as well after you're done with it, or else you're going to end up with a memory leak. So you could do the following:

#include <string>
#include <cstring>

char* return_buffer(const std::string& string)
{
   char* return_string = new char[string.length() + 1];
   strcpy(return_string, string.c_str());

   return return_string;
}

//now use in code
int main()
{
    std::string some_string = "Stuff";
    char* buffer = return_buffer(some_string);

    //...do something with buffer

    //...after you're done with the buffer to prevent memory leak
    delete [] buffer;

    return 0;
}

C++ memory stream to C file stream

If you're on a GNU system, (Linux, Glibc), you can use fopencookie to create a FILE * wrapper for your decoding operation:

FILE *fopencookie(void *cookie, const char *mode, cookie_io_functions_t io_funcs);

[...]

The fopencookie() function serves a purpose similar to fopen(3): it
opens a new stream and returns a pointer to a FILE object that is
used to operate on that stream.

The cookie argument is a pointer to the caller's cookie structure
that is to be associated with the new stream. This pointer is
supplied as the first argument when the standard I/O library invokes
any of the hook functions described below.

The mode argument serves the same purpose as for fopen(3). The
following modes are supported: r, w, a, r+, w+, and a+. See fopen(3)
for details.

The io_funcs argument is a structure that contains four fields
pointing to the programmer-defined hook functions that are used to
implement this stream. The structure is defined as follows

typedef struct {
    cookie_read_function_t  *read;
    cookie_write_function_t *write;
    cookie_seek_function_t  *seek;
    cookie_close_function_t *close;
} cookie_io_functions_t;

[...]

(I didn't want to copy the entire manpage to my answer).

Basically then you can do:

ssize_t my_read(void *cookie, char *buf, size_t size) {
    std::istream *the_stream = static_cast<std::istream*>(cookie);
    // insert magic
    return bytes_read;
}

cookie_io_functions_t my_functions = {
    my_read,
    NULL,
    NULL,
    NULL,
};

...

FILE *wrapped = fopencookie(static_cast<void *>&stream, "rb", my_functions);

On BSD/OSX, you'd be equally lucky, because it comes with funopen which is just a slightly different API to achieve exactly the same thing.

And if you want to support Windows, well, poor you.

Could I use stringstream as a memory stream like `MemoryStream` in C#?

When storing character sequences in a std::string you can have included null characters. Correspondingly, a std::stringstream can deal with embedded null characters as well. However, the various formatted operations on streams won't pass through the null characters. Also, when using a built-in string to assign values to a std::string the null characters will matter, i.e., you'd need to use the various overloads taking the size of the character sequence as argument.

What exactly are you trying to achieve? There may be an easier approach than traveling in string streams. For example, if you want to read the stream interface to interact with a memory buffer, a custom stream buffer is really easy to write and setup:

struct membuf
    : std::streambuf 
{
        membuf(char* base, std::size_t size) {
        this->setp(base, base + size);
        this->setg(base, base, base + size);
    }
    std::size_t written() const { return this->pptr() - this->pbase(); }
    std::size_t read() const    { return this->gptr() - this->eback(); }
};

int main() {
    // obtain a buffer starting at base with size size
    membuf       sbuf(base, size);
    std::ostream out(&sbuf);
    out.write("1\08\09\0", 6); // write three digits and three null chars
}

---

Related Topics