Initializing a C++ Std::Istringstream from an in Memory Buffer

initializing a C++ std::istringstream from an in memory buffer?

Look at std::istrstream it has a constructor

 istrstream( char* pch, int nLength );

This class is sort of depreciated or at least you are normally told to use other classes.

The issue with strstream is that it is more complex to manage the memory of the char* buffer so in general you would prefer stringstream as it does the memory management for you. However in this case you are already managing the memory of the char* so the normal benefit is in this case a cost. In fact in this case strstream does exactly what you want with minimal overhead in code or speed. This is similar to the discussion of ostrsteram by Herb Sutter

initializing a C++ std::istringstream from an in memory buffer?

Look at std::istrstream it has a constructor

 istrstream( char* pch, int nLength );

This class is sort of depreciated or at least you are normally told to use other classes.

The issue with strstream is that it is more complex to manage the memory of the char* buffer so in general you would prefer stringstream as it does the memory management for you. However in this case you are already managing the memory of the char* so the normal benefit is in this case a cost. In fact in this case strstream does exactly what you want with minimal overhead in code or speed. This is similar to the discussion of ostrsteram by Herb Sutter

Does stringstream initializing with big length buffer doubles memory usage

So I would like to learn if my program uses 200MB in total at runtime
or not?

If you construct a stringstream from a char array it will at least double the memory usage. From the reference of std::basic_stringstream constructor:

Uses a copy of str as initial contents of the underlying string
device.

You could write your own stream buffer to create a non-owning stringstream, but the boost::iostreams library already provides that via array devices.

namespace io = boost::iostreams;

char str[100*1024*1024];

// No copy of str is made here! The stream just stores pointer and size of array.
io::stream<io::array_source> strm( str, sizeof(str) );

// Do something with the stream as usual. It is fully compatible with standard streams.
int x;
strm >> x;

There is a nice online compiler that shows peak memory usage.
In the following examples I'm creating a stream from a 10 MiB array.

Live example using std::stringstream. Peaks at 31(!) MiB.

Live example using boost::array_source. Peaks at only 11 MiB.

Why there is a peak memory usage of even 3x the char array size when using std::stringstream? Because we must create a temporary string from the char array first as std::stringstream doesn't have a constructor that takes a char pointer.

How to initialise std::istringstream from const unsigned char* without cast or copy?

Your answer is still copying.

Have you considered something like this?

const unsigned char *p;
size_t len;

std::istringstream str;
str.rdbuf()->pubsetbuf(
    reinterpret_cast<char*>(const_cast<unsigned char*>(p)), len);

How to do initialization and assignment separately for std::istringstream and move in below case?

You cannot move a string into a stringstream at all. Yes, you can type std::istringstream iss(std::move(result_string)); all you want, but it will not move the string into the stream. It will copy it. Not unless your stringstream implementation has some non-standard addition to it.

You can re-initialize a stringstream at any time by using the str function:

istringstream theStream(...);

//stuff

theStream.str(result_string); //Stream reinitialized.

If you want to wrap that in std::move to make yourself feel better, you can. But it still won't move from it. move is not a magical salve that causes things to be moved; it's just a fancy type-cast. The actual movement happens via constructors, and stringstreams do not have constructors that can move from std::string objects.

How to initialize a std::stringstream?

stringstream ss << "Number of people is " << numPeople;

Why can't I assign the stringstream value at the same time I declare it?

This is similar to hoping this would work...

int x + 3 + 9;

...but this doesn't parse as a variable definition, let alone a definition and assignment.

The legal way to define and initialise an object

The only ways to define and initialise a variable with a non-default value are (in a grammar sense - this isn't code):

type identifier(...args...);
type identifier{...args...};
type identifier = ...expression...;

The last notation is equivalent to the first - i.e. type identifier(arg) where arg is passed the ...expression....

Trying to use the legal notation for int and stringstream

For int, you can easily correct the code:

int x = 3 + 9;

...and it works because "3 + 9" can be evaluated independently first to give a sane value to store in x. The compiler's behaviour for operator + on ints does what we want: it produces the int result we then want to store in x. You can think of the above as:

int x = (int)3 + (int)9; // notate the implicit types
        (int)12;         // evaluate expression => value to assign
int x((int)12);          // construct a valid value

It works! But if you try that for stringstream...

stringstream ss = "Number of people is " << numPeople;  // BROKEN
                  "Number of people is " << numPeople; // bitshift?!

...it won't work, because "Number of people is " << numPeople needs to be evaluated first but is illegal - you'll get an error like:

error C2296: '<<' : illegal, left operand has type 'const char [20]'

The problem is that the compiler's still trying to apply the bitwise shift operation, which only makes sense for numbers, because the overloads for << that we want to use require that any "X << Y" code has the left-hand part "X" be - or be implicitly convertible to - an ostream&. A string literal can't be converted. At this point, the compiler is oblivious to the stringstream to which the result of the expression will be passed.

A solution for stringstream

It's a bit of a chicken-and-egg problem, because you need to combine the right-hand values you want in the stringstream to call the stringstream's constructor, but for that you need... a stringstream. You can actually pull that off with a temporary stringstream:

static_cast<std::ostringstream&&>(std::ostringstream{} << "Number of people is " << numPeople)

The cast is unfortunately needed because the operator<< overloads handle stringstreams via references to their ostream base class, returning an ostream&, so you need to cast back to the stringstream type manually, so you can then invoke the std::stringstream move constructor...

The complete one-liner construction is then...

std::ostringstream ss(static_cast<std::ostringstream&&>(std::ostringstream{} << "Number of people is " << numPeople));
...or...
auto&& ss = static_cast<std::ostringstream&&>(std::ostringstream{} << "Number of people is " << numPeople);

...but that's too hideous to contemplate.

Making the solution (arguably) less hideous with macros

Yes, you read that right. Depending on your sensibilities, you may feel a macro helps or is worse...

#define OSS(VALUES) \
    static_cast<std::ostringstream&&>(std::ostringstream{} << VALUES)

auto&& ss = OSS("Number of people is " << numPeople);

FWIW, you could also use the macro to create strings...

auto&& s = OSS("Number of people is " << numPeople).str(); 

...or create a dedicated macro...

#define STR(VALUES) \
    static_cast<std::ostringstream&&>(std::ostringstream{} << VALUES).str()
auto&& s = STR("Number of people is " << numPeople);

An (arguably) better practice - separate construction and initialisation

Just create the stringstream - optionally providing a single string to the constructor - then use operator<< in a second statement:

std::stringstream ss;
ss << "Number of people is " << numPeople;

This is much easier to read, and there are no weird macros required.

An alternative

C++11 introduced to_string() overloads which are convenient if you have an integral value or two to concatentate with or into a string:

auto&& s = "Number of people is " + std::to_string(numPeople);

This may be inefficient though (check your compiler(s) optimisation abilities if you care): each std::to_string() is likely to dynamically allocate a buffer for an independent std::string instance, then the individual concatenations may involve extra copying of text, and the original dynamically-allocated buffers may need to be enlarged, then most of those temporary std::strings will take time to deallocate during destruction.

Discussion

Ideally, std::stringstream would have a constructor accepting an arbitrary number of constructor arguments (A, B, C...) to be formatted into the stringstream as if by a subsequent << A << B << C.... There are already constructors with arguments (e.g. (std::ios_base::openmode, const Allocator&)), so we'd need a placeholder to distinguish such arguments from values we're trying to format into the stream, or a weirder workaround like requiring the values to be formatted into the stream be passed in as an initialiser list.

Still, it looks and feels very weird using strings with , instead of <<:

std::stringstream ss{"n:", std::setw(4), std::hex, '\n');

And then if during code maintenance you find you need to move the streaming values to a point after construction, you'd need to change the separator. Breaking it out into two lines to start with - construction then streaming - simplifies that maintenance.

It was worse in C++03

C++03 lacked move constructors, so it was necessary to use the std::ostringstream::str() member function on the temporary to get an extra deep-copy of the std::string with which to construct the named stringsteam...

stringstream ss(static_cast<std::ostringstream&>(std::ostringstream() << "Number of people is " << numPeople).str());

With this C++03 code, there's a likelihood of duplicate dynamic memory allocations (unless the strings are short enough to fit inside the string object, a commonly provided std::string technique called "Short String Optimisation" or SSO). There's also a deep copy of textual content. Construction-followed-by-streaming was a better approach.

non-copying istringstream

It's fairly trivial to write a basic std::streambuf class that reads from a given memory area. You can then construct an istream from this and read from that.

initializing a C++ std::istringstream from an in memory buffer?

Note that the lifetime of the buffer pointed to be c_str() is very limited, though, and there's no guarantee that a call to c_str() want cause some copying although I don't know of any implementations where it does.

Setting the internal buffer used by a standard stream (pubsetbuf)

After some more research on this problem, and scrutiny of my code, I came across a post suggesting the use of a hand-coded std::streambuf class. The idea behind this code is to create a streambuf that initializes its internals to refer to the given buffer. The code is as follows.

#include <streambuf>

template <typename char_type>
struct ostreambuf : public std::basic_streambuf<char_type, std::char_traits<char_type> >
{
    ostreambuf(char_type* buffer, std::streamsize bufferLength)
    {
        // set the "put" pointer the start of the buffer and record it's length.
        setp(buffer, buffer + bufferLength);
    }
};

Now if you look at my original code, you will notice that I didn't really need a stringstream to begin with. All I really needed was a way to write to an external buffer using the IOStream library and std::ostream is a much better type to address this problem. Incidentally, I suspect this is how the array_sink type from Boost.IOStreams is implemented.

Here is the modified code that uses my ostreambuf type.

#include <ostream>
#include "ostreambuf.h"  // file including ostreambuf struct from above.

void FillBuffer(char* buffer, unsigned int size)
{
    ostreambuf<char> ostreamBuffer(buffer, size);
    std::ostream messageStream(&ostreamBuffer);

    messageStream << "Hello" << std::endl;
    messageStream << "World!" << std::endl;
}

---

Related Topics