What Exactly Is Streambuf? How to Use It

What exactly is streambuf? How do I use it?

Stream buffers represent input or output devices and provide a low level interface for unformatted I/O to that device. Streams, on the other hand, provide a higher level wrapper around the buffer by way of basic unformatted I/O functions and especially via formatted I/O functions (i.e., operator<< and operator>> overloads). Stream objects may also manage a stream buffer's lifetime.

For example a file stream has an internal file stream buffer. The stream manages the lifetime of the buffer and the buffer is what provides actual read and write capabilities to a file. The stream's formatting operators ultimately access the stream buffer's unformatted I/O functions, so you only ever have to use the stream's I/O functions, and don't need to touch the buffer's I/O functions directly.

Another way to understand the differences is to look at the different uses they make of locale objects. Streams use the facets that have to do with formatting such as numpunct and num_get. You can also expect that the overloads of stream operator<< and operator>> for custom time or money data types will use the time and money formatting facets. Stream buffers, however, use the codecvt facets in order to convert between the units their interface uses and bytes. So, for example, the interface for basic_streambuf<char16_t> uses char16_t and so basic_streambuf<char16_t> internally uses codecvt<char16_t, char, mbstate_t> by default to convert the formatted char16_t units written to the buffer to char units written to the underlying device. So you can see that streams are mostly for formatting and stream buffers provide a low level interface for unformatted input or output to devices which may use a different, external encoding.

You can use a stream buffer when you want only unformatted access to an I/O device. You can also use stream buffers if you want to set up multiple streams that share a stream buffer (though you'll have to carefully manage the lifetime of the buffer). There are also special purpose stream buffers you might want to use, such as wbuffer_convert in C++11 which acts as a façade for a basic_streambuf<char> to make it look like a wide character stream buffer. It uses the codecvt facet it's constructed with instead of using the codecvt facet attached to any locale. You can usually achieve the same effect by simply using a wide stream buffer imbued with a locale that has the appropriate facet.

boost streambuf consume and commit, what is it

Boost ASIO streambuf is more than just a character array. From the basic_streambuf documentation:

The basic_streambuf class is derived from std::streambuf to associate the streambuf's input and output sequences with one or more character arrays. These character arrays are internal to the basic_streambuf object, but direct access to the array elements is provided to permit them to be used efficiently with I/O operations. Characters written to the output sequence of a basic_streambuf object are appended to the input sequence of the same object.

The documentation goes on to talk about possible implementation strategies. The streambuf object may simply use a single contiguous character array with pointers to manage the input and output sequences. But the interface allows for more complicated schemes.

You asked what the code snippet actually does, but this depends on the underlying implementation. In short, prepare() ensures that the underlying buffer is big enough to hold what you are trying to put into it. It also gives you access to the buffer via a mutuable_buffers object. Once data has been written to the streambuf (presumably, when the read handler is called), commit() makes that data available to the input sequence. You can access those bytes using data(). After you are finished with the data in the buffer (because you copied it, processed it, or whatever), consume() removes the data from the input sequence. A subsequent call to data() would not contain the bytes from the previous call.

You also asserted that the underlying buffer used to store the data is never mentioned, and that is correct. It is up to the ASIO authors to decide how to store the actual data.

C++ streams confusion: istreambuf_iterator vs istream_iterator?

IOstreams use streambufs to as their source / target of input / output. Effectively, the streambuf-family does all the work regarding IO and the IOstream-family is only used for formatting and to-string / from-string transformation.

Now, istream_iterator takes a template argument that says what the unformatted string-sequence from the streambuf should be formatted as, like istream_iterator<int> will interpret (whitespace-delimited) all incoming text as ints.

On the other hand, istreambuf_iterator only cares about the raw characters and iterates directly over the associated streambuf of the istream that it gets passed.

Generally, if you're only interested in the raw characters, use an istreambuf_iterator. If you're interested in the formatted input, use an istream_iterator.

All of what I said also applies to ostream_iterator and ostreambuf_iterator.

istream and ostream with shared streambuf mutually thread-safe for duplex I/O?

There is no special guarantee given for std::streambuf (or std::basic_streambuf<...>) which gives more guarantees than what is generally given. That is, you can have multiple threads reading the object's state at any time but if there is one thread modifying the object's state there shall be no other thread accessing the object. Both reading and writing characters modify the stream buffer's state, i.e. from a formal point of view you can't use them without external synchronization.

Internally the two buffers are entirely separate and have nothing to do with each other. The operations on stream buffers modify them in a rather structured way and I can't imagine that any implementation would have an explicit interaction between the two sets of pointers. That is, in practical terms I don't think there is any synchronization necessary between reading and writing. However, I hadn't realized before that the two sets of buffer pointers may actually share the same cache lines which may at least cause performance problems. I don't think this should cause any correctness problems.

The only resource possibly shared between the two stream buffers is the std::locale object which is meant to be stateless, however. Also, std::streambuf doesn't make any use of this object itself: it is your stream buffer which may use some of the facets (e.g. the std::codecvt<...> facet). As the locale is changed via a call to the virtual function imbue() you would be able to intercept this change and do whatever synchronization is needed if your stream buffer uses the locale.

In summary, the standard doesn't make any guarantee that it would work to use concurrent threads to read and write using the same stream buffer. In practice, the DS9k is probably the only system where it would fail and the two threads may end up effectively synchronized due to the buffer pointers ending up in shared cache lines.

initializing a C++ std::istringstream from an in memory buffer?

Look at std::istrstream it has a constructor

 istrstream( char* pch, int nLength );

This class is sort of depreciated or at least you are normally told to use other classes.

The issue with strstream is that it is more complex to manage the memory of the char* buffer so in general you would prefer stringstream as it does the memory management for you. However in this case you are already managing the memory of the char* so the normal benefit is in this case a cost. In fact in this case strstream does exactly what you want with minimal overhead in code or speed. This is similar to the discussion of ostrsteram by Herb Sutter

how I can read pointer data from boost::streambuf but without copy data

boost::asio::streambuf is no more than buffer queue. Use streambuf::data(). It return a list of buffer witch represents committed buffers. After successful send/write use streambuf::consume(size_t) to remove/reuse buffers.

streambuf b;
size_t size;
size= read( _socket, b.prepare( 1024 ) ); 
b.commit( size );   // after this function you may call read again
size= write( socket_, b.data() ); // you can check size() if there is anything
b.consume(size);  // remove size bytes from data() buffers  

initializing a C++ std::istringstream from an in memory buffer?

Look at std::istrstream it has a constructor

 istrstream( char* pch, int nLength );

This class is sort of depreciated or at least you are normally told to use other classes.

The issue with strstream is that it is more complex to manage the memory of the char* buffer so in general you would prefer stringstream as it does the memory management for you. However in this case you are already managing the memory of the char* so the normal benefit is in this case a cost. In fact in this case strstream does exactly what you want with minimal overhead in code or speed. This is similar to the discussion of ostrsteram by Herb Sutter

Copy a streambuf's contents to a string

I mostly don't like answers that say "You don't want X, you want Y instead and here's how to do Y" but in this instance I'm pretty sure I know what tstenner wanted.

In Boost 1.66, the dynamic string buffer type was added so async_read can directly resize and write to a string buffer.

How to create a C++ streambuf object from a C FILE*, sharing its buffer (and buffer state) with the FILE object pointed to?

There exists a (non-standard, looking for the reference now) C++ streambuf that is a layer on top of stdio, then it would be sufficient to constantly flush the stream buffer.

The automatic-flush part is portably accomplished using std::unitbuf:

mysb << std::unitbuf;

---

Ahh, it's named stdio_filebuf and comes with gcc's C++ library: __gnu_cxx::stdio_filebuf::stdio_filebuf constructor

---

Related Topics