How to Read from Memory Just Like from a File Using iOStream

How can I read from memory just like from a file using iostream?

I found a solution that works on VC++ since Nim solution works only on GCC compiler (big thanks, though. Thanks to your answer I found other answers which helped me!).

It seems that other people have similar problem too. I did exactly as here and here.

So to read from a piece of memory just like form a istream you have to do this:

class membuf : public streambuf
{
    public:
        membuf(char* p, size_t n) {
        setg(p, p, p + n);
    }
};

int main()
{
    char buffer[] = "Hello World!\nThis is next line\nThe last line";  
    membuf mb(buffer, sizeof(buffer));

    istream istr(&mb);
    string line;
    while(getline(istr, line))
    {
        cout << "line:[" << line << "]" << endl;
    }
}

EDIT: And if you have '\r\n' new lines do as Nim wrote:

if (*line.rbegin() == '\r') line.erase(line.end() - 1);

I'm trying to treat this memory as as wistream. Does anybody know how to do this? I asked separate question for this.

How can I read from memory just like from a file using wistream?

I assume that your data is already converted into the desired encoding (see @detunized answer).

Using my answer to your previous question the conversion is straight forward:

namespace io = boost::iostreams;

io::filtering_wistream in;
in.push(warray_source(array, arraySize));

If you insist on not using boost then the conversion goes as follows (still straight forward):

class membuf : public wstreambuf // <- !!!HERE!!!
{
public:
    membuf(wchar_t* p, size_t n) { // <- !!!HERE!!!
        setg(p, p, p + n);
    }
};

int main()
{
    wchar_t buffer[] = L"Hello World!\nThis is next line\nThe last line";  
    membuf mb(buffer, sizeof(buffer)/sizeof(buffer[0]));

    wistream istr(&mb);
    wstring line;
    while(getline(istr, line))
    {
        wcout << L"line:[" << line << L"]" << endl;
    }
}

Also consider this for why use plain char UTF-8 streams.

Does constructing an iostream (c++) read data from the hard drive into memory?

No, it will not read the entire file into memory when you open it. It will read your file in chunks though, but I believe this process will not start until you read the first byte. Also these chunks are relatively small (on the order of 4-128 kibibytes in size), and the fact it does this will speed things up greatly if you are reading the file sequentially.

In a test on my Linux box (well, Linux VM) simply opening the file only results in the OS open system call, but no read system call. It doesn't start reading anything from the file until the first attempt to read from the stream. And then it reads 8191 (why 8191? that seems a very strange number) byte chunks as I read the file in.

How to read file content into istringstream?

std::ifstream has a method rdbuf(), that returns a pointer to a filebuf. You can then "push" this filebuf into your stringstream:

#include <fstream>
#include <sstream>

int main()
{
    std::ifstream file( "myFile" );

    if ( file )
    {
        std::stringstream buffer;

        buffer << file.rdbuf();

        file.close();

        // operations on the buffer...
    }
}

EDIT: As Martin York remarks in the comments, this might not be the fastest solution since the stringstream's operator<< will read the filebuf character by character. You might want to check his answer, where he uses the ifstream's read method as you used to do, and then set the stringstream buffer to point to the previously allocated memory.

Read file to memory, loop through data, then write file

So pardon the potentially blatantly obvious, but if you want to process this line by line, then...

#include <iostream>
#include <fstream>
#include <string>
using namespace std;

int main(int argc, char *argv[])
{
    // read lines one at a time
    ifstream inf("S050508-v3.txt");
    string line;
    while (getline(inf, line))
    {
        // ... process line ...
    }
    inf.close();

    return 0;
}

And just fill in the body of the while loop? Maybe I'm not seeing the real problem (a forest for the trees kinda thing).

EDIT

The OP is inline with using a custom streambuf which may not necessarily be the most portable thing in the world, but he's more interested in avoiding flipping back and forh between input and output files. With enough RAM, this should do the trick.

#include <iostream>
#include <fstream>
#include <iterator>
#include <memory>
using namespace std;

struct membuf : public std::streambuf
{
    membuf(size_t len)
        : streambuf()
        , len(len)
        , src(new char[ len ] )
    { 
        setg(src.get(), src.get(), src.get() + len);
    }

    // direct buffer access for file load.
    char * get() { return src.get(); };
    size_t size() const { return len; };

private:
    std::unique_ptr<char> src;
    size_t len;
};

int main(int argc, char *argv[])
{
    // open file in binary, retrieve length-by-end-seek
    ifstream inf(argv[1], ios::in|ios::binary);
    inf.seekg(0,inf.end);
    size_t len = inf.tellg();
    inf.seekg(0, inf.beg);

    // allocate a steam buffer with an internal block
    //  large enough to hold the entire file.
    membuf mb(len+1);

    // use our membuf buffer for our file read-op.
    inf.read(mb.get(), len);
    mb.get()[len] = 0;

    // use iss for your nefarious purposes
    std::istream iss(&mb);
    std::string s;
    while (iss >> s)
        cout << s << endl;

    return EXIT_SUCCESS;
}

How to read large files in segments?

The problem is that you don't override the buffer's content. Here's what your code does:

It reads the beginning of the file
When reaching the 'YZ', it reads it and only overrides the buffer's first two characters ('U' and 'V') because it has reached the end of the file.

One easy fix is to clear the buffer before each file read:

#include <iostream>
#include <fstream>
#include <array>

int main()
{
    std::ifstream bigFile("bigfile.txt", std::ios::binary | std::ios::ate);
    int fileSize = bigFile.tellg();
    std::cout << bigFile.tellg() << " Bytes" << '\n';

    bigFile.seekg(0);
    
    constexpr size_t bufferSize = 4;
    std::array<char, bufferSize> buffer;
    
    while (bigFile)
    {
        for (int i(0); i < bufferSize; ++i)
            buffer[i] = '\0';
        bigFile.read(buffer.data(), bufferSize);
        // Print the buffer data
        std::cout.write(buffer.data(), bufferSize) << '\n';
    }
}

I also changed:

The std::unique_ptr<char[]> to a std::array since we don't need dynamic allocation here and std::arrays's are safer that C-style arrays
The printing instruction to std::cout.write because it caused undefined behavior (see @paddy's comment). std::cout << prints a null-terminated string (a sequence of characters terminated by a '\0' character) whereas std::cout.write prints a fixed amount of characters
The second file opening to a call to the std::istream::seekg method (see @rustyx's answer).

Another (and most likely more efficient) way of doing this is to read the file character by character, put them in the buffer, and printing the buffer when it's full. We then print the buffer if it hasn't been already in the main for loop.

#include <iostream>
#include <fstream>
#include <array>

int main()
{
    std::ifstream bigFile("bigfile.txt", std::ios::binary | std::ios::ate);
    int fileSize = bigFile.tellg();
    std::cout << bigFile.tellg() << " Bytes" << '\n';

    bigFile.seekg(0);
    
    constexpr size_t bufferSize = 4;
    std::array<char, bufferSize> buffer;
    
    int bufferIndex;
    for (int i(0); i < fileSize; ++i)
    {
        // Add one character to the buffer
        bufferIndex = i % bufferSize;
        buffer[bufferIndex] = bigFile.get();
        // Print the buffer data
        if (bufferIndex == bufferSize - 1)
            std::cout.write(buffer.data(), bufferSize) << '\n';
    }
    // Override the characters which haven't been already (in this case 'W' and 'X')
    for (++bufferIndex; bufferIndex < bufferSize; ++bufferIndex)
        buffer[bufferIndex] = '\0';
    // Print the buffer for the last time if it hasn't been already
    if (fileSize % bufferSize /* != 0 */)
        std::cout.write(buffer.data(), bufferSize) << '\n';
}

Read file line by line using ifstream in C++

First, make an ifstream:

#include <fstream>
std::ifstream infile("thefile.txt");

The two standard methods are:

Assume that every line consists of two numbers and read token by token:
```
int a, b;
while (infile >> a >> b)
{
    // process pair (a,b)
}
```

Line-based parsing, using string streams:

#include <sstream>
#include <string>

std::string line;
while (std::getline(infile, line))
{
    std::istringstream iss(line);
    int a, b;
    if (!(iss >> a >> b)) { break; } // error

    // process pair (a,b)
}

You shouldn't mix (1) and (2), since the token-based parsing doesn't gobble up newlines, so you may end up with spurious empty lines if you use getline() after token-based extraction got you to the end of a line already.