Most Efficient Way of Copying a Raw Byte Array into an Empty Byte Vector

How do you copy the contents of an array to a std::vector in C++ without looping?

If you can construct the vector after you've gotten the array and array size, you can just say:

std::vector<ValueType> vec(a, a + n);

...assuming a is your array and n is the number of elements it contains. Otherwise, std::copy() w/resize() will do the trick.

I'd stay away from memcpy() unless you can be sure that the values are plain-old data (POD) types.

Also, worth noting that none of these really avoids the for loop--it's just a question of whether you have to see it in your code or not. O(n) runtime performance is unavoidable for copying the values.

Finally, note that C-style arrays are perfectly valid containers for most STL algorithms--the raw pointer is equivalent to begin(), and (ptr + n) is equivalent to end().

C++ Writing to file vector of byte

To store a vector in a file, you have to write the contents of the vector, not the vector itself. You can access the raw data with &vector[0], address of the first element (given it contains at least one element).

ofstream outfile(filename, ios::out | ios::binary); 
outfile.write(&data[0], data.size());

This should be fairly efficient at writing. fstream is generic, use ofstream if you are going to write.

Is there a function to copy an array in C/C++?

Since C++11, you can copy arrays directly with std::array:

std::array<int,4> A = {10,20,30,40};
std::array<int,4> B = A; //copy array A into array B

Here is the documentation about std::array

Is it possible to extract data from std::vector without copying it? (and make the vector forget it)

No, It is not possible to extract part of data from vector as far as I know.

It is not compatible with structure of vector that provides its data in a continuous part of memory. std::vector memory is continues, so if it was possible to move part of its memory to another place, you need to shift reminder of memory to keep it continuous. It will be a huge burden itself.

I personally suggest to pass main vector by pointer/reference and use required parts directly as needed.

If you need to move whole data of std::vector to another place, you can just use std::move() to do so. You can even use std::swap() to swap contents of 2 vector together.

I need copyable buffer, as light as possible (e.g. not zero initialized)?

Std vector of a struct that is packed to 1 byte and whose default constructor leaves then contents uninitialized (ie does nothing) should work.

Note that you can do mass inserts into an empty vector, and the only initialization will be from the incoming data. Ie, vect.insert( vect.end(), random_access_iterator_begin, blah_end ) might have the performance you want.

How to copy the contents of std::vector to c-style static array,safely?

The problem is that you're adding things to the vector so it ends up with more elements than were in the myarr array that you initialized it with.

If you want to copy the vector back into the array, you'll need to size it down:

myvec.resize( MAX_SIZE);

Or you could limit the number of elements you copy back:

copy( myvec.begin(), myvec.begin()+MAX_SIZE, myarr);

If you want the myarr array to contain all the elements, then it needs to be larger than MAX_SIZE, and you've found out why people suggest to use vector rather than raw arrays (vectors know how to grow, arrays do not).

Note that while you don't want 'Any answer that resembles:"You use c++, drop the c style array implementation. Use only vector for all array implementation"', you can often get away with using a vector and passing &myvec[0] to routines that expect a raw array. vector is required to store its elements contiguously just like a raw array for just this reason.

Since you're getting the 'unsafe operation' warning, you're using Microsoft's compiler. To fix the problem safely, you're supposed to use the checked_copy algorithm instead of copy. As Evgeny Lazin indicates, you can create a checked iterator for your array to pass to the checked_copy algorithm.

Other options to make the copy safe that do not require Microsoft extensions would be to wrap the array in a class (possibly templated) that keeps track of the array size and provides methods to copy data into the array in a safe manner. Something like STLSoft's array_proxy template or Boost's boost::array might help.

Force file copy with very lower resources requirements

Re-reading this - "Now from time to time i like the program to copy/sync the files to our server. The challenge im facing is that i don't want the copy to use many system resources so it's not slowing the application or the writing of new files."

This means that you're not actually trying to avoid working the box hard; you just don't want to fall behind on your other file work.

I'd create a background thread that does all your disk access; and provide 2 queues for it to work on. The first being the new files to write; the 2nd being the files to copy. This thread can then use the disk at full speed by working on chunks at a time; with the priority being getting a chunk from the new file area. This will allow you to start doing a copy; and then stop copying entirely while there's lots of files to process (in case there's a burst) and then copy as fast as you can once it's all been processed.

---

Related Topics