Encrypting/Decrypting Large Files (.Net)

encrypting and/or decrypting large files (AES) on a memory and storage constrained system, with catastrophe recovery

It is doable, provided you have a means to save the AES status vector together with the file position.

1. Save AES status and file position P to files STAGE1 and STAGE2
2. Read one chunk (say, 10 megabytes) of encrypted/decrypted data
3. Write the decrypted/encrypted chunk to external scratch SCRATCH
4. Log the fact that SCRATCH is completed
5. Write SCRATCH over the original file at the same position
6. Log the fact that SCRATCH has been successfully copied
7. Goto 1

If you get a hard crash after stage 1, and STAGE1 and STAGE2 disagree, you just restart and assume the stage with the earliest P to be good.
If you get a hard crash during or after stage 2, you lose 10 megabytes worth of work: but the AES and P are good, so you just repeat stage 2.
If you crash at stage 3, then on recovery you won't find the marker of stage 4, and so will know that SCRATCH is unreliable and must be regenerated. Having STAGE1/STAGE2, you are able to do so.
If you crash at stage 4, you will BELIEVE that SCRATCH must be regenerated, even if you could avoid this -- but you lose nothing in regenerating except a little time.
By the same token, if you crash during 5, or before 6 is committed to disk, you just repeat stages 5 and 6. You know you don't have to regenerate SCRATCH because stage 4 was committed to disk. If you crash after stage 1, you will still have a good SCRATCH to copy.

All this assumes that 10 MB is more than a cache's (OS + hard disk if writeback) worth of data. If it is not, raise to 32 or 64 MB. Recovery will be proportionately slower.

It might help to flush() and sync(), if these functions are available, after every write-stage has been completed.

Total write time is a bit more than twice normal, because of the need of "writing twice" in order to be sure.

Decryption of large file with AES

Try using streams, especially CryptoStream. The Microsoft example at the bottom of this page actually perform file based encryption with RijndaelManaged so you're in luck. You would first need to extract the IV from the file stream of course, e.g. by reading exactly 16 bytes byte-by-byte. Only wrap the stream after reading the IV.

That way there is no need for memory consumption other than a buffer size, which should range in a few KiB maximum.

Flutter encrypt large files

Generally speaking, you need a temporary buffer to hold your data. If your RAM is not large enough (very likely on mobile devices) it has to be the disk.

So create a second file, and read the first file in batches of bytes that are small enough your memory will be able to handle it. Your encryption method should be able to handle this, as it's a very common occurrence. Write the resulting batches of encrypted content to the second file. Once you are done, delete/overwrite the original.

---

Related Topics