Filereader API on Big Files

filereader api on big files

Your application is failing for big files because you're reading the full file into memory before processing it. This inefficiency can be solved by streaming the file (reading chunks of a small size), so you only need to hold a part of the file in memory.

A File objects is also an instance of a Blob, which offers the .slice method to create a smaller view of the file.

Here is an example that assumes that the input is ASCII (demo: http://jsfiddle.net/mw99v8d4/).

function findColumnLength(file, callback) {
    // 1 KB at a time, because we expect that the column will probably small.
    var CHUNK_SIZE = 1024;
    var offset = 0;
    var fr = new FileReader();
    fr.onload = function() {
        var view = new Uint8Array(fr.result);
        for (var i = 0; i < view.length; ++i) {
            if (view[i] === 10 || view[i] === 13) {
                // \n = 10 and \r = 13
                // column length = offset + position of \r or \n
                callback(offset + i);
                return;
            }
        }
        // \r or \n not found, continue seeking.
        offset += CHUNK_SIZE;
        seek();
    };
    fr.onerror = function() {
        // Cannot read file... Do something, e.g. assume column size = 0.
        callback(0);
    };
    seek();

    function seek() {
        if (offset >= file.size) {
            // No \r or \n found. The column size is equal to the full
            // file size
            callback(file.size);
            return;
        }
        var slice = file.slice(offset, offset + CHUNK_SIZE);
        fr.readAsArrayBuffer(slice);
    }
}

The previous snippet counts the number of bytes before a line break. Counting the number of characters in a text consisting of multibyte characters is slightly more difficult, because you have to account for the possibility that the last byte in the chunk could be a part of a multibyte character.

Read huge files in browser using FileReader (web api)

An ArrayBuffer is really a buffer, an in-memory buffer. That's how buffers work. Your code tries to load the whole file into memory. To access specific ranges of a file without loading the whole into memory, you must use Blob.slice (Files implement all the methods of Blobs) as you suspected.

How to load large local files using JavaScript?

FileReader enables you to read contents of files asynchronously. With respect to large file (2GB in your case), you can use function/method FileReader.readAsArrayBuffer() to read a certain chunk size of a file in the memory hence this won't crash your browser, this blog is a good example.

Using Javascript FileReader with huge files

This will only read the first 10 mb:

var file = form.getEl().down('input[type=file]').dom.files[0];
var reader = new FileReader();

reader.onload = function(e) {
    var data = e.target.result;
    form.displayedData = data;
};

reader.readAsText(file.slice(0, 10 * 1024 * 1024));

How to read large video files in JavaScript using FileReader?

The FileReader class in JavaScript contains multiple methods to read files:

• readAsText(): This reads a file and returns its content as text. Suitable for small text files.
• readAsBinaryString(): This reads a file and returns its content as a binary string. Suitable for small files of any type.
• readAsDataURL(): This reads a file and returns a Data URL referencing it. This is inefficient for large files as the file is loaded into memory as a whole before being processed.
• readAsArrayBuffer(): This reads a file and returns an ArrayBuffer containing the input file 'chopped up in smaller pieces'. This works for very large files, too.

In the question, the readAsDataURL() method is used as it is usually most convenient. However, for very large video files (and very large files in general) it does not work for the reason described above leading to an empty result. Instead, you should use readAsArrayBuffer():

let reader = new  FileReader();
reader.readAsArrayBuffer(file);

Now, the file reader returns an ArrayBuffer after loading the file. In order to be able to show the video in HTML, we have to convert this buffer to a blob, that can then give us a URL to the video file:

reader.onload = function(e) {
  // The file reader gives us an ArrayBuffer:
  let buffer = e.target.result;

  // We have to convert the buffer to a blob:
  let videoBlob = new Blob([new Uint8Array(buffer)], { type: 'video/mp4' });

  // The blob gives us a URL to the video file:
  let url = window.URL.createObjectURL(videoBlob);

  video.src = url;
}

Chrome FileReader returns empty string for big files (= 300MB)

Is this a chrome bug?

As I said in my answer to Chrome, FileReader API, event.target.result === "", this a V8 (Chrome's but also node-js's and others' JavaScript JS engine) limitation.

It is intentional and thus can't really qualify as "a bug".

The technicalities are that what actually fails here is to build a String of more than 512MB (less the header) on 64bit systems because in V8 all heap objects must fit in a Smi (Small Integer), (cf this commit).

Why is there neither an error nor an exception?

That, might be a bug... As I also show in my linked answer, we get a RangeError when creating such a string directly:

const header = 24;
const bytes = new Uint8Array( (512 * 1024 * 1024) - header );
let txt = new TextDecoder().decode( bytes );
console.log( txt.length ); // 536870888
txt += "f"; // RangeError

---

Related Topics