JavaScript Filereader - Parsing Long File in Chunks

filereader api on big files

Your application is failing for big files because you're reading the full file into memory before processing it. This inefficiency can be solved by streaming the file (reading chunks of a small size), so you only need to hold a part of the file in memory.

A File objects is also an instance of a Blob, which offers the .slice method to create a smaller view of the file.

Here is an example that assumes that the input is ASCII (demo: http://jsfiddle.net/mw99v8d4/).

function findColumnLength(file, callback) {
    // 1 KB at a time, because we expect that the column will probably small.
    var CHUNK_SIZE = 1024;
    var offset = 0;
    var fr = new FileReader();
    fr.onload = function() {
        var view = new Uint8Array(fr.result);
        for (var i = 0; i < view.length; ++i) {
            if (view[i] === 10 || view[i] === 13) {
                // \n = 10 and \r = 13
                // column length = offset + position of \r or \n
                callback(offset + i);
                return;
            }
        }
        // \r or \n not found, continue seeking.
        offset += CHUNK_SIZE;
        seek();
    };
    fr.onerror = function() {
        // Cannot read file... Do something, e.g. assume column size = 0.
        callback(0);
    };
    seek();

    function seek() {
        if (offset >= file.size) {
            // No \r or \n found. The column size is equal to the full
            // file size
            callback(file.size);
            return;
        }
        var slice = file.slice(offset, offset + CHUNK_SIZE);
        fr.readAsArrayBuffer(slice);
    }
}

The previous snippet counts the number of bytes before a line break. Counting the number of characters in a text consisting of multibyte characters is slightly more difficult, because you have to account for the possibility that the last byte in the chunk could be a part of a multibyte character.

How to read large video files in JavaScript using FileReader?

The FileReader class in JavaScript contains multiple methods to read files:

• readAsText(): This reads a file and returns its content as text. Suitable for small text files.
• readAsBinaryString(): This reads a file and returns its content as a binary string. Suitable for small files of any type.
• readAsDataURL(): This reads a file and returns a Data URL referencing it. This is inefficient for large files as the file is loaded into memory as a whole before being processed.
• readAsArrayBuffer(): This reads a file and returns an ArrayBuffer containing the input file 'chopped up in smaller pieces'. This works for very large files, too.

In the question, the readAsDataURL() method is used as it is usually most convenient. However, for very large video files (and very large files in general) it does not work for the reason described above leading to an empty result. Instead, you should use readAsArrayBuffer():

let reader = new  FileReader();
reader.readAsArrayBuffer(file);

Now, the file reader returns an ArrayBuffer after loading the file. In order to be able to show the video in HTML, we have to convert this buffer to a blob, that can then give us a URL to the video file:

reader.onload = function(e) {
  // The file reader gives us an ArrayBuffer:
  let buffer = e.target.result;

  // We have to convert the buffer to a blob:
  let videoBlob = new Blob([new Uint8Array(buffer)], { type: 'video/mp4' });

  // The blob gives us a URL to the video file:
  let url = window.URL.createObjectURL(videoBlob);

  video.src = url;
}

Related Topics