Linux/Perl Mmap Performance

Linux/perl mmap performance

Ok, found the problem. As suspected, neither linux or perl were to blame. To open and access the file I do something like this:

#!/usr/bin/perl
# Create 1 GB file if you do not have one:
# dd if=/dev/urandom of=test.bin bs=1048576 count=1000
use strict; use warnings;
use Sys::Mmap;

open (my $fh, "<test.bin")
    || die "open: $!";

my $t = time;
print STDERR "mmapping.. ";
mmap (my $mh, 0, PROT_READ, MAP_SHARED, $fh)
    || die "mmap: $!";
my $str = unpack ("A1024", substr ($mh, 0, 1024));
print STDERR " ", time-$t, " seconds\nsleeping..";

sleep (60*60);

If you test that code, there are no delays like those I found in my original code, and after creating the minimal sample (always do that, right!) the reason suddenly became obvious.

The error was that I in my code treated the $mh scalar as a handle, something which is light weight and can be moved around easily (read: pass by value). Turns out, it's actually a GB long string, definitively not something you want to move around without creating an explicit reference (perl lingua for a "pointer"/handle value). So if you need to store in in a hash or similar, make sure you store \$mh, and deref it when you need to use it like ${$hash->{mh}}, typically as the first parameter in a substr or similar.

Linux: Large int array: mmap vs seek file?

I'd say performance should be similar if access is truly random. The OS will use a similar caching strategy whether the data page is mapped from a file or the file data is simply cached without an association to RAM.

Assuming cache is ineffective:

• You can use fadvise to declare your access pattern in advance and disable readahead.
• Due to address space layout randomization, there might not be a contiguous block of 4 TB in your virtual address space.
• If your data set ever expands, the address space issue might become more pressing.

So I'd go with explicit reads.

Linux non-persistent backing store for mmap()

Yes, and I do this all the time...

open the file, unlink it, use ftruncate or (better) posix_fallocate to make it the right size, then use mmap with MAP_SHARED to map it into your address space. You can then close the descriptor immediately if you want; the memory mapping itself will keep the file around.

For speed, you might find you want to help Linux manage its page cache. You can use posix_madvise with POSIX_MADV_WILLNEED to advise the kernel to page data in and POSIX_MADV_DONTNEED to advise the kernel to release the pages.

You might find that last does not work the way you want, especially for dirty pages. You can use sync_file_range to explicitly control flushing to disk. (Although in that case you will want to keep the file descriptor open.)

All of this is perfectly standard POSIX except for the Linux-specific sync_file_range.

What is the fastest way to read 10 GB file from the disk?

Most of the time you will be I/O bound not CPU bound, thus just read this file through normal Perl I/O and process it in single thread. Unless you prove that you can do more I/O than your single CPU work, don't waste your time with anything more. Anyway, you should ask: Why on Earth is this in one huge file? Why on Earth don't they split it in a reasonable way when they generate it? It would be magnitude more worth work. Then you can put it in separate I/O channels and use more CPU's (if you don't use some sort of RAID 0 or NAS or ...).

Measure, don't assume. Don't forget to flush caches before each test. Remember that serialized I/O is a magnitude faster than random.

Increasing a file's size using mmap

On POSIX systems at least, mmap() cannot be used to increase (or decrease) the size of a file. mmap()'s function is to memory map a portion of a file. It's logical that the thing you request to map should actually exist! Frankly, I'm really surprised that you would actually be able to do this under MS Windows.

If you want to grow a file, just ftruncate() it before you mmap() it.

Python multiprocessing text extraction performance issue vs Perl equivalent

EDIT

Forgot to add thanks to the contributors on this thread:

Python slow read performance issue

who helped me solve this.

EDIT

It all boiled down in the end to the order of the directory read, this applied to my main application as well as the tests.

Basically Perl sorts lexographically (i.e. 1,11,2,22) by default, Python sorts by directory order (ls -U) and the files are created in a natural order (1,2,3,4) so I took the original Python slurp and created a slurpNatural after some searching Stackoverflow for a simple natural sort:

import glob, sys, re

def natural_sort_key(s, _nsre=re.compile('([0-9]+)')):
    return [int(text) if text.isdigit() else text.lower()
            for text in re.split(_nsre, s)]

for file in sorted(glob.iglob(sys.argv[1] + '/*.xml'), key=natural_sort_key):
    with open(file) as x:
        f = x.read()

I then ran all 3 against the 50K docs and got:

$ sync; sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
$ /usr/bin/time perl slurp.pl 1
1.21user 2.17system 0:12.70elapsed 26%CPU (0avgtext+0avgdata 9140maxresident)k
1234192inputs+0outputs (22major+2466minor)pagefaults 0swaps

$ sync; sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
$ /usr/bin/time python slurp.py 1
2.88user 6.13system 4:48.00elapsed 3%CPU (0avgtext+0avgdata 8020maxresident)k
1236304inputs+0outputs (35major+52252minor)pagefaults 0swaps

$ sync; sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
$ /usr/bin/time python slurpNatural.py 1
1.25user 2.82system 0:10.70elapsed 38%CPU (0avgtext+0avgdata 22408maxresident)k
1237360inputs+0outputs (35major+56531minor)pagefaults 0swaps

The natural sort which mirrors the creation order is clearly the fastest and in this case mirrors how my actual data is created and so have now changed the Python to sort the directory contents before processing.

Thanks for all the help, I honestly didn't think the order of reading the files would make such a big difference!

Si

---

Related Topics