Determine How Many Times File Is Mapped into Memory

Determine how many times file is mapped into memory

You can check all the /proc/*/maps files and count the number of times the memory-mapped file is mentioned.

For example, this is how the memory mapped "/tmp/delme" is mentioned

7fdb737d0000-7fdb737d1000 rw-s 00000000 08:04 13893648                   /tmp/delme
7fdb737d8000-7fdb737d9000 rw-s 00000000 08:04 13893648                   /tmp/delme

when using the following code:

// g++ -std=c++11 delme.cc && ./a.out
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <sys/mman.h>
#include <unistd.h>
#include <fstream>
#include <iostream>

int main() {
  int fileDescriptor = open("/tmp/delme", O_RDWR, 0664);
  if (fileDescriptor < 0) return false;
  auto memorymap = mmap (NULL, 123, PROT_READ | PROT_WRITE, MAP_SHARED, fileDescriptor, 0);
  auto memorymap2 = mmap (NULL, 123, PROT_READ | PROT_WRITE, MAP_SHARED, fileDescriptor, 0);
  close (fileDescriptor);
  std::ifstream mappings ("/proc/self/maps");
  std::cout << mappings.rdbuf() << std::endl;
}

See also Understanding Linux /proc/id/maps.

If you issue a global lock, preventing any mappings and unmappings from happening while the counting is taking the place, then this counter will be 100% correct.

Can I have a memory mapped file, mapped to two or more processes at the same time (windows)?

You don't need to unmap the file after reading or writing at all. Windows guarantees that the data "visible" in the mapping in two processes will be the same when the local file is mapped on one computer.

Frequently Accessing File Mapped Memory

You ask about accessing repeatedly a 1-2 GB range of a memory-mapped file. The short answer is that yes, it will have a performance downside, especially if your pages are small.

The first problem is that repeatedly accessing 1 GB of any memory will (by default) tend to evict whatever other data you have, and CPU data cache hit rates will suffer. Consider that a typical CPU data cache size is just a few megabytes.

The second problem is the Translation Lookaside Buffer. If you're using the default page size of 4 KB, then 1 GB is 250k pages. On Linux you should enable "huge pages" to reduce the TLB load imposed by accessing such a large region of memory.

The third problem is that it seems you haven't tested it. Testing is very important for this sort of thing. Without it, you'll never know if you have a real problem or only a hypothetical one.

Ensure memory mapped page is in memory

If you need to just guarantee there will be no network access, you can read required regions to a private memory using ReadFile (may be done asynchronously).

Another (hacky) way is to use copy-on-write (FILE_MAP_COPY) mapping and cause fake page modification. This allows you to have a contiguous memory mapping that might be easier to navigate, but with certain pages being kept in a private memory. This has disadvantage of causing increased memory pressure as OS reserves memory for the whole mapping.

How to find the end of a memory mapped file in Windows platform without previously knowing the size of the file?

Call GetFileSizeEx to get the size of a file, and use this in combination with the base address and the current read address to determine where the end address is.

what is the suggested number of bytes each time for files too large to be memory mapped at one time?

There is no optimal size. With a 32-bit process, there is only 4 GB of address space total, and usually only 2 GB is available for user mode processes. This 2 GB is then fragmented by code and data from the exe and DLL's, heap allocations, thread stacks, and so on. Given this, you will probably not find more than 1 GB of contigous space to map a file into memory.

The optimal number depends on your app, but I would be concerned mapping more than 512 MB into a 32-bit process. Even with limiting yourself to 512 MB, you might run into some issues depending on your application. Alternatively, if you can go 64-bit there should be no issues mapping multiple gigabytes of a file into memory - you address space is so large this shouldn't cause any issues.

You could use an API like VirtualQuery to find the largest contigous space - but then your actually forcing out of memory errors to occur as you are removing large amounts of address space.

EDIT: I just realized my answer is Windows specific, but you didn't which platform you are discussing. I presume other platforms have similar limiting factors for memory-mapped files.

---

Related Topics