What's the Max File Mapping Size in 64Bits MAChine

CreateFileMapping size limit in a 64-bit process

There is no 2GB limit for file mappings. You can boot 32-bit Windows with the 3GB option or when a 32-bit process running on a 64-bit system, you get the full 4GB if the correct PE flag is set. All these limits are theoretical and you will never reach them in practice.

How large of a view you can map depends on two things;

1. The contiguous range of free addresses in your processes address space.
2. Available kernel memory for keeping track of the memory pages.

The first one is the big limit on 32-bit systems since the address space of your process is shared with system libraries, 3rd-party libraries (anti-virus, injected "tweaking" tools etc.), the PEB and TEBs, the system region, thread stacks and memory reserved by hardware. This will often put you well below 2GB. Any design requiring more than 500MB should probably be changed to only map in specific smaller ranges as needed.

For a 64-bit process on 64-bit Windows, the virtual address space is the 128-terabyte range 0x000'00000000 through 0x7FFF'FFFFFFFF (KB889654 claims 8 TB but that only applies to < Windows 8.1). Any usable range is going to be smaller but you can assume a couple of terabyte at least. 40GB is no problem and not enough to run into problems with low system resources either.

How big can a memory-mapped file be?

You're being too conservative: A memory-mapped file can be larger than the address space. The view of the memory-mapped file is limited by OS memory constraints, but that's only the part of the file you're looking at at one time. (And I guess technically you could map multiple views of discontinuous parts of the file at once, so aside from overhead and page length constraints, it's only the total # of bytes you're looking at that poses a limit. You could look at bytes [0 to 1024] and bytes [2⁴⁰ to 2⁴⁰ + 1024] with two separate views.)

In MS Windows, look at the MapViewOfFile function. It effectively takes a 64-bit file offset and a 32-bit length.

what is the maximum size of a PE file on 64-bit Windows?

According to the spec it is 32-bit unsigned value for a PE32+ image just like a PE32 image.

However, in my testing with both 32-bit and 64-bit applications (PE32/PE32+ files) on Windows 7 SP1 Home Premium x64, the maximum file size for either is between 1.8-1.85GB.

I tested by creating a very basic C executable with Visual Studio (~8K for 32-bit and 9K for 64-bit) and the added an empty code section to the PE header until Windows would no longer load it, and then binary searched for the limit. Looking at the process with vmmap showed that almost all of the entire first 2GB of address space were the image (including any subsequently loaded DLLs such as kernel32.dll). The limit was the same for me with both 32 and 64-bit processes. The 64-bit process did have the flag set in it's NT Header's File Header's section stating that it could handle addresses >2GB. It also could allocate memory for non-image sections above the 2GB limit.

It seems like the image is required to fit in it's entirety in the lower 2GB of VA space for the process, which means the SizeOfImage is being treated a signed 32-bit integer by the loader effectively.

Memory mapping a huge file in 32 bit software running on 64 bit OS

You can't. A 32-bit program uses 32-bit pointers that can't go past 4GB, even when running on a 64-bit OS.

One thing that may help a little, though, is to link your program with the /LARGEADDRESSAWARE option. By default, 32-bit Windows programs are only able to use 2GB of the address space, because some programs assume that a pointer's uppermost bit is always zero and use it as a flag to store additional information. That'd break if Windows started allocating memory above 2GB, so you have to link your program with a special option that tells Windows your program doesn't abuse that bit. This lets your program use the full 4GB of address space instead of being limited to 2GB.

What is the largest possible heap size with a 64-bit JVM?

If you want to use 32-bit references, your heap is limited to 32 GB.

However, if you are willing to use 64-bit references, the size is likely to be limited by your OS, just as it is with 32-bit JVM. e.g. on Windows 32-bit this is 1.2 to 1.5 GB.

Note: you will want your JVM heap to fit into main memory, ideally inside one NUMA region. That's about 1 TB on the bigger machines. If your JVM spans NUMA regions the memory access and the GC in particular will take much longer. If your JVM heap start swapping it might take hours to GC, or even make your machine unusable as it thrashes the swap drive.

Note: You can access large direct memory and memory mapped sizes even if you use 32-bit references in your heap. i.e. use well above 32 GB.

Compressed oops in the Hotspot JVM

Compressed oops represent managed pointers (in many but not all places in the JVM) as 32-bit values which must be scaled by a factor of 8 and added to a 64-bit base address to find the object they refer to. This allows applications to address up to four billion objects (not bytes), or a heap size of up to about 32Gb. At the same time, data structure compactness is competitive with ILP32 mode.

What is the largest file size that can be opened by Vim? (64-bit)

The documentation just states this:

Maximum file size 2147483647 bytes (2 Gbyte) when a long integer is
32 bits. Much more for 64 bit longs. Also limited
by available disk space for the |swap-file|.

so "much more" will most likely run into your hard drive limitation since it goes to exabytes of alocatable size.

Memory-mapping huge files in Java

Take a look at Using a memory mapped file for a huge matrix code which shows how to create a list of MappedByteBuffer, each smaller then 2 GB, to map the entire file:

private static final int MAPPING_SIZE = 1 << 30;
...
long size = 8L * width * height;
for (long offset = 0; offset < size; offset += MAPPING_SIZE) {
    long size2 = Math.min(size - offset, MAPPING_SIZE);
    mappings.add(raf.getChannel().map(FileChannel.MapMode.READ_WRITE, offset, size2));
}

As per JDK-6347833 (fs) Enhance MappedByteBuffer to support sizes >2GB on 64 bit platforms the reason for the 2 GB limit is:

A MappedByteBuffer is a ByteBuffer with additional operations to support memory-mapped file regions. To support mapping a region larger than Integer.MAX_VALUE would require a parallel hierarchy of classes. For now the only solution is create multiple MappedByteBuffers where each corresponds to a region that is no larger than 2GB.

---

Related Topics