Python 32-Bit Memory Limits on 64Bit Windows

Python 32-bit memory limits on 64bit windows

I don't know why you think your process should be able to access 4GB. According to Memory Limits for Windows Releases at MSDN, on 64-bit Windows 7, a default 32-bit process gets 2GB.* Which is exactly where it's running out.

So, is there a way around this?

Well, you could make a custom build of 32-bit Python that uses the IMAGE_FILE_LARGE_ADDRESS_AWARE flag, and rebuild numpy and all of your other extension modules. I can't promise that all of the relevant code really is safe to run with the large-address-aware flag; there's a good chance it is, but unless someone's already done it and tested it, "a good chance" is the best anyone is likely to know.

Or, more obviously, just use 64-bit Python instead.

---

The amount of physical RAM is completely irrelevant. You seem to think that you have
an "8GB limit" with 8GB of RAM, but that's not how it works. Your system takes all of your RAM plus whatever swap space it needs and divides it up between apps; an app may be able to get 20GB of virtual memory without getting a memory error even on an 8GB machine. And meanwhile, a 32-bit app has no way of accessing more than 4GB, and the OS will use up some of that address space (half of it by default, on Windows), so you can only get 2GB even on an 8GB machine that's not running anything else. (Not that it's possible to ever be "not running anything else" on a modern OS, but you know what I mean.)

---

So, why does this work on your linux box?

Because your linux box is configured to give 32-bit processes 3.5GB of virtual address space, or 3.99GB, or… Well, I can't tell you the exact number, but every distro I've seen for many years has been configured for at least 3.25GB.

---

* Also note that you don't even really get that full 2GB for your data; your program. Most of what the OS and its drivers make accessible to your code sits in the other half, but some bits sit in your half, along with every DLL you load and any space they need, and various other things. It doesn't add up to too much, but it's not zero.

How much memory can a 32 bit process access on a 64 bit operating system?

2 GB by default. If the application is large address space aware (linked with /LARGEADDRESSAWARE), it gets 4 GB (not 3 GB, see http://msdn.microsoft.com/en-us/library/aa366778.aspx)

They're still limited to 2 GB since many application depends on the top bit of pointers to be zero.

32 bit applications memory limit on a 64 bit OS (Windows)

Processes do not need to use the same address space. That means multiple 32-bit applications running simultaneously can effectively use all the memory available on a 64-bit machine.

Python speed 32 v 64 bit Windows issue

Problem is caused by use of virtural machine software on the windows server.

How do I get Python 3.7 to use more than 2 GB memory? (64bit setup)

Thank's to many helpful comments (geckos, jammin0921, Óscar López, and Heap Overflow) it looks like what I was observing was not a limitation of Python, but rather that the apparently clever data-management by Python/Pandas meant that once the 12 GB of .txt-files had been read into DataFrames, their total size was actually below 2 GB, by looking at the memory usage of the dataframe (df): df.memory_usage(True, True).sum() which gave 1.9 GB

Having tried to manipulate this further by further increasing size of the data I read in, I do see RAM usage above 2 GB from python3.7 process.

---

Related Topics