Python Subprocess.Popen "Oserror: [Errno 12] Cannot Allocate Memory"

Python subprocess.Popen erroring with OSError: [Errno 12] Cannot allocate memory after period of time

when you use popen you need to hand in close_fds=True if you want it to close extra file descriptors.

creating a new pipe, which occurs in the _get_handles function from the back trace, creates 2 file descriptors, but your current code never closes them and your eventually hitting your systems max fd limit.

Not sure why the error you're getting indicates an out of memory condition: it should be a file descriptor error as the return value of pipe() has an error code for this problem.

Python script on ubuntu - OSError: [Errno 12] Cannot allocate memory

This error message...

    restore_signals, start_new_session, preexec_fn)
OSError: [Errno 12] Cannot allocate memory

...implies that the operating system was unable to allocate memory to initiate/spawn a new session.

Additionally, this error message...

urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='www.bandsintown.com', port=443): Max retries exceeded with url: /en/c/san-francisco-ca?page=6 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f90945757f0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',))

...implies that your program have successfully iterated till Page 5 and while on Page 6 you see this error.

I don't see any issues in your code block as such. I have taken your code, made some minor adjustments and here is the execution result:

Code Block:

options = webdriver.ChromeOptions() 
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
base_url = 'https://www.bandsintown.com/en/c/san-francisco-ca?page='
for i in range(1,10):
    #cycle through pages in range
    driver.get(base_url + str(i))
    pageURL = base_url + str(i)
    print(pageURL)

Console Output:

https://www.bandsintown.com/en/c/san-francisco-ca?page=1
https://www.bandsintown.com/en/c/san-francisco-ca?page=2
https://www.bandsintown.com/en/c/san-francisco-ca?page=3
https://www.bandsintown.com/en/c/san-francisco-ca?page=4
https://www.bandsintown.com/en/c/san-francisco-ca?page=5
https://www.bandsintown.com/en/c/san-francisco-ca?page=6
https://www.bandsintown.com/en/c/san-francisco-ca?page=7
https://www.bandsintown.com/en/c/san-francisco-ca?page=8
https://www.bandsintown.com/en/c/san-francisco-ca?page=9

Deep dive

This error is coming from subprocess.py:

self.pid = _posixsubprocess.fork_exec(
    args, executable_list,
    close_fds, tuple(sorted(map(int, fds_to_keep))),
    cwd, env_list,
    p2cread, p2cwrite, c2pread, c2pwrite,
    errread, errwrite,
    errpipe_read, errpipe_write,
    restore_signals, start_new_session, preexec_fn)

However, as per the discussion in OSError: [Errno 12] Cannot allocate memory this error OSError: [Errno 12] Cannot allocate memory is related to RAM / SWAP.

Swap Space

Swap Space is the memory space in the system hard drive that has been designated as a place for the os to temporarily store data which it can no longer hold with in the RAM. This gives you the ability to increase the amount of data your program can keep in its working memory. The swap space on the hard drive will be used primarily when there is no longer sufficient space in RAM to hold in-use application data. However, the information written to I/O will be significantly slower than information kept in RAM, but the operating system will prefer to keep running application data in memory and use swap space for the older data. Deploying swap space as a fall back for when your system’s RAM is depleted is a safety measure against out-of-memory issues on systems with non-SSD storage available.

System Check

To check if the system already has some swap space available, you need to execute the following command:

$ sudo swapon --show

If you don’t get any output, that means your system does not have swap space available currently. You can also verify that there is no active swap using the free utility as follows:

$ free -h

If there is no active swap in the system you will see an output as:

Output
               total        used       free        shared      buff/cache  available
Mem:           488M         36M        104M        652K        348M        426M
Swap:            0B          0B          0B

Creating Swap File

In these cases you need to allocate space for swap to use as a separate partition devoted to the task and you can create a swap file that resides on an existing partition. To create a 1 Gigabyte file you need to execute the following command:

$ sudo fallocate -l 1G /swapfile

You can verify that the correct amount of space was reserved by executing the following command:

$ ls -lh /swapfile

#Output
$ -rw-r--r-- 1 root root 1.0G Mar 08 10:30 /swapfile

This confirms the swap file has been created with the correct amount of space set aside.

Enabling the Swap Space

Once the correct size file is available we need to actually turn this into swap space. Now you need to lock down the permissions of the file so that only the users with specific privileges can read the contents. This prevents unintended users from being able to access the file, which would have significant security implications. So you need to follow the steps below:

Make the file only accessible to specific user e.g. root by executing the following command:
```
$ sudo chmod 600 /swapfile
```
Verify the permissions change by executing the following command:
```
$ ls -lh /swapfile

#Output
-rw------- 1 root root 1.0G Apr 25 11:14 /swapfile
```
This confirms only the root user has the read and write flags enabled.

Now you need to mark the file as swap space by executing the following command:

$ sudo mkswap /swapfile

#Sample Output
Setting up swapspace version 1, size = 1024 MiB (1073737728 bytes)
no label, UUID=6e965805-2ab9-450f-aed6-577e74089dbf

Next you need to enable the swap file, allowing the system to start utilizing it executing the following command:
```
$ sudo swapon /swapfile
```

You can verify that the swap is available by executing the following command:

$ sudo swapon --show

#Sample Output
NAME      TYPE  SIZE USED PRIO
/swapfile file 1024M   0B   -1

Finally check the output of the free utility again to validate the settings by executing the following command:

$ free -h

#Sample Output
          total        used        free      shared  buff/cache   available
Mem:           488M         37M         96M        652K        354M        425M
Swap:          1.0G          0B        1.0G

Conclusion

Once the Swap Space has been set up successfully the underlying operating system will begin to use it as necessary.

OSError: [Errno 12] Cannot allocate memory when using python multiprocessing Pool

os.fork() makes a copy of a process, so if you're sitting at about 18 GB of usage, and want to call fork, you need another 18 GB. Twice 18 is 36 GB, which is well over 32 GB. While this analysis is (intentionally) naive—some things don't get copied on fork—it's probably sufficient to explain the problem.

The solution is either to make the pools earlier, when less memory needs to be copied, or to work harder at sharing the largest objects. Or, of course, add more memory (perhaps just virtual memory, i.e., swap space) to the system.

Python multiprocessing - Debugging OSError: [Errno 12] Cannot allocate memory

We had this a couple of time. According to my sys admin, there is "a bug" in unix, which will raise the same error if you are out of memory, of if your process reach the max file descriptor limit.

We had a leak of file descriptor, and the error raising was [Errno 12] Cannot allocate memory#012OSError.

So you should look at your script and double check if the problem is not the creation of too many FD instead