Setting Stacksize in a Python Script

Setting stacksize in a python script

I have good experience with the following code. It doesn't require any special user permissions:

import resource, sys
resource.setrlimit(resource.RLIMIT_STACK, (2**29,-1))
sys.setrecursionlimit(10**6)

It does however not seem to work with pypy.

If resource.setrlimit doesn't work, you can also try using threads:

sys.setrecursionlimit(10**6)
import threading
threading.stack_size(2**26)
threading.Thread(target=main).start()

How do I increase the stack size in python

AFAIK a program can only change the stack size of new threads or processes (Windows' CreateThread function, for example). As Python (and the Win32 API for Python) does not expose such functionality, you should rather replace the stack allocation with heap memory. Or is there a specific reason for using the stack?? If you really have to use alloca you might want to create a separate thread for execution of DLL code (which is overkill I think).

EDIT: Correction - Python does allow for setting the stack size when creating new threads (see thread.stack_size)

How to change the stack size of subprocess in Python

by sheer luck I found out that although the same CMakeLists.txt is used for creating the projects (locally and on Jenkins) the resulting projects generated from them differ.

My local projects (and the GUI-VisualStudio solution I manually created on the Server for error finding) had 10MB of stack reserved whereas the python-based call to CMake was only using the default value of 1MB which is not enough for the project.

This strange behavior of Cmake may be compensated for by adding this line to CMakeLists.txt:

# Set linker to use 10MB of stack memory for all gtest executables, the default of 1MB is not enough!
SET( CMAKE_EXE_LINKER_FLAGS  "${CMAKE_EXE_LINKER_FLAGS} /STACK:\"10000000\"")

sorry to bother you :)
I'm glad it's over

Increase recursion limit and stack size in python 2.7

To my experience you have a problem not with stack size, but with an algorithm itself.

It's possible to implement tree traversal procedure without recursion at all. You should implement stack-based depth/breadth first search algorithm.
Python-like pseudo-code might look like this:

stack = []
def traverse_tree(root):
  stack.append(root)
  while stack:
    cur = stack.pop()
    cur.do_some_awesome_stuff()
    stack.append(cur.get_children())

This approach is incredibly scalable and allows you to deal with any trees.

As further reading you can try this and that.

What is the maximum recursion depth in Python, and how to increase it?

It is a guard against a stack overflow, yes. Python (or rather, the CPython implementation) doesn't optimize tail recursion, and unbridled recursion causes stack overflows. You can check the recursion limit with sys.getrecursionlimit:

import sys
print(sys.getrecursionlimit())

and change the recursion limit with sys.setrecursionlimit:

sys.setrecursionlimit(1500)

but doing so is dangerous -- the standard limit is a little conservative, but Python stackframes can be quite big.

Python isn't a functional language and tail recursion is not a particularly efficient technique. Rewriting the algorithm iteratively, if possible, is generally a better idea.

Change default stack size for spark driver running from jupyter?

You can customize the Java options used for the driver by passing spark.driver.extraJavaOptions as a configuration value into the SparkConf, eg:

from pyspark import SparkConf, SparkContext
conf = (SparkConf()
     .setMaster("spark://spark-master:7077")
     .setAppName("MyApp")
     .set("spark.driver.extraJavaOptions", "-Xss4M"))
sc = SparkContext.getOrCreate(conf = conf)

Note that in http://spark.apache.org/docs/latest/configuration.html it states about spark.driver.extraJavaOptions:

Note: In client mode, this config must not be set through the SparkConf directly in your application, because the driver JVM has already started at that point. Instead, please set this through the --driver-java-options command line option or in your default properties file.

However this is talking about the JVM SparkConf class. When it’s set in the PySpark Python SparkConf, that passes it as a command-line parameter to spark-submit, which then uses it when instantiating the JVM, so that comment in the Spark docs does not apply.

Python threading.stack_size() has no effect

The minimum thread stack size on Windows is probably at least 64kB. Quoting:

The operating system rounds up the specified size to the nearest multiple of the
system's allocation granularity (typically 64 KB). To retrieve the allocation
granularity of the current system, use the GetSystemInfo function.

So trying to set it to 32kB will probably look a lot like trying to set it to 64kB.

Additionally, CPython implements threading.stack_size on Windows so that it only controls the initially committed stack. It does not try to control the reserve memory for the stack. From the same location:

The reserved memory size represents the total stack allocation in virtual memory.

This means that each of your threads uses up the reserve memory size in virtual memory. You didn't mention how many threads you manage to create before encountering an error but I suspect it's just enough to exhaust addressable memory in your process (which is probably a 32 bit process even though you're running it on Windows 7 x86-64 because the CPython build/distribution is x86(-32)).

That is, even though you (your threads) are not using the memory and even though you have more physical memory on the system, Python can't address the extra memory with its tiny 32 bit pointers and so new threads beyond the limit you're encountering can't have their reserve memory allocated (because there are no addresses left to assign to it).

If you want to be able to change the per-thread reserved memory then you probably need to call CreateThread or _beginthreadex differently than CPython calls it. This probably means need to get CPython changed.

That said, and at the risk of having you yell at me, I seriously doubt you need more than the 1500 threads you can already create.

---

Related Topics