Upper Memory Limit

Upper memory limit?

(This is my third answer because I misunderstood what your code was doing in my original, and then made a small but crucial mistake in my second—hopefully three's a charm.

Edits: Since this seems to be a popular answer, I've made a few modifications to improve its implementation over the years—most not too major. This is so if folks use it as template, it will provide an even better basis.

As others have pointed out, your MemoryError problem is most likely because you're attempting to read the entire contents of huge files into memory and then, on top of that, effectively doubling the amount of memory needed by creating a list of lists of the string values from each line.

Python's memory limits are determined by how much physical ram and virtual memory disk space your computer and operating system have available. Even if you don't use it all up and your program "works", using it may be impractical because it takes too long.

Anyway, the most obvious way to avoid that is to process each file a single line at a time, which means you have to do the processing incrementally.

To accomplish this, a list of running totals for each of the fields is kept. When that is finished, the average value of each field can be calculated by dividing the corresponding total value by the count of total lines read. Once that is done, these averages can be printed out and some written to one of the output files. I've also made a conscious effort to use very descriptive variable names to try to make it understandable.

try:
    from itertools import izip_longest
except ImportError:    # Python 3
    from itertools import zip_longest as izip_longest

GROUP_SIZE = 4
input_file_names = ["A1_B1_100000.txt", "A2_B2_100000.txt", "A1_B2_100000.txt",
                    "A2_B1_100000.txt"]
file_write = open("average_generations.txt", 'w')
mutation_average = open("mutation_average", 'w')  # left in, but nothing written

for file_name in input_file_names:
    with open(file_name, 'r') as input_file:
        print('processing file: {}'.format(file_name))

        totals = []
        for count, fields in enumerate((line.split('\t') for line in input_file), 1):
            totals = [sum(values) for values in
                        izip_longest(totals, map(float, fields), fillvalue=0)]
        averages = [total/count for total in totals]

        for print_counter, average in enumerate(averages):
            print('  {:9.4f}'.format(average))
            if print_counter % GROUP_SIZE == 0:
                file_write.write(str(average)+'\n')

file_write.write('\n')
file_write.close()
mutation_average.close()

Increasing (or decreasing) the memory available to R processes

From:

http://gking.harvard.edu/zelig/docs/How_do_I2.html (mirror)

Windows users may get the error that R
has run out of memory.

If you have R already installed and
subsequently install more RAM, you may
have to reinstall R in order to take
advantage of the additional capacity.

You may also set the amount of
available memory manually. Close R,
then right-click on your R program
icon (the icon on your desktop or in
your programs directory). Select
``Properties'', and then select the
``Shortcut'' tab. Look for the
``Target'' field and after the closing
quotes around the location of the R
executible, add

--max-mem-size=500M

as shown in the figure below. You may
increase this value up to 2GB or the
maximum amount of physical RAM you
have installed.

If you get the error that R cannot
allocate a vector of length x, close
out of R and add the following line to
the ``Target'' field:

--max-vsize=500M

or as appropriate. You can always
check to see how much memory R has
available by typing at the R prompt

memory.limit()

which gives you the amount of available memory in MB. In previous versions of R you needed to use: round(memory.limit()/2^20, 2).

Docker container memory upper limit

Ref. the user memory constraints:

Examples:

$ docker run -it ubuntu:14.04 /bin/bash

We set nothing about memory, this means the processes in the container can use as much memory and swap memory as they need.

Unless you explicitly limit the memory (--memory) then your limit will be the amount of memory assigned to the host. docker stats <container> displays the memory limits (among other stats):

Why does the (Oracle) JVM have a fixed upper limit for memory usage (-Xmx)?

Hm, I'll try summarizing the answers so far.

There is no technical reason why the JVM needs to have a hard limit for its heap size. It could have been implemented without one, and in fact many other dynamic languages do not have this.

Therefore, giving the JVM a heap size limit was simply a design decision by the implementors. Second-guessing why this was done is a bit difficult, and there may not be a single reason. The most likely reason is that it helps protect a system from a Java program with a memory leak, which might otherwise exhaust all RAM and cause other apps to crash or the system to thrash.

Sun could have omitted the feature and simply told people to use the OS-native resource limiting mechanisms, but they probably wanted to always have a limit, so they implemented it themselves.
At any rate, the JVM needs to be aware of any such limit (to adapt its GC strategy), so using an OS-native mechanism would not have saved much programming effort.

Also, there is one reason why such a built-in limit is more important for the JVM than for a "normal" program without GC (such as a C/C++ program):

Unlike a program with manual memory management, a program using GC does not really have a well-defined memory requirement, even with fixed input data. It only has a minimum requirement, i.e. the sum of the sizes of all objects that are actually live (reachable) at a given point in time. However, in practice a program will need additional memory to hold dead, but not yet GCed objects, because the GC cannot collect every object right away, as that would cause too much GC overhead. So GC only kicks in from time to time, and therefore some "breathing room" is required on the heap, where dead objects can await the GC.

This means that the memory required for a program using GC is really a compromise between saving memory and having good througput (by letting the GC run less often). So in some cases it may make sense to set the heap limit lower than what the JVM would use if it could, so save RAM at the expense of performance. To do this, there needs to be a way to set a heap limit.

How to increase Jupyter notebook Memory limit?

Jupyter notebook has a default memory limit size. You can try to increase the memory limit by following the steps:

1) Generate Config file using command:

jupyter notebook --generate-config

2) Open jupyter_notebook_config.py file situated inside 'jupyter' folder and edit the following property:

NotebookApp.max_buffer_size = your desired value

Remember to remove the '#' before the property value.

3) Save and run the jupyter notebook.
It should now utilize the set memory value.
Also, don't forget to run the notebook from inside the jupyter folder.

---

Alternatively, you can simply run the Notebook using below command:

 jupyter notebook --NotebookApp.max_buffer_size=your_value

JavaScript memory limit

In Chrome and Chromium OS, the memory limit is defined by the browser, and you can inspect the limit with the following command in the Developer Tools command-line by hitting F12:

> window.performance.memory.jsHeapSizeLimit
1090519040

On my Windows 10 OS, it is about 1 GB.

On Chrom(e/ium), you can get around the heap size limit by allocating native arrays:

var target = []
while (true) {
    target.push(new Uint8Array(1024 * 1024)); // 1Meg native arrays
}

This crashes the tab at around 2GB, which happens very rapidly. After that Chrom(e/ium) goes haywire, and repeating the test is not possible without restarting the browser.

I also recommend reading TrackJS's blogpost about Monitoring JavaScript Memory before you get deep into the weeds trying to diagnose or measure anything memory related in the browser.

You can also search comp.lang.javascript for javascript memory limit.

See also these Stack Overflow posts:

1. Maximum size of an Array in Javascript, which suggests you can store up to 2³²-1 = 4,294,967,295 = 4.29 billion elements.

2. Maximum number of arguments a JavaScript function can accept

There is additional knowledge on the JS9 astronomical image display library website: Dealing with Memory Limitations.

(I was trying to find a good answer, and the "there is no upper limit" answer provided here was just silly to me. I cannot run into a production issue for a multi-million dollar project and say to management, "Well, I assumed there is no upper limit and everything would be okay." Try to do a proof-of-concept, e.g. loading lots of combobox controls in your JavaScript UI framework of choice, etc. You may discover your framework has some performance degradation.)

Here are some components that I've found scale very well both in CPU performance and memory performance:

1. Microsoft Monaco editor
   
   
   • This is used by several commercial projects:
     
     
     1. Postman, as of v7.1.1-canary08
     2. VS Code

Here are some examples of frameworks with well-known performance degradation:

1. Angular: Poor change detection approach.
   
   
   • For each async event, compare each of the bindings (Model-Dom binding) to its old value to decide if to re-render.
     
     
     1. NG1: >2500 watchers, performance grinds to a halt
     2. NG2: the same problem remains but you have a long tiring workaround: Switch to immutables and spread ChangeDetectionStrategy.onPush all over your app to turn off the default problematic strategy
2. React
   
   
   • Again, Immutable collections of JS objects only scale so far.
     
     
     1. create-react-app internally uses Immutable.JS, and Immutable.JS can only create about 500k immutable collections before it dies.

Here are some other things to think about:

1. Use array.slice for manipulating arrays to minimize additional array allocations; array.slice will modify the array in place, which will reduce garbage collection and overall heap size.

---

Related Topics