Linux Ulimit with Java Does Not Work Properly

linux ulimit with java does not work properly

You get the same result if you run the same command from the command line:

$ "/bin/sh" "-c" "ulimit" "-n"
unlimited

This is because -c only looks at the argument immediately following it, which is ulimit. The -n is not part of this argument, and is instead instead assigned as a positional parameter ($0).

To run ulimit -n, the -n needs to be part of that argument:

$ "/bin/sh" "-c" "ulimit -n"
1024

In other words, you should be using:

new String[]{"/bin/sh", "-c", "ulimit -n"}

How to use ulimit with java correctly?

It's really hard to effectively ulimit java. Many pools are unbounded, and the JVM fails catastrophically when an allocation attempt fails. Not all the memory is actually committed, but much of it is reserved thus counting toward the virtual memory limit imposed by ulimit.

After much investigation, I've uncovered many of the different categories of memory java uses. This answer applies to OpenJDK and Oracle 8.x on a 64-bit system:

Heap

This is the most well understood portion of the JVM memory. It is where the majority of your program memory is used. It can be controlled with the -Xmx and -Xms options.

Metaspace

This appears to hold metadata about classes that have been loaded. I could not find out whether this category will ever release memory to the OS, or if it will only ever grow. The default maximum appears to be 1g. It can be controlled with the -XX:MaxMetaspaceSize option. Note: specifying this might not do anything without also specifying the Compressed class space as well.

Compressed class space

This appears related to the Metaspace. I could not find out whether this category will ever release memory to the OS, or if it will only ever grow. The default maximum appears to be 1g. It can be controlled with the '-XX:CompressedClassSpaceSize` option.

Garbage collector overhead

There appears to be a fixed amount of overhead depending on the selected garbage collector, as well as an additional allocation based on the size of the heap. Observation suggests that this overhead is about 5% of the heap size. There are no known options for limiting this (other than to select a different GC algorithm).

Threads

Each thread reserves 1m for its stack. The JVM appears to reserve an additional 50m of memory as a safety measure against stack overflows. The stack size can be controlled with the -Xss option. The safety size cannot be controlled. Since there is no way to enforce a maximum thread count and each thread requires a certain amount of memory, this pool of memory is technically unbounded.

Jar files (and zip files)

The default zip implementation will use memory mapping for zip file access. This means that each jar and zip file accessed will be memory mapped (requiring an amount of reserved memory equal to the sum of file sizes). This behavior can be disabled by setting the sun.zip.disableMemoryMapping system property (as in -Dsun.zip.disableMemoryMapping=true)

NIO Direct Buffers

Any direct buffer (created using allocateDirect) will use that amount of off-heap memory. The best NIO performance comes with direct buffers, so many frameworks will use them.

The JVM provides no way to limit the total amount of memory allowed for NIO buffers, so this pool is technically unbounded.

Additionally, this memory is duplicated on-heap for each thread that touches the buffer. See this for more details.

Native memory allocated by libraries

If you are using any native libraries, any memory they allocate will be off-heap. Some core java libraries (like java.util.zip.ZipFile) also use native libraries that consume-heap memory.

The JVM provides no way to limit the total amount of memory allocated by native libraries, so this pool is technically unbounded.

malloc arenas

The JVM uses malloc for many of these native memory requests. To avoid thread contention issues, the malloc function uses multiple pre-allocated pools. The default number of pools is equal to 8 x cpu but can be overridden by setting the environment variable MALLOC_ARENAS_MAX. Each pool will reserve a certain amount of memory even if it's not all used.

Setting MALLOC_ARENAS_MAX to 1-4 is generally recommend for java, as most frequent allocations are done from the heap, and a lower arena count will prevent wasted virtual memory from counting towards the ulimit.

This category is not technically it's own pool, but it explains the virtual allocation of extra memory.

Java and virtual memory ulimit

This is a well known giant bug in all available JVMs (oracle or openjdk, version 6,7,8). Reported here:
https://bugs.launchpad.net/ubuntu/+source/openjdk-7/+bug/1241926
https://bugs.openjdk.java.net/browse/JDK-8071445

Unfortunately without solution by the Java developers.
Solution: Play desperately with the various java options or stop using java and java based applications :-(

About causing too many open files with java?

I presume that the garbage collector is running, finding lots of unreachable BufferedReader objects and collecting them. That causes the underlying stream objects to be finalized ... which closes them.

To make this code break, add the BufferedReader objects to a list, so that they remain reachable.

---

And here's why I think that changing 7168 to MAXINT is working.

When a JVM starts, it will use a relatively small heap. One of the things that happens during GC is that the JVM decides if it needs to resize the heap. So here is what could be happening:

• The JVM starts with a heap that is too small to hold 7168 open files + BufferedReader objects. (Remember that each of the latter probably has a preallocated buffer!)

• You start opening files.

• At about N = 7168 - 2538, the heap fills up with all of the BufferedReader objects + FileInputStream objects + various detritus from JVM startup / warmup.

• The GC runs, and causes a (probably) all of the BufferedReader objects to be collected / finalized / closed.

• Then the GC decides that it needs to expand the heap. You now have enough heap space for more open BufferedReader objects than your ulimit allows.

• You resume opening files ... and then hit the open file limit.

That's one possible pattern.

---

If you really want to investigate this, I advise you turn on GC logging, and see if you can correlate the number of FDs reported by lsof with GC runs.

(You could try adding sleep calls between each open to make it easier to get lsof measurements. But that could change the JVM behavior in other ways ...)

Too many open files ( ulimit already changed )

I just put the line ulimit -n 8192 inside the catalina.sh, so when I do the catalina start, java runs with the specified limit above.

---

Related Topics