"Java.Lang.Outofmemoryerror:Unable to Create New Native Thread"

MongoDB ops manager java.lang.OutOfMemoryError: unable to create native thread

SUGGESTION: focus on your JVM;

• Ensure you have a 64-bit version of Java
• Try tuning your JVM parameters:

https://docs.opsmanager.mongodb.com/current/reference/troubleshooting/system/

1. Open mms.conf in your preferred text editor.

2. Find this line:

   JAVA_MMS_UI_OPTS="${JAVA_MMS_UI_OPTS} -Xss228k -Xmx4352m -Xms4352m -XX:NewSize=600m -Xmn1500m -XX:ReservedCodeCacheSize=128m -XX:-OmitStackTraceInFastThrow"
   

3. Change the -Xmx and -Xms values to a larger value. Both parameters should be set to the same value to remove any performance impact from
   the VM constantly reclaiming memory from the heap.

The value is specified as #k|m|g: a number followed by

k (kilobytes), m (megabytes), or g (gigabytes)

By default, Xmsx and Xms are both set to 4,352 MB (4352m).

EXAMPLE: To set the Java heap to 10 GB, set this value to:

-Xmx10g -Xms10g

STRONG SUGGESTION: I would continue to focus on JVM settings, However, this link might also be relevant:

https://stackoverflow.com/a/31445282/421195

I encountered a similar issue in our Test Ops Manager deployment when
we upgraded to Ops Manager 1.8.0. I ultimately opened up a ticket with
MongoDB Support and this was the resolution for our issue:

The Ops Manager components are launched using the default username
"mongodb-mms". Please adjust the ulimit settings for this user to
match those of the "mongodb" user, currently defined in
/etc/security/limits.d/99-mongodb-mms-automation-agent.conf.

You may wish to add a separate file under /etc/security/limits.d/
for the mongodb-mms user.

More information can be found
here.

New information:

So I tried a fresh install with the same version of MongoDB (4.4.3)
and Ops Manager(4.4.8.100) to check if something was wrong with the
newest versions. Throws the same error.

I tried running jconsole -debug ->

[1,323s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached

This suggests you might be running out of threads.

Relevant links:

https://github.com/elastic/elasticsearch/issues/31982

Elasticsearch version (bin/elasticsearch --version):
6.3.1

JVM version (java -version):10

OS version:centos

java.lang.OutOfMemoryError: unable to create native thread: possibly
out of memory or process/resource limits reached but my os has free
80g memory

i used docker.elastic.co/elasticsearch/elasticsearch:6.3.1,

jvm config:
-Xms32g
-Xmx32g

...

[I had] a similar (but likely unrelated) issue in our app (which is
using the ES client). For whatever reason, it had gone berserk during
the weekend, spawning 9400 threads which made the machine fail in new
thread creation for the same user account.

ps -o nlwp,pid -fe helped me spot this, so I could kill the bad
process and get the system back to a usable state. Greatly
appreciated!

Here is an example ps -o nlwp,pid -fe from my Ubuntu system (an AWS VM). I suspect your "ps" will look very, very different:

# ps -o nlwp,pid -fe
NLWP   PID
   1 13409
   1 13410
   1 13418
   1   915
   1   911

Addendum:

I switched the OS (from Ubuntu 18.04 LTS 64bit) to CentOS 8 and now its working perfectly.

Troubleshooting irregular OutOfMemoryError: unable to create new native thread

The issue went away after migrating to Java 11.

Related Topics