Tomcat Guice/Jdbc Memory Leak

Tomcat Guice/JDBC Memory Leak

I just dealt with this problem myself. Contrary to some other answers, I do not recommend issuing the t.stop() command. This method has been deprecated, and for good reason. Reference Oracle's reasons for doing this.

However there is a solution for removing this error without needing to resort to t.stop()...

You can use most of the code @Oso provided, just replace the following section

Set<Thread> threadSet = Thread.getAllStackTraces().keySet();
Thread[] threadArray = threadSet.toArray(new Thread[threadSet.size()]);
for(Thread t:threadArray) {
if(t.getName().contains("Abandoned connection cleanup thread")) {
synchronized(t) {
t.stop(); //don't complain, it works
}
}
}

Replace it using the following method provided by the MySQL driver:

try {
AbandonedConnectionCleanupThread.shutdown();
} catch (InterruptedException e) {
logger.warn("SEVERE problem cleaning up: " + e.getMessage());
e.printStackTrace();
}

This should properly shutdown the thread, and the error should go away.

Guice + Tomcat potential memory leak

If you are getting this when you shutdown the webapp, I wouldn't worry too much. This type of resource leak on app. shutdown is common. They do become a problem when you frequently do hot deploys (i.e. un-deploy many times without killing the JVM), but they won't be problematic when a cold deploy is done (un-deploy/deploy is done while killing the JVM before re-deploy).

A common tactic is that you do hot-deploys during development (as they are typically faster than cold-deploys), and only do a cold deploy when the resource leak starts to affect your performance. Then, in production you do a cold deploy on every deploy. Given the number of code/libraries that has this type of leak, trying to eliminate them would be hard IMO.

Suspected memory leak with Jdbc and Tomcat

Since Tomcat 6.0 , there has been a feature to detect classloader memory leaks. Read more here. The above messages by tomcat are purely for information purpose and tomcat already took sufficient measures to avoid classloader leak by de-registering the Driver.

To prevent it as you have rightly pointed out , you can either move the jar completely to tomcat's lib folder where it wont be affected by a application context reload. Or you can explicitly call a DriverManager.deregister(driver). (Read here)

To understand more about ClassLoader leaks (here)

To understand why its suggested to move it to tomcat lib from applications WEB_INF/lib you can read more here.

Edit in response to query in the comment

No its not recommended to have multiple jars as it can lead to classcastexception. Each class is identified by combination of class name and classloader. Look at this for a clearer understanding with an example.

The better approach according to me would be to write a servlet context listener and explicitly de-register your driver at context destroyed. This way you can keep you jdbc driver in your web-inf/lib and need not move it to tomcat/lib.

Even if you keep jars at multiple locations, according to the classloader hierarchy that tomcat follows which is different from what java delegation model (more info here) , tomcat will pick up the jar in your web-inf/lib first.

Memory leak in web-app, tomcat+hibernate+Guice

I'v found solution. It turns up that connection pool c3p0 0.9.2.1 has memory leak. And hibernate 4.3.1-Final uses that version of c3p0. Switching to hibernate 4.1.12 and c3p0 0.9.1 solve my problem.

Tomcat complaining about memory leak due to open connection after shut due to JDBC driver

It appears you are having this problem:

tomcat7 - jdbc datasource - This is very likely to create a memory leak

Specifically, you are including your jdbc driver as part of the war instead of the tomcat lib folder.

Tomcat 7+ memory leak on stop/redeploy. Spring Data, JPA, Hibernate, MySQL

Short answer - hopefully the same problem for you...

Those two com.test.config.dao.JPAConfiguration$$...CGLIB$$... classes were being referenced indirectly by the Abandoned connection cleanup thread in MySQL:

20-Jun-2018 21:25:22.987 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [test-1.0-SNAPSHOT] appears to have started a thread named [Abandoned connection cleanup thread] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:43)

The following answer enabled me to resolve the problem. E.g. in tomcat/conf/server.xml, look for the JreMemoryLeakPreventionListener line and replace it with this:

<Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" 
classesToInitialize="com.mysql.jdbc.Driver" />

This forces the MySQL JDBC driver, and its cleanup thread, to be loaded outside the classloader for the web application. This means the cleanup thread won't hold a reference to the webapp classloader as its context class loader.


Expanded answer - how to trace the leak in your environment...

Hopefully the above is all you need - it was enough to reproduce and solve the problem against https://github.com/egotovko/tomcat-leak

However there are many other causes of a leaked reference to a web application that can stop it undeploying. E.g. other threads still running (Tomcat is good at warning about these) or references from outside the web application.

To properly trace the cause, you can chase the reference in a heap dump. If this is not familiar, you can get a heap dump from jmap -dump:file=dump.hprof <pid>, or by directly connecting from such as jvisualvm (also included in the JDK).

With the heap dump open in jvisualvm:

  • Select the Classes button for the heap dump
  • Sort the list of classes by name
  • Look for classes in the web application - e.g. com.test.config.dao.JPAConfiguration$$EnhancerBySpringCGLIB$$ in this example
  • This should be showing with an instance count of 2 or so
  • Double click to show these in the Instances View
  • In the References pane for one of these instances, right click and Show Nearest GC Root
  • E.g. for that Abandoned connection cleanup thread in MySQL:
    Referencing thread in JVisualVM

Note how the AbandonedConnectionCleanupThread has a contextClassLoader, which is the ParallelWebappClassLoader for the web application. Tomcat needs to be able to release the class loader to undeploy the web application.

Once you've tracked down what's holding the reference, it's then normally a case of investigating how better to configure that library in Tomcat, or perhaps someone else has seen that memory leak. It's also not uncommon to have to repeat the exercise, when there are several references to clear up.



Related Topics



Leave a reply



Submit