Waiting at Sun.Misc.Unsafe.Park(Native Method)

WAITING at sun.misc.Unsafe.park(Native Method)

unsafe.park is pretty much the same as thread.wait, except that it's using architecture specific code (thus the reason it's 'unsafe'). unsafe is not made available publicly, but is used within java internal libraries where architecture specific code would offer significant optimization benefits. It's used a lot for thread pooling.

So, to answer your question, all the thread is doing is waiting for something, it's not really using any CPU. Considering that your original stack trace shows that you're using a lock I would assume a deadlock is going on in your case.

Yes I know you have almost certainly already solved this issue by now. However, you're one of the top results if someone googles sun.misc.unsafe.park. I figure answering the question may help others trying to understand what this method that seems to be using all their CPU is.

Analysing thread dump - lot of blocked threads on sun.misc.Unsafe.park

sun.misc.Unsafe.park(...) is basically like Thread.wait, but it uses os code, so it is not exposed to us.

You can see in the stack traces that the threads being parked are from thread pools related to blocking queues. Threads that are "parked" coming from pools are simply waiting for a task to be assigned. Also, they really consume 0% CPU so I would doubt this is your issue.

It is possible though that you have a deadlock or concurrency issue making it so your queue is blocking itself forever...

Also, the only thread in there that has anything mentioning I/O is the one with ID 63135.

What is the difference between Thread.State: WAITING (parking) vs BLOCKED at sun.misc.Unsafe.park()

Thread states - here is a little explanation of the thread states.

NEW
The thread has not yet started.

RUNNABLE
The thread is executing in the JVM.

BLOCKED
The thread is blocked waiting for a monitor lock.

WAITING
The thread is waiting indefinitely for another thread to perform a particular action.

TIMED_WAITING
The thread is waiting for another thread to perform an action for up to a specified waiting time.

TERMINATED
The thread has exited.

The BLOCKED state should be concerning if it's there for a long period for the same threads.
This of course depends on your case - how you process data, how you create threads (and thread pools), what are your critical sections and how all that interacts with each other.

Single thread dump of the production is not enough - you should take several dumps and
- compare what happens and
- for how long the threads are running/waiting
- is this occurs on high load or after high load
- does your thread count increases over long time, etc.

So there is no way to tell that 500 blocked threads at this particular point in time is good or bad but for sure it's concerning. One thread takes around ~2MB just to initialize & allocate so it's 1GB of memory.

It's highly likely that there are some critical sections that are held by some threads that cause your problem and unresponsiveness of your application. You can potentially have some really complex situation reading from queues using blocking methods, etc.

Possible course of action:

• Make several dumps and compare - what has changed? What threads are still blocked?
• Check if you can pinpoint the invocations in the blocked threads (is somewhere your package prefix or java's/hazelcast's packges only) in the stacktrace in the dump.
• Check with tracking tools (flight-recorder / jvisualvm) the growth of the threads and when they (which are getting blocked) are created - what the app is doing at that moment?
• Analyse your codebase on the potential misuse of the blocking calls and synchronized methods/uses.
• Check thread pools maximum sizes & worker queue implementations and strategies when the limit is reached (e.g. to understand look at implementations of RejectedExecutionHandler)

sun.misc.Unsafe.park triggered by JDBC calls

sun.misc.Unsafe.park is pretty much the same as thread.wait, but it uses architecture dependent code, due to the fact that park is native call, it can get the benefit in performance (it's a popular pattern in java.util.concurrent, e.g. you can see the usage of park in ConcurrentLinkedQueue and thread pooloing). Obviously that you have a multithread environment and db connections use thread pool for own purposes. In the jdk parking brings significant optimization.

About your situation with JDBC, I would suggest a deadlock is going on in your case. All threads are waiting for some event. You can investigate a deadlock, I would like to advise you to read this articles:

jdbc-deadlock-avoidance

jdbc and deadlocks manual

HTH

---

Related Topics