Com.Sun.Tools.Attach.Attachnotsupportedexception: Unable to Open Socket File: Target Process Not Responding or Hotspot Vm Not Loaded

com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file: target process not responding or HotSpot VM not loaded

Work around for now.

Adding '-XX:+StartAttachListener' to jvm argument fixed the issue.

A similar issue is discussed here at https://code.google.com/p/jmockit/issues/detail?id=136 and http://mail.openjdk.java.net/pipermail/macosx-port-dev/2013-October/006098.html (which talks about a possible regression in jdk7 build)

Java - AttachNotSupportedException: Unable to open socket file: HotSpot VM not loaded

Common reasons for this problem:

• Attach socket /tmp/.java_pid1234 has been removed (e.g. by a scheduled job that periodically cleans up /tmp).
• Target JVM is started with -XX:+DisableAttachMechanism option.
• Garbage Collection or other long VM operation (e.g. Heap Dump) is in progress.
• JVM cannot reach safepoint within attach timeout. This happens rarely, and the problem is typically intermittent.

AttachNotSupportedException when trying to start a JFR recording

One of the probable reasons is that /tmp/.java_pid1234 file has been deleted (where 1234 is PID of a Java process).

Tools that depend on Dynamic Attach Mechanism (jstack, jmap, jcmd, jinfo) communicate to JVM through a UNIX domain socket created at /tmp.
This socket is created by JVM lazily on the first attach attempt or eagerly at JVM initialization if -XX:+StartAttachListener flag is specified.

Once the file corresponding to the socket is deleted, tools cannot connect to the target process, and unfortunately there is no way to re-create communication socket without restarting JVM.

For the description of Dynamic Attach Mechanism see this answer.

AttachNotSupportedException due to missing java_pid file in Attach API

I experienced this same issue.

Exception in thread "main" com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file: target process not responding or HotSpot VM not loaded

The solution was found doing some heavy googling.

First answer came http://www.jvmmonitor.org/doc/index.html . Appears there is a bug:

If you see the additional message "Unable to open socket file: target
process not responding or Hotspot VM not loaded", either your
application didn't respond creating a socket file like
/tmp/.java_pid1234 (e.g. due to hangup, file system permission), or
JVM Monitor was not able to find the created socket file (e.g. due to
the bug 7009828).

Then after some more searching I found a conversation on github for another tool which had the same symptom "Unable to open socket file" (https://github.com/rhuss/jolokia/issues/34):

jgreen: Caused by: com.sun.tools.attach.AttachNotSupportedException:
Unable to open socket file: target process not responding or HotSpot VM not loaded

jgreen: right I have it working but only when lauching as the exact same user as activemq.
root does not work

This last piece was the solution. The only way this .attach call would be successful was through running the java code that calls attach as the same user as the one who owned the process running the jvm. In my case it was the activemq user.

System.out.println("HEAP: " + ManagementFactory.getMemoryMXBean().getHeapMemoryUsage());

HEAP: init = 27127296(26491K) used = 3974200(3881K) committed = 26345472(25728K) max = 675086336(659264K)

Running jmap getting Unable to open socket file

jmap vs. jmap -F, as well as jstack vs. jstack -F use completely different mechanisms to communcate with the target JVM.

jmap / jstack

When run without -F these tools use Dynamic Attach Mechanism. This works as follows.

1. Before connecting to Java process 1234, jmap creates a file .attach_pid1234 at the working directory of the target process or at /tmp.

2. Then jmap sends SIGQUIT to the target process. When JVM catches the signal and finds .attach_pid1234, it starts AttachListener thread.

3. AttachListener thread creates UNIX domain socket /tmp/.java_pid1234 to listen to commands from external tools.

4. For security reasons when a connection (from jmap) is accepted, JVM verifies that credentials of the socket peer are equal to euid and egid of JVM process. That's why jmap will not work if run by different user (even by root).

5. jmap connects to the socket, and sends dumpheap command.

6. This command is read and executed by AttachListener thread of the JVM. All output is sent back to the socket. Since the heap dump is made in-process directly by JVM, the operation is really fast. However, JVM can do this only at safepoints. If a safepoint cannot be reached (e.g. the process is hung, not responding, or a long GC is in progress), jmap will timeout and fail.

Let's summarize the benefits and the drawbacks of Dynamic Attach.

Pros.

• Heap dump and other operations are run collaboratively by JVM at the maximum speed.
• You can use any version of jmap or jstack to connect to any other version of JVM.

Cons.

• The tool should be run by the same user (euid/egid) as the target JVM.
• Can be used only on live and healthy JVM.
• Will not work if the target JVM is started with -XX:+DisableAttachMechanism.

---

jmap -F / jstack -F

When run with -F the tools switch to special mode that features HotSpot Serviceability Agent. In this mode the target process is frozen; the tools read its memory via OS debugging facilities, namely, ptrace on Linux.

1. jmap -F invokes PTRACE_ATTACH on the target JVM. The target process is unconditionally suspended in response to SIGSTOP signal.

2. The tool reads JVM memory using PTRACE_PEEKDATA. ptrace can read only one word at a time, so too many calls required to read the large heap of the target process. This is very and very slow.

3. The tool reconstructs JVM internal structures based on the knowledge of the particular JVM version. Since different versions of JVM have different memory layout, -F mode works only if jmap comes from the same JDK as the target Java process.

4. The tool creates heap dump itself and then resumes the target process.

Pros.

• No cooperation from target JVM is required. Can be used even on a hung process.
• ptrace works whenever OS-level privileges are enough. E.g. root can dump processes of all other users.

Cons.

• Very slow for large heaps.
• The tool and the target process should be from the same version of JDK.
• The safepoint is not guaranteed when the tool attaches in forced mode. Though jmap tries to handle all special cases, sometimes it may happen that target JVM is not in a consistent state.

Note

There is a faster way to take heap dumps in forced mode. First, create a coredump with gcore, then run jmap over the generated core file. See the related question.

---

Related Topics