What's an Alternative for Dtrace on Linux

DTrace on Ubuntu, how-to?

As author of dtrace4linux, let me answer.

In essence, dtrace on Linux/MacOS/FreeBSD/Solaris is the same - we are all based on the same source code with the same goals. Because there is no central maintainer, the codes are effectively forks, with Solaris being considered the master. The major source code difference is the glue for each platform.

DTrace is a combination of a number of things:

• kernel driver
• user space "dtrace" command
• probe function mechanisms (eg syscall, fbt)
• scripting language

Look at syscall tracing its pretty straightforward:

$ dtrace -n syscall::open:
.....

This traps every open system call, no matter who executed it. If you know Unix, you know the syscall is approximately:

open(char *filename, int flags, [int perms])

So, arg0 is the string "filename". But this is where things differ. The C lib function is as above, but this maps to a system call, which is something like:

open(int someflags, char *filename, int userflags, int perms)

So the filename is not in arg0, but arg1. (Apologies if the above is wrong - I think open() is the same in the kernel and user space, but this is not true, e.g. for the stat() family of functions).

This is where some of the "portability" issues of DTrace arise - if you use a Solaris guide to dtrace, and try to run some of the scripts or examples, you might find they dont work the same. In theory, this is Linux (my) fault, and dtrace4linux should be modified to hide this.

That all applies to syscalls.

Now lets look at fbt. Fbt is just function tracing - any function - with any parameters. You can trace the linux function which implements the open() syscall (its called sys_open [possibly]). If you trap this function:

$ dtrace -n fbt::sys_open:

then you have to look at the kernel source code to see what arg0, arg1, arg2, etc is. And almost certainly is different to Solaris or MacOS - its Linux's implementation detail.

But you may want access to some argument, e.g. to get hold of some internal kernel data structure (TCP, Disk driver, USB driver, etc). Solaris provides "providers" which are higher level ways to access data structures than knowing "arg3 is a 'struct foo *'". Without these providers, scripts would be totally opsys dependent and no portability. Most people dont care what a "tcp" structure looks like, but want access to key fields like pktin, pktout, rcvbytes, sndbytes.

In summary dtrace4linux and Solaris dtrace provides a portability layer to allow access to these features or structures, but neither dtrace4linux or Solaris attempts to do a complete job to provide portability across the thousands of structures in each kernel.

In general, you can take solaris tutorial scripts and use them to try and understand what doesnt work, but attempting to use them 'as-is' will frustrate you if you do not know what to look for.

I consider dtrace4linux as "not bad" and "not good enough" to hide these differences. (dtrace4linux is about on par with MacOS - if you use the Solaris tutorials, some may not work on a Mac or FreeBSD).

How can I run this DTrace script to profile my application?

If all you want to do is run this particular DTrace script, simply save it to a .d script file and use a command like the following to run it against your compiled executable:

sudo dtrace -s dtracescript.d -c [Path to executable]

where you replace dtracescript.d with your script file name.

This assumes that you have DTrace as part of your system (I'm running Mac OS X, which has had it since Leopard).

If you're curious about how this works, I wrote a two-part tutorial on using DTrace for MacResearch a while ago, which can be found here and here.

strace or procmon equivalent for java

After messing around a lot, it looks like it is back to strace.

strace -f java xxx | egrep openat 1> files.log 2>&1

Then look at the files.log. What I didn't appreciate was that java launches the executable as a different process. strace -f traces the forked processes as well.

openat is the call for opening files. There are lots opens on class, jar and .so files. Once those have been taken out, what you are left with is all the other files.

End to end profiling of web/J2EE and client applications

I am not aware of any profiling agent that provides an out-of-the-box feature to profile applications end-to-end. At least, not at the time of writing this.

However, you could consider using DTrace to collect profiling information (not necessarily all the info provided by a typical profiler) from both the client and the server at the same time. An example on how to use this in a Java web-application (with Firefox as the client, and Tomcat as the server) is available in this article. The trick lies in having DTrace instrumentation built into the JVM (running the server) and the client, and in writing a DTrace script that writes the collected information from the traces into a parseable output. Since, the question isn't clear about whether the client is in Java I'll assume that the client is also in Java; if not, the application/executable must support DTrace instrumentation.

A few caveats are to be mentioned though:

• DTrace is not available in all operating systems.
  
  • It is currently available in Solaris 10 and Mac OS Leopard.
  • I'm not aware of Dtrace support in Linux distributions, where SystemTap is the recommended alternative. FreeBSD has an experimental implementation of DTrace. It should be noted that SystemTap is not an implementation of DTrace for Linux, but a wholly different entity in itself.
  • On MSFT Windows, you're left to fend for yourself, atleast for the moment.
• Assumptions about the JVM and the client (since I know that the server runs on a JVM)
  
  • If your client is a JVM then you must use a JVM that has DTrace instrumentation or equivalent (I'm referring to SystemTap here) built into it, or your DTrace/SystemTap scripts will not work. This means that unless the authors of the JRE/JDK for your distribution, have added code to support DTrace/SystemTap, you are constrained to use a JVM that has such a support built in. If it is of importance in your context, OpenJDK 1.6.0 appears have instrumentation support for SystemTap, and the Oracle/Sun Java 6 distributions for Solaris have support for DTrace. On Java 1.4 and 5 on Solaris, you'll need to install a JVMTI agent.
  • If your client is a web-browser or a thick-client written in a different language, then you must ensure that the process it is running under can support DTrace. For instance, DTrace instrumented Firefox is available on Solaris 10, and presumably on Solaris Express 11 (I have verified DTrace support only on OpenSolaris 2009.06). SystemTap probes haven't been added in Firefox for a lot of Linux distributions, so you'll often need to build Firefox from source, with the appropriate flags in the Firefox build script to add the probes; my previous experience with this process leads me to believe that this is doable, but isn't easy. If you are running a different application (neither Firefox, nor a JVM) as the client, then you'll need to verify DTrace/SystemTap support for it.
• Writing good DTrace/SystemTap scripts to collect profiling information is not easy. You'll need to learn another language, and you'll need to ensure that the scripts do not add their own overhead (and validating the Heisenberg Uncertainty Principle in the context of profiling).

You can certainly write DTrace/SystemTap scripts that operate over remote clients and servers, but it is not advisable to write the collected instrumentation data to a socket (to avoid the afore-mentioned overhead).

---

Related Topics