How Does Ltrace (Library Tracing Tool) Work

How does ltrace (library tracing tool) work?

Dynamic executables have a symbol table used by the linker when resolving references that need to be connected to library functions. (You can see this yourself by running objdump -T /path/to/binary).

This symbol table is accessible by other tools -- such as ltrace -- as well, so it's trivial to determine which functions need to be hooked and walk that list individually.

See a talk on ltrace internals presented at the Ottowa Linux Symposium, which provides a detailed, function-by-function breakdown; to follow along the source, see the official repository, or a third-party github mirror.

Some newer releases (more recent than that talk) also hook the dlopen() call, to be able to trace invocation of dynamically loaded libraries as well. The mechanism there should be rather obvious on a moment's thought -- if one can replace dlopen() with a shim (when dlopen() itself is dynamically linked as above), one can then set a breakpoint on any function pointer it returns.

How to trace dynamically loaded library calls with ltrace

You need to add a special flag -x pattern to force tracing of symbols in dlopen-ed libraries (the easiest would be -x '*', see @nayana's answer for more details.)

Older versions of ltrace do not include the relevant patch from https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=537781 (follow link to timetobleed.com) so they can't trace dlopen-ed libs.

Tracing calls to a shared library

The tool you are looking for is called ltrace. It allows to trace any call from the program to all (or a set of given) libraries.

For example, the following call will list any call to an external function loaded by a shared library:

$> ltrace ls /
__libc_start_main(0x4028c0, 2, 0x7fff1f4e72d8, 0x411e60 <unfinished ...>
strrchr("ls", '/')                               = nil
setlocale(LC_ALL, "")                            = "en_US.UTF-8"
bindtextdomain("coreutils", "/usr/share/locale") = "/usr/share/locale"
textdomain("coreutils")                          = "coreutils"
__cxa_atexit(0x40a200, 0, 0, 0x736c6974756572)   = 0
isatty(1)                                        = 0
getenv("QUOTING_STYLE")                          = nil
getenv("COLUMNS")                                = nil
ioctl(1, 21523, 0x7fff1f4e6e80)                  = -1
getenv("TABSIZE")                                = nil
getopt_long(2, 0x7fff1, "abcdfghiklmnopqrstuvw:xABCDFGHI:"..., 0x413080, -1) = -1
...
+++ exited (status 0) +++

If you want to focus on a particular library, then you should use the --library=pattern option:

-l, --library library_pattern
    Display only calls to functions implemented by libraries that match
    library_pattern. Multiple library patterns can be specified with several
    instances of this option. Syntax of library_pattern is described in 
    section FILTER EXPRESSIONS.

    Note that while this option selects calls that might be directed to the 
    selected libraries, there's no actual guarantee that the call won't be 
    directed elsewhere due to e.g. LD_PRELOAD or simply dependency ordering.
    If you want to make sure that symbols in given library are actually called,
    use -x @library_pattern instead.

So, for example, getting the list of calls to libselinux.so.1 is done like this:

$ ltrace -l libselinux.so.1 ls /
ls->freecon(0, 0xffffffff, 0x7f78c4eee628, 0)                           = 0
bin dev media root sbin sys usr boot etc home lib lost+found proc run tmp
+++ exited (status 0) +++

Only one call to the function freecon() is taken out this run.

Tool to trace local function calls in Linux

Assuming you only want to be notified for specific functions, you can do it like this:

compile with debug informations (as you already have symbol informations, you probably also have enough debugs in)

given

#include <iostream>

int fac(int n) {
    if(n == 0)
        return 1;
    return n * fac(n-1);
}

int main()
{
    for(int i=0;i<4;i++)
        std::cout << fac(i) << std::endl;
}

Use gdb to trace:

[js@HOST2 cpp]$ g++ -g3 test.cpp
[js@HOST2 cpp]$ gdb ./a.out
(gdb) b fac
Breakpoint 1 at 0x804866a: file test.cpp, line 4.
(gdb) commands 1
Type commands for when breakpoint 1 is hit, one per line.
End with a line saying just "end".
>silent
>bt 1
>c
>end
(gdb) run
Starting program: /home/js/cpp/a.out
#0  fac (n=0) at test.cpp:4
1
#0  fac (n=1) at test.cpp:4
#0  fac (n=0) at test.cpp:4
1
#0  fac (n=2) at test.cpp:4
#0  fac (n=1) at test.cpp:4
#0  fac (n=0) at test.cpp:4
2
#0  fac (n=3) at test.cpp:4
#0  fac (n=2) at test.cpp:4
#0  fac (n=1) at test.cpp:4
#0  fac (n=0) at test.cpp:4
6

Program exited normally.
(gdb)

Here is what i do to collect all function's addresses:

tmp=$(mktemp)
readelf -s ./a.out | gawk '
{ 
  if($4 == "FUNC" && $2 != 0) { 
    print "# code for " $NF; 
    print "b *0x" $2; 
    print "commands"; 
    print "silent"; 
    print "bt 1"; 
    print "c"; 
    print "end"; 
    print ""; 
  } 
}' > $tmp; 
gdb --command=$tmp ./a.out; 
rm -f $tmp

Note that instead of just printing the current frame(bt 1), you can do anything you like, printing the value of some global, executing some shell command or mailing something if it hits the fatal_bomb_exploded function :) Sadly, gcc outputs some "Current Language changed" messages in between. But that's easily grepped out. No big deal.

How to use ltrace for mpi programs?

You should be able to simply use:

$ mpiexec -n 4 -other_mpiexec_options ltrace ./executable

But that will create a huge mess since the outputs from the different ranks will merge. A much better option is to redirect the output of ltrace to a separate file for each rank. Getting the rank is easy with some MPI implementations. For example, Open MPI exports the world rank in the environment variable OMPI_COMM_WORLD_RANK. The following wrapper script would help:

#!/bin/sh

ltrace --output trace.$OMPI_COMM_WORLD_RANK $*

Usage:

$ mpiexec -n 4 ... ltrace_wrapper ./executable

This will produce 4 trace files, one for each rank: trace.0, trace.1, trace.2, and trace.3.

For MPICH and other MPI implementations based on it and using the Hydra PM exports PMI_RANK and the above given script has to be modified and OMPI_COMM_WORLD_RANK replaced with PMI_RANK. One could also write an universal wrapper that works with both families of MPI implementations.

ltrace does not show sin() in the output

This is because your sin call is a constant value and gcc optimizes it out (even when compiling with -O0 and without -lm). This is the result of running disass main in gdb:

   0x0000000000400580 <+0>:     push   %rbp
   0x0000000000400581 <+1>:     mov    %rsp,%rbp
   0x0000000000400584 <+4>:     sub    $0x10,%rsp
   0x0000000000400588 <+8>:     mov    0xee(%rip),%eax        # 0x40067c
   0x000000000040058e <+14>:    mov    %eax,-0x4(%rbp)
   0x0000000000400591 <+17>:    mov    $0x400660,%edi
   0x0000000000400596 <+22>:    callq  0x400450 <puts@plt>
   0x000000000040059b <+27>:    mov    0xdf(%rip),%eax        # 0x400680
   0x00000000004005a1 <+33>:    mov    %eax,-0x4(%rbp)
   0x00000000004005a4 <+36>:    movss  -0x4(%rbp),%xmm0
   0x00000000004005a9 <+41>:    cvtps2pd %xmm0,%xmm0
   0x00000000004005ac <+44>:    mov    $0x40066e,%edi
   0x00000000004005b1 <+49>:    mov    $0x1,%eax
   0x00000000004005b6 <+54>:    callq  0x400460 <printf@plt>
   0x00000000004005bb <+59>:    mov    $0x0,%eax
   0x00000000004005c0 <+64>:    leaveq 
   0x00000000004005c1 <+65>:    retq  

There is no call for sin here.

Changing your code to read:

#include<stdio.h>
#include<math.h>

int main()
{
    float x, y;
    scanf("%f", &x);
    y=sin(x);
    printf("sin(%f)=%f\n", x, y);
    return 0;
}

will make you need -lm when compiling:

$ gcc -Wall -Wextra -O0 -g 1.c -lm

and now you'll see this disassembled output:

   ...
   0x00000000004006c9 <+25>:    callq  0x4005b0 <__isoc99_scanf@plt>
   0x00000000004006ce <+30>:    movss  -0x8(%rbp),%xmm0
   0x00000000004006d3 <+35>:    unpcklps %xmm0,%xmm0
   0x00000000004006d6 <+38>:    cvtps2pd %xmm0,%xmm0
   0x00000000004006d9 <+41>:    callq  0x4005a0 <sin@plt>
   ...

and the call in ltrace:

__libc_start_main(0x4006b0, 1, 0x7fffd25ecff8, 0x400720 <unfinished ...>
__isoc99_scanf(0x4007b0, 0x7fffd25ecf08, 0x7fffd25ed008, 0x400720) = 1
sin(0x7fffd25ec920, 0x7fa1a6388a20, 1, 16)                         = 0x7fa1a643b780
printf("sin(%f)=%f\n", 3.000000, 0.141120sin(3.000000)             =0.141120
)                                                                  = 23
+++ exited (status 0) +++

---

Related Topics