Debugging Shared Libraries with Gdbserver

Debugging shared libraries remotely with gdb/gdbserver

Wild guess: you loaded the shared library into host GDB at incorrect address.

Instead of explicitly loading it into GDB, use "set stop-on-solib-event on", wait for the library to get loaded (info shared will tell you current list of loaded libraries), and then set the breakpoints.

A way to have GDB load libraries from local sysroot and remote gdbserver

GDB does not support multiple sysroots, but there are some work-arounds.

• One way to fix the issue is by patching GDB to actually support multiple sysroots. Function solib_find_1 in gdb/solib.c (link to source code) handles the library lookup. Currently, it checks the sysroot (and if that starts with target:, uses the libraries from the target). Otherwise it takes the basename (i.e. takes the file name from the given absolute path) and looks up the libraries in solib-search-path.
  
  To get the desired behavior, the solib_find_1 function should instead fall back to a different directory, e.g. this patch: https://gitlab.com/gbenson/binutils-gdb/commit/0ebe17076406a85a35eb0c4f362850ed9efb843e
• A simpler approach without patching GDB: Copy the target libraries to the host. Variants include:
  
  
  • Directly inside the local sysroot. This is quick and simple.
  • With symlinks inside the local sysroot to the location of the target libraries.
  • Use a filesystem that merges multiple locations, such as OverlayFS.
  • Use a bind mount to mount the directories or files over the sysroot.
• If you don't want to copy files: Establish an automatic local mirror of libraries from the target (e.g. with sshfs), and use the approaches from the previous point to fall back to the target when needed.
• If you don't have the ability to alter the filesystem, then you can put a proxy between your GDB client and the GDB server:
  
  
  1. Use set sysroot target:
  2. Create a proxy for GDB's File-I/O Remote Protocol, and let it transparently forward every message, except for requests for target files. If the files of interest are available locally, reply with that. Otherwise forward the message as-is.
  3. Attach the debugger using target remote :[your_proxy_port] instead of target remote :[actual_gdbserver_port], to let your proxy connect to the gdbserver on your behalf.

I found that a combination of the "Copy the target libraries to the host" variants was the most effective, as it has minimal requirements (I only need a way to copy/receive files from the target once):

• I copied the system libraries from the target to a local directory. I actually copied all of them so I didn't have to find out which one I really needed.
• I recreated the directory structure on the target of the application and added a symlink to the build output directory of my project. The use of symlinks efforts to get the setup working without recurring maintenance.

The content of the sysroot didn't actually have to be identical to the target's remote files: The target had stripped libraries without debugging symbols, while I used the non-stripped libraries in the sysroot.

GDB remote debugging: influences on execution on remote target

How does the usage of a wrong ld.so binary on the debugging host influence the execution of the application within gdbserver on the target?

Good question.

One possible explanation: in order to properly keep track of e.g. loaded shared libraries on the target, GDB sets a number of internal breakpoints (these are visible in maintenance info breakpoints output -- they have negative breakpoint number).

When you don't supply a local file at all, GDB has no idea where to set these breakpoints, and so it doesn't (you can't e.g. debug library initializers without them).

When you supply an incorrect local file, GDB does set the breakpoint ... in the wrong place (by overwriting what GDB thinks is an instruction, but what in reality is a PLT relocation). When the loader then encounters this overwritten relocation, it complains.

GDB: Error while remotely loading shared libraries

Finally i found a solution for my problem.
The problem was in step 2 shown above. Somehow gdb was not loading all the symbols and PATH to shared libraries. Although PATH shown with ldd command was accurate. But still it was not working for me if I simply run the gdb ./MY_Application
So I executed one more command after this i.e.

file MY_Application

and also copied library file from
/srv/chroot/loc/usr/local/davis/lib/ to /usr/local/davis/lib/ after this I connect it to remote server and now it is working fine.

gdb: how to learn which shared library loaded a shared library in question

Is there any way to learn it in gdb or in some other way?

There are a few ways.

You can run the program with env LD_DEBUG=files /path/to/exe.

This will produce output similar to:

 LD_DEBUG=files /bin/date
     76042:
     76042:     file=libc.so.6 [0];  needed by /bin/date [0]

It is the needed by part that you most care about.

You could also use GDB and use set set stop-on-solib-events 1. This will produce output similar to:

Stopped due to shared library event:
  Inferior loaded /lib/x86_64-linux-gnu/libc.so.6

At that point, you could execute where command and observe which dlopen() call caused the new library to be loaded.

You could also set a breakpoint on dlopen() and do the same.

The stop-on-solib-events may be better if your executable repeatedly dlopen()s the same library -- the library set will not change when you dlopen() the same library again, and you'll stop when you don't care to stop. Setting stop-on-solib-events avoids such unnecessary stops.

---

Related Topics