Debugging Shared Libraries Remotely with Gdb/Gdbserver

Debugging shared libraries remotely with gdb/gdbserver

Wild guess: you loaded the shared library into host GDB at incorrect address.

Instead of explicitly loading it into GDB, use "set stop-on-solib-event on", wait for the library to get loaded (info shared will tell you current list of loaded libraries), and then set the breakpoints.

A way to have GDB load libraries from local sysroot and remote gdbserver

GDB does not support multiple sysroots, but there are some work-arounds.

• One way to fix the issue is by patching GDB to actually support multiple sysroots. Function solib_find_1 in gdb/solib.c (link to source code) handles the library lookup. Currently, it checks the sysroot (and if that starts with target:, uses the libraries from the target). Otherwise it takes the basename (i.e. takes the file name from the given absolute path) and looks up the libraries in solib-search-path.
  
  To get the desired behavior, the solib_find_1 function should instead fall back to a different directory, e.g. this patch: https://gitlab.com/gbenson/binutils-gdb/commit/0ebe17076406a85a35eb0c4f362850ed9efb843e
• A simpler approach without patching GDB: Copy the target libraries to the host. Variants include:
  
  
  • Directly inside the local sysroot. This is quick and simple.
  • With symlinks inside the local sysroot to the location of the target libraries.
  • Use a filesystem that merges multiple locations, such as OverlayFS.
  • Use a bind mount to mount the directories or files over the sysroot.
• If you don't want to copy files: Establish an automatic local mirror of libraries from the target (e.g. with sshfs), and use the approaches from the previous point to fall back to the target when needed.
• If you don't have the ability to alter the filesystem, then you can put a proxy between your GDB client and the GDB server:
  
  
  1. Use set sysroot target:
  2. Create a proxy for GDB's File-I/O Remote Protocol, and let it transparently forward every message, except for requests for target files. If the files of interest are available locally, reply with that. Otherwise forward the message as-is.
  3. Attach the debugger using target remote :[your_proxy_port] instead of target remote :[actual_gdbserver_port], to let your proxy connect to the gdbserver on your behalf.

I found that a combination of the "Copy the target libraries to the host" variants was the most effective, as it has minimal requirements (I only need a way to copy/receive files from the target once):

• I copied the system libraries from the target to a local directory. I actually copied all of them so I didn't have to find out which one I really needed.
• I recreated the directory structure on the target of the application and added a symlink to the build output directory of my project. The use of symlinks efforts to get the setup working without recurring maintenance.

The content of the sysroot didn't actually have to be identical to the target's remote files: The target had stripped libraries without debugging symbols, while I used the non-stripped libraries in the sysroot.

GDB remote debugging: influences on execution on remote target

How does the usage of a wrong ld.so binary on the debugging host influence the execution of the application within gdbserver on the target?

Good question.

One possible explanation: in order to properly keep track of e.g. loaded shared libraries on the target, GDB sets a number of internal breakpoints (these are visible in maintenance info breakpoints output -- they have negative breakpoint number).

When you don't supply a local file at all, GDB has no idea where to set these breakpoints, and so it doesn't (you can't e.g. debug library initializers without them).

When you supply an incorrect local file, GDB does set the breakpoint ... in the wrong place (by overwriting what GDB thinks is an instruction, but what in reality is a PLT relocation). When the loader then encounters this overwritten relocation, it complains.

GDB: Error while remotely loading shared libraries

Finally i found a solution for my problem.
The problem was in step 2 shown above. Somehow gdb was not loading all the symbols and PATH to shared libraries. Although PATH shown with ldd command was accurate. But still it was not working for me if I simply run the gdb ./MY_Application
So I executed one more command after this i.e.

file MY_Application

and also copied library file from
/srv/chroot/loc/usr/local/davis/lib/ to /usr/local/davis/lib/ after this I connect it to remote server and now it is working fine.

The difference between gdbserver and remote gdb

gdbserver is smaller than GDB and has fewer dependencies, so it can be run on a system which is resource constrained such that GDB itself can't be run. Or maybe GDB just can't be run for some other reason, e.g. the target platform doesn't have Python, and you want to use some of the Python features of GDB.

Next, but kind of related, the target platform might not be the same architecture as the host platform (on which GDB runs), maybe compiling GDB, Python, syntax-highlighting libraries, etc for a small target isn't what you want to do with your time, instead you can just compile gdbserver, and then run GDB on your local machine.

Finally, though gdbserver is useful in itself, it also serves as a useful mechanism to test GDB's remote serial protocol support. This protocol is documented in the GDB manual and allows software other than gdbserver to interact with GDB, for example OpenOCD, and QEMU, both support GDB's remote protocol, this allows GDB to debug baremetal targets on which neither GDB, nor gdbserver could ever run.

You are correct to observe that this basically means that GDB has two methods of debug, its native target support, and its remote target support. If we wanted to be ideologically pure then we could (in theory) remove the native target support from GDB, and just make use of the gdbserver in all cases, i.e. when you try to debug a program on the local machine, GDB would automatically start gdbserver, connect to it, and debug through that interface. I don't expect to see that change any time soon though.

---

Related Topics