Where Are Gdb Symbols Coming From

Where are GDB symbols coming from?

TL;DR:

1. There is a special .gnu_debugdata compressed section in Fedora binaries that GDB reads, and which contains mini-symbols.
2. Contents of that section can be conveniently printed with eu-readelf -Ws --elf-section /usr/bin/ls

---

readelf -S /usr/bin/ls | grep abformat

That command is dumping sections. You want symbols instead:

readelf -s /usr/bin/ls | grep abformat
readelf --all /usr/bin/ls | grep abformat

strings /usr/bin/ls | grep abformat

Strings tries to guess what you want, and doesn't output all strings found in the binary. See this blog post and try:

strings -a /usr/bin/ls | grep abformat

Update: I confirmed the results you've observed: abformat does not appear anywhere, yet GDB knows about it.

Turns out, there is a .gnu_debugdata compressed section (described here), which has mini-symbols.

To extract this data, normally you would do:

objcopy -O binary -j .gnu_debugdata /usr/bin/ls ls.mini.xz

However, that is broken on my system (produces empty output), so instead I used dd:

# You may need to adjust the numbers below from "readelf -WS /usr/bin/ls"
dd if=/usr/bin/ls of=ls.mini.xz bs=1 skip=151896 count=3764
xz -d ls.mini.xz
nm ls.mini | grep abformat

This produced:

00000000000005db0 t abformat_init

QED.

Additional info:

1. Confusing GDB no debugging symbols is addressed in this bug.
2. objcopy refusing to copy .gnu_debugdata is the subject of this bug.

3. There is a tool that can conveniently dump this info:

   eu-readelf -Ws --elf-section /usr/bin/ls | grep abformat
37: 0000000000005db0 593 FUNC LOCAL DEFAULT 14 abformat_init

no debugging symbols found when using gdb

Some Linux distributions don't use the gdb style debugging symbols. (IIRC they prefer dwarf2.)

In general, gcc and gdb will be in sync as to what kind of debugging symbols they use, and forcing a particular style will just cause problems; unless you know that you need something else, use just -g.

Why is gdb not finding any debugging symbols, even though I have -g at every stage of compilation and linkage in my Makefile?

place the source code, the object file, and the executable file in the same directory as where you are running gdb

There are other ways, which require some simple commands into gdb before entering run, however; they are a nuisance to redo over and over.

How to know the name and/or path of the debug symbol file which is linked to a binary executable?

how can we know same without actually running gdb?

Either of these commands will tell you:

readelf -x.gnu_debuglink foo
objdump -sj.gnu_debuglink foo

The objcopy --add-gnu-debuglink simply adds .gnu_debuglink section containing the given path, and a checksum of the debug file.

More info here.

gdb behaves differently for symbols in the .bss, vs. symbols in .data

glibc includes a symbol named buf, as well.

(gdb) info variables ^buf$
All variables matching regular expression "^buf$":

File strerror.c:
static char *buf;

Non-debugging symbols:
0x000000000060108b  buf            <-- this is our buf
0x00007ffff7dd6400  buf            <-- this is glibc's buf

gdb happens to choose the symbol from glibc over the symbol from the executable. This is why ptype buf shows char *.

Using a different name for the buffer avoids the problem, and so does a global buf to make it a global symbol. You also wouldn't have a problem if you wrote a stand-alone program that didn't link libc (i.e. define _start and make an exit system call instead of running a ret)

---

Note that 0x00007ffff7dd6400 (address of buf on my system; different from yours) is not actually a stack address. It visually looks like a stack address, but it's not: it has a different number of f digits after the 7. Sorry for that confusion in comments and an earlier edit of the question.

Shared libraries are also loaded near the top of the low 47 bits of virtual address space, near where the stack is mapped. They're position-independent, but a library's BSS space has to be in the right place relative to its code. Checking /proc/PID/maps again more carefully, gdb's &buf is in fact in the rwx block of anonymous memory (not mapped to any file) right next to the mapping for libc-2.21.so.

7ffff7a0f000-7ffff7bcf000 r-xp 00000000 09:7f 17031175       /lib/x86_64-linux-gnu/libc-2.21.so
7ffff7bcf000-7ffff7dcf000 ---p 001c0000 09:7f 17031175       /lib/x86_64-linux-gnu/libc-2.21.so
7ffff7dcf000-7ffff7dd3000 r-xp 001c0000 09:7f 17031175       /lib/x86_64-linux-gnu/libc-2.21.so
7ffff7dd3000-7ffff7dd5000 rwxp 001c4000 09:7f 17031175       /lib/x86_64-linux-gnu/libc-2.21.so
7ffff7dd5000-7ffff7dd9000 rwxp 00000000 00:00 0        <--- &buf is in this mapping
...
7ffffffdd000-7ffffffff000 rwxp 00000000 00:00 0              [stack]     <---- more FFs before the first non-FF than in &buf.

A normal call instruction with a rel32 encoding can't reach a library function, but it doesn't need to because GNU/Linux shared libraries have to support symbol interposition, so calls to library functions actually jump to the PLT, where an indirect jmp (with a pointer from the GOT) goes to the final destination.

gdb symbols loaded but no symbols shown for seg fault

Is the problem the fact that the core was created on another system?

Yes, exactly.

See this answer for possible solutions.

Update:

So does this mean I can only debug the program on the system where it is both built and crashes?

It is certainly not true that you can only debug a core on the system where the binary was both built and crashed -- I debug core dumps from different systems every day, and in my case the build host, the host where the program crashed, and the host on which I debug are all separate.

One thing I just noticed: your style of loading the core: gdb -c core followed by symbol-file, doesn't work for PIE executables (at least when using GDB 10.0) -- this may be a bug in GDB.

The "regular" way of loading the core is:

gdb ./ovcc core

See if that gives you better results. (You still need to arrange for matching DSOs, as linked answer shows how to do.)

Debugging: my symbols don't get loaded in gdb?

but none of any help (most of these relate to a forgotten -g flag, or an added -s, stripping down the symbols).

It is almost certain that you either have a stray -s somewhere on your link line, or you run stip on the binary during installation.

Look at your link command line and install command carefully, there is strip somewhere in there.

P.S. As Tom Tromey already said, GDB is rarely effective in helping with a problem like this. Using Valgrind or Address Sanitizer will likely get you to the root cause much faster.

---

Related Topics