How to Analyse a Crash Dump File Using Gdb

How to analyse a crash dump file using GDB

The first thing to look for is the error message that you get when the program crashes. This will often tell you what kind of error occurred. For example "segmentation fault" or "SIGSEGV" almost certainly mean that your program has de-referenced a NULL or otherwise invalid pointer. If the program is written in C++, then the error message will often tell you the name of any uncaught exception.

If you aren't seeing the error message, then run the program from the command line, or pipe its output into a file.

In order for a core file to be really useful, you need to compile your program without optimisation and with debugging information. GCC needs the following options: -g -O0. (Make sure your build doesn't have any other -O options.)

Once you have the core file, then open it in gdb with:

gdb YOUR-APP COREFILE

Type where to see the point where the crash occurred. You are basically in a normal debugging session - you can examine variables, move up and down the stack, switch between threads and whatever.

If your program has crashed, then it's probably an invalid memory access - so you need to look for a pointer that has zero-value, or that points to bad looking data. You might not find the problem at the very bottom of the stack, you might have to move up the stack a few levels before you find the problem.

Good luck!

How do I analyze a program's core dump file with GDB when it has command-line parameters?

You can use the core with GDB in many ways, but passing parameters which is to be passed to the executable to GDB is not the way to use the core file. This could also be the reason you got that error. You can use the core file in the following ways:

gdb <executable> <core-file> or gdb <executable> -c <core-file> or

gdb <executable>
...
(gdb) core <core-file>

When using the core file you don't have to pass arguments. The crash scenario is shown in GDB (checked with GDB version 7.1 on Ubuntu).

For example:

$ ./crash -p param1 -o param2
Segmentation fault (core dumped)
$ gdb ./crash core
GNU gdb (GDB) 7.1-ubuntu
...
Core was generated by `./crash -p param1 -o param2'. <<<<< See this line shows crash scenario
Program terminated with signal 11, Segmentation fault.
#0  __strlen_ia32 () at ../sysdeps/i386/i686/multiarch/../../i586/strlen.S:99
99    ../sysdeps/i386/i686/multiarch/../../i586/strlen.S: No such file or directory.
    in ../sysdeps/i386/i686/multiarch/../../i586/strlen.S
(gdb)

If you want to pass parameters to the executable to be debugged in GDB, use --args.

For example:

$ gdb --args ./crash -p param1 -o param2
GNU gdb (GDB) 7.1-ubuntu
...
(gdb) r
Starting program: /home/@@@@/crash -p param1 -o param2

Program received signal SIGSEGV, Segmentation fault.
__strlen_ia32 () at ../sysdeps/i386/i686/multiarch/../../i586/strlen.S:99
99    ../sysdeps/i386/i686/multiarch/../../i586/strlen.S: No such file or directory.
    in ../sysdeps/i386/i686/multiarch/../../i586/strlen.S
(gdb)

Man pages will be helpful to see other GDB options.

Most useful commands are:

• bt (backtrace)
• info locals (show values of local variables)
• info registers (show values of CPU registers)
• frame X (change to stack frame X)
• up and down (navigate in the stack frame (call chain))

How to analyse a coredump file of GDB

GDB can get you started:

$ gdb --help
This is the GNU debugger.  Usage:

    gdb [options] [executable-file [core-file or process-id]]
    gdb [options] --args executable-file [inferior-arguments ...]

[snip extended docs]

So, you'll invoke it like this:

gdb myprog core

GDB will then start in the usual way, but the state will be as if you'd stopped at a breakpoint. You can then use "print", "examine", "list", "backtrace", "up", "down", etc. to investigate what caused the crash.

In fact, you can use any GDB command except "continue", "step", "next", or anything else that requires an actual running program.

gdb debugging of core.# file - getting the full command which caused the crash

You can load core dump with matching binary (the one for which core dump was generated) and print argv values in the frame where main function resides.

Something like this:

gdb /tools/graphmap/bin/Linux-x64/graphmap /4thExp/core.82912

Then go up in stack trace to initial frame where int main(int argc, char *argv[]) resides. Now you can print the number of arguments and their values from gdb prompt.

Update:

It appears that your binary is multithreaded and crash happened in some auxiliary thread. You should therefore find main thread and switch to it. Here is an example of how to do it for Firefox with many threads:

(gdb) t a a bt -1

Thread 59 (Thread 0x7f691deff700 (LWP 25924)):
#12 0x00007f69dce93f6f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
..........
..........
many threads are listed here
..........
..........
Thread 1 (Thread 0x7f69de01a740 (LWP 4143)):
#17 0x000056374cb38817 in main ()
(gdb) t 1
[Switching to thread 1 (Thread 0x7f69de01a740 (LWP 4143))]
#0  0x00007f69dce8800d in poll () at ../sysdeps/unix/syscall-template.S:84
84  T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)

Now gdb is switched to main thread (Thread 1).

Core dump analysis using gdb

1. You need to configure Cygwin to produce core dumps by including

   error_start=x:\path\to\dumper.exe
   
   in your CYGWIN environment variable (see here in section "dumper" for more information). If you didn't do this, you will only get a stacktrace -- which may also help you in diagnosing the problem, though.

2. Start gdb as follows to attach it to a core dump file:

   gdb myexecutable --core=mycorefile
   
   You can now use the usual gdb commands to print a stacktrace, examine the values of variables, and so on.

using gdb to analyze core dump - generated by an erlang application

This GDB was configured as "x86_64-apple-darwin15.4.0".
"/Users/sad/projects/core" is not a core dump: File format not recognized

$ file core
/Users/sad/projects/core: ELF 64-bit LSB core file x86-64, version 1 (SYSV), ...

Mac OS does not use ELF file format. We can safely assume that this core came from some other system, not the one you are trying to analyse it on.

It is still possible to analyse that core on the Mac OS system, but you need:

1. a cross-gdb (i.e. one that can run on Mac OS host, but can deal with ELF files for your target; it is likely that you'll have to build such GDB yourself) and
2. (unless you have a fully-static executable), you need complete set of shared libraries from the host on which the crash happened. See this answer.

In general, it is much easier to do the post-mortem analysis on the host where the crash happened.

---

Related Topics