Core Dump in Linux

How to generate a core dump in Linux on a segmentation fault?

This depends on what shell you are using. If you are using bash, then the ulimit command controls several settings relating to program execution, such as whether you should dump core. If you type

ulimit -c unlimited

then that will tell bash that its programs can dump cores of any size. You can specify a size such as 52M instead of unlimited if you want, but in practice this shouldn't be necessary since the size of core files will probably never be an issue for you.

In tcsh, you'd type

limit coredumpsize unlimited

How to generate core dump file in Linux?

The generation of core dump files it is not always enabled. Try with the ulimit command.

How do I analyze a program's core dump file with GDB when it has command-line parameters?

You can use the core with GDB in many ways, but passing parameters which is to be passed to the executable to GDB is not the way to use the core file. This could also be the reason you got that error. You can use the core file in the following ways:

gdb <executable> <core-file> or gdb <executable> -c <core-file> or

gdb <executable>
...
(gdb) core <core-file>

When using the core file you don't have to pass arguments. The crash scenario is shown in GDB (checked with GDB version 7.1 on Ubuntu).

For example:

$ ./crash -p param1 -o param2
Segmentation fault (core dumped)
$ gdb ./crash core
GNU gdb (GDB) 7.1-ubuntu
...
Core was generated by `./crash -p param1 -o param2'. <<<<< See this line shows crash scenario
Program terminated with signal 11, Segmentation fault.
#0  __strlen_ia32 () at ../sysdeps/i386/i686/multiarch/../../i586/strlen.S:99
99    ../sysdeps/i386/i686/multiarch/../../i586/strlen.S: No such file or directory.
    in ../sysdeps/i386/i686/multiarch/../../i586/strlen.S
(gdb)

If you want to pass parameters to the executable to be debugged in GDB, use --args.

For example:

$ gdb --args ./crash -p param1 -o param2
GNU gdb (GDB) 7.1-ubuntu
...
(gdb) r
Starting program: /home/@@@@/crash -p param1 -o param2

Program received signal SIGSEGV, Segmentation fault.
__strlen_ia32 () at ../sysdeps/i386/i686/multiarch/../../i586/strlen.S:99
99    ../sysdeps/i386/i686/multiarch/../../i586/strlen.S: No such file or directory.
    in ../sysdeps/i386/i686/multiarch/../../i586/strlen.S
(gdb)

Man pages will be helpful to see other GDB options.

Most useful commands are:

bt (backtrace)
info locals (show values of local variables)
info registers (show values of CPU registers)
frame X (change to stack frame X)
up and down (navigate in the stack frame (call chain))

What is a core dump file in Linux? What information does it provide?

It's basically the process address space in use (from the mm_struct structure which contains all the virtual memory areas), and any other supporting information^*a, at the time it crashed.

For example, let's say you try to dereference a NULL pointer and receive a SEGV signal, causing you to exit. As part of that process, the operating system tries to write your information to a file for later post-mortem analysis.

You can load the core file into a debugger along with the executable file (for symbols and other debugging information, for example) and poke around to try and discover what caused the problem.

^*a: in kernel version 2.6.38, fs/exec.c/do_coredump() is the one responsible for core dumps and you can see that it's passed the signal number, exit code and registers. It in turn passes the signal number and registers to a binary-format-specific (ELF, a.out, etc) dumper.

The ELF dumper is fs/binfmt_elf.c/elf_core_dump() and you can see that it outputs non-memory-based information, like thread details, in fs/binfmt_elf.c/fill_note_info(), then returns to output the process space.

Who generate a core dump file? Kernel or glibc?

The kernel itself generates the coredump. See the core handling routines in the linux kernel source here:

http://lxr.linux.no/linux+v3.12.6/fs/coredump.c

If the process receives any of the following signals 1, the kernel responds by attempting a coredump.

#define SIG_KERNEL_COREDUMP_MASK (\
rt_sigmask(SIGQUIT)   |  rt_sigmask(SIGILL)    | \
rt_sigmask(SIGTRAP)   |  rt_sigmask(SIGABRT)   | \
rt_sigmask(SIGFPE)    |  rt_sigmask(SIGSEGV)   | \
rt_sigmask(SIGBUS)    |  rt_sigmask(SIGSYS)    | \
rt_sigmask(SIGXCPU)   |  rt_sigmask(SIGXFSZ)   | \
SIGEMT_MASK

This coredump is configurable, and can be disabled or controlled in several ways, including the file /proc/sys/kernel/core_pattern, and ulimit. One can also control the delivery of these signals through the signal handling mechanisms.

Core dump in Linux

The usual solution is to build with -g and to strip off the debug information before releasing the file. Look for the 'strip' command.
You keep the file with debug information and use it to debug core dumps you get from customers.

If you want to print the human readable backtrace on the users machine you'll need to distribute binaries with (some) debug information.
Look for the 'backtrace()' function in glibc.

Note that core dumps will be created (if ulimit is set appropriately) even if your binary doesn't contain debug information.

The best way to ensure the creation of a core dump is probably to execute your binary from a script which sets ulimit before running the binary.

Enabling core dumps on Debian 9

Maybe your system has been messed by systemd. In that case you have to install systemd-coredump:

# apt-get install systemd-coredump

Next you can inspect the core dumps with coredumpctl.

# sleep 100
^\Quit (core dumped)
# coredumpctl list
TIME                            PID   UID   GID SIG COREFILE  EXE
Tue 2021-06-15 18:00:58 CEST  21065     0     0   3 present   /bin/sleep

They are placed in /var/lib/systemd/coredump.

# ll /var/lib/systemd/coredump/
total 68
-rw-r----- 1 root root 26605 Jun 15 18:00 core.sleep.0.af2fa0f571f84f279c4510ee3281a787.21065.1623772857000000.lz4

Find out when core dump is finished

I was able to get my script to wait for the core dump write to finish by using inotifywait. See the following snippet:

core_file="core.$app_pid"
core_path=/path/core/file/dir
core_complete="false"
# Setup inotifywait loop to wait until core file has been complety written
inotifywait -e close_write --format '%f' $core_path | while read line; do
  echo "File $line was closed"
  # Check to see if the line we read was the core file
  if [[ "$line" == "$core_file"  ]]; then
    echo "Core file write complete"
    core_complete="true"
  fi
done

This so far has seemed to do the trick.