How to Debug Linux Kernel Modules with Qemu

How to debug Linux kernel modules with QEMU?

The easiest way in my opinion is to use buildroot
http://buildroot.uclibc.org/

clone it, configure it to use your custom kernel (default userspace is fine for a start, you might want to change it later).

it will build your kernel and root filesystem. the entire process takes about half an hour, twenty minutes of which is compiling the monster

my run line looks something:
qemu-system-i386
-hda rootfs.ext2
-kernel bzImage
-m 512M
-append "root=/dev/sda console=ttyS0"
-localtime
-serial stdio

and some more options regarding a tap device

How to debug the Linux kernel with GDB and QEMU?

I'd try:

(gdb) target remote localhost:1234
(gdb) continue

Using the '-s' option makes qemu listen on port tcp::1234, which you can connect to as localhost:1234 if you are on the same machine. Qemu's '-S' option makes Qemu stop execution until you give the continue command.

Best thing would probably be to have a look at a decent GDB tutorial to get along with what you are doing. This one looks quite nice.

debugging kernel with qemu and gdb, breakpoint not working?

This was solve not long after my posting the question and I forgot to put an answer.

It was because of the KASLR (kernel address space location randomization). You should disable it in the kernel configuration, or give option in the boot parameter. (without it, the kernel image is located in random location, causing mismach between debug symbol location and actual code location). This KASLR is turned on by default for aarch64.

In my case I did it with :

${QEMU_DIR}/qemu-system-aarch64 -M ${QMACHINE} -cpu cortex-a72 -kernel ${LINUX_DIR}/arch/arm64/boot/Image -initrd ${BUSYBOX_DIR}/initramfs.cpio.gz --append "root=/dev/ram init=/init nokaslr" -m 2048M -nographic

And I had to use 'hb'(or hbreak (hardware break)) instead of 'b'(or break).

How to debug the init_module() call of a Linux kernel module?

Yes, the init_module function gets called as soon as you load it to the kernel using insmod. You can just add a line of printk and verify it being printed as soon as you insert the module.

You cannot pass a parameter such as -m to debug the kernel module.

You can only pass parameters that are intended to be handled within the kernel module that you have written, using MODULE_PARAMS.

Issue with qemu and gdb

So, if I understand the problem correctly, you have a kernel module that needs no specific hardware. When you are working with the module, the system freezes but the kernel log contains nothing special.

The following may be helpful.

Getting the log

The symptoms you described may still be a result of a kernel oops or panic. The logging facilities sometimes die before they can output the information about the error to the log file. You may try to output the log via a serial port, this should be more reliable.

As your kernel module does not need any specific hardware, the easiest way is probably to install the same Linux distro as you use to a virtual machine and connect the virtual serial port (COM) of that machine to a pipe on your host system.

This is usually quite easy to do. For example, this blog post contains the detailed instructions in case the host OS and the guest OS are Ubuntu 11.10.

VirtualBox is used there to manage the virtual machines. If you prefer QEMU, this should be possible as well. I suppose it is a bit easier to go with VirtualBox though but it is a matter of personal preference.

Basically, you need to perform the following steps.

Create a virtual machine and install the Linux distro you need as a guest OS there.
Enable a serial port (COM1, ...) in the configuration of the virtual machine and configure it to connect to a special file on the host ("host pipe"), say /tmp/vbox_serial.
Start the guest OS and adjust its boot options: at least, add console=ttyS0,115200 or something like that to the kernel options in the boot loader menu.
On the host, start minicom, socat or whatever else to read from /tmp/vbox_serial.
That is it. Now you should get the kernel log of the guest OS pouring to your host system via /tmp/vbox_serial. If the guest system crashes then, you will get the log even if it is not saved into a file on the guest itself.

To make things easier, you may use socat on your host system rather than minicom that the author of that blog post suggests. The power of minicom is probably not needed here.

This way, you can use socat and tee to save the log to guest.log file while still outputting it to the console:

socat /tmp/vbox_serial - | tee guest.log

If there was a kernel oops or panic, the backtrace in the log usually helps to find out what
has gone wrong.

Detecting Deadlocks

If you have obtained the full log via a serial connection or some other means and still there is nothing suspicious there and you suspect there has been a deadlock in the kernel,
lockdep tool may help. It is included into the kernel (but you may need to rebuild the kernel with CONFIG_LOCKDEP_SUPPORT=y).

Lockdep detects the potential deadlocks and outputs the results to the kernel log. This presentation may help you analyse its output.

Tracing Facilities

If you need tracing of some events in the kernel to debug your system, there are some tools that could be handy.

Kprobes - a kind of breakpoints you can set in almost arbitrary place in the kernel. Can be used to trace function calls among other things, with a moderate performance impact.
SystemTap - a powerful system to analyze what is going on in the kernel. Part of it is based on Kprobes.
Ftrace - a tracing system included into the kernel, incurs less overhead than Kprobes if that matters.

How is Linux kernel live debugging done and what tools are used?

Another option is to use an ICE or JTAG controller, and GDB. This 'hardware' solution is especially used with embedded systems.

But for instance QEMU offers similar features:

start QEMU with a GDB 'remote' stub which listens on 'localhost:1234' : qemu -s ...,
then with GDB, you open the kernel file vmlinux compiled with debug information (you can take a look a this mailing list thread where they discuss the unoptimization of the kernel).
connect GDB and QEMU: target remote localhost:1234

see your live kernel:

  (gdb) where
  #0  cpu_v7_do_idle () at arch/arm/mm/proc-v7.S:77
  #1  0xc0029728 in arch_idle () atarm/mach-realview/include/mach/system.h:36
  #2  default_idle () at arm/kernel/process.c:166
  #3  0xc00298a8 in cpu_idle () at arch/arm/kernel/process.c:199
  #4  0xc00089c0 in start_kernel () at init/main.c:713

Unfortunately, user-space debugging is not possible so far with GDB (no task list information, no memory management unit reprogramming to see different process contexts, ...), but if you stay in kernel-space, that's quite convenient.

info threads will give you the list and states of the different CPUs

You can get more details about the procedure in this PDF:

Debugging Linux systems using GDB and QEMU.

How to Debug Linux Kernel Modules with Qemu