What Is the Meaning of Question Marks '' in Linux Kernel Panic Call Traces

What is the meaning of question marks '?' in Linux kernel panic call traces?

'?' means that the information about this stack entry is probably not reliable.

The stack output mechanism (see the implementation of dump_trace() function) was unable to prove that the address it has found is a valid return address in the call stack.

'?' itself is output by printk_stack_address().

The stack entry may be valid or not. Sometimes one may simply skip it.
It may be helpful to investigate the disassembly of the involved module to see which function is called at ClearFunctionName+0x88 (or, on x86, immediately before that position).

Concerning reliability

On x86, when dump_stack() is called, the function that actually examines the stack is print_context_stack() defined in arch/x86/kernel/dumpstack.c. Take a look at its code, I'll try to explain it below.

I assume DWARF2 stack unwind facilities are not available in your Linux system (most likely, they are not, if it is not OpenSUSE or SLES). In this case, print_context_stack() seems to do the following.

It starts from an address ('stack' variable in the code) that is guaranteed to be an address of a stack location. It is actually the address of a local variable in dump_stack().

The function repeatedly increments that address (while (valid_stack_ptr ...) { ... stack++}) and checks if what it points to could also be an address in the kernel code (if (__kernel_text_address(addr)) ...). This way it attempts to find the functions' return addresses pushed on stack when these functions were called.

Of course, not every unsigned long value that looks like a return address is actually a return address. So the function tries to check it. If frame pointers are used in the code of the kernel (%ebp/%rbp registers are employed for that if CONFIG_FRAME_POINTER is set), they can be used to traverse the stack frames of the functions. The return address for a function lies just above the frame pointer (i.e. at %ebp/%rbp + sizeof(unsigned long)). print_context_stack checks exactly that.

If there is a stack frame for which the value 'stack' points to is the return address, the value is considered a reliable stack entry. ops->address will be called for it with reliable == 1, it will eventually call printk_stack_address() and the value will be output as a reliable call stack entry. Otherwise the address will be considered unreliable. It will be output anyway but with '?' prepended.

[NB] If frame pointer information is not available (e.g. like it was in Debian 6 by default), all call stack entries will be marked as unreliable for this reason.

The systems with DWARF2 unwinding support (and with CONFIG_STACK_UNWIND set) is a whole another story.

How to read, understand, analyze, and debug a Linux kernel panic?

It's just an ordinary backtrace, those functions are called in reverse order (first one called was called by the previous one and so on):

unwind_backtrace+0x0/0xf8
warn_slowpath_common+0x50/0x60
warn_slowpath_null+0x1c/0x24
ocal_bh_enable_ip+0xa0/0xac
bdi_register+0xec/0x150

The bdi_register+0xec/0x150 is the symbol + the offset/length there's more information about that in Understanding a Kernel Oops and how you can debug a kernel oops. Also there's this excellent tutorial on Debugging the Kernel

Note: as suggested below by Eugene, you may want to try addr2line first, it still needs an image with debugging symbols though, for example

addr2line -e vmlinux_with_debug_info 0019594c(+offset)

Linux kernel panic. Understanding oops message - pgd?

pgd is short for "page global directory", the kernel's name for the top level of a page table.

How to interpret addresses in a kernel oops

All the information you need is right there:

[10991.880354] BUG: unable to handle kernel NULL pointer dereference at   (null)

That's the reason.

[10991.880359] IP: [<c06969d4>] iret_exc+0x7d0/0xa59

That's the instruction pointer at the time of fault. We'll get back to this momentarily.

[10991.880365] *pdpt = 000000002258a001 *pde = 0000000000000000

These are physical page table entries. the descriptor table, and the page descriptor entry. Naturally, the latter is NULL, since it's a NULL pointer. The above values are rarely useful (only in cases where physical memory mapping is required)

[10991.880368] Oops: 0002 [#1] PREEMPT SMP

That's the oops code. PREEMPT SMP shows you the kernel is preemptible, and compiled for SMP, rather than UP. This is important for cases where the bug is from some race condition, etc.

[10991.880371] last sysfs file: /sys/devices/platform/coretemp.3/temp1_input

That's not necessarily the culprit, but oftentimes is. sys files are exported by various kernel modules, and oftentimes an I/O operation on the sys file leads to the faulty module code execution.

[10991.880374] Modules linked in: ... [last unloaded: preloadtrace]

The kernel doesn't necessarily know which module is to blame, so it's giving you all of them. Also, it may very well be that a recently unloaded module didn't clean up and left some residue (like some timer, or callback) in the kernel - which is a classic case for oops or panic. So the kernel reports the last unloaded one, as well.

[10991.880402] Pid: 4487, comm: python Tainted: GF           2.6.37.1-1.2-desktop #1 To be filled by O.E.M. To be filled by O.E.M./To be filled by O.E.M.

If the faulting thread is a user mode thread, you get the PID and command line. "Tainted" flags are the kernel's way of saying it's not a kernel fault (the kernel source is open and "pure". "Taint" comes from the blasphemous non-GPL modules, and others.

[10991.880408] EIP: 0060:[<c06969d4>] EFLAGS: 00210246 CPU: 0
[10991.880411] EIP is at iret_exc+0x7d0/0xa59

That gives you the faulting instruction pointer, both directly and in symbol+offset form. The part after the slash is the size of the function.

[10991.880413] EAX: 00000000 EBX: 00000000 ECX: 0000018c EDX: b7837000
[10991.880415] ESI: b7837000 EDI: 00000000 EBP: b7837000 ESP: e2a81ee0
[10991.880417]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068

The registers are shown here. Your NULL is likely EAX.

[10991.880420] Process python (pid: 4487, ti=e2a80000 task=df940530 task.ti=e2a80000)
[10991.880422] Stack:
[10991.880423]  00000000 0000018c 00000000 0000018c e5e903dc e4616353 00000009 df99735c
[10991.880428]  df900a7c df900a7c b7837000 df80ad80 df99735c 00000009 e46182a4 e2a81f70
[10991.880433]  e28cd800 e09fc840 e28cd800 fffffffb e09fc888 c03718c1 e4618290 0000018c

The area near the stack pointer is displayed. The kernel has no idea what these values mean, but they are the same output as you'd get from gdb displaying the $rsp. So it's up to you to figure what they are. (For example, c03718c1 is a kernel return address, likely - so you can go to /proc/kallsyms to figure it out, or rely on it being in the trace, as it is , next). This tells you that all the data up to it is the stack frame

Now, because you have the stack call trace, you can put the fragments together:

[10991.880423]  00000000 0000018c 00000000 0000018c e5e903dc e4616353 --> back to write_func

[            ]  ..................................................... 00000009 df99735c
[10991.880428]  df900a7c df900a7c b7837000 df80ad80 df99735c 00000009 e46182a4 e2a81f70
[10991.880433]  e28cd800 e09fc840 e28cd800 fffffffb e09fc888 c03718c1  --> back to proc_file_write

[10991.882046] Code: f3 aa 58 59 e9 5a f9 d7 ff 8d 0c 88 e9 12 fa d7 ff 01 d9 e9 7b fa d7 ff 8d 0c 8b e9 73 fa d7 ff 01 d9 eb 03 8d 0c 8b 51 50 31 c0 <f3> aa 58 59 e9 cf fa d7 ff 01 d9 e9 38 fb d7 ff 8d 0c 8b e9 30

Again, kernel can't disassemble for you (it's oopsing, and might very well panic, give it a break!). But you can use gdb to disassemble these values.

So now you know everything. You can actually disassemble your own module and figure out where exactly in write_func the NULL pointer is dereferenced. (You're probably passing it as an argument to some function).

---

Related Topics