How to Get Perf to Find Symbols in My Program

perf can find symbol in the kernel ,but can not find symbol in my program. How to fix it?

Today, when I run perf test, I got a message saying vmlinux symtab matches kallsyms: Failed.

When I was looking for the reason, I found that the reason is that the value of /proc/sys/kernel/kptr_restrict is 1. When we set it to 0, we will get the symbol in our program.

perf cannot find external module symbols

I had to make it a kernel module, then perf could find its symbols:

IN_TREE_DIR=/lib/modules/`uname -r`/kernel/modulename
mkdir -p $IN_TREE_DIR
cp modulename.ko $IN_TREE_DIR
depmod -a

Using debug symbols with perf

You are using an old version of perf which do not support -g dwarf but only -g (without argument) i.e. it does not support DWARF unwinding.

how to get rid of the unknown section in the perf

The [unknown] block in the perf report output refers to the name of the dynamic shared object (DSO). perf report could not resolve the DSO path and hence it prints [unknown]. Per the latest kernel source code tree (which is 5.3.9 at the time of writing), you can see this here.

It is important to know that the determination of the DSO symbols happen with the help of the sampled event address. The function thread__resolve is responsible for doing exactly that. In older kernels, the thread__resolve method had another name - perf_event__preprocess_sample_addr.

Given the snapshot of your output, it looks like the address of the event that was sampled during perf record, is 0. This means that the address could not be resolved at all. It is an address in the kernel space (looking at the symbol [k] 0000000000000000) and perf in your case, could not resolve it.

The comments highlight setting perf_event_paranoid to a suitable value so that you can probe both kernel and user-space events successfully. Setting perf_event_paranoid to a value that allows you to correctly probe events in the kernel space should "be a step" towards correctly resolving the address.

how to get perf to decode symbols for hadoop java code

It turns out that the perf-$pid.map objects that show up in the perf output are associated with JIT compiled java code. In order for perf to be able to decode those symbols, the java code needs to produce /tmp/perf-$pid.map symbol map files.

There is a perf-map-agent library on github that can be used to generate the symbol map files for JITed code. With the library in place, add -agentpath:<dir>/libperfmap.so to the java command line.

To make hadoop jobs generate the symbol maps, add lines such as the following to hadoop-env.sh:

export HADOOP_JAVA_PLATFORM_OPTS="-agentpath:/usr/lib/oprofile/libjvmti_oprofile.so $HADOOP_JAVA_PLATFORM_OPTS"
export JAVA_TOOL_OPTIONS="-agentpath:/usr/lib/libperfmap.so $JAVA_TOOL_OPTIONS"

Can't see kernel symbols in 'perf report' when trying to trace a probe on CentOS 7.7

Perf in RHEL 7.7 has a bug that prevents it from reporting kernel symbols: https://access.redhat.com/solutions/4797281

I strongly suspect CentOS 7.7 has the exact same bug. Following the workaround suggested in the page above, I've downgraded perf to the version in CentOS 7.6, and it seems to work:

# give access to packages from 7.6
yum-config-manager --add-repo=http://vault.centos.org/centos/7.6.1810/updates/x86_64/

# reinstall older perf
yum remove perf
yum install perf-3.10.0-957.1.3.el7

# optional: disable the vault repo
yum-config-manager --disable vault.centos.org_centos_7.6.1810_updates_x86_64_

# test that it now works
perf record -a -g -- sleep 3
perf report --stdio
# looks good!