Attaching Ebpf to Kprobe

Attaching eBPF to KPROBE?

In your eBPF program:

#include <errno.h>
#include <bpf/bpf.h>
#include <stdio.h>
#include <string.h>

#include "bpf_load.h"
#include "bpf_util.h"
#include "libbpf.h"

#define SEC(NAME) __attribute__((section(NAME), used))

SEC("kprobe/execve")
int bpf_prog1(struct pt_regs *ctx)
{
        char m[]="hello world";
        bpf_trace_printk(m,sizeof(m));
        
        return 0;
}

You use bpf_trace_printk() correctly (although you might want to add a \n at the end of your message or your output will be messy), but it turns out none of the files you include contains the definition for this helper.

bpf_trace_printk() is compiled as part of the kernel and won't ever be compiled into your BPF object file. When trying to load your program, the function load_bpf_file() does a relocation step where it places the number associated to bpf_trace_printk() (in user API) in the relevant instruction of the eBPF bytecode.

But it needs to find this number somewhere. It is defined in header linux/bpf.h (pulled from several of your includes) as FN(trace_printk) (some macro magic going on), resulting de facto in a #define BPF_FUNC_trace_prink 6. But you need to tell your loading function that it corresponds to the bpf_trace_prink() you're calling!

Two solutions:

• Manually declare it:
  

  static int (*bpf_trace_printk)(const char *fmt, int fmt_size, ...) =
          (void *) BPF_FUNC_trace_printk;
  

• Or add a header that contains it, e.g. tools/lib/bpf/bpf_helpers.h in kernel repo. In your case:
  

  #include <bpf/bpf_helpers.h>
  

  (Note that this header is generated when compiling libbpf, it is not present in the repository by default.)

which kprobe hooks can I attach eBPF programs to?

You can attach a kprobe to nearly all functions of your kernel (provided they have not been inlined when compiling the kernel). One way to list those functions is through cat /proc/kallsyms. In your case, grep for tcp on that file? As for the signatures, I don't believe there is a place to get them other than by reading the kernel sources for your kernel version.

Note that, because kernel functions are not part of the user API, there is no guarantee regarding the stability of their signature (which could be a reason why a list of signatures would make little sense—other than the huge number of signatures to list). If you want your eBPF programs to be more robust and portable between different kernel versions, you should have a look at CO-RE.

How can I attached a BPF program to a kernel function via a kprobe?

TL;DR You can use the traditional kprobe API to trace a function, then perf_event_open + ioctl to attach a BPF program.

This is implemented in the load_and_attach function of file load_bpf.c in the kernel, and in the bpf_attach_kprobe and bpf_attach_tracing_event function of file libbpf.c in bcc.

---

You can see this in action when tracing the hello_world.py from bcc:

$ strace -s 100 python examples/hello_world.py
[...]
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_KPROBE, insn_cnt=15, insns=0x7f35716217d0, license="GPL", log_level=0, log_size=0, log_buf=0, kern_version=265728}, 72) = 3
openat(AT_FDCWD, "/sys/bus/event_source/devices/kprobe/type", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/sys/bus/event_source/devices/kprobe/format/retprobe", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/sys/kernel/debug/tracing/kprobe_events", O_WRONLY|O_APPEND) = 4
getpid()                                = 8121
write(4, "p:kprobes/p_sys_clone_bcc_8121 sys_clone", 40) = 40
close(4)                                = 0
openat(AT_FDCWD, "/sys/kernel/debug/tracing/events/kprobes/p_sys_clone_bcc_8121/id", O_RDONLY) = 4
read(4, "1846\n", 4096)                 = 5
close(4)                                = 0
perf_event_open({type=PERF_TYPE_TRACEPOINT, size=0 /* PERF_ATTR_SIZE_??? */, config=1846, ...}, -1, 0, -1, PERF_FLAG_FD_CLOEXEC) = 4
mmap(NULL, 36864, PROT_READ|PROT_WRITE, MAP_SHARED, 4, 0) = 0x7f356c58b000
ioctl(4, PERF_EVENT_IOC_SET_BPF, 0x3)   = 0
ioctl(4, PERF_EVENT_IOC_ENABLE, 0)      = 0
openat(AT_FDCWD, "/sys/kernel/debug/tracing/trace_pipe", O_RDONLY) = 5
fstat(5, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
fstat(5, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
read(5, 

1. The first syscall (bpf) loads the BPF program in the kernel.
2. Then bcc follows the kprobe API to trace sys_clone by writing p:kprobes/p_sys_clone_bcc_8121 sys_clone in p:kprobes/p_sys_clone_bcc_8121 sys_clone.
3. bcc retrieves, in p:kprobes/p_sys_clone_bcc_8121 sys_clone, an ID to use in perf_event_open.
4. bcc calls perf_event_open with type PERF_TYPE_TRACEPOINT
5. and attaches the loaded BPF program (referenced by fd 0x3) to that perf_event, with an PERF_EVENT_IOC_SET_BPF ioctl.

Tried to learn eBPF tail call, fails to attach kprobe

So I didn't have time to finish typing my answer before the cause of the issue was found by Roadowl in the comments :). Since there was a second part in the question about references for tail calls, and I have written that bit anyway, I post it just in case it can be helpful.

• Just for future reference, bcc documentation has a paragraph on tail calls, but by the look of your code you found it already :).

• If you want to understand how tail calls work, I would suggest having a look at Cilium's documentation, in particular the section on tail calls. Keep in mind that bcc provides some wrappers (such as the .call() function) that won't be covered in Cilium's doc, but it should help you understand what is happening under the hood anyway.

• bcc itself does not seem to use tail calls a lot, I could only find one networking example that seems to use it (although I did not search thoroughly).

• You can find some simple example programs using tail calls in the iproute2 repository (a simple one, one that loops). You can also find some in kernel samples or selftests: grep for tail_call.

Exception: Failed to attach BPF to kprobe when executing sudo opensnoop-bpfcc

I had to compile bcc from source code instead of installing it using the package.

1. Install linux kernel headers

sudo apt install linux-headers-$(uname -r)

2. Install required dependencies

sudo apt-get -y install bison build-essential cmake flex git libedit-dev \ libllvm6.0 llvm-6.0-dev libclang-6.0-dev python zlib1g-dev libelf-dev

3. Compile bcc from source code

git clone https://github.com/iovisor/bcc.git
mkdir bcc/build; cd bcc/build
cmake .. -DCMAKE_INSTALL_PREFIX=/usr
make
sudo make install

failing to attach eBPF `kretprobes` to `napi_poll()` with bcc tools

Most of the time the Failed to attach BPF to kprobe error is caused by an inlined function. As explained in the Kprobes documentation (section Kprobes Features and Limitations), Kprobes will fail to attach if the target function was inlined. Since napi_poll is static, it might have been inlined at compile time.

You can check in kernel symbols if napi_poll was inlined or not:

$ cat /boot/System.map-`uname -r` | grep " napi_poll"
$
$ cat /boot/System.map-`uname -r` | grep " net_rx_action"
ffffffff817d8110 t net_rx_action

On my system, napi_poll is inlined while net_rx_action is not.

---

There are several workarounds for this problem, depending on your goal.

1. If you don't mind recompiling your kernel, you can use the Linux inline attribute to ensure napi_poll is not inlined.
2. If you can't change your kernel, the usual workaround is to find a calling function of napi_poll that provides the same information. A function called by napi_poll can also work if it provides enough information and is not inlined itself.

`do_sys_open` vs `__x86_sys_open` when attaching kprobe

A BPF program attached to __x86_sys_open won't have the same result if you attach it to do_sys_open instead because those two functions don't have the same prototype:

long do_sys_open(int dfd, const char __user *filename, int flags, umode_t mode);
long sys_open(const char __user *filename, int flags, umode_t mode);

So the filename argument, for example, won't be stored in the same register depending on which function you trace. You will need to edit the BPF program as well to fix this.

---

Related Topics