Getting BPF programs working with USDT probes (Dtrace) in Linux

3

So I'm following this link to attach a BPF program to user space probes, Dtrace format (see section User Statically Defined Tracepoints).

C program:

#include <sys/sdt.h>
int main() {
    DTRACE_PROBE("hello-usdt", "probe-main");
}

The checks to ensure the probe info is included:

readelf -n hello_usdt
  stapsdt              0x00000033   NT_STAPSDT (SystemTap probe descriptors)
    Provider: "hello_usdt"
    Name: "probe-main"
    Location: 0x0000000000400535, Base: 0x00000000004005d4, Semaphore: 0x0000000000000000
    Arguments: 

Also with tplist:

sudo /usr/share/bcc/tools/tplist -l /path/to/hello_usdt
/path/to/hello_usdt "hello_usdt":"probe-main"

Content of the BPF program (usdt.py):

from bcc import BPF, USDT

bpf_source = """
#include <uapi/linux/ptrace.h>
int trace_binary_exec(struct pt_regs *ctx) {
  u64 pid = bpf_get_current_pid_tgid();
  bpf_trace_printk("New hello_usdt process running with PID: %d", pid);
}
"""

usdt = USDT(path = "./hello_usdt")
usdt.enable_probe(probe = "probe-main", fn_name = "trace_binary_exec")
bpf = BPF(text = bpf_source, usdt = usdt)
bpf.trace_print()

Failure description:

sudo ./usdt.py 
Traceback (most recent call last):
  File "./usdt.py", line 13, in <module>
    usdt.enable_probe(probe = "probe-main", fn_name = "trace_binary_exec")
  File "/usr/lib/python3/dist-packages/bcc/usdt.py", line 154, in enable_probe
    probe
bcc.usdt.USDTException: failed to enable probe 'probe-main';
        a possible cause can be that the probe requires a pid to enable

What I tried also:

  1. Giving the full executable (hello_usdt) in the BPF program:

usdt = USDT(path = "/full/path/to/hello_usdt")

  1. Replacing above line with the PID value (./hello_usdt &), fetch pid directly in the BPF program:

usdt = USDT(pid=1234)

  1. Tried with both python2.7 and 3.7, after installing python bcc package.

No luck so far. Any help much appreciated. Thanks.

linux
dtrace
bpf
ebpf
probe
asked on Stack Overflow Jun 29, 2020 by mdaniel • edited Jun 29, 2020 by Qeole

1 Answer

4

I think i figured out what was wrong in the original link. I patched the source file of bcc-tools (function bcc_usdt_enable_probe and deeper), compiled and installed this custom version with some printfs to trace why it's failing and it turns out that when parsing the probes list, the probe defined in my program was seen as string with additional quotes, something like this: ""probe-main"". So I replaced the line:

DTRACE_PROBE("hello-usdt", "probe-main");

with

DTRACE_PROBE("hello-usdt", probe-main);

The 2nd change I did was in the python script. I didn't go into details, but it looks that this line was wrong (syntax):

bpf = BPF(text = bpf_source, usdt = usdt)

The python compiler was complaining that usdt is not known, I found then other implementations using usdt_contexts=[usdt]. So I replaced above line with:

bpf = BPF(text = bpf_source, usdt_contexts = [usdt])

Replaced then in the source code also the probe macro to include an additional timestamp:

DTRACE_PROBE1("hello_usdt", probe-main, tv.tv_sec);

Updated also the python script:

#!/usr/bin/python3
from bcc import BPF, USDT

bpf_source = """
#include <uapi/linux/ptrace.h>
int trace_binary_exec(struct pt_regs *ctx) {
  u64 pid = bpf_get_current_pid_tgid();
  u64 ts=0;
  bpf_usdt_readarg(1, ctx, &ts);
  bpf_trace_printk("PROBE-HIT (PID: %d), TS: %lu\\n", pid, ts);
}
"""

usdt = USDT(path = "/path/to/hello_usdt")
usdt.enable_probe(probe = "probe-main", fn_name = "trace_binary_exec")
bpf = BPF(text = bpf_source, usdt_contexts = [usdt])
bpf.trace_print()

And got it running:

sudo ./usdt.py
b'      hello_usdt-18920 [000] .... 300066.568941: 0x00000001: PROBE-HIT (PID: 18920), TS: 1593460390'
b'      hello_usdt-18920 [000] .... 300067.569284: 0x00000001: PROBE-HIT (PID: 18920), TS: 1593460391'
b'      hello_usdt-18920 [000] .... 300068.569509: 0x00000001: PROBE-HIT (PID: 18920), TS: 1593460392'
b'      hello_usdt-18920 [000] .... 300069.569935: 0x00000001: PROBE-HIT (PID: 18920), TS: 1593460393'
b'      hello_usdt-18920 [000] .... 300070.570362: 0x00000001: PROBE-HIT (PID: 18920), TS: 1593460394'
b'      hello_usdt-18920 [000] .... 300083.574478: 0x00000001: PROBE-HIT (PID: 18920), TS: 1593460407'
b'      hello_usdt-18920 [000] .... 300084.574782: 0x00000001: PROBE-HIT (PID: 18920), TS: 1593460408'
b'      hello_usdt-18920 [000] .... 300085.575214: 0x00000001: PROBE-HIT (PID: 18920), TS: 1593460409'
b'      hello_usdt-18920 [000] .... 300086.575630: 0x00000001: PROBE-HIT (PID: 18920), TS: 1593460410'

I don't know why the implementation in the original link is different. Maybe a matter of BPF/Dtrace version, didn't go in details. However, I didn't find other implementations that cover all steps for a basic C based program, but found snippets which helped me solving the issue.

answered on Stack Overflow Jun 29, 2020 by mdaniel

User contributions licensed under CC BY-SA 3.0