Version 5.6 of the Systemtap tracing tool has been released.
BPF LSM hooks and XDP packet-processing probes for the --bpf runtime, BTF-based kernel.tracepoint probes, statement execution tracing, a new @enumname() operator, richer runtime error context, dyninst hardware watchpoints, modern systemd service templates, and broad Linux 7.2 runtime/tapset compatibility work. Multithreaded speedups throughout.
| From: | | "Frank Ch. Eigler" <fche-AT-elastic.org> |
| To: | | systemtap-AT-sourceware.org, lwn-AT-lwn.net |
| Subject: | | systemtap 5.6 release |
| Date: | | Thu, 17 Sep 2026 19:34:21 -0400 |
| Message-ID: | | <aqx4_ewIxjMFmYOi@elastic.org> |
The SystemTap team announces release 5.6
BPF LSM hooks and XDP packet-processing probes for the --bpf runtime,
BTF-based kernel.tracepoint probes, statement execution tracing, a new
@enumname() operator, richer runtime error context, dyninst hardware
watchpoints, modern systemd service templates, and broad Linux 7.2
runtime/tapset compatibility work. Multithreaded speedups throughout.
= Where to get it
https://sourceware.org/systemtap/ - our project page
https://sourceware.org/ftp/systemtap/releases/
https://koji.fedoraproject.org/koji/packageinfo?packageID...
git tag release-5.6 (commit 211e84349504077f0c84f759601b02f946465d29)
There have been around 260 commits since the last release.
There have been around 26 bugs fixed / features added since the last
release.
= SystemTap frontend (stap) changes
- New `--debug` option builds the generated kernel module (.ko) or
dyninst module (.so) with maximum debugging information
(CONFIG_DEBUG_INFO, -g, gcc -save-temps=obj) and runs pahole(1)
when available. (PR34169)
- Statement-by-statement script execution tracing via
`stap -D STP_EXECTRACE` (not yet available with --bpf). (PR34166)
- New `--semantic-keep-going` option: continue pass-2 elaboration after
semantic errors, dump a tab-separated SEMANTIC_ERROR catalog at the end
of the failing pass, and still exit non-zero. Aimed at machine testing.
- Optimizer work for multiple probe handlers on the same probe point.
(PR34161)
- Pass-2 elaboration derives DWARF probe points concurrently within each
stapfile and shares per-CU/function caches; capped by STAP_NTHREADS,
if elfutils is compiled with multithreading support. (PR34432)
- DW_OP_entry_value location expressions are now honored, improving
$parm$/$var$ pretty-printing and @entry() access for entry-only locals.
= SystemTap backend changes
- LSM (Linux Security Module) hook support for the BPF runtime
(lsm.bprm_check_security, lsm.file_open, lsm.socket_create, and many
others). Use `$ctx` with `@cast()` and `$return` to allow/deny.
Requires kernel 5.7+, CONFIG_BPF_LSM=y. (PR34126)
- XDP packet-processing probes for the BPF runtime: attach to network
interfaces to inspect incoming packets and return a pass/drop/tx verdict.
Requires kernel 4.8+, CONFIG_BPF_SYSCALL=y. (PR34583)
- BTF-based `kernel.tracepoint("name")` and
`module("mod").tracepoint("name")` probes discover tracepoints from
`btf_trace_*` typedefs without trace headers/tracefs. Handler args are
positional ($arg1, ...). Separate from `kernel.trace("system:name")`.
Requires vmlinux.h / CONFIG_DEBUG_INFO_BTF for kernel probes.
(PR34632)
- Kernel build IDs are now extracted from compressed/non-ELF vmlinuz on
aarch64 and s390x, feeding dwfl/debuginfod. (PR34488)
- Runtime handler errors now include a short call-site stack through
nested functions and the active probe point. (PR34320)
- Runtime/tapset porting for Linux 7.2-rc kernels (and continued
enterprise back-compat). (PR34318)
- Python HelperSDT modernization; deprecate/remove Python 2 probing
support. (PR34292, PR34293)
= SystemTap script-language changes
- `@cast()` operations now see through typedefs. (PR34199)
- New `@enumname()` operator: map an integral value to its DWARF
enumerator name string (the reverse of `@enum()`). Unknown values
render as decimal. (PR34499)
- Short-circuit evaluation of `@defined()` ternaries during variable
expansion. (PR34414)
= SystemTap tapset changes
- `tp_syscall.*` / `syscall_any` prefer
`kernel.tracepoint("sys_enter")` / `sys_exit` when BTF vmlinux
tracepoints are available, with `kernel.trace(...)` fallbacks;
`tp_syscall("name")' includes amazingly fast dispatch for
arbitrary probed-syscall subsets; `tp_syscall("name").return`
supports `@entry()`. (PR34153)
- Python probing split into per-minor-version tapsets (3.9 through
3.15); Python 2 probes removed. Python 3.13 tapset added and py3execdir
probing fixed. (PR33037)
- process.data hardware watchpoints now work under --runtime=dyninst,
including process.data("SYMBOL") with run-time name resolution;
print_ubacktrace* uses a third-party stackwalk symbolized with elfutils.
- Numerous NFS/VFS/socket/SCSI/signal/futex adaptations for Linux 7.2
API drift.
= Developer / packaging changes
- REMOVED legacy `systemtap-service` / `systemtap.service` initscript
infrastructure. Replaced with systemd templates:
`stap@.service` (compile/run `.stp` from /etc/systemtap/script.d/)
and `staprun@.service` (run precompiled `.ko` modules), plus
standalone `stap-onboot` for initramfs embedding. (PR34128)
- Optional ML-DSA module signing (Secure Boot / MOK and staprun privilege
signatures) alongside default RSA; set SYSTEMTAP_MOK_KEY_TYPE /
SYSTEMTAP_SIGN_KEY_TYPE. Requires OpenSSL 3.5+ / NSS CKM_ML_DSA.
- New `--enable-debug` configure option and `--enable-glibcxx-debug`
command-line option. (PR31683)
- Probe javac against HelperSDT before enabling Java support. (PR34475)
- Testsuite speedups and listing_mode timeout fixes; show SystemTap
version in generated docs. (PR34152, PR34303, PR33105)
= SystemTap sample scripts
All 209 examples can be found at
https://sourceware.org/systemtap/examples/
New sample scripts:
bpf_xdp.stp
Count incoming packets per protocol via XDP: decode ethertypes on a
loopback XDP program and accumulate per-protocol packet/byte counts.
bpf_xdp_guru.stp
Bump IPv6 hop limits for proxy health checks: an XDP program rewrites
the hop limit of hop-limit-1 packets so proxied container health
checks succeed.
cve-2026-31431bpf.stp
EXPERIMENTAL emergency security band-aid using BPF LSM hooks;
blocks AF_ALG AEAD socket binds.
cve-2026-64600.stp
EXPERIMENTAL emergency security band-aid (RefluXFS); for
reference/education only.
= Examples of tested kernel versions
Based on Sourceware buildbot / Bunsen testruns for the release-tip
4.18.0 (RHEL8 x86_64)
5.14.0 (CentOS Stream 9 / RHEL9 x86_64)
6.12.0 (CentOS Stream 10 / RHEL10 x86_64)
7.2 (Fedora 44 x86_64)
7.3.0-rc* (Fedora rawhide x86_64 gcc + clang, s390x, riscv64, aarch64, ppc64le)
Test results are stored in bunsen, see
https://sourceware.org/systemtap/links.html
= Contributors for this release
Aaron Merey, Frank Ch. Eigler, Marco Benatto, Martin Cermak, Mikhail
Dmitrichenko*, Miro Hrončok*, Sv. Lockal*, proprietary and open-source AI
Special thanks to new contributors, marked with '*' above.
= Known issues with this release
- BTF `kernel.tracepoint` / `module().tracepoint` probes are not yet
available with `--runtime=bpf`.
- `STP_EXECTRACE` statement tracing is not yet available with `--bpf`.
- Legacy `systemtap-service` / `systemtap.service` are gone; migrate to
`stap@.service` / `staprun@.service` or `stap-onboot`.
= Bugs fixed for this release <https://sourceware.org/PR#####>
PR23360 replace sys_open boilerplate probe point with do_exit
PR30965 update SYNOPSIS section of the probe::* manpages
PR31683 add --enable-debug configury option
PR32105 remove bashisms from interactive-notebook/Makefile.am
PR32108 remove obsolete -Wno-implicit-function-declaration flag
PR32767 KFAIL classic kernel.trace("*") census misses
PR33037 add Python 3.13 tapset and fix py3execdir probing
PR33105 show systemtap version in generated docs
PR34126 add linux LSM hook support to the bpf runtime
PR34128 replace initscript with systemd service templates
PR34152 speed up the *syscall*.exp family
PR34153 high-performance tp_syscall() dispatcher; @entry() support
PR34161 optimize multiple probe handlers for the same probe point
PR34166 add -DSTP_EXECTRACE statement tracing
PR34169 add --debug option
PR34199 have @cast() operations see through typedefs
PR34214 fix autosprintf recursion and defer rlimit application
PR34283 clang build compatibility tweaks
PR34292 stop using deprecated imp in python helper
PR34293 modernize python extension build, remove python2 support
PR34303 fix listing_mode.exp buildbot timeouts
PR34318 runtime port for linux kernel 7.2-rc
PR34320 improve runtime error messages: active probe context
PR34414 short-circuit @defined ternaries during var expansion
PR34432 harden pass-2 parallelism: shared dwarf lock, STAP_NTHREADS
PR34475 probe javac against HelperSDT before enabling Java
PR34488 extract kernel build-ids from aarch64/s390x vmlinuz
PR34499 support @enumname() reverse enum mapping
PR34583 add XDP probe support to the bpf runtime
PR34632 fix BTF tracepoint discovery for void* and _tp-named events
Instagram Engineering
Shopify Engineering