4 ms·

Relevant read from TrailOfBits on the pitfalls of using eBPF for security monitoring: https://blog.trailofbits.com/2023/09/25/pitfalls-of-relying-on-ebpf-for-se

Relevant read from TrailOfBits on the pitfalls of using eBPF for security monitoring: https://blog.trailofbits.com/2023/09/25/pitfalls-of-relying-on-ebpf-for-security-monitoring-and-some-solutions/ https://blog.trailofbits.com/2023/09/25/pitfalls-of-relying-... (Pitfalls of relying on eBPF for security monitoring (and some solutions)) > eBPF is a powerful tool for Linux observability and monitoring, but it was not designed for security and comes with inherent limitations. Developers need to be aware of pitfalls like probe unreliability, data truncation, instruction limits, concurrency issues, event overload, and page faults. Workarounds exist, but they are imperfect and often add complexity. Maybe eBPF tooling should come with warning labels on what pitfalls are accounted for.

insanitybit 3y ago
Some of these apply to the audit framework as well.
happy-dude 3y ago
While the TrailOfBits post is informative, they extrapolate the wrong premise from Brendan Gregg's original post[1]. Brendan's statement was originally about the particular set of eBPF tools created for observability (especially his own). These tools (like bcc and bpftrace) are specifically geared towards observability and were not built for security in mind, nor will they be "patched" or fixed for these purposes. Does this imply that eBPF is a bad foundation for security-focused tooling? Brendan states the following at the close of this blog post: > There is potential for an awesome eBPF security product, and it's not just the visibility that's valuable (all those arrows) it's also the low overhead. These slides included our overhead evaluation showing bcc/eBPF was far more efficient than auditd or go-audit. (It was pioneering work, but unfortunately the slides are all we have: Alex, I, and others left Netflix before open sourcing it.) There are now other eBPF security products, including open source projects (e.g., tetragon), but I don't know enough about them all to have a recommendation. > Note that I'm talking about the observability tools here and not the eBPF kernel runtime itself, which has been designed as a secure sandbox. Nor am I talking about privilege escalation, since to run the tools you already need root access (that car has sailed!). This reads to me that security tooling built on top of eBPF is possible and various organizations are in-flight making it happen (such as Falco[2], Tetragon[3], and Tracee[4]). These teams have recognized the shortcomings of eBPF and are layering other kernel-instrumentation capabilities such as kprobes and LSM hooks into their solutions. Additionally, the TrailOfBits blog post states: > Developers need to be aware of pitfalls like probe unreliability, data truncation, instruction limits, concurrency issues, event overload, and page faults. Workarounds exist, but they are imperfect and often add complexity. These inherent limitations exist primarily because eBPF is a virtual machine within kernel space. Many of these constraints exist because eBPF programs should _never_ lock up the kernel. The eBPF verifier[5] does some checks on the possible code paths the program can take, such as finite bounded loops, null checks on variables, etc. The foundational aspect here is that the eBPF virtual machine is designed to protect the kernel while running programs in kernel space, and that imperfect/complex workarounds may be needed by security-focused projects to respect that foundation. [1] https://www.brendangregg.com/blog/2023-04-28/ebpf-security-issues.html https://www.brendangregg.com/blog/2023-04-28/ebpf-security-i... [2] https://github.com/falcosecurity/falco/ https://github.com/falcosecurity/falco/ [3] https:/github.com/cilium/tetragon [4] https://github.com/aquasecurity/tracee https://github.com/aquasecurity/tracee [5] https://docs.kernel.org/bpf/verifier.html https://docs.kernel.org/bpf/verifier.html
tptacek 3y ago
This is a great post, but there aren't non-problematic alternatives to eBPF for most of these pitfalls. If you're building a reference monitor out of tricky eBPF primitives --- if you're relying on eBPF to somehow make go/no-go decisions about system usage --- those pitfalls are super-problematic, the same way seccomp filters are. But if you're just generating events for monitoring... well, it's good to keep in mind the limitations of all monitoring systems.
lcvw 3y ago
As someone who uses bpf for security monitoring every day, every one of those problems is a has a workaround. And the only real competitor is writing a kernel module which has serious safety issues.
tyingq 3y ago
None of that sounds worse than commercial offerings in the space.

insanitybit 3y ago

Some of these apply to the audit framework as well.

happy-dude 3y ago

While the TrailOfBits post is informative, they extrapolate the wrong premise from Brendan Gregg's original post[1]. Brendan's statement was originally about the particular set of eBPF tools created for observability (especially his own). These tools (like bcc and bpftrace) are specifically geared towards observability and were not built for security in mind, nor will they be "patched" or fixed for these purposes. Does this imply that eBPF is a bad foundation for security-focused tooling? Brendan states the following at the close of this blog post: > There is potential for an awesome eBPF security product, and it's not just the visibility that's valuable (all those arrows) it's also the low overhead. These slides included our overhead evaluation showing bcc/eBPF was far more efficient than auditd or go-audit. (It was pioneering work, but unfortunately the slides are all we have: Alex, I, and others left Netflix before open sourcing it.) There are now other eBPF security products, including open source projects (e.g., tetragon), but I don't know enough about them all to have a recommendation. > Note that I'm talking about the observability tools here and not the eBPF kernel runtime itself, which has been designed as a secure sandbox. Nor am I talking about privilege escalation, since to run the tools you already need root access (that car has sailed!). This reads to me that security tooling built on top of eBPF is possible and various organizations are in-flight making it happen (such as Falco[2], Tetragon[3], and Tracee[4]). These teams have recognized the shortcomings of eBPF and are layering other kernel-instrumentation capabilities such as kprobes and LSM hooks into their solutions. Additionally, the TrailOfBits blog post states: > Developers need to be aware of pitfalls like probe unreliability, data truncation, instruction limits, concurrency issues, event overload, and page faults. Workarounds exist, but they are imperfect and often add complexity. These inherent limitations exist primarily because eBPF is a virtual machine within kernel space. Many of these constraints exist because eBPF programs should _never_ lock up the kernel. The eBPF verifier[5] does some checks on the possible code paths the program can take, such as finite bounded loops, null checks on variables, etc. The foundational aspect here is that the eBPF virtual machine is designed to protect the kernel while running programs in kernel space, and that imperfect/complex workarounds may be needed by security-focused projects to respect that foundation. [1] https://www.brendangregg.com/blog/2023-04-28/ebpf-security-issues.html https://www.brendangregg.com/blog/2023-04-28/ebpf-security-i... [2] https://github.com/falcosecurity/falco/ https://github.com/falcosecurity/falco/ [3] https:/github.com/cilium/tetragon [4] https://github.com/aquasecurity/tracee https://github.com/aquasecurity/tracee [5] https://docs.kernel.org/bpf/verifier.html https://docs.kernel.org/bpf/verifier.html

tptacek 3y ago

This is a great post, but there aren't non-problematic alternatives to eBPF for most of these pitfalls. If you're building a reference monitor out of tricky eBPF primitives --- if you're relying on eBPF to somehow make go/no-go decisions about system usage --- those pitfalls are super-problematic, the same way seccomp filters are. But if you're just generating events for monitoring... well, it's good to keep in mind the limitations of all monitoring systems.

lcvw 3y ago

As someone who uses bpf for security monitoring every day, every one of those problems is a has a workaround. And the only real competitor is writing a kernel module which has serious safety issues.

tyingq 3y ago

None of that sounds worse than commercial offerings in the space.