4 ms·
As the kind-of-sort-of maintainer of Linux's syscall infrastructure on x86, I have a public service announcement: the syscall auditing infrastructure is awful.
by amluto 10y ago
As the kind-of-sort-of maintainer of Linux's syscall infrastructure on x86, I have a public service announcement: the syscall auditing infrastructure is awful.
It is inherently buggy in numerous ways. It hardcodes the number of arguments a syscall has incorrectly. It screws up compat handling. It doesn't robustly match entries to returns. It has an utterly broken approach to handling x32 syscalls. It has terrifying code that does bizarre things involving path names (!). It doesn't handle containerization sensibly at all. I wouldn't be at all surprised if it contains major root holes. And last, but certainly not least, it's eminently clear that no one stress tests it.
If you really want to use it for production, invest the effort to fix it, please. (And cc me.) Otherwise do yourself a favor and stay away from it. Use the syscall tracing infrastructure instead.
- viraptor 10y agoThanks for this information! I have a few questions though: > It doesn't robustly match entries to returns Do you mean it fails to match the call/return (annoying), or that it may mismatch them? (pretty bad...) > It doesn't handle containerization sensibly at all. Do you mean it's just oblivious to namespaces (but works as far as I know) or something actually not working?
- amluto 10y ago>> It doesn't robustly match entries to returns > Do you mean it fails to match the call/return (annoying), or that it may mismatch them? (pretty bad...) __audit_syscall_entry() and __audit_syscall_exit() have some interesting checks in them, and I was never convinced that every entry would get paired with the corresponding exit and logged correctly. >> It doesn't handle containerization sensibly at all. > Do you mean it's just oblivious to namespaces (but works as far as I know) or something actually not working? As far as I know, there is one global audit daemon and audit log, and you have to be globally privileged to use it.
- arunk-s 10y agoTrue that, I've been writing [libaudit-go](https://github.com/mozilla/libaudit-go/ https://github.com/mozilla/libaudit-go/) which aims to provide a replacement of the c version of auditd libraries and is in constant development. During this period I looked very closely in the auditd source code and I can say it looks like a bunch of things are patched together without much prior thought to make it work.
- base698 10y agoIs Syscall tracing == perf trace? Ive been doing the same sort of thing with perf. I hadn't heard of the auditing tools.
- amluto 10y agoYes, more or less. There are bunch of different "tracing" mechanisms in the kernel, and perf trace is the common way to use it. Syscalls trigger tracepoints, and anything that can see tracepoints can see them. Using eBPF to trace syscalls is probably quite useful.
- rhuber 10y agoHi - Ryan from the blog post here. eBPF is great, and we plan to support it as our collection mechanism, but not today and maybe not soon. When we wrote go-audit, there were few, (if any?), distros that had eBPF kernel support. Now Ubuntu 16.04 has support, as do many others, and that is great! But.. Releasing a tool that will work for most people today is very different to releasing one that will work for most people in a couple of years. We are sharing this imperfect tool that uses an using an imperfect syscall monitoring mechanism, both of which were written by imperfect humans. We have found plenty of problems with auditd during development (some real facepalm stuff, to be sure), but it also does many useful things. We have worked around limitations and very definitely stress tested it in our environment. A lot. A lot lot. If you don't trust auditd, that's a totally valid opinion to hold. I welcome criticism of our approach, but a PSA that doesn't involve having testing our tool is rather unfair.
- gbrown_ 10y ago> When we wrote go-audit, there were few, (if any?), distros that had eBPF kernel > support. Now Ubuntu 16.04 has support, as do many others, and that is great! This! I'm rather tired of people preaching about eBPF when in reality many of us have to run long term supported kernels/ distros like RedHat or Ubuntu LTS releases.
- sanxiyn 10y agoWhere did amluto "preach" eBPF? What is said is "If Slack really want to use syscall auditing for production, invest the effort to fix it, please. (And cc me.)", which seems totally reasonable to me. Slack's answer is "We have worked around limitations and very definitely stress tested it in our environment. A lot. A lot lot." As I understand, this means "No, Slack will not invest any effort to fix Linux syscall auditing in upstream kernel, because we have already worked around limitations." Which is kind of sad and expected.
- amluto 10y ago> If you don't trust auditd I know basically nothing about auditd. It's the kernel code I don't trust. Go-auditd may well be fantastic, but treating the kernel part as a reliable black box seems unwise to me. Edit: you might not need eBPF to get something better. Plain old "perf script" and the underlying ringbuffer API should work decently well on older kernels. There's a performance hit, but Steven Rostedt has a fix, and it should get backported to RHEL at least.
- hawski 10y agoOfftopic (and forgive me if I mess up some details): I was reading about eBPF and how it's instruction set is easily JITable across several architectures. I was wondering if it would be good as a VM for retro games. Something like Pico-8 [1], but where one could write games in this restricted C or eBPF-ASM. As I understand it the VM calls specific functions with limits of calling convention (10 64-bit registers shared between arguments, returns etc.). VM can also expose helper functions. It seems quite capable for such a purpose. Where could I find more materials to explore this side of eBPF? I found a user space eBPF VM [2] - probably a good start. EDIT: In case anyone is wondering: Why? For fun! [1] http://www.lexaloffle.com/pico-8.php http://www.lexaloffle.com/pico-8.php [2] https://github.com/rlane/ubpf https://github.com/rlane/ubpf
- amluto 10y agoIf nothing else, you'll need to rework the verifier. The time complexity of verification is super-linear right now. Awhile back I proposed a linear time algorithm that was a bit stricter, but it didn't get implemented. Also, from memory, there aren't real stack frames right now. (This is a limitation of the implementation and the verifier, not a fundamental issue.)
- coredog64 10y agoWe (unintentionally) stress test it. Our Linux admins vociferously insist using it to audit everything that is happening on a 120 core box that spawns tens of heavy processes every second.