5 ms·
What's the advantage of monitoring from the kernel layer? It's not jumping out at me...
by throwaway894345 6y ago
What's the advantage of monitoring from the kernel layer? It's not jumping out at me...
- SEJeff 6y agoTracing from user to kernel back to userspace in one single pane of glass. Have you never heard anyone hype about DTrace on Solaris? This allows building similar things on production systems with little to no impact on running production applications.
- suthakamal 6y agohadn't thought of this like DTrace. Thx for that!
- SEJeff 6y agoAnytime! eBPF is Linux’s equivalent of DTrace
- cyphar 6y agoStrictly speaking, one of the uses of eBPF is a DTrace-like tracing tool but it's also used in quite a few other places (for instance it's also used for devices cgroup policy in cgroupv2).
- SEJeff 6y agoAnd filtering syscalls for seccomp or literally packet filtering. I believe Jens Axboe and team were looking at using eBPF for some of the low level IO subsystems. But I was explaining the most common and obvious (for a user) use of eBPF. They should just look at the io visor project to see some of the stuff that can be done with it (disclaimer, I work with one of the io visor maintainers)
- 0xEFF 6y agodtrace has also been on macOS since 10.5, with a pretty nice GUI app as well. I’ve used it to trace ruby and python code to isolate slow API requests from the end user’s perspective.
- colechristensen 6y agoThat sounds really interesting, do you have any references to share about macOS gui dtrace?
- SEJeff 6y agoThe app is called Instruments. Check it out at /Developer/Applications/Instruments https://developer.apple.com/library/archive/documentation/AnalysisTools/Conceptual/instruments_help-collection/Chapter/Chapter.html https://developer.apple.com/library/archive/documentation/An...
- seneca 6y agoThe kernel has a "god's eye view" of everything happening on its OS. With event based tracing in the kernel there's no chance of missing an occurrence because your sampling rate is too low, for example. You can also correlate and enrich data that just isn't available in userspace.
- justicezyx 6y agoA few things: * Non intrusive: meaning one can snoop info of application without changing application code. * Deep visibility: function level and syscall/kernel functions reveal more context and are more accurate in a lot of cases. * Low overhead: everything runs inside kernel space, no context switching compared to other kernel based/aided tracing. * Expressiveness: eBPF is fairly expressive, can do many things that usually are exclusive to high level programming languages.
- deleted 6y ago[deleted]
- lbotos 6y agoAt GitLab we've built and used strace-parser to help us get to quick and deep debugging, but running strace is EXPENSIVE: https://gitlab.com/gitlab-com/support/toolbox/strace-parser https://gitlab.com/gitlab-com/support/toolbox/strace-parser We've been eagerly awaiting some customers to adopt newer kernels so we can start leveraging eBPF because of the performance gains in these type of scenarios. Getting down the the kernel often can help find problems with disk access or network issues. One of our staff engineers is exploring it now for NFS stats: https://gitlab.com/wchandler/tracing-tools/-/blob/master/nfsqueue https://gitlab.com/wchandler/tracing-tools/-/blob/master/nfs... In Support Engineering we often straddle the line of 'SRE style stare at graphs and configuration as code' and 'log on to the box and look at syscalls'. We are very very excited about eBPF.
- justicezyx 6y agoTake a look at Pixie dynamic logging [1]. Deploys eBPF kprobes (based on bpftrace) and uprobe (based a custom front-end language) and instantly get rich data (arts, return value, latency), query data in a Pandas-like scripting language and visual dashboard. [1] https://docs.pixielabs.ai/using-pixie/code-tracing/ https://docs.pixielabs.ai/using-pixie/code-tracing/
- boston_sre87 6y agoIt's extremely quick to do it there as well. Originally, bpf was created to replace tcpdump with a quicker less impactful alternative. People saw that it was a pretty neat alternative so they started extending it (the e in eBPF). Think they might have finally come round again and just call it BPF now again.