4 ms·
'Linux might get DTrace-like capabilities via SystemTap' Might? SystemTap's been stable & out of the box on RHEL 5 for years. The extensive list of default tap
by nailer 13y ago
'Linux might get DTrace-like capabilities via SystemTap'
Might? SystemTap's been stable & out of the box on RHEL 5 for years. The extensive list of default tap points are supported on production by RH. When I was doing systems stuff 5 years ago we used it for double checking TCP socket options that weren't normally exposed to userspace tools.
Dtrace can do some awesome stuff stap can't, like trace something from userspace down to kernel space. But systemtap does provide live instrumentation on running kernels and has for years.
PS: Just saw p93. Sun got in trouble for the obvious "Setting up SystemTap is difficult on distros that don't maintain, support, use or like SystemTap" years ago. This is repeated on p 93. Stop it, this makes Joyent look stupid, you're better than this.
And p100. Re: 'several million dollar E10K' performance? Red Hat were replacing them with Xeon's by 2003 (with Sun's old sales staff) and crushing them. Though that was SPARC's fault not Solaris's.
- bcantrill 13y ago(Disclosure: I work at Joyent.) Are you accusing Brendan of misrepresenting SystemTap on slide 93? If so, could you be specific as to what that misrepresentation is? Brendan has a ton of experience earnestly using SystemTap (or rather, trying to use it), and it's hard to see how sharing that experience makes either Brendan or his employer "look stupid." For more details on Brendan's explorations of SystemTap: http://dtrace.org/blogs/brendan/2011/10/15/using-systemtap/ That's several years old at this point, but the architectural limitations highlighted by Brendan haven't changed.
- nailer 13y agoYes, I am accusing the author of deliberately misrepresenting facts. I'm not sure what's vague, let me recap: 1. 'might' implies the future. As you can tell from the post you're replying to, Systemtap is present thing. I think I actually put most of the tapsets shown on the Wikipedia page around 2009, and I was using them then. 2. Also, as the author partially acknowledges in the next slide, Ubuntu is the dumbest place to illustrate a typical Systemtap experience. I suspect that's deliberate, rather than ignorance as Sun tried that a few years back and I don't think anyone with any interest in the matter is ignorant enough not to remember the shitstorm. As I've written elsewhere: if your customers mostly run Ubuntu, then test dtrace on Ubuntu too. It'd give you just as accurate impression about what the dtrace 'experience' is as installing SystemTap on Ubuntu. Edit: actually looks like Joyent wrote the shitstorm post for a few years ago, you've still got it up, you still don't see anything wrong, and you just linked to it. christ read the HN comments from 2011. https://news.ycombinator.com/item?id=3118416 https://news.ycombinator.com/item?id=3118416 Edit 2: you're Bryan Cantrill. You didn't seem to care much about performance a while ago: http://cryptnet.net/mirrors/texts/kissedagirl.html http://cryptnet.net/mirrors/texts/kissedagirl.html Edit 3: more Cantrill classiness http://www.quora.com/Node-js/Why-did-Ben-Noordhuis-decide-to-leave-Node-js-development http://www.quora.com/Node-js/Why-did-Ben-Noordhuis-decide-to... Sorry to anyone who thinks I've gotten personal, but this guy is a well known... unsavoury character.
- brendangregg 13y agoDemonstrating on Ubuntu was indeed deliberate -- it's because most of my customers, and most of the performance issues I'm debugging, are on Ubuntu.
- nailer 13y agoFor the sake of consistency, then, I suggest you install dtrace on Ubuntu.
- _delirium 13y agoI thought that was sort of his point, that you can't? I didn't read it as a DTrace vs. Systemtap discussion, but as an observability-on-SmartOS versus observability-on-Ubuntu discussion. If Ubuntu shipped with DTrace (or it could be easily installed), there'd be no real observability difference. It's a good point that this is a SmartOS vs. Ubuntu rather than SmartOS vs. all-Linuxes comparison, though.
- nailer 13y agoRunning dtrace on Ubuntu is possible: much like SystemTap, it's unsupported and involves lots of messing around. If someone wrote an article called 'Using dtrace' and wrote about the poor experience they had on Ubuntu it'd be just as (in)valid.
- brendangregg 13y agoNot really; DTrace has never been advertised as a tool for Linux. I've talked about the in-development ports, and that they aren't complete, and that they can panic/freeze systems. OTOH, SystemTap is advertised as tracing for Linux, and that it's safe for production use. My customers are running Linux, usually Ubuntu, and so I tried using SystemTap. I wouldn't have written that blog post if the SystemTap homepage simply said: "SystemTap is a tracing tool for RHEL. There are ports in progress for Ubuntu and others, which may panic or freeze systems." I disagree with the logic for not warning customers of this, and I think it's bad for Red Hat. If the homepage had stated that it's a RHEL tool, those frustrated users would have known to try RHEL, instead of giving up entirely.
- the_mitsuhiko 13y ago> Stop it, this makes Joyent look stupid, you're better than this. How does it make joyent look stupid? Dtrace is miles ahead of Systemtap, especially when it comes to installation because dtrace is generally embraced where it's being used as a default installation which systemtap is not and I don't have to make a monkey dance if I upgrade the kernel. On a system with dtrace like OS X tracing tools like dtruss are basically just wrappers around dtrace. It's very natural to the platform.
- brendangregg 13y agoIt would be great if all Linux distributions had SystemTap out-of-the-box like RHEL. Or something with a sufficient feature set to solve problems (which may become ktap, or perf_events, or lttng). However, most of the Linux performance issues I'm debugging are on Ubuntu systems. So the question remains: when will Linux-at-large get this capability? Red Hat can certainly say that SystemTap is obviously a priority on their OS, and they can only make best effort for others. I said this in the talk. Indeed, a working SystemTap should be an incentive to run RHEL. It is a compelling argument -- the benefits of a working SystemTap are enormous (which I also explained in the talk). Sun made basically the same argument for switching to Solaris: a working DTrace. Although, I'm not sure that using SystemTap is entirely safe for production yet, which I said was the most important priority. I notice that this related kernel panic is still not fixed: http://sourceware.org/bugzilla/show_bug.cgi?id=2725 http://sourceware.org/bugzilla/show_bug.cgi?id=2725 Although some recent and promising progress has happened in the past few months, as noted by the bug! I only care about this bug because I sometimes trace large subsets of the kernel (eg, all functions in a module or driver), and I think I've hit the same related issue. I don't actually need to trace everything. You said that DTrace can trace from userspace to kernel, which SystemTap can't. I'm not sure if there's a specific case you're referring to, but SystemTap nowadays can indeed trace userspace to kernel (via uprobes), and a lot of work has been happening to get SystemTap userspace tapsets to work. As for setting up SystemTap being difficult on distros that don't support or maintain SystemTap. Well, almost all of my customers run Ubuntu. If they hit perf issues, do I convince them to switch to RHEL? (The approach that Sun made for DTrace.) Or is this a problem to be taking up with the other Linux distro maintainers? Which goes back to my original point: when will Linux get this? :-)
- ema_rocca 13y agoIs it problematic to install SystemTap on Ubuntu? On Debian systems it's straightforward. apt-get install systemtap linux-image-amd64-dbg
- _delirium 13y agoIt's not too much harder, but requires a non-default respository, so mildly harder. Quasi-official instructions here: https://wiki.ubuntu.com/Kernel/Systemtap https://wiki.ubuntu.com/Kernel/Systemtap The difference is that, as of the past few releases, Ubuntu doesn't ship a -dbg kernel package in the main repository like Debian does. Instead they've come up with a new system to ship 'debug symbol' variants of all packages, with a -dbgsym suffix. But they ship them in a separate repository that isn't enabled by default, I guess to avoid confusing end-users (ddebs.ubuntu.com). So you need to configure that repository, install its key, then pull linux-image-$(uname -r)-dbgsym. I guess you might also want to pull the debug-symbol versions of the main libraries as well.
- rodgerd 13y ago> And p100. Re: 'several million dollar E10K' performance? Red Hat were replacing them with Xeon's by 2003 (with Sun's old sales staff) and crushing them. Though that was SPARC's fault not Solaris's Possibly I missed the golden era of Sun field support, or perhaps it was more a regional thing, but I consistently get vastly superior support from Red Hat than I've ever had from Sun, even when running M-class Sun kit. And let us not speak of Oracle.
- khc 13y agoI've played with systemtap around 2011/2012, with RHEL 5 and versions of Fedora at the time, and I could easily get it to kernel panic. I went on to their irc channel to report the problems and got some fixes, but in the end it still wasn't stable and I gave up due to lack of time. I really wanted to get it working. One of my colleagues (who was from Sun) actually left and went to work for Joyent, presumably at least partly because of the frustration with the lack of a dtrace like tool.