7 ms·

Linux Performance

536 points by pablode 10y ago

arca_vorago 10y ago
This is awesome, I haven't seen all the perf tools so clearly laid out in usecase ever.
- SEJeff 10y ago
  You think this is awesome search YouTube for his talks and prepare to be blown away. Incredible engineer and funny guy.
  - nazri1 10y ago
    One thing that I learned from him and have never forgotten is that you can reduce a harddisk (the spindle kind) performance just by screaming at it https://www.youtube.com/watch?v=tDacjrSCeq4 https://www.youtube.com/watch?v=tDacjrSCeq4
    - pablode 10y ago
      This is awesome! Reminds me of how "IBM Tech Uses Hard Drives to Predict Earthquakes" [1]. And then it's just one guy screaming at the HDDs. [1] http://www.pcmag.com/article2/0,2817,2371337,00.asp http://www.pcmag.com/article2/0,2817,2371337,00.asp
    - sdiepend 10y ago
      When it happens in a datacenter: http://motherboard.vice.com/read/a-loud-sound-just-shut-down-a-banks-data-center-for-10-hours http://motherboard.vice.com/read/a-loud-sound-just-shut-down...
    - SEJeff 10y ago
      Yup that is what Brendan is famous/infamous for and it is pretty hilarious. His talks at conferences like say Monitorama are always equal parts amusing and deeply technical. I've been lucky enough to learn a ton from his work on pcc / perf to find a DTrace equiv on Linux.
whatupmd 10y ago
This site is a great resource and I hope Brendan keeps posting amazing content.
- digi_owl 10y ago
  It is interesting to see a former Solaris developer embrace Linux like this, when others seems to consider it an abomination.
  - boomboomsubban 10y ago
    It's a tremendous boon to Linux that it happened, it would have been very easy for him to transition into Illumos while with Joyent or FreeBSD at Netflix. Both already ran DTrace. He stuck with Linux and spent time making their performance tools competitive.
    - gghh 10y ago
      I love Brendan's writeups and scripts, and use them everyday multiple times a day, but he didn't "make Linux performance tools competitive". He's spreading and making available knowledge about perf, ftrace, eBPF that would otherwise remain relegated in Linux kernel development circles. He's also contributing to bcc (a tool in the eBPF ecosystem), but his main merit in my opinion is marketing, with his books and blog (I'm using the term in positive sense).
      - boomboomsubban 10y ago
        The lack of DTrace like tools has long been a criticism of Linux. The entire BPF team deserves a lot of credit for their work, but his decision to highlight this problem and help solve it rather than move to FreeBSD likely got it into 4.9 rather than some kernel in the future.
  - Thaxll 10y ago
    Well it's the #1 platform, not sure how you can avoid using it. And I don't see why it would be such "abomination". Linux is still very clean imo
digi_owl 10y ago
503, seems we torched the place...
- iso-8859-1 10y ago
  http://archive.is/fQDKB http://archive.is/fQDKB
  - saycheese 10y ago
    Please use non-tiny-URLs when possible: https://web.archive.org/web/20161206124221/http://www.brendangregg.com/linuxperf.html https://web.archive.org/web/20161206124221/http://www.brenda...
    - wfn 10y ago
      Archive.is is not related to web.archive.org, and the short URL is not a 'tiny' version of the latter. (That said, I agree with the general advice and think that tiny-fied URLs break the web.)
      - josteink 10y ago
        Shortened urls are gods way of giving you surprise goatse. Avoid. Always.
        iso-8859-1 10y ago
        As if non-short URL's can be trusted... https://archive.is/umsFk#http://disney.com https://archive.is/umsFk#http://disney.com Or all the phishing domains...
- mrsernine 10y ago
  Seems like posting to HN is a good way to trigger a DDOS
  - 10y ago
Sarkie 10y ago
https://web.archive.org/web/20161206124221/http://www.brendangregg.com/linuxperf.html https://web.archive.org/web/20161206124221/http://www.brenda...
smcl 10y ago
That "Linux Performance Observability Tools" diagram detailing the utils to inspect every part of the stack is excellent!
RodrigoT 10y ago
A couple months ago I was looking for some courses that could cover the new tools available on linux to measure performance: in kernel and user space. My main frustration is that there was no sign of any structured course that instruct on all these tools, and as importantly, when to and not to use each of them. Before you tell me again: no, youtube videos and blog posts are not good enough: I need to be able to sing up multiple members of different teams and be sure we all receive a homogeneous knowledge. Can Anyone share any pointers about where I could find this kind of training?
- homarp 10y ago
  say how much you're willing to pay for it, and I'm sure somehow will build the course for you.
- blakesterz 10y ago
  >> Before you tell me again: no, youtube videos and blog posts... I've been hunting for such a thing as well, though I'm guessing your hunt is for much higher quality than mine. I've not been able to find anything BUT based on what I've been watching and reading (yes, youtube and blogs) it wouldn't be much work to assemble a reading/viewing list from youtube/blogs and that might just work for you given that there might not be a perfect solution out there. The market for advanced high quality dedicated training on really advanced things is probably rather small (just my guess).
- abusque 10y ago
  I used to work at a consulting firm called EfficiOS[0], specialized in OS and application efficiency and performance. They maintain the LTTng[1] kernel and userspace tracers and related tools, and offer training. I'm not sure whether that responds to your specific needs, but if you get in touch they'll certainly be able to guide you. [0] http://www.efficios.com/ http://www.efficios.com/ [1] http://lttng.org/ http://lttng.org/
cimi_ 10y ago
I recommend Brendan's book on Systems Performance[1], describes a structured approach to solving performance problems and is a good reference for various types of resource performance profiles. [1] http://www.brendangregg.com/sysperfbook.html http://www.brendangregg.com/sysperfbook.html
- thesmallestcat 10y ago
  Has anybody else found this book a little too high-level? I've had a hard time getting useful, practical advice out of it but haven't read it cover-to-cover yet. For instance there's a bit about the impacts of using a GC'd language, but there's nothing new, just common knowledge. Does anybody know of a lower-level/more advanced resource like this book?
  - CaptSpify 10y ago
    IMHO: I use it more as a reference book. "Oh, I'm having I/O problems, let me look up the sections on Filesystems and Disks" It is very dry, and hard to read from cover to cover, but it is filled with a mountain of info. I unfortunately don't know of a more introductory book to recommend, but would also be curious
    - effie 10y ago
      The parent asked for more advanced book, not for more introductory book.
      - CaptSpify 10y ago
        You are correct. <insert joke about me needing more coffee here>
  - brendangregg 10y ago
    I wanted to give an overview & awareness of all performance areas, not just the focus of the book (systems). So topics like runtimes (and GC), queueing theory, etc, are summarized in less than a page in a 600+ page book. A big problem with performance is gaps in people's knowledge (unknown unknowns -- which I referenced in the intro), so I wanted to at least create awareness of them (to become known unknowns). But I bet that's frustrating if I've hit an area you want to know a lot more about. Areas like systems performance methodology I've written a lot of new material for in the book in more depth.
shakencrew 10y ago
Previous discussion on Hacker News: https://news.ycombinator.com/item?id=8205057 https://news.ycombinator.com/item?id=8205057
billsmithaustin 10y ago
His site is a great information source. I used some of his work to diagnose a JVM performance problem (discussed in https://news.ycombinator.com/item?id=12505517 https://news.ycombinator.com/item?id=12505517).

Linux Performance

536 points by pablode 10y ago

arca_vorago 10y ago

This is awesome, I haven't seen all the perf tools so clearly laid out in usecase ever.

SEJeff 10y ago
You think this is awesome search YouTube for his talks and prepare to be blown away. Incredible engineer and funny guy.
- nazri1 10y ago
  One thing that I learned from him and have never forgotten is that you can reduce a harddisk (the spindle kind) performance just by screaming at it https://www.youtube.com/watch?v=tDacjrSCeq4 https://www.youtube.com/watch?v=tDacjrSCeq4
  - pablode 10y ago
    This is awesome! Reminds me of how "IBM Tech Uses Hard Drives to Predict Earthquakes" [1]. And then it's just one guy screaming at the HDDs. [1] http://www.pcmag.com/article2/0,2817,2371337,00.asp http://www.pcmag.com/article2/0,2817,2371337,00.asp
  - sdiepend 10y ago
    When it happens in a datacenter: http://motherboard.vice.com/read/a-loud-sound-just-shut-down-a-banks-data-center-for-10-hours http://motherboard.vice.com/read/a-loud-sound-just-shut-down...
  - SEJeff 10y ago
    Yup that is what Brendan is famous/infamous for and it is pretty hilarious. His talks at conferences like say Monitorama are always equal parts amusing and deeply technical. I've been lucky enough to learn a ton from his work on pcc / perf to find a DTrace equiv on Linux.

whatupmd 10y ago

This site is a great resource and I hope Brendan keeps posting amazing content.

digi_owl 10y ago
It is interesting to see a former Solaris developer embrace Linux like this, when others seems to consider it an abomination.
- boomboomsubban 10y ago
  It's a tremendous boon to Linux that it happened, it would have been very easy for him to transition into Illumos while with Joyent or FreeBSD at Netflix. Both already ran DTrace. He stuck with Linux and spent time making their performance tools competitive.
  - gghh 10y ago
    I love Brendan's writeups and scripts, and use them everyday multiple times a day, but he didn't "make Linux performance tools competitive". He's spreading and making available knowledge about perf, ftrace, eBPF that would otherwise remain relegated in Linux kernel development circles. He's also contributing to bcc (a tool in the eBPF ecosystem), but his main merit in my opinion is marketing, with his books and blog (I'm using the term in positive sense).
    - boomboomsubban 10y ago
      The lack of DTrace like tools has long been a criticism of Linux. The entire BPF team deserves a lot of credit for their work, but his decision to highlight this problem and help solve it rather than move to FreeBSD likely got it into 4.9 rather than some kernel in the future.
- Thaxll 10y ago
  Well it's the #1 platform, not sure how you can avoid using it. And I don't see why it would be such "abomination". Linux is still very clean imo

digi_owl 10y ago

503, seems we torched the place...

iso-8859-1 10y ago
http://archive.is/fQDKB http://archive.is/fQDKB
- saycheese 10y ago
  Please use non-tiny-URLs when possible: https://web.archive.org/web/20161206124221/http://www.brendangregg.com/linuxperf.html https://web.archive.org/web/20161206124221/http://www.brenda...
  - wfn 10y ago
    Archive.is is not related to web.archive.org, and the short URL is not a 'tiny' version of the latter. (That said, I agree with the general advice and think that tiny-fied URLs break the web.)
    - josteink 10y ago
      Shortened urls are gods way of giving you surprise goatse. Avoid. Always.
      - iso-8859-1 10y ago
        As if non-short URL's can be trusted... https://archive.is/umsFk#http://disney.com https://archive.is/umsFk#http://disney.com Or all the phishing domains...
mrsernine 10y ago
Seems like posting to HN is a good way to trigger a DDOS
- 10y ago

Sarkie 10y ago

https://web.archive.org/web/20161206124221/http://www.brendangregg.com/linuxperf.html https://web.archive.org/web/20161206124221/http://www.brenda...

smcl 10y ago

That "Linux Performance Observability Tools" diagram detailing the utils to inspect every part of the stack is excellent!

RodrigoT 10y ago

A couple months ago I was looking for some courses that could cover the new tools available on linux to measure performance: in kernel and user space. My main frustration is that there was no sign of any structured course that instruct on all these tools, and as importantly, when to and not to use each of them. Before you tell me again: no, youtube videos and blog posts are not good enough: I need to be able to sing up multiple members of different teams and be sure we all receive a homogeneous knowledge. Can Anyone share any pointers about where I could find this kind of training?

homarp 10y ago
say how much you're willing to pay for it, and I'm sure somehow will build the course for you.
blakesterz 10y ago
>> Before you tell me again: no, youtube videos and blog posts... I've been hunting for such a thing as well, though I'm guessing your hunt is for much higher quality than mine. I've not been able to find anything BUT based on what I've been watching and reading (yes, youtube and blogs) it wouldn't be much work to assemble a reading/viewing list from youtube/blogs and that might just work for you given that there might not be a perfect solution out there. The market for advanced high quality dedicated training on really advanced things is probably rather small (just my guess).
abusque 10y ago
I used to work at a consulting firm called EfficiOS[0], specialized in OS and application efficiency and performance. They maintain the LTTng[1] kernel and userspace tracers and related tools, and offer training. I'm not sure whether that responds to your specific needs, but if you get in touch they'll certainly be able to guide you. [0] http://www.efficios.com/ http://www.efficios.com/ [1] http://lttng.org/ http://lttng.org/

cimi_ 10y ago

I recommend Brendan's book on Systems Performance[1], describes a structured approach to solving performance problems and is a good reference for various types of resource performance profiles. [1] http://www.brendangregg.com/sysperfbook.html http://www.brendangregg.com/sysperfbook.html

thesmallestcat 10y ago
Has anybody else found this book a little too high-level? I've had a hard time getting useful, practical advice out of it but haven't read it cover-to-cover yet. For instance there's a bit about the impacts of using a GC'd language, but there's nothing new, just common knowledge. Does anybody know of a lower-level/more advanced resource like this book?
- CaptSpify 10y ago
  IMHO: I use it more as a reference book. "Oh, I'm having I/O problems, let me look up the sections on Filesystems and Disks" It is very dry, and hard to read from cover to cover, but it is filled with a mountain of info. I unfortunately don't know of a more introductory book to recommend, but would also be curious
  - effie 10y ago
    The parent asked for more advanced book, not for more introductory book.
    - CaptSpify 10y ago
      You are correct. <insert joke about me needing more coffee here>
- brendangregg 10y ago
  I wanted to give an overview & awareness of all performance areas, not just the focus of the book (systems). So topics like runtimes (and GC), queueing theory, etc, are summarized in less than a page in a 600+ page book. A big problem with performance is gaps in people's knowledge (unknown unknowns -- which I referenced in the intro), so I wanted to at least create awareness of them (to become known unknowns). But I bet that's frustrating if I've hit an area you want to know a lot more about. Areas like systems performance methodology I've written a lot of new material for in the book in more depth.

shakencrew 10y ago

Previous discussion on Hacker News: https://news.ycombinator.com/item?id=8205057 https://news.ycombinator.com/item?id=8205057

billsmithaustin 10y ago

His site is a great information source. I used some of his work to diagnose a JVM performance problem (discussed in https://news.ycombinator.com/item?id=12505517 https://news.ycombinator.com/item?id=12505517).