5 ms·

Linux Performance

186 points by Walkman 12y ago

nisa 12y ago
I'm just a lowly student assistant that deals with a lot of Linux machines running Hadoop but I'm totally in love with these presentations from Joyent and the SmartOS guys. I seriously considered moving to SmartOS as ZFS and Zones and likely Dtrace as these features that would make my job, that largely comes down to organizing running software on machines and debugging problems, far easier..and would allow me to use the machines to better degree but in reality it's not going to happen. Nobody is familiar with Solaris userland. I'm not a sophisticated and educated systems engineer at Joyent I'm just a stressed guy trying to fix problems. Unfortunatly Linux is pretty good at making it work because a lot people are in a similar situation and someone will fix it for me. I just don't have the time and knowledge and energy to e.g. fix native Hadoop libraries in the ecosystem to build with another libc or make my own or other applications able to run without some Linux specific crap.. That beeing said I really thought about pushing SmartOS/Solaris but as a lone fighter It would be suicide in a world where everyone knows apt-get install <whatever> and get his shit done in a reasonable way.. Maybe it's something for specialised application and not academia I've came pretty far with just strace and perf top and most problems I had in my own application where better analyzed by valgrind and kcachegrind or massif and the visualizer...
- adamnemecek 12y ago
  You might be aware of this but FreeBSD has all of those features that you mention. The objections that you have against Solaris might still apply though.
  - nisa 12y ago
    Yes. I'm running ZFS on Linux and while ZFS is really great it's not really good integrated in the kernel and sometimes pretty unstable at least in my rather esoteric scenario... Despite other claims FreeBSD suffers similar problems.. (https://clusterhq.com/blog/complexity-freebsd-vfs-using-zfs-example-part-1-2/ https://clusterhq.com/blog/complexity-freebsd-vfs-using-zfs-...). Other problems are that jails are fine but there is no disk I/O limitation possible... I've also thought about FreeBSD and while pkgng is really great it's a similar problem.. I'm stumbling upon bugs or untested things and I'm unable to contribute time to fixing them.
    - tachion 12y ago
      Do you mind pointing to your PR's with the bugs you've found or at least mention what they were? It sound like you've found a hell of a bugs/problems in a system (and I am thinking about FreeBSD now) that me and huge number of other people are running without any issues, so it would be beneficial for everyone, if you'd share your problems with PR's - there is active community around it that can fix issues if you are unable to do it.
      - nisa 12y ago
        Sorry if I was unclear. I stumbled upon a few issues running ZFS on Linux that are known and on the development roadmap. Things like ARC integration and better failure handling in case of disk problems. I don't run anything big on FreeBSD and ZFS. I have not experienced problems on a raidz2 ZFS fileserver that runs FreeBSD except that disks drop out quite randomly but I've yet been unable to pinpoint that and it's likely that these are hardware issues as the system runs on budget hardware. Sorry if my comment spreaded FUD.
        ryao 12y ago
        This might be of use to you: http://open-zfs.org/w/index.php?title=Hardware#Error_recovery_control http://open-zfs.org/w/index.php?title=Hardware#Error_recover...
    - XorNot 12y ago
      The biggest problem I've had with ZFS on Linux is that the arc_cache won't back off from memory quick enough. There's still an edge case in their somewhere where it hits swap and effectively locks up (its still running, just amazingly slowly). The fix is to set the arc_limit low (like 1GB) - it's not a hard limit, but it stops it eating all the RAM on my 8gb ram box.
      - ryao 12y ago
        Swap on zvols is not fully stable on Linux yet. I have patches in the works for 0.6.4 that should improve things.
  - evandena 12y ago
    How's KVM on FreeBSD these days?
    - ryao 12y ago
      The port was abandoned as no one was willing to finish it. FreeBSD developers have since written their own hypervisor called byhve.
- shmerl 12y ago
  Sorry for off-topic. Do you have any opinion on HPCC? It seems to be aiming at the same goals as Hadoop but claims to be able to do more and have better performance. I tried to find more first hand experience reports, but it's rather obscure and not used that widely.
- bcantrill 12y ago
  You might be interested in our ongoing work to execute Linux binaries in a SmartOS zone.[1] It's still an area of very active development (one needs to unfortunately follow the source[2] to follow its progress), but we have a ton working -- and it's all being done in the open. (And it's certainly very nice to be able to "apt-get install" something and then be able to DTrace the resulting application!) We're close to having enough working to be able to document where we are and get others kicking the tires, so stay tuned to the SmartOS discussion list if this is something you're interested in! [1] https://www.mail-archive.com/smartos-discuss@lists.smartos.org/msg00475.html https://www.mail-archive.com/smartos-discuss@lists.smartos.o... [2] https://github.com/joyent/illumos-joyent https://github.com/joyent/illumos-joyent
  - nisa 12y ago
    This looks great!
- donavanm 12y ago
  It's no dtrace, but check out systemtap "stap". It provides a lot of similar functionality on modern linux. It can actually use existing dtrace probe points if you have a binary like mysqld. The ability to follow call graphs, and insert arbitrary logic, is amazing for chasing down system issues.
wmf 12y ago
It looks like this page was recently updated and coincidentally today he gave a talk on Linux performance at LinuxCon: http://events.linuxfoundation.org/events/linuxcon-north-america/program/schedule http://events.linuxfoundation.org/events/linuxcon-north-amer... http://events.linuxfoundation.org/sites/events/files/slides/LinuxPerfTools_0.pdf http://events.linuxfoundation.org/sites/events/files/slides/...
- brendangregg 12y ago
  Yes, I gave the talk this morning. I hope people found it useful! In case slideshare is quicker to load, the slides are also at: http://www.slideshare.net/brendangregg/linux-performance-tools http://www.slideshare.net/brendangregg/linux-performance-too...
  - jaryd 12y ago
    Do you expect video of the presentation to be available any time in the near future?
    - brendangregg 12y ago
      Sorry, like other talks at LinuxCon, it was not videoed. I think it would be a useful to have a video of it, so, much as I hate to give the same talk twice, I'll probably do it again at some point for the video.
      - rodgerd 12y ago
        Don't tell me you didn't submit for linux.conf.au next year...
josephyu0305 12y ago
Nice link it show lot of Linux performance help when it comes the system crash/ and very much learning i gotfrom his presentation.
fasteo 12y ago
I would recommend his book [1] to anyone interested in systems performance. What really caught my attention is the focus he puts in having a goal and applying a method to solve performance issues. Many times, I have found myself "lost" while isolating a performance issue. Not anymore. [1] http://www.brendangregg.com/sysperfbook.html http://www.brendangregg.com/sysperfbook.html

186 points by Walkman 12y ago

nisa 12y ago

I'm just a lowly student assistant that deals with a lot of Linux machines running Hadoop but I'm totally in love with these presentations from Joyent and the SmartOS guys. I seriously considered moving to SmartOS as ZFS and Zones and likely Dtrace as these features that would make my job, that largely comes down to organizing running software on machines and debugging problems, far easier..and would allow me to use the machines to better degree but in reality it's not going to happen. Nobody is familiar with Solaris userland. I'm not a sophisticated and educated systems engineer at Joyent I'm just a stressed guy trying to fix problems. Unfortunatly Linux is pretty good at making it work because a lot people are in a similar situation and someone will fix it for me. I just don't have the time and knowledge and energy to e.g. fix native Hadoop libraries in the ecosystem to build with another libc or make my own or other applications able to run without some Linux specific crap.. That beeing said I really thought about pushing SmartOS/Solaris but as a lone fighter It would be suicide in a world where everyone knows apt-get install <whatever> and get his shit done in a reasonable way.. Maybe it's something for specialised application and not academia I've came pretty far with just strace and perf top and most problems I had in my own application where better analyzed by valgrind and kcachegrind or massif and the visualizer...

adamnemecek 12y ago
You might be aware of this but FreeBSD has all of those features that you mention. The objections that you have against Solaris might still apply though.
- nisa 12y ago
  Yes. I'm running ZFS on Linux and while ZFS is really great it's not really good integrated in the kernel and sometimes pretty unstable at least in my rather esoteric scenario... Despite other claims FreeBSD suffers similar problems.. (https://clusterhq.com/blog/complexity-freebsd-vfs-using-zfs-example-part-1-2/ https://clusterhq.com/blog/complexity-freebsd-vfs-using-zfs-...). Other problems are that jails are fine but there is no disk I/O limitation possible... I've also thought about FreeBSD and while pkgng is really great it's a similar problem.. I'm stumbling upon bugs or untested things and I'm unable to contribute time to fixing them.
  - tachion 12y ago
    Do you mind pointing to your PR's with the bugs you've found or at least mention what they were? It sound like you've found a hell of a bugs/problems in a system (and I am thinking about FreeBSD now) that me and huge number of other people are running without any issues, so it would be beneficial for everyone, if you'd share your problems with PR's - there is active community around it that can fix issues if you are unable to do it.
    - nisa 12y ago
      Sorry if I was unclear. I stumbled upon a few issues running ZFS on Linux that are known and on the development roadmap. Things like ARC integration and better failure handling in case of disk problems. I don't run anything big on FreeBSD and ZFS. I have not experienced problems on a raidz2 ZFS fileserver that runs FreeBSD except that disks drop out quite randomly but I've yet been unable to pinpoint that and it's likely that these are hardware issues as the system runs on budget hardware. Sorry if my comment spreaded FUD.
      - ryao 12y ago
        This might be of use to you: http://open-zfs.org/w/index.php?title=Hardware#Error_recovery_control http://open-zfs.org/w/index.php?title=Hardware#Error_recover...
  - XorNot 12y ago
    The biggest problem I've had with ZFS on Linux is that the arc_cache won't back off from memory quick enough. There's still an edge case in their somewhere where it hits swap and effectively locks up (its still running, just amazingly slowly). The fix is to set the arc_limit low (like 1GB) - it's not a hard limit, but it stops it eating all the RAM on my 8gb ram box.
    - ryao 12y ago
      Swap on zvols is not fully stable on Linux yet. I have patches in the works for 0.6.4 that should improve things.
- evandena 12y ago
  How's KVM on FreeBSD these days?
  - ryao 12y ago
    The port was abandoned as no one was willing to finish it. FreeBSD developers have since written their own hypervisor called byhve.
shmerl 12y ago
Sorry for off-topic. Do you have any opinion on HPCC? It seems to be aiming at the same goals as Hadoop but claims to be able to do more and have better performance. I tried to find more first hand experience reports, but it's rather obscure and not used that widely.
bcantrill 12y ago
You might be interested in our ongoing work to execute Linux binaries in a SmartOS zone.[1] It's still an area of very active development (one needs to unfortunately follow the source[2] to follow its progress), but we have a ton working -- and it's all being done in the open. (And it's certainly very nice to be able to "apt-get install" something and then be able to DTrace the resulting application!) We're close to having enough working to be able to document where we are and get others kicking the tires, so stay tuned to the SmartOS discussion list if this is something you're interested in! [1] https://www.mail-archive.com/smartos-discuss@lists.smartos.org/msg00475.html https://www.mail-archive.com/smartos-discuss@lists.smartos.o... [2] https://github.com/joyent/illumos-joyent https://github.com/joyent/illumos-joyent
- nisa 12y ago
  This looks great!
donavanm 12y ago
It's no dtrace, but check out systemtap "stap". It provides a lot of similar functionality on modern linux. It can actually use existing dtrace probe points if you have a binary like mysqld. The ability to follow call graphs, and insert arbitrary logic, is amazing for chasing down system issues.

wmf 12y ago

It looks like this page was recently updated and coincidentally today he gave a talk on Linux performance at LinuxCon: http://events.linuxfoundation.org/events/linuxcon-north-america/program/schedule http://events.linuxfoundation.org/events/linuxcon-north-amer... http://events.linuxfoundation.org/sites/events/files/slides/LinuxPerfTools_0.pdf http://events.linuxfoundation.org/sites/events/files/slides/...

brendangregg 12y ago
Yes, I gave the talk this morning. I hope people found it useful! In case slideshare is quicker to load, the slides are also at: http://www.slideshare.net/brendangregg/linux-performance-tools http://www.slideshare.net/brendangregg/linux-performance-too...
- jaryd 12y ago
  Do you expect video of the presentation to be available any time in the near future?
  - brendangregg 12y ago
    Sorry, like other talks at LinuxCon, it was not videoed. I think it would be a useful to have a video of it, so, much as I hate to give the same talk twice, I'll probably do it again at some point for the video.
    - rodgerd 12y ago
      Don't tell me you didn't submit for linux.conf.au next year...

josephyu0305 12y ago

Nice link it show lot of Linux performance help when it comes the system crash/ and very much learning i gotfrom his presentation.

fasteo 12y ago

I would recommend his book [1] to anyone interested in systems performance. What really caught my attention is the focus he puts in having a goal and applying a method to solve performance issues. Many times, I have found myself "lost" while isolating a performance issue. Not anymore. [1] http://www.brendangregg.com/sysperfbook.html http://www.brendangregg.com/sysperfbook.html