6 ms·

Linux Profiling at Netflix

134 points by 2510c39011c5 12y ago

planckscnst 12y ago
The SCALE 13x videos are going up here: https://www.youtube.com/channel/UCN2nbMPLJWv3Y__4JuF_hMQ https://www.youtube.com/channel/UCN2nbMPLJWv3Y__4JuF_hMQ Does anyone know in what room and day Brendan spoke?
- rollardurby 12y ago
  https://www.youtube.com/watch?v=dhDpncTDDvI#t=892 https://www.youtube.com/watch?v=dhDpncTDDvI#t=892
  - teddyh 12y ago
    You can format the time like this: https://www.youtube.com/watch?v=dhDpncTDDvI#t=14m52s https://www.youtube.com/watch?v=dhDpncTDDvI#t=14m52s
- brendangregg 12y ago
  While it's great to have all the talks captured like this (and I get to go see some other talks I missed), the quality is a bit fuzzy. If that's annoying you too much, there was a second camera in the room. I'll tweet the video when it's posted (@brendangregg).
- irabinovitch1 12y ago
  We're in the process of editing all the videos down to individual session recordings. We're all volunteer, so apologies for the delays. If anyone out there would like we're happy to get extra hands with the editing work.
icantthinkofone 12y ago
So some aren't confused. Content delivery is on FreeBSD, not Linux.
- ThreeFinger 12y ago
  ZFS drives?
rogerbinns 12y ago
Can anyone speak as to why Netflix needs so much computing power? As far as I can tell, content delivery is handled separately, so all these talks are about the non-content side. Sure we need to see various personalised lists when first connecting, but no activity while actually watching things. The personalisation seems very light - eg their lists of comedies look like they show the same to everyone. And it doesn't seem intelligent at all, like for example hiding items you have already seen. https://news.ycombinator.com/item?id=9083196 https://news.ycombinator.com/item?id=9083196
- Thaxll 12y ago
  Because they have thousand Linux nodes for Cassandra.
- cpwatson 12y ago
  The microservice architecture results in a larger footprint than you might find with a more monolithic app stack; which provides us with improved engineering velocity and reliability. Although the content playback is served by our CDN, all other device interactions are handled by the services on AWS. We also have some blogposts which provide a glimpse into some of the work we do behind the scenes (http://techblog.netflix.com/2015/01/netflixs-viewing-data-how-we-know-where.html http://techblog.netflix.com/2015/01/netflixs-viewing-data-ho...). Oh yeah, we also encode and run our big data services on AWS...
  - rogerbinns 12y ago
    This still doesn't answer why it is so large. Or more accurately my experience as a Netflix customer doesn't reflect the apparent size of all these backends. Compare to something like Reddit where every page is customised, I go through lots of them, and my interactions are frequent. My Netflix interactions look like recently viewed being custom, and everything else being generic (same for all users). If you told me my recommendations are recalculated once a month I'd believe you. The majority of time is spent viewing which is just periodic progress updates. In other words my user experience suggests Reddit having an infrastructure many times larger than Netflix.
krakensden 12y ago
It often feels like the only users of perf are its developers and Brendan Gregg.
- scott_s 12y ago
  I use it - perf is wonderful. Back in the dark days of grad school, getting performance counter data for your program required hacking your code to call into PAPI (http://icl.cs.utk.edu/papi/overview/ http://icl.cs.utk.edu/papi/overview/). Being able to tell a program, "Hey, run this for me, and tell me how many branch mispredictions there are. And where they happen" is like magic. If you do any performance analysis of software, learn perf. Official manual: https://perf.wiki.kernel.org/index.php/Main_Page https://perf.wiki.kernel.org/index.php/Main_Page Brendan Gregg's perf page: http://www.brendangregg.com/perf.html http://www.brendangregg.com/perf.html
- s_kanev 12y ago
  I beg to disagree. For instance, it's been running for years sampling Google's fleet [1] (the paper mentions oprofile, but that has been replaced quite a while ago). [1] http://courses.cs.washington.edu/courses/cse590g/11sp/36575.pdf http://courses.cs.washington.edu/courses/cse590g/11sp/36575....
- jschwartzi 12y ago
  I used it to debug some performance issues in a graphics engine we use at my company. It helped me understand what the low-hanging fruit was.
X-Istence 12y ago
This talk was absolutely fantastic at SCALE13x. Lot of good content that will help a lot of people get started with CPU profiling.
distantsounds 12y ago
Those flame graphs look hella useful, hopefully the JDK patches make their way in so we don't have to futz around with customizing too much. Thanks for this!
- MrBuddyCasino 12y ago
  Absolutely, I had no idea there is a way to export the symbols and make them known to perf. Its magic, and I want it.
shmerl 12y ago
Surely they use Linux, yet they still can't offer a proper Linux client. DRMed much.
ctb_mg 12y ago
I really like the workflow chart on slide 22. I can't help but think this is a useful way to visualize common workflow sequences of commands (think git add, git commit, git push). Honestly I wouldn't mind if man pages came with ascii art to visualize this type of documentation.
api 12y ago
Why FreeBSD for content delivery?
- emaste 12y ago
  From https://openconnect.itp.netflix.com/software/index.html https://openconnect.itp.netflix.com/software/index.html: For the operating system, we use FreeBSD version 10. This was selected for its balance of stability and features, a strong development community and staff expertise. All code improvements, feature additions, and bug fixes are contributed directly back to the open source community via the FreeBSD committers on our team. We also strive to stay at the front of the FreeBSD development process, allowing us to have a tight feedback loop with other community and partner developers. The result has been a positive open source ecosystem that lowers our development costs and multiplies the effectiveness of our efforts..
nwmcsween 12y ago
Omitting the frame pointer is a default on x86_64 its specified in the abi so it's going to be mighty tough to change that. I don't understand why you recommend not using the debug info when you already have to use it for symbol name resolution.
- brendangregg 12y ago
  Are you talking about http://www.x86-64.org/documentation/abi.pdf http://www.x86-64.org/documentation/abi.pdf, and footnote 7 on page 16, which reads: "7 The conventional use of %rbp as a frame pointer for the stack frame may be avoided by using %rsp (the stack pointer) to index into the stack frame. This technique saves two instructions in the prologue and epilogue and makes one additional general-purpose register (%rbp) available." %rbp is mentioned elsewhere, eg, "optionally used as frame pointer" This ABI does not appear to specify that the frame pointer is omitted. It uses the words "may" and "optionally". Which means, the gcc engineers can choose to do what they want. And they should choose to keep the frame pointer _by default_, and _optionally_ omit it (for when the sacrifice is deemed worth it): -fomit-frame-pointer. This would better follow the language used in the ABI! :-) We've found debuginfo to be large (size, bytes), which can slow down instance creation, which means we must run with a higher headroom to accommodate. A stripped-down debuginfo that just enabled stack walking might do.
  - nwmcsween 12y ago
    Compress the debug info? Gdb handles it no idea about perf though, there is also dwo to remove redundant debug info.
deirdres 12y ago
For those who missed today's post https://news.ycombinator.com/item?id=9121015 https://news.ycombinator.com/item?id=9121015 because it got downvoted (wrongly) for being a duplicate: accompanying blog post and video now available: http://www.brendangregg.com/blog/2015-02-27/linux-profiling-at-netflix.html http://www.brendangregg.com/blog/2015-02-27/linux-profiling-...

134 points by 2510c39011c5 12y ago

The SCALE 13x videos are going up here: https://www.youtube.com/channel/UCN2nbMPLJWv3Y__4JuF_hMQ https://www.youtube.com/channel/UCN2nbMPLJWv3Y__4JuF_hMQ Does anyone know in what room and day Brendan spoke?

rollardurby 12y ago
https://www.youtube.com/watch?v=dhDpncTDDvI#t=892 https://www.youtube.com/watch?v=dhDpncTDDvI#t=892
- teddyh 12y ago
  You can format the time like this: https://www.youtube.com/watch?v=dhDpncTDDvI#t=14m52s https://www.youtube.com/watch?v=dhDpncTDDvI#t=14m52s
brendangregg 12y ago
While it's great to have all the talks captured like this (and I get to go see some other talks I missed), the quality is a bit fuzzy. If that's annoying you too much, there was a second camera in the room. I'll tweet the video when it's posted (@brendangregg).
irabinovitch1 12y ago
We're in the process of editing all the videos down to individual session recordings. We're all volunteer, so apologies for the delays. If anyone out there would like we're happy to get extra hands with the editing work.

icantthinkofone 12y ago

So some aren't confused. Content delivery is on FreeBSD, not Linux.

ThreeFinger 12y ago
ZFS drives?

rogerbinns 12y ago

Can anyone speak as to why Netflix needs so much computing power? As far as I can tell, content delivery is handled separately, so all these talks are about the non-content side. Sure we need to see various personalised lists when first connecting, but no activity while actually watching things. The personalisation seems very light - eg their lists of comedies look like they show the same to everyone. And it doesn't seem intelligent at all, like for example hiding items you have already seen. https://news.ycombinator.com/item?id=9083196 https://news.ycombinator.com/item?id=9083196

Thaxll 12y ago
Because they have thousand Linux nodes for Cassandra.
cpwatson 12y ago
The microservice architecture results in a larger footprint than you might find with a more monolithic app stack; which provides us with improved engineering velocity and reliability. Although the content playback is served by our CDN, all other device interactions are handled by the services on AWS. We also have some blogposts which provide a glimpse into some of the work we do behind the scenes (http://techblog.netflix.com/2015/01/netflixs-viewing-data-how-we-know-where.html http://techblog.netflix.com/2015/01/netflixs-viewing-data-ho...). Oh yeah, we also encode and run our big data services on AWS...
- rogerbinns 12y ago
  This still doesn't answer why it is so large. Or more accurately my experience as a Netflix customer doesn't reflect the apparent size of all these backends. Compare to something like Reddit where every page is customised, I go through lots of them, and my interactions are frequent. My Netflix interactions look like recently viewed being custom, and everything else being generic (same for all users). If you told me my recommendations are recalculated once a month I'd believe you. The majority of time is spent viewing which is just periodic progress updates. In other words my user experience suggests Reddit having an infrastructure many times larger than Netflix.

krakensden 12y ago

It often feels like the only users of perf are its developers and Brendan Gregg.

scott_s 12y ago
I use it - perf is wonderful. Back in the dark days of grad school, getting performance counter data for your program required hacking your code to call into PAPI (http://icl.cs.utk.edu/papi/overview/ http://icl.cs.utk.edu/papi/overview/). Being able to tell a program, "Hey, run this for me, and tell me how many branch mispredictions there are. And where they happen" is like magic. If you do any performance analysis of software, learn perf. Official manual: https://perf.wiki.kernel.org/index.php/Main_Page https://perf.wiki.kernel.org/index.php/Main_Page Brendan Gregg's perf page: http://www.brendangregg.com/perf.html http://www.brendangregg.com/perf.html
s_kanev 12y ago
I beg to disagree. For instance, it's been running for years sampling Google's fleet [1] (the paper mentions oprofile, but that has been replaced quite a while ago). [1] http://courses.cs.washington.edu/courses/cse590g/11sp/36575.pdf http://courses.cs.washington.edu/courses/cse590g/11sp/36575....
jschwartzi 12y ago
I used it to debug some performance issues in a graphics engine we use at my company. It helped me understand what the low-hanging fruit was.

X-Istence 12y ago

This talk was absolutely fantastic at SCALE13x. Lot of good content that will help a lot of people get started with CPU profiling.

distantsounds 12y ago

Those flame graphs look hella useful, hopefully the JDK patches make their way in so we don't have to futz around with customizing too much. Thanks for this!

MrBuddyCasino 12y ago
Absolutely, I had no idea there is a way to export the symbols and make them known to perf. Its magic, and I want it.

shmerl 12y ago

Surely they use Linux, yet they still can't offer a proper Linux client. DRMed much.

ctb_mg 12y ago

I really like the workflow chart on slide 22. I can't help but think this is a useful way to visualize common workflow sequences of commands (think git add, git commit, git push). Honestly I wouldn't mind if man pages came with ascii art to visualize this type of documentation.

api 12y ago

Why FreeBSD for content delivery?

emaste 12y ago
From https://openconnect.itp.netflix.com/software/index.html https://openconnect.itp.netflix.com/software/index.html: For the operating system, we use FreeBSD version 10. This was selected for its balance of stability and features, a strong development community and staff expertise. All code improvements, feature additions, and bug fixes are contributed directly back to the open source community via the FreeBSD committers on our team. We also strive to stay at the front of the FreeBSD development process, allowing us to have a tight feedback loop with other community and partner developers. The result has been a positive open source ecosystem that lowers our development costs and multiplies the effectiveness of our efforts..

nwmcsween 12y ago

Omitting the frame pointer is a default on x86_64 its specified in the abi so it's going to be mighty tough to change that. I don't understand why you recommend not using the debug info when you already have to use it for symbol name resolution.

brendangregg 12y ago
Are you talking about http://www.x86-64.org/documentation/abi.pdf http://www.x86-64.org/documentation/abi.pdf, and footnote 7 on page 16, which reads: "7 The conventional use of %rbp as a frame pointer for the stack frame may be avoided by using %rsp (the stack pointer) to index into the stack frame. This technique saves two instructions in the prologue and epilogue and makes one additional general-purpose register (%rbp) available." %rbp is mentioned elsewhere, eg, "optionally used as frame pointer" This ABI does not appear to specify that the frame pointer is omitted. It uses the words "may" and "optionally". Which means, the gcc engineers can choose to do what they want. And they should choose to keep the frame pointer _by default_, and _optionally_ omit it (for when the sacrifice is deemed worth it): -fomit-frame-pointer. This would better follow the language used in the ABI! :-) We've found debuginfo to be large (size, bytes), which can slow down instance creation, which means we must run with a higher headroom to accommodate. A stripped-down debuginfo that just enabled stack walking might do.
- nwmcsween 12y ago
  Compress the debug info? Gdb handles it no idea about perf though, there is also dwo to remove redundant debug info.

deirdres 12y ago

For those who missed today's post https://news.ycombinator.com/item?id=9121015 https://news.ycombinator.com/item?id=9121015 because it got downvoted (wrongly) for being a duplicate: accompanying blog post and video now available: http://www.brendangregg.com/blog/2015-02-27/linux-profiling-at-netflix.html http://www.brendangregg.com/blog/2015-02-27/linux-profiling-...