5 ms·

This is great news, and we're already using ZFS in production on Ubuntu in a few areas at Netflix (not widespread yet). Ubuntu 16.04 also comes with enhanced B

This is great news, and we're already using ZFS in production on Ubuntu in a few areas at Netflix (not widespread yet). Ubuntu 16.04 also comes with enhanced BPF, the new Linux tracing & programming framework that is builtin to the kernel, and is a huge leap forward for Linux tracing. Eg, we can start using tools like these: https://github.com/iovisor/bcc#tracing https://github.com/iovisor/bcc#tracing

2trill2spill 11y ago
How does BPF compare to dtrace?
- brendangregg 11y ago
  If you browse some of the _example.txt files in https://github.com/iovisor/bcc/tree/master/tools https://github.com/iovisor/bcc/tree/master/tools , you'll see it's solving the same problems we used to solve with DTrace, plus a few extra. Here's a couple of the ZFS examples (since we're talking ZFS): # ./zfsslower Tracing ZFS operations slower than 10 ms TIME COMM PID T BYTES OFF_KB LAT(ms) FILENAME 06:31:28 dd 25570 W 131072 38784 303.92 data1 06:31:34 dd 25686 W 131072 38784 388.28 data1 06:31:35 dd 25686 W 131072 78720 519.66 data1 06:31:35 dd 25686 W 131072 116992 405.94 data1 06:31:35 dd 25686 W 131072 153600 433.52 data1 [...] # ./zfsdist Tracing ZFS operation latency... Hit Ctrl-C to end. ^C operation = 'read' usecs : count distribution 0 -> 1 : 0 | | 2 -> 3 : 0 | | 4 -> 7 : 4479 |****************************************| 8 -> 15 : 1028 |********* | 16 -> 31 : 14 | | 32 -> 63 : 1 | | 64 -> 127 : 2 | | 128 -> 255 : 6 | | 256 -> 511 : 1 | | 512 -> 1023 : 1256 |*********** | 1024 -> 2047 : 9 | | 2048 -> 4095 : 1 | | 4096 -> 8191 : 2 | | [...] The current BPF interface we're using (bcc) is Python for the frontend, and C for the backend. It's currently much more verbose than DTrace, and involves writing 10x the lines of code. For some immediate use cases at Netflix, that's not a big problem, as staff will be using BPF via a GUI (Vector), not writing these tools directly. There's also high level features it's still missing (like tracepoints and sampling), so what will be in Ubuntu 14.04 won't do everything, but it will do a fair amount: most of those _example.txt's. Some use a newer BPF interface (linux 4.5), and we've been putting the legacy versions in an /old directory specifically for Ubuntu 16.04 users.
  - cthalupa 11y ago
    Does the current publicly released version of Vector support BPF? Or is there perhaps a PMDA that allows BPF support? I'm following along with all of this pretty excitedly, and crossing my fingers for a Linux tracing book with BPF, ftrace, perf, etc. to read through and keep on my shelf next to your performance and dtrace books ;)
    - brendangregg 11y ago
      Vector doesn't have BPF support yet. When it does we should be open sourcing it. As for a book, I'll try to get it done. :)
tiffanyh 11y ago
What about FreeBSD? Does that imply Netflix is transitioning away from FreeBSD? If so, why?
- brendangregg 11y ago
  Netflix cloud: AWS EC2, tens of thousands of cloud instances, mostly Ubuntu. Netflix CDN (Open Connect Appliance): lots of physical boxes, FreeBSD.
  - tiffanyh 11y ago
    Any reason why you don't use FreeBSD everywhere (or Ubuntu everywhere). Why two different OS'es?
    - brendangregg 11y ago
      It's really two questions: Why choose Ubuntu for the cloud, and, why choose FreeBSD for the CDN. We believe that's the best choice for both environments. I was trying to type in an explanation here, but that's really something that will take a lot to explain (maybe a Netflix tech blog post).
      - vageli 11y ago
        That would be a great post.
      - tzs 11y ago
        If you do write that blog post, it would be cool if you not only covered the FreeBSD vs. Ubuntu aspects of the choice, but also the Ubuntu vs. other Linux aspects (particularly Debian).
diptanu 11y ago
So good to see ZOL has come so far!

2trill2spill 11y ago

How does BPF compare to dtrace?

brendangregg 11y ago
If you browse some of the _example.txt files in https://github.com/iovisor/bcc/tree/master/tools https://github.com/iovisor/bcc/tree/master/tools , you'll see it's solving the same problems we used to solve with DTrace, plus a few extra. Here's a couple of the ZFS examples (since we're talking ZFS): # ./zfsslower Tracing ZFS operations slower than 10 ms TIME COMM PID T BYTES OFF_KB LAT(ms) FILENAME 06:31:28 dd 25570 W 131072 38784 303.92 data1 06:31:34 dd 25686 W 131072 38784 388.28 data1 06:31:35 dd 25686 W 131072 78720 519.66 data1 06:31:35 dd 25686 W 131072 116992 405.94 data1 06:31:35 dd 25686 W 131072 153600 433.52 data1 [...] # ./zfsdist Tracing ZFS operation latency... Hit Ctrl-C to end. ^C operation = 'read' usecs : count distribution 0 -> 1 : 0 | | 2 -> 3 : 0 | | 4 -> 7 : 4479 |****************************************| 8 -> 15 : 1028 |********* | 16 -> 31 : 14 | | 32 -> 63 : 1 | | 64 -> 127 : 2 | | 128 -> 255 : 6 | | 256 -> 511 : 1 | | 512 -> 1023 : 1256 |*********** | 1024 -> 2047 : 9 | | 2048 -> 4095 : 1 | | 4096 -> 8191 : 2 | | [...] The current BPF interface we're using (bcc) is Python for the frontend, and C for the backend. It's currently much more verbose than DTrace, and involves writing 10x the lines of code. For some immediate use cases at Netflix, that's not a big problem, as staff will be using BPF via a GUI (Vector), not writing these tools directly. There's also high level features it's still missing (like tracepoints and sampling), so what will be in Ubuntu 14.04 won't do everything, but it will do a fair amount: most of those _example.txt's. Some use a newer BPF interface (linux 4.5), and we've been putting the legacy versions in an /old directory specifically for Ubuntu 16.04 users.
- cthalupa 11y ago
  Does the current publicly released version of Vector support BPF? Or is there perhaps a PMDA that allows BPF support? I'm following along with all of this pretty excitedly, and crossing my fingers for a Linux tracing book with BPF, ftrace, perf, etc. to read through and keep on my shelf next to your performance and dtrace books ;)
  - brendangregg 11y ago
    Vector doesn't have BPF support yet. When it does we should be open sourcing it. As for a book, I'll try to get it done. :)

tiffanyh 11y ago

What about FreeBSD? Does that imply Netflix is transitioning away from FreeBSD? If so, why?

brendangregg 11y ago
Netflix cloud: AWS EC2, tens of thousands of cloud instances, mostly Ubuntu. Netflix CDN (Open Connect Appliance): lots of physical boxes, FreeBSD.
- tiffanyh 11y ago
  Any reason why you don't use FreeBSD everywhere (or Ubuntu everywhere). Why two different OS'es?
  - brendangregg 11y ago
    It's really two questions: Why choose Ubuntu for the cloud, and, why choose FreeBSD for the CDN. We believe that's the best choice for both environments. I was trying to type in an explanation here, but that's really something that will take a lot to explain (maybe a Netflix tech blog post).
    - vageli 11y ago
      That would be a great post.
    - tzs 11y ago
      If you do write that blog post, it would be cool if you not only covered the FreeBSD vs. Ubuntu aspects of the choice, but also the Ubuntu vs. other Linux aspects (particularly Debian).

diptanu 11y ago

So good to see ZOL has come so far!