3 ms·

Have they ever talked about why the content isn't stored/read from ZFS just the root pool?

iv42 2y ago
Yes. Because sendfile(2) is not zero-copy on ZFS.
shrubble 2y ago
Yes, it is because they can do zero-copy sending of content that they can't do under ZFS (yet). Some links to Netflix papers and video talks on this older Reddit thread: https://www.reddit.com/r/freebsd/comments/ltjv8m/zfs_is_overwhelminly_praised_yet_why_does_netflix/ https://www.reddit.com/r/freebsd/comments/ltjv8m/zfs_is_over...
- gigatexal 2y ago
  bravo thank you for that
8fingerlouie 2y ago
Besides not being able to do zero copy on ZFS (yet), it probably also have to do with them not using RAID for content, and single drive ZFS doesn't make much sense in that scenario. Single drive ZFS brings snapshots and COW, as well as bitrot detection, but for Netflix OCAs, snapshots are not used, and it's mostly read-only content, so not much use for COW, and bitrot is less of a problem with media. Yes, you may get a corrupt pixel every now and then (assuming it's not caught by the codec), but the media will be reseeded again every n days, so the problem will solve itself. I assume they have ample redundancy in the rest of the CDN, so if a drive fails, they can simply redirect traffic to the next nearest mirror, and when a drive is eventually replaced, it will be seeded again by normal content seeding.
- _zoltan_ 2y ago
  yet? you write "yet" as it's something that would be almost readily available, yet it's been at least 2 years now and it's still not there. am I missing something or that "yet" is more like "maybe sometime if ever"?
  - toast0 2y ago
    Netflix has been developing in kernel and userland for this; if ZFS for content was a priority, they could make 0-copy sendfile work. Yes, it's not trivial at all. It will probably happen eventually, by someone who needs 0-copy and ZFS; or by ZFS people who want to support more use cases.
  - sophacles 2y ago
    Yet does not suggest any sort of impending completion. We haven't been to the center of the galaxy yet. We haven't achieved immortality yet. Both are valid sentences without any fixed timeline, and in the case of the first, a date that is hundreds of thousands of years in the future at soonest. "yet" just means "up until this time" (in conext, sometimes it means "by the time you're talking about" - e.g. I'm scheduled on-call on the 13th, but i won't be back from my PTO yet).
drewg123 2y ago
In addition to the lack of zero-copy sendfile from ZFS, we also have the problem the ZFS is also lacking async sendfile support (eg, the ability to continue the sendfile operation from the disk interrupt handler when the blocks arrive from disk).

iv42 2y ago

Yes. Because sendfile(2) is not zero-copy on ZFS.

Yes, it is because they can do zero-copy sending of content that they can't do under ZFS (yet). Some links to Netflix papers and video talks on this older Reddit thread: https://www.reddit.com/r/freebsd/comments/ltjv8m/zfs_is_overwhelminly_praised_yet_why_does_netflix/ https://www.reddit.com/r/freebsd/comments/ltjv8m/zfs_is_over...

gigatexal 2y ago
bravo thank you for that

8fingerlouie 2y ago

Besides not being able to do zero copy on ZFS (yet), it probably also have to do with them not using RAID for content, and single drive ZFS doesn't make much sense in that scenario. Single drive ZFS brings snapshots and COW, as well as bitrot detection, but for Netflix OCAs, snapshots are not used, and it's mostly read-only content, so not much use for COW, and bitrot is less of a problem with media. Yes, you may get a corrupt pixel every now and then (assuming it's not caught by the codec), but the media will be reseeded again every n days, so the problem will solve itself. I assume they have ample redundancy in the rest of the CDN, so if a drive fails, they can simply redirect traffic to the next nearest mirror, and when a drive is eventually replaced, it will be seeded again by normal content seeding.

_zoltan_ 2y ago
yet? you write "yet" as it's something that would be almost readily available, yet it's been at least 2 years now and it's still not there. am I missing something or that "yet" is more like "maybe sometime if ever"?
- toast0 2y ago
  Netflix has been developing in kernel and userland for this; if ZFS for content was a priority, they could make 0-copy sendfile work. Yes, it's not trivial at all. It will probably happen eventually, by someone who needs 0-copy and ZFS; or by ZFS people who want to support more use cases.
- sophacles 2y ago
  Yet does not suggest any sort of impending completion. We haven't been to the center of the galaxy yet. We haven't achieved immortality yet. Both are valid sentences without any fixed timeline, and in the case of the first, a date that is hundreds of thousands of years in the future at soonest. "yet" just means "up until this time" (in conext, sometimes it means "by the time you're talking about" - e.g. I'm scheduled on-call on the 13th, but i won't be back from my PTO yet).

drewg123 2y ago

In addition to the lack of zero-copy sendfile from ZFS, we also have the problem the ZFS is also lacking async sendfile support (eg, the ability to continue the sendfile operation from the disk interrupt handler when the blocks arrive from disk).