3 ms·
The thing I never see mentioned when the pros and cons of ZFS are discussed is that ZFS is not zero-copy for things like sendfile. This is one reason why we (N
by drewg123 6y ago
The thing I never see mentioned when the pros and cons of ZFS are discussed is that ZFS is not zero-copy for things like sendfile. This is one reason why we (Netflix) use UFS for serving content rather than ZFS.
This is because ZFS is cached by the ARC, not the normal page cache. ARC is weird, and operates in 8K blocks (like sparc page size), rather than 4K pages. Zero-copy things like sendfile depend on referencing pages in the page cache, and have never been adapted to deal with ARC. So making sendfile zero-copy with ZFS is a hard project that would involve either teaching ARC to "loan" pages to sendfile, or ripping out ARC caching from ZFS and making it use the same page cache that all other filesystems use.
- louwrentius 6y agoThanks for the comment. I think it's a bigger issue: people advocating ZFS only promote the good features and aren't open about the downsides (or even try to downplay them with clear bullshit). It all depends on the circumstances and requirements: (small) business application or some home-build NAS? In your case, how much does it matter that some node experiences bitrot, and how big are those risks?
- drewg123 6y agoOur use case is far different than a home NAS (hundreds of TB of disk), I replied to that simply because it was also talking about potential downsides to ZFS. For our use, bit rot is pretty low risk. We have tooling to catch corrupted files (and it happens surprisingly rarely). We don't care about any of the raid like features (if a drive dies, we tell clients to get their video elsewhere). For our use case, ZFS would be attractive mainly because of the ability to keep metadata in the L2 ARC. One of our bigger sources of P99 latency is uncached metadata reads from mechanical drives. Our FS guys are currently solving that problem in other ways.
- louwrentius 6y agoThank you for this explanation, very interesting. If that's something you can share, would be a nice blogpost.
- e40 6y agoUFS on what OS?
- anthk 6y agoFreeBSD.
- giantrobot 6y agoNetFlix is a heavy FreeBSD shop, so I'd assume FreeBSD.
- throw0101a 6y ago> NetFlix is a heavy FreeBSD shop, so I'd assume FreeBSD. Kind of: their edge-cache appliances run FreeBSD. IIRC they run Linux on their Amazon cloud for all their 'internal' stuff. If you do some searches there's some good presentations on their work on getting encrypted streaming to go Very Fast: * https://www.phoronix.com/scan.php?page=news_item&px=Netflix-NUMA-FreeBSD-Optimized https://www.phoronix.com/scan.php?page=news_item&px=Netflix-... * https://2019.eurobsdcon.org/talk-speakers/#numa https://2019.eurobsdcon.org/talk-speakers/#numa * https://netflixtechblog.com/serving-100-gbps-from-an-open-connect-appliance-cdb51dda3b99 https://netflixtechblog.com/serving-100-gbps-from-an-open-co...
- drewg123 6y agoUFS on FreeBSD. And that's me :)