5 ms·
Netflix kernel engineer here.. We use FreeBSD's async sendfile() and not aio, so it would be a bit harder for us to fix open latency, since we're not using aio
by drewg123 8y ago
Netflix kernel engineer here.. We use FreeBSD's async sendfile() and not aio, so it would be a bit harder for us to fix open latency, since we're not using aio.
I had not thought about open latency being an issue, that's fascinating. Looking at one of our busy 100G servers w/NVME storage, I see an openat syscall latency of no more than 8ms sampling over a few minute period, with most everything being below 65us. However, the workload is probably different than yours (more longer lived connections, fewer but larger files, fewer opens in general). Eg, we probably don't have nearly the "long tail" issue you do..
- khc 8y agoright I suspect you have way fewer files than we do and everything is in the dentry cache. Pretty sure that most of your files are bigger than 60KB too :-) (which is our p90)
- drewg123 8y agoWow, it is a different world :)
- khc 8y agoour machines also run many different services (the CDN product is just one of them) and isolating I/O from different products is difficult. I'd also love to have NVMe.
- solarengineer 8y agoAny chance you could use dtrace to isolate process specific I/O?
- khc 8y ago"isolate" here I meant prevent one process's IO from affecting another one
- the8472 8y agoAt my job we have to open many small files from NFS. The latency of open() absolutely murders sequential performance (>80 seconds just to open a scene descriptions). Prewawrming the fairly shortlived NFS access caches in parallel evaporates most of the performance penalty.
- pram 8y agoTIL what a 'dentry' is!
- pg314 8y agoHave you looked into using something like SQLite instead of the filesystem? [1] [1] https://www.sqlite.org/fasterthanfs.html https://www.sqlite.org/fasterthanfs.html
- Kalium 8y agoSQLite makes a ton of sense for systems that don't need to worry about concurrent writes. It's possible that a CDN's cache system might need to concern itself with concurrent writes.
- khc 8y agoOther than the concurrent write that another comment mentioned, looks like this test is done with a data set that fits entirely in RAM. I wish we have enough RAM for the entire internet :-(
- ioquatix 8y ago8ms actually seems quite slow at that level.
- drewg123 8y agoYes, that's the absolute worst one out of a large sample size (far less than a fraction of a percent). I suspect that openat() was particularly unlucky, and interrupted multiple times..
- crest 8y agoA few more aio_* syscalls would really simplify things a lot. I suspect the most important missing aio_* syscalls are aio_open()/close() and aio_stat(). The semantics for async open/close would be tricky.