13 ms·
Non blocking disk I/O is one thing where NT is really ahead of all nix OSes. Unlike say network IO where we have all sorts of platforms(go, node) which allow yo
by enitihas 8y ago
Non blocking disk I/O is one thing where NT is really ahead of all nix OSes. Unlike say network IO where we have all sorts of platforms(go, node) which allow you to scale by doing async IO, there aren't much options for disk I/O, primarily because of lack of nix options.
- zzzcpan 8y agoIf you mean IOCP API that requires you to hold memory hostage for it, then no, NT sucks at it. The idea behind that API only works well if there is no switch between kernel space and user space. Otherwise you can emulate that awful API with threads on unix-like systems, it will perform the same. Don't get me wrong though, caching/file serving in nginx was always very hacky. And blowing latency by blocking on filesystem and yet relying on vfs cache underneath is just one side effect of that.
- acdha 8y ago> If you mean IOCP API that requires you to hold memory hostage for it, then no, NT sucks at it. “hold memory hostage” really doesn't help make your point. It's inflammatory wording one step above spelling it Micro$oft and it makes it sound like you don't understand the engineering trade-offs which the Microsoft engineers made. It'd be much better if you explained _why_ you disagree with their choice and especially whether there are differences in the type of work which you've used it for which might explain that opinion.
- brian-armstrong 8y agoThe preferred nomenclature is "Micro$haft".
- acdha 8y agoI remember Slashdot in 1997, too, and would like to avoid repeating it. HN is poorer for substance-free jokes.
- smt88 8y agoI've found reddit to be substantive, sometimes on par with HN, but filled with jokes. Depending on the subreddit, you may be able to scratch your itch there.
- 2038AD 8y agoI've found most jokes on reddit seem to be attempts to get upvotes. Then again I'm not a fan of the upvote system generally. One thing I dislike is that downvoted comments are hidden. It equates disagreement with spam and discourages people from expressing genuine beliefs and reactions. I was going to make sure I haven't downvoted anything here but I can't find if there's a way to check.
- orf 8y agoI get the feeling you've had a bad experience with the API and that colours your opinion of the overall design of IOCP. It's really quite well thought out. Also in my experience it handles slightly more concurrent connections with a fair bit less CPU usage. There are some weird and not very nice parts + limitations to the API that makes it pretty hard to write cross-platform things with it, granted.
- johncolanduoni 8y agoThe memory is either held hostage in your buffer or the kernel’s. For that matter, if you’re making a blocking read/write call the memory is just as much a hostage.
- halayli 8y agoFreebsd have had kqueue for the longest time. Unfortunately Linux didn't adopt it for political reasons and osx did a half port and dropped disk io support.
- tedunangst 8y agokqueue doesn't work for disk. It always reports ready to read.
- johncolanduoni 8y agoIf you try to use the readiness-based kqueue events that is the case, but the completion-based aio events do work with disk files on kqueue.
- tedunangst 8y agoAh, that slipped my mind.
- crest 8y agoOn FreeBSD kqueue does work for normal files if you use aio_* and SIGEV_KEVENT.
- the8472 8y agoWell, there's opportunistic nonblocking IO which you can do on your event loop with readaheads (assuming you know your access patterns) and then RWF_NOWAIT and fall back to a thread pool when that fails. Of course that only helps if you're reading from a file into a userspace buffer. If you want to sendfile that doesn't help.
- tatersolid 8y agoHow can you use sendfile and TLS at the same time? Aren’t they incompatible?
- crest 8y agoThere are patches for FreeBSD and Linux sendfile respectively to perform TLS symmetric encryption inside the kernel and use out of band signaling for the key exchange.
- khc 8y agoAuthor of the post (and the engineer who did the work) here. There are ways to do non-blocking disk I/O in *nix (aio/io_submit in linux) but all of which requires you to have an open file descriptor first. Does NT allow you to open a file in an async fashion?
- drewg123 8y agoNetflix kernel engineer here.. We use FreeBSD's async sendfile() and not aio, so it would be a bit harder for us to fix open latency, since we're not using aio. I had not thought about open latency being an issue, that's fascinating. Looking at one of our busy 100G servers w/NVME storage, I see an openat syscall latency of no more than 8ms sampling over a few minute period, with most everything being below 65us. However, the workload is probably different than yours (more longer lived connections, fewer but larger files, fewer opens in general). Eg, we probably don't have nearly the "long tail" issue you do..
- khc 8y agoright I suspect you have way fewer files than we do and everything is in the dentry cache. Pretty sure that most of your files are bigger than 60KB too :-) (which is our p90)
- drewg123 8y agoWow, it is a different world :)
- khc 8y agoour machines also run many different services (the CDN product is just one of them) and isolating I/O from different products is difficult. I'd also love to have NVMe.
- solarengineer 8y agoAny chance you could use dtrace to isolate process specific I/O?
- sbjs 8y agoThought you were emphasizing a lot of your text, until I realized you just have two \*nix in your comment but didn't escape the asterisk!
- hyperman1 8y agohttps://blogs.msdn.microsoft.com/oldnewthing/20180725-00/?p=99335 https://blogs.msdn.microsoft.com/oldnewthing/20180725-00/?p=... This is an old new thing post which demonstrates it is easy tot fall off the happy path on nt. Im not sure the windows world is much better off here.