44 ms·

\Device\Afd, or, the Deal with the Devil that makes async Rust work on Windows

100 points by zinekeller 4y ago

yccs27 4y ago
I love it when hacky programming is compared to witchcraft, because it describes the feeling of reading and writing it. Reminds me of the "technical interview" [0] from yesterday‘s front page. [0] https://news.ycombinator.com/item?id=35120084 https://news.ycombinator.com/item?id=35120084
- doublerabbit 4y ago
  I can name a few through my journeys. Perl is a temple. You construct some ritualistic temple of a project, leave it be for years and will stand happily without any issues. Come back many months later, add a single comment; total Indiana Jones scenario. TCL is voodoo, it does stuff that shouldn't work but does awesomely. How it does it is just voodoo. Ruby is just magic. You create a thing, change one thing and the next thing you know you've suddenly got two features for the price of one. PHP is a sheep.
  - tracker1 4y ago
    PHP is a sheep with one droopy eye and one of the ears actually looks like it belongs to a donkey, but the fur/wool is at first glance, well-conditioned.
dblohm7 4y ago
Good god. I like nothing about this piece. I hate the claim that Windows networking is "duct-taped" in. I hate that they're using undocumented stuff. I hate that they're trying to shoehorn Windows into a Unix-y model; in my experience this is often symptomatic of failing to consider all platforms when originally designing the higher-level abstractions. Just blah. > To make a long story that I don’t fully understand short, you need to have buffers that are pinned in memory and aren’t touched as long as the operation is in progress. I don't understand why this is so confusing. > I doubt that Microsoft is really rushing to make it so that you can’t run Javascript on Windows anymore. Nah, but don't be surprised if suddenly a bunch of apps receive compatibility shims.
- pornel 4y ago
  Rust sometimes is very Linux-centric. This problem has been considered when async was in the draft stage, and shrugged off, because it's just Windows. macOS in Rust is also treated as "it's just a weird Linux, right?".
  - dblohm7 4y ago
    That's how things have always been done at Mozilla (I spent several years there dealing with Windows-specific stuff for Firefox)... I kinda wonder if Rust inherited a lot of that attitude from there.
- Dylan16807 4y ago
  > I don't understand why this is so confusing. Pinning a buffer is easy. Dealing with ownership ramifications in all situations is confusing.
  - dblohm7 4y ago
    That's fair, but how is that any different than async I/O on any other OS?
    - Dylan16807 4y ago
      My understanding is the API they want notifies when data is ready, then you make a buffer and fill it immediately. In that situation a significantly smaller set of errors is possible and you don't have to figure out disposal of a pinned and possibly-in-use buffer if you decide to cancel.
- fxtentacle 4y ago
  I belive the issue here is that the author very much likes Rust and is not willing to openly admit that Rust's needs Windows-specific features. I found it odd that even the examples for low-level C APIs were presented through Rust wrappers. "the overarching idea is that IOCP doesn’t gel well with not only the Future model, but also with Rust’s borrow checker as a whole" comes pretty close, though, but it's still lacking the insight that it's an issue in Rust's language design and not in Windows. Also, I've heard a lot of praise about Microsoft's networking integration which made things like mounting network drives possible and reliable as early as Windows 2000. But of course, if you refuse to use the recommended APIs, you'll suffer for it. I wonder if the author would describe it in similar detail, if they open a Linux device like /dev/video0 and send a V4L syscall to it. Because that's the equivalent to using the Afd device. That said, I enjoyed their word choices and writing style.
josephcsible 4y ago
I can't believe Windows is more than 20 years behind Linux in offering an official API for such an important feature.
- anaisbetts 4y ago
  Windows does offer an official API for this, as the article states. Unfortunately, everyone wants an API that works exactly like epoll, and IOCP just Doesn't Work Like That. It's actually way closer to how most people would expect async I/O to work, were they to design it from scratch without the well being poisoned by knowledge of epoll. However, every *nix / macOS person writes their epoll code, then finds out that it Doesn't Work That Way on Windows, so this gives them back their epoll-like API.
- zdimension 4y ago
  From a Windows development point of view, Linux is way behind. Async I/O has been at the core of the NT kernel since the very beginning (since VMS, even), whereas on Linux (and Unix) everything has been designed with blocking I/O in mind. IOCP match the way people usually think about asynchronous processing, but people have just become used to the weird way poll and epoll do things. It's like saying Windows is behind Linux because it doesn't support Unix permissions.
anaisbetts 4y ago
A story I heard while working on Kernel and Drivers at MS, is that this acronym originally stood for "Another Fucking Driver", named by Dave Cutler once a kernel engineer told him that it was necessary and asked him for a good name. Obviously before they shipped, they needed to come up with a more Palatable name, so "Auxiliary function driver" was the name they came up with.

100 points by zinekeller 4y ago

I love it when hacky programming is compared to witchcraft, because it describes the feeling of reading and writing it. Reminds me of the "technical interview" [0] from yesterday‘s front page. [0] https://news.ycombinator.com/item?id=35120084 https://news.ycombinator.com/item?id=35120084

doublerabbit 4y ago
I can name a few through my journeys. Perl is a temple. You construct some ritualistic temple of a project, leave it be for years and will stand happily without any issues. Come back many months later, add a single comment; total Indiana Jones scenario. TCL is voodoo, it does stuff that shouldn't work but does awesomely. How it does it is just voodoo. Ruby is just magic. You create a thing, change one thing and the next thing you know you've suddenly got two features for the price of one. PHP is a sheep.
- tracker1 4y ago
  PHP is a sheep with one droopy eye and one of the ears actually looks like it belongs to a donkey, but the fur/wool is at first glance, well-conditioned.

dblohm7 4y ago

Good god. I like nothing about this piece. I hate the claim that Windows networking is "duct-taped" in. I hate that they're using undocumented stuff. I hate that they're trying to shoehorn Windows into a Unix-y model; in my experience this is often symptomatic of failing to consider all platforms when originally designing the higher-level abstractions. Just blah. > To make a long story that I don’t fully understand short, you need to have buffers that are pinned in memory and aren’t touched as long as the operation is in progress. I don't understand why this is so confusing. > I doubt that Microsoft is really rushing to make it so that you can’t run Javascript on Windows anymore. Nah, but don't be surprised if suddenly a bunch of apps receive compatibility shims.

pornel 4y ago
Rust sometimes is very Linux-centric. This problem has been considered when async was in the draft stage, and shrugged off, because it's just Windows. macOS in Rust is also treated as "it's just a weird Linux, right?".
- dblohm7 4y ago
  That's how things have always been done at Mozilla (I spent several years there dealing with Windows-specific stuff for Firefox)... I kinda wonder if Rust inherited a lot of that attitude from there.
Dylan16807 4y ago
> I don't understand why this is so confusing. Pinning a buffer is easy. Dealing with ownership ramifications in all situations is confusing.
- dblohm7 4y ago
  That's fair, but how is that any different than async I/O on any other OS?
  - Dylan16807 4y ago
    My understanding is the API they want notifies when data is ready, then you make a buffer and fill it immediately. In that situation a significantly smaller set of errors is possible and you don't have to figure out disposal of a pinned and possibly-in-use buffer if you decide to cancel.
fxtentacle 4y ago
I belive the issue here is that the author very much likes Rust and is not willing to openly admit that Rust's needs Windows-specific features. I found it odd that even the examples for low-level C APIs were presented through Rust wrappers. "the overarching idea is that IOCP doesn’t gel well with not only the Future model, but also with Rust’s borrow checker as a whole" comes pretty close, though, but it's still lacking the insight that it's an issue in Rust's language design and not in Windows. Also, I've heard a lot of praise about Microsoft's networking integration which made things like mounting network drives possible and reliable as early as Windows 2000. But of course, if you refuse to use the recommended APIs, you'll suffer for it. I wonder if the author would describe it in similar detail, if they open a Linux device like /dev/video0 and send a V4L syscall to it. Because that's the equivalent to using the Afd device. That said, I enjoyed their word choices and writing style.

josephcsible 4y ago

I can't believe Windows is more than 20 years behind Linux in offering an official API for such an important feature.

anaisbetts 4y ago
Windows does offer an official API for this, as the article states. Unfortunately, everyone wants an API that works exactly like epoll, and IOCP just Doesn't Work Like That. It's actually way closer to how most people would expect async I/O to work, were they to design it from scratch without the well being poisoned by knowledge of epoll. However, every *nix / macOS person writes their epoll code, then finds out that it Doesn't Work That Way on Windows, so this gives them back their epoll-like API.
zdimension 4y ago
From a Windows development point of view, Linux is way behind. Async I/O has been at the core of the NT kernel since the very beginning (since VMS, even), whereas on Linux (and Unix) everything has been designed with blocking I/O in mind. IOCP match the way people usually think about asynchronous processing, but people have just become used to the weird way poll and epoll do things. It's like saying Windows is behind Linux because it doesn't support Unix permissions.

anaisbetts 4y ago

A story I heard while working on Kernel and Drivers at MS, is that this acronym originally stood for "Another Fucking Driver", named by Dave Cutler once a kernel engineer told him that it was necessary and asked him for a good name. Obviously before they shipped, they needed to come up with a more Palatable name, so "Auxiliary function driver" was the name they came up with.