7 ms·

Curiosity got the better of me recently when I re-read Russinovich's [NT and VMS - The Rest Of The Story](http://windowsitpro.com/windows-client/windows-nt-and-

Curiosity got the better of me recently when I re-read Russinovich's [NT and VMS - The Rest Of The Story](http://windowsitpro.com/windows-client/windows-nt-and-vms-rest-story http://windowsitpro.com/windows-client/windows-nt-and-vms-re...), and I bought a copy of [VMS Internals and Data Structures](http://www.amazon.com/VAX-VMS-Internals-Data-Structures/dp/1555580599 http://www.amazon.com/VAX-VMS-Internals-Data-Structures/dp/1...). Side-by-side, comparing VMS to UNIX, and VMS's approach to a few key areas like I/O, ASTs and tiered interrupt levels are simply just more sophisticated. NT inherited all of that. It was fundamentally superior, as a kernel, to UNIX, from day 1. I haven't met a single person that has understood NT and Linux/UNIX, and still thinks UNIX is superior as far as the kernels go. I have definitely alienated myself the more I've discovered that though, as it's such a wildly unpopular sentiment in open source land. Cutler got a call from Gates in 89, and from 89-93, NT was built. He was 47 at the time, and was one of the lead developers of VMS, which was a rock-solid operating system. In 93, Linus was 22, and starting "implementing enough syscalls until bash ran" as a fun project to work on. Cutler despised the UNIX I/O model. "Getta byte getta byte getta byte byte byte." The I/O request packet approach to I/O (and tiered interrupts) is one of the key reasons behind NT's superiority. And once you've grok'd things like APCs and structured exception handling, signals just seem absolutely ghastly in comparison.

jen20 10y ago
I've never met a single person who understood what they were talking about and referred to a "UNIX kernels". It may be true that Linux was once less advanced than NT - this is no longer the case, despite egregious design flaws in things like epoll. It has simply never been true (for example) for the Illumos (nee Solaris) kernel.
- trentnelson 10y ago
  I qualified it as "Linux/UNIX kernel" because I wanted to emphasize the kernel and not userspace. Solaris event ports are good, but they're still ultimately backed by a readiness-oriented I/O model, and can't be used for asynchronous file I/O.
  - niels_olson 10y ago
    Here's a nice graphical comparison of syscalls between Linux and Windows http://www.visualcomplexity.com/vc/project.cfm?id=392 http://www.visualcomplexity.com/vc/project.cfm?id=392 Are you saying the Windows flow looks like spaghetti only because the software tested software (Apache) wasn't designed for Windows?
    - trentnelson 10y ago
      Heh, 10 years old, original link doesn't work, image is tiny. And it sounds like they were comparing Linux and Apache to IIS and Windows. It's hard to evaluate this in any way more than "yeah that's a cute spaghetti diagram". If I wanted to drag Linux through the mud visually I'd depict how much time every socket I/O op spends in vfs/fsync stuff. (i.e. you can depict anything to make your point)
      - niels_olson 10y ago
        That might be a cool diagram to show! I honestly know nothing about kernel programming, just happened to come across that a long time ago and bookmarked it.
    - jsmeaton 10y ago
      The second image is of IIS running on Windows, not Apache. Different software on different OSes. Regardless, it doesn't seem like parent is making the argument that the NT kernel isn't complicated - just that it is superior.
      - bitwize 10y ago
        It is more complicated because -- surprise -- I/O usage patterns do not fit in one neat little box, and you have to provide special handling of special cases that do occur frequently in the real world.
  - binarycrusader 10y ago
    Solaris event ports most certainly can be and are used for async I/O. I'm not sure how you can claim otherwise: https://blogs.oracle.com/dap/entry/libevent_and_solaris_event_ports https://blogs.oracle.com/dap/entry/libevent_and_solaris_even... https://blogs.oracle.com/praks/entry/file_events_notification https://blogs.oracle.com/praks/entry/file_events_notificatio... And Solaris, (unlike Linux historically at least), supports async I/O on both files and sockets. Linux (historically) only supported it for sockets. I have no idea if Linux generally supports async I/O for files at this point.
- ambrop7 10y ago
  Which design faults do you think epoll specifically has? I know there are lots of file descriptors not usable with epoll or rather async i/o in general and that sucks (e.g. regular files). For networking, I find epoll/sockets nicer to work with than Windows' IOCP, because with IOCP you need to keep your buffers around until the kernel deems your operation complete. I think you have 3 options: 1) Design the whole application to manage buffers like IOCP likes (this propagates to client code because now e.g. they need to have their ring buffer reference-counted). 2) You handle it transparently in the socket wrapper code by using an intermediate buffer and expose a simple read()/write() interface which doesn't require the user to keep a buffer around when they don't want the socket anymore. 3) You handle it by synchronously waiting for I/O to be cancelled after using CancelIo. This sounds risky with potential to lock up the application for an unknown amount of time. It's also non-trivial because in that time IOCP will give you completion results for unrelated I/Os which you will need to buffer and process later. On the other hand, with Linux such issues don't exist by design, because data is only ever copied in read/write calls which return immediately (in non-blocking mode).
  - geocar 10y ago
    The cool thing is that by gifting buffers to the kernel as policy, the gets to make that choice. In Linux, you don't, and userspace and kernelspace both end up doing unnecessary copying, and it means with shared buffers, something might poll and not-block, but actually block by the time you get around to using the buffer. This is annoying, and it generally means you need more than two system calls on average for every IO operation in performance servers. As a rule, you can generally detect "design faults" by the number of competing and overlapping designs (select, poll, epoll, kevent, /dev/poll, aio, sigio, etc, etc, etc). I personally would have preferred a more fleshed out SIGIO model, but we got what we got... One specific fault of epoll (compared to near-relatives) is that the epoll_data_t is very small. In Kqueue you can store both the file descriptor with activity (ident) as well as a few bytes of user data. As a result, people use a heap pointer which causes an extra stall to memory. Memory is so slow...
    - ambrop7 10y ago
      > something might poll and not-block, but actually block by the time you get around to using the buffer On Linux if a socket is set to non-blocking it will not block. I don't really understand your point with shared buffers. You wouldn't typically share a TCP socket since that would result in unpredictable splitting/joining of data. > In Linux, you don't, and userspace and kernelspace both end up doing unnecessary copying I'm not so sure the Linux design where copies are done in syscalls must be inherently less efficient. I'm pretty sure with either design, you generally need at least one memcpy - for RX, from the in-kernel RX buffers to the user memory, and for TX, from user memory to the in-kernel TX buffers. I think getting rid of either copy is extremely hard and would need extremely smart hardware, especially the RX copy (because the Ethernet hardware would need to analyze the packet and figure out where the final destination of the data is!). Getting rid of TX copy might be easier but still hard because it'd need complex DMA support on the Ethernet card that could access potentially unaligned memory addresses. On the other hand, I also don't think you need more than one copy, if you design the network stack with that in mind. > you need more than two system calls on average for every IO operation in performance servers. True but it's not obvious that this is a performance bottleneck. Consider that a single epoll wait can return many ready sockets. I think theoretically it would hurt latency rather than throughput. > As a rule, you can generally detect "design faults" by the number of competing and overlapping designs. On Linux, I think for sockets, there are only: blocking, select, poll, epoll. And the latter three are just different ways to do the same thing. On Windows, it's much more complicated - see this list of different methods to use sockets (my own answer): http://stackoverflow.com/questions/11830839/when-using-iocp-should-i-set-wsaoverlappeds-hevent-to-null-or-to-a-valid-handl/11831450#11831450 http://stackoverflow.com/questions/11830839/when-using-iocp-... > One specific fault of epoll (compared to near-relatives) is that the epoll_data_t is very small. Pretty much universally when you're dealing with a socket, you have some nontrivial amount of data associated with it that you will need to access when it's ready for I/O, typically a struct which at least holds the fd number. Naturally you put a pointer to such a struct into the epoll_data_t. I don't see how one could do it more efficiently outside of very specialized cases.
adamnemecek 10y ago
Can I ask what else is on your reading list? I ended up buying the VMS Internals book. Also do you have an opinion on BeOS?
- trentnelson 10y ago
  I was fascinated by BeOS in the late 90s when I had a lot of enthusiasm (and little clue). All their threading claims just sounded so cool. I was also really into FreeBSD from around 2.2.5 so I got to see how all the SMPng stuff (and kqueue!) evolved, as well as all the different threading models people in UNIX land were trying (1:1, 1:m, m:n). NT solves it properly. Efficient multithreading support and I/O (especially asynchronous I/O) are just so intrinsically related. Trying to bend UNIX processes and IPC and signals and synchronous I/O into an efficient threading implementation is just trying to fit a square peg in a round hole in my opinion. As for reading list... I've bought so many old books lately. Here's my "makes the short list" bookshelf: http://imgur.com/DfTUVQx http://imgur.com/DfTUVQx And the more ridiculous one that I use as a cover page on my resume: http://imgur.com/0u9OZcN http://imgur.com/0u9OZcN What things in particular are you interested in?
  - adamnemecek 10y ago
    Lol, I've been saying something like this for some time. Thanks for the answer. I guess what I'm interested is somewhat obscure/historical operating systems and also HW that are in some way superior to currently popular solutions. The more comparative the better. Also your reading list has quite a few Oracle SQL entries so I'm guessing it's your preferred DB of choice. What features are you using that aren't available in MySQL or Postgres?
    - trentnelson 10y ago
      As far as commercial vendors go I really preferred SQL Server from about version 2000 onwards. I got into Oracle again recently for a consulting project with a finance client (who basically have unlimited Oracle licenses) and really quite enjoyed it since the Oracle 7/8 days. You can do some phenomenally sophisticated things... I extensively leveraged things like partitioning, parallel execution (dbms_parallel_execute!), lots of PL/SQL using the pipelined table cursor stuff, data mining stuff (dbms_frequent_itemset!), index-organized tables, and my god, bitmap indexes were a godsend, direct insert tricks for bulk data loading, external tables were fantastic (you can wrap a .csv in an external table and interact with it in parallel just like any other table -- great for ingesting large amounts of janky .csv data from other parts of the business). The parallel execution and robust partitioning options were probably the most critical pieces that have no particularly good counterpart in open source land.
1024core 10y ago
The joke used to be: VMS++ --> WNT
- CamperBob2 10y ago
  Yep, that'd be the spiritual predecessor to Bing Is Not Google.
  - rmu09 10y ago
    First there was HAL, superior to IBM.
filereaper 10y ago
Since we're going into the history of Windows NT, VMS and Dave Cutler. I'd like to highlight this classic book on the history of all three of the above[1] It follows the same line of narrative as The Soul of a New Machine [1] https://www.amazon.com/Showstopper-Breakneck-Windows-Generation-Microsoft-ebook/dp/B00J5X5E9U https://www.amazon.com/Showstopper-Breakneck-Windows-Generat...
- trentnelson 10y ago
  I freaking love Showstopper! Such a great book. It's probably the most information available on David Cutler anywhere. The author also e-mailed me saying thanks when I tweeted him how much I liked the book, which I thought was super nice.

jen20 10y ago

I've never met a single person who understood what they were talking about and referred to a "UNIX kernels". It may be true that Linux was once less advanced than NT - this is no longer the case, despite egregious design flaws in things like epoll. It has simply never been true (for example) for the Illumos (nee Solaris) kernel.

trentnelson 10y ago
I qualified it as "Linux/UNIX kernel" because I wanted to emphasize the kernel and not userspace. Solaris event ports are good, but they're still ultimately backed by a readiness-oriented I/O model, and can't be used for asynchronous file I/O.
- niels_olson 10y ago
  Here's a nice graphical comparison of syscalls between Linux and Windows http://www.visualcomplexity.com/vc/project.cfm?id=392 http://www.visualcomplexity.com/vc/project.cfm?id=392 Are you saying the Windows flow looks like spaghetti only because the software tested software (Apache) wasn't designed for Windows?
  - trentnelson 10y ago
    Heh, 10 years old, original link doesn't work, image is tiny. And it sounds like they were comparing Linux and Apache to IIS and Windows. It's hard to evaluate this in any way more than "yeah that's a cute spaghetti diagram". If I wanted to drag Linux through the mud visually I'd depict how much time every socket I/O op spends in vfs/fsync stuff. (i.e. you can depict anything to make your point)
    - niels_olson 10y ago
      That might be a cool diagram to show! I honestly know nothing about kernel programming, just happened to come across that a long time ago and bookmarked it.
  - jsmeaton 10y ago
    The second image is of IIS running on Windows, not Apache. Different software on different OSes. Regardless, it doesn't seem like parent is making the argument that the NT kernel isn't complicated - just that it is superior.
    - bitwize 10y ago
      It is more complicated because -- surprise -- I/O usage patterns do not fit in one neat little box, and you have to provide special handling of special cases that do occur frequently in the real world.
- binarycrusader 10y ago
  Solaris event ports most certainly can be and are used for async I/O. I'm not sure how you can claim otherwise: https://blogs.oracle.com/dap/entry/libevent_and_solaris_event_ports https://blogs.oracle.com/dap/entry/libevent_and_solaris_even... https://blogs.oracle.com/praks/entry/file_events_notification https://blogs.oracle.com/praks/entry/file_events_notificatio... And Solaris, (unlike Linux historically at least), supports async I/O on both files and sockets. Linux (historically) only supported it for sockets. I have no idea if Linux generally supports async I/O for files at this point.
ambrop7 10y ago
Which design faults do you think epoll specifically has? I know there are lots of file descriptors not usable with epoll or rather async i/o in general and that sucks (e.g. regular files). For networking, I find epoll/sockets nicer to work with than Windows' IOCP, because with IOCP you need to keep your buffers around until the kernel deems your operation complete. I think you have 3 options: 1) Design the whole application to manage buffers like IOCP likes (this propagates to client code because now e.g. they need to have their ring buffer reference-counted). 2) You handle it transparently in the socket wrapper code by using an intermediate buffer and expose a simple read()/write() interface which doesn't require the user to keep a buffer around when they don't want the socket anymore. 3) You handle it by synchronously waiting for I/O to be cancelled after using CancelIo. This sounds risky with potential to lock up the application for an unknown amount of time. It's also non-trivial because in that time IOCP will give you completion results for unrelated I/Os which you will need to buffer and process later. On the other hand, with Linux such issues don't exist by design, because data is only ever copied in read/write calls which return immediately (in non-blocking mode).
- geocar 10y ago
  The cool thing is that by gifting buffers to the kernel as policy, the gets to make that choice. In Linux, you don't, and userspace and kernelspace both end up doing unnecessary copying, and it means with shared buffers, something might poll and not-block, but actually block by the time you get around to using the buffer. This is annoying, and it generally means you need more than two system calls on average for every IO operation in performance servers. As a rule, you can generally detect "design faults" by the number of competing and overlapping designs (select, poll, epoll, kevent, /dev/poll, aio, sigio, etc, etc, etc). I personally would have preferred a more fleshed out SIGIO model, but we got what we got... One specific fault of epoll (compared to near-relatives) is that the epoll_data_t is very small. In Kqueue you can store both the file descriptor with activity (ident) as well as a few bytes of user data. As a result, people use a heap pointer which causes an extra stall to memory. Memory is so slow...
  - ambrop7 10y ago
    > something might poll and not-block, but actually block by the time you get around to using the buffer On Linux if a socket is set to non-blocking it will not block. I don't really understand your point with shared buffers. You wouldn't typically share a TCP socket since that would result in unpredictable splitting/joining of data. > In Linux, you don't, and userspace and kernelspace both end up doing unnecessary copying I'm not so sure the Linux design where copies are done in syscalls must be inherently less efficient. I'm pretty sure with either design, you generally need at least one memcpy - for RX, from the in-kernel RX buffers to the user memory, and for TX, from user memory to the in-kernel TX buffers. I think getting rid of either copy is extremely hard and would need extremely smart hardware, especially the RX copy (because the Ethernet hardware would need to analyze the packet and figure out where the final destination of the data is!). Getting rid of TX copy might be easier but still hard because it'd need complex DMA support on the Ethernet card that could access potentially unaligned memory addresses. On the other hand, I also don't think you need more than one copy, if you design the network stack with that in mind. > you need more than two system calls on average for every IO operation in performance servers. True but it's not obvious that this is a performance bottleneck. Consider that a single epoll wait can return many ready sockets. I think theoretically it would hurt latency rather than throughput. > As a rule, you can generally detect "design faults" by the number of competing and overlapping designs. On Linux, I think for sockets, there are only: blocking, select, poll, epoll. And the latter three are just different ways to do the same thing. On Windows, it's much more complicated - see this list of different methods to use sockets (my own answer): http://stackoverflow.com/questions/11830839/when-using-iocp-should-i-set-wsaoverlappeds-hevent-to-null-or-to-a-valid-handl/11831450#11831450 http://stackoverflow.com/questions/11830839/when-using-iocp-... > One specific fault of epoll (compared to near-relatives) is that the epoll_data_t is very small. Pretty much universally when you're dealing with a socket, you have some nontrivial amount of data associated with it that you will need to access when it's ready for I/O, typically a struct which at least holds the fd number. Naturally you put a pointer to such a struct into the epoll_data_t. I don't see how one could do it more efficiently outside of very specialized cases.

adamnemecek 10y ago

Can I ask what else is on your reading list? I ended up buying the VMS Internals book. Also do you have an opinion on BeOS?

trentnelson 10y ago
I was fascinated by BeOS in the late 90s when I had a lot of enthusiasm (and little clue). All their threading claims just sounded so cool. I was also really into FreeBSD from around 2.2.5 so I got to see how all the SMPng stuff (and kqueue!) evolved, as well as all the different threading models people in UNIX land were trying (1:1, 1:m, m:n). NT solves it properly. Efficient multithreading support and I/O (especially asynchronous I/O) are just so intrinsically related. Trying to bend UNIX processes and IPC and signals and synchronous I/O into an efficient threading implementation is just trying to fit a square peg in a round hole in my opinion. As for reading list... I've bought so many old books lately. Here's my "makes the short list" bookshelf: http://imgur.com/DfTUVQx http://imgur.com/DfTUVQx And the more ridiculous one that I use as a cover page on my resume: http://imgur.com/0u9OZcN http://imgur.com/0u9OZcN What things in particular are you interested in?
- adamnemecek 10y ago
  Lol, I've been saying something like this for some time. Thanks for the answer. I guess what I'm interested is somewhat obscure/historical operating systems and also HW that are in some way superior to currently popular solutions. The more comparative the better. Also your reading list has quite a few Oracle SQL entries so I'm guessing it's your preferred DB of choice. What features are you using that aren't available in MySQL or Postgres?
  - trentnelson 10y ago
    As far as commercial vendors go I really preferred SQL Server from about version 2000 onwards. I got into Oracle again recently for a consulting project with a finance client (who basically have unlimited Oracle licenses) and really quite enjoyed it since the Oracle 7/8 days. You can do some phenomenally sophisticated things... I extensively leveraged things like partitioning, parallel execution (dbms_parallel_execute!), lots of PL/SQL using the pipelined table cursor stuff, data mining stuff (dbms_frequent_itemset!), index-organized tables, and my god, bitmap indexes were a godsend, direct insert tricks for bulk data loading, external tables were fantastic (you can wrap a .csv in an external table and interact with it in parallel just like any other table -- great for ingesting large amounts of janky .csv data from other parts of the business). The parallel execution and robust partitioning options were probably the most critical pieces that have no particularly good counterpart in open source land.

1024core 10y ago

The joke used to be: VMS++ --> WNT

CamperBob2 10y ago
Yep, that'd be the spiritual predecessor to Bing Is Not Google.
- rmu09 10y ago
  First there was HAL, superior to IBM.

filereaper 10y ago

Since we're going into the history of Windows NT, VMS and Dave Cutler. I'd like to highlight this classic book on the history of all three of the above[1] It follows the same line of narrative as The Soul of a New Machine [1] https://www.amazon.com/Showstopper-Breakneck-Windows-Generation-Microsoft-ebook/dp/B00J5X5E9U https://www.amazon.com/Showstopper-Breakneck-Windows-Generat...

trentnelson 10y ago
I freaking love Showstopper! Such a great book. It's probably the most information available on David Cutler anywhere. The author also e-mailed me saying thanks when I tweeted him how much I liked the book, which I thought was super nice.