5 ms·

Original author here. Back in September, I got the recommendation to read the "Showstopper!: The Breakneck Race to Create Windows NT and the Next Generation at

by jmmv 3y ago

Original author here. Back in September, I got the recommendation to read the "Showstopper!: The Breakneck Race to Create Windows NT and the Next Generation at Microsoft" book from some other comment thread here in HN. I don't exactly remember which thread that was though... but learning about the history of Windows NT sounded interesting, so I bought the book right away and have been slowly reading through it over the last three months. The story in the book seemed so exciting (and messed up) that I ended up writing a "book review" with commentary on various topics that caught my attention. The details might not be super-precise, but they are hopefully enough to paint the right picture in its context. I thought you might enjoy it due to where I got the suggestion from!

HankB99 3y ago
Is the book labelled as fiction? I understand that early versions of NT emitted error messages identifying itself as OS/2. Did the book explain when NT became "brand new OS from scratch". I think there's been a bit of revisionist history WRT the roots of NT. MS wants to make it something "we did without their help." I didn't read your entire blog so apologies if you addressed this.
- jmmv 3y ago
  My understanding is that the original collaboration between Microsoft and IBM was around certain parts of OS/2, not the whole system, so it is plausible that Microsoft worked simultaneously on OS/2 and NT as the book describes. Note that NT is heavily based on the work previously done at Digital on VMS (by people from the same team), so there are some pre-existing foundations there, at least in terms of design principles and ideas. https://en.wikipedia.org/wiki/OS/2 https://en.wikipedia.org/wiki/OS/2 provides a recap of the history but I cannot find details on exactly who did what in that page. And my memory from what I read in the book is fuzzy at this point...
- dboreham 3y ago
  It's been a few decades since I read the book, but note it was published just after NT was released, when everyone knew that it had been called OS/2-NT for a while. So probably no wool pulling going on there. Note also that this was a naming move. NT was never based on OS/2. otoh in my experience (I've worked for two companies that had books written about them), book authors tend to a) focus on drama and b) get their information from a small number of senior participants.
- trealira 3y ago
  The book doesn't shy away from describing the collaboration between Microsoft and IBM. Also, the book depicts Microsoft knowingly screwing over IBM in their collaboration. So Gates sided with Maritz. NT would assume the Windows “personality,” by assuming those of its attributes that were visible to the customer. NT also would be redesigned to run applications written specifically for Windows. But Gates added a wrinkle to his position. Given the depth of IBM’s attachment to OS/2, Gates reasoned, abandoning the program altogether would only trigger a nasty split with IBM. Why not maintain a pretense of cooperation by convincing IBM of the merits of an operating system that hosted both Windows and OS/2? NT could be designed to do this, Gates reasoned. If IBM accepted this, Microsoft could continue to describe Cutler’s program as “NT OS/2.” Even though it contradicted his rhetoric about the need for a single software standard, Gates was prepared to tell IBM: “There’s no reason to think two [program personalities] can’t be of equal importance” to customers. This was a bold stroke. By denying reality, Gates’s tactic would freeze IBM long enough for Microsoft to install Windows as the standard for computing. Gates would peddle this apparent compromise to IBM and rival software makers, while Maritz and Cutler’s team pursued the real objective internally. ... Cutler asked about Ballmer’s promise that NT would still sport an OS/2 personality. Not to worry, Ballmer assured him, someone else could take care of that. Indeed, OS/2 was now so unimportant to Microsoft that Ballmer had assigned a tiny team in Israel to create an OS/2 variant of NT. The Microsofties hated OS/2 so much they couldn’t even bear to have someone working on it in the same continent.
- voidfunc 3y ago
  Dave Cutler has elaborated on a lot of this stuff in some recent YT interviews that are excellent: https://www.youtube.com/watch?v=9K3eMzF6x28 https://www.youtube.com/watch?v=9K3eMzF6x28 Part of it is Cutler really really hated working with IBM... it starts around 3:00 minute mark.
  - canucker2016 3y ago
    Several MSFT employees I talked to mentioned that they didn't like working with IBM - too much of a culture clash from what I could gather.
    - EvanAnderson 3y ago
      An earlier comment mentioned Gordon Letwin[0] (lead architect of OS/2 at MSFT), and that led me down the rabbit hole to an "infamous" USENET post he made re: OS/2[1]. It sounds like there was acrimony all around: Another problem that came up here was that IBM didn't want us to use the windows API for the graphical environment under OS/2. Many key folks inside IBM had always hated Windows. IBM had this crazy thing called TopView, it was a character oriented windowing scheme and not very good. Bill Gates, myself, and some other folks made several trips to Boca Raton to try to explain to those guys why a character oriented windowing scheme was obsolete before it was even written, but to no avail. One of IBM's most major problems is that although their top guys may be smart, they aren't techically savvy. [0] https://en.wikipedia.org/wiki/Gordon_Letwin https://en.wikipedia.org/wiki/Gordon_Letwin [1] http://gunkies.org/wiki/Gordon_Letwin_OS/2_usenet_post http://gunkies.org/wiki/Gordon_Letwin_OS/2_usenet_post
      - canucker2016 3y ago
        You cut off the best bits of Letwin's rant: "...although their top guys may be smart, they aren't techically savvy. And their low level guys are often neither. IBM doesn't promote on the basis of your skills and ability; they promote on the basis of seniority and other secondary factors. So the guy who makes these decisions often doesn't know what he's doing. And he doesn't know that he doesn't know, because his peers are equally butt-ignorant too. So these guys can never figure out how other folks, including but not limited to Microsoft, keep doing better! Must be dumb luck, they think. I always agreed that it was dumb luck. If you catch my drift... :-) "
- chasil 3y ago
  The NT kernel, a rewrite in C of the VMS kernel (which itself was written in assembler), presented three APIs to userland: Win32, POSIX, and OS/2. It seems reasonable that the OS/2 emulator/API/userland inside of early Windows would identify itself as truly OS/2, which it was likely seen to be internally. The VMS kernel itself would also emulate programs written for the earlier RSX-11 operating system that ran on the PDP-11, so there is a history of emulation between these operating systems. "Broad software compatibility was initially achieved with support for several API 'personalities', including Windows API, POSIX, and OS/2 APIs – the latter two were phased out starting with Windows XP... "Mark Russinovich wrote in 1998 that while creating Windows NT, Microsoft developers rewrote VMS in C. Although they added the Win32 API, NTFS file system, GUI, and backwards compatibility with DOS, OS/2, and Win16, DEC engineers almost immediately noticed the two operating systems' internal similarities; parts of VAX/VMS Internals and Data Structures, published by Digital Press, accurately describe Windows NT internals using VMS terms. Instead of a lawsuit, Microsoft agreed to pay DEC $65–100 million, help market VMS, train Digital personnel on Windows NT, and continue Windows NT support for the DEC Alpha." https://en.wikipedia.org/wiki/Windows_NT https://en.wikipedia.org/wiki/Windows_NT Information on the Windows POSIX subsystem: https://en.wikipedia.org/wiki/Microsoft_POSIX_subsystem https://en.wikipedia.org/wiki/Microsoft_POSIX_subsystem
  - nrr 3y ago
    "The NT kernel, a rewrite in C of the VMS kernel (which itself was written in assembler), presented three APIs to userland: Win32, POSIX, and OS/2." Not quite. I'd rephrase this as "NT presented three userland personalities—the CSR (Win32), OS/2, and POSIX—through a composition of kernel primitives implemented as userspace subsystems." The core binary image ntoskrnl.exe only exposes the primitives needed to get the CSR, OS2, and PSX subsystems started, and those (mostly, save for some code that lives in win32k.sys) run in userspace. They more or less act as translators between the native NT ABI/API and the respective subsystems' ABIs/APIs. PSX in particular did a lot of its own PE image mapping and parsing (something that you'd perhaps think would be done in kernel space) so that it could defer to the other subsystems to run foreign code among other things.
    - 3y ago
- mschaef 3y ago
  > Is the book labelled as fiction? > I understand that early versions of NT emitted error messages identifying itself as OS/2. Did the book explain when NT became "brand new OS from scratch". Windows NT started out as Microsoft's effort to do OS/2 NT. There was a 32-bit Microsoft OS/2 2.0 (distinct from IBM OS/2 2.0), and OS/2 NT was to be OS/2 3.0. OS/2 NT wasn't rebranded Windows NT until Windows 3.0 eliminated OS/2 from relevance, and with it, the Microsoft/IBM partnership. That's when IBM introduced their own (very nice) OS/2 2.0 and started a years long effort to eliminate Microsoft code from the OS/2 codebase (and the Microsoft copyright message from all over the UI)
borissk 3y ago
The part about NTFS is slightly incorrect. It wasn't really a new file system. It was a copy of High Performance File System (HPFS) for OS/2, that was originally implemented in assembler by Microsoft's best programmer Gordon Letwin. NTFS and HPFS even used the same partition id.
- jmmv 3y ago
  Oh thanks. I must have forgotten about that; if you have read the book, do you know if this was explained? Will need to read more and try to update the text!
- jvert 3y ago
  NTFS was definitely a new filesystem and supported many capabilities not in HPFS. Windows NT also shipped with HPFS support which was a complete reimplementation of the HPFS filesystem which shipped with OS/2.
- chasil 3y ago
  The DEC people added some stuff. "The MFT structure supports algorithms which minimize disk fragmentation... While this strongly resembles the W_FID of Files-11, other NTFS structures radically differ." https://en.wikipedia.org/wiki/NTFS https://en.wikipedia.org/wiki/NTFS "ODS-2 is the standard VMS file system, and remains the most common file system for system disks (the disk on which the operating system is installed)... NTFS - Has many structural and metadata similarities with Files-11 and is almost certainly conceptually derived from it." https://en.wikipedia.org/wiki/Files-11 https://en.wikipedia.org/wiki/Files-11
- AshamedCaptain 3y ago
  They use the same partition id because that was the partition id used for IFSes in OS/2 (and therefore all non-DOS filesystems). This should not be taken as a sign of their similarities (or differences).
- kimixa 3y ago
  Also some of the features listed (Compression, ACLs, volume sizes) were incrementally added over years and not available at first release, which somewhat invalidates the comparisons with other filesystems "Of the era". The NTFS of today is rather different to the NTFS of 1993.
chasil 3y ago
I have just been dealing with problems of VMS this week, and I have Dave Cutler to thank. We have a temporary emulated VAX running VMS 7 and Rdb [a SQL database now owned by Oracle]. After several tries to move the Rdb backup files through our QA firewall with FTP, we finally had to use Multinet SFTP2 to push the backup instead (upping the transfer time from 3 to 40 minutes), which autodetected "VMS mode." There are some secret record blocking/delimiter capabilities in the ODS-2 filesystem used by VMS that an FTP binary transfer will not preserve, and which the Rdb backup/restore process absolutely needs. Thank heavens that UNIX thinks a file is a single stream of bytes. Windows continues the feature experimentation with NTFS "alternate data streams." My, how clever that is [not]. EDIT: from the article, the claim "The open-source BSDs and Linux had just seen the light of day." is not quite correct. Bill Joy was involved in the earliest BSD repacking of v6 Research UNIX, shipped in 1978. "In 1975, Ken Thompson took a sabbatical from Bell Labs and came to Berkeley as a visiting professor. He helped to install Version 6 Unix and started working on a Pascal implementation for the system. Graduate students Chuck Haley and Bill Joy improved Thompson's Pascal and implemented an improved text editor, ex. Other universities became interested in the software at Berkeley, and so in 1977 Joy started compiling the first Berkeley Software Distribution (1BSD), which was released on March 9, 1978." https://en.wikipedia.org/wiki/Berkeley_Software_Distribution https://en.wikipedia.org/wiki/Berkeley_Software_Distribution ...I have also read elsewhere that a conscious decision was made in Windows to never, ever support big-endian architectures (but I am not certain that this is correct). NetBSD and Linux can run big-endian. "Now, I know you will complain that NTFS is slow… but you know what? It almost-certainly is not... NTFS on its own is fine..." XFS appears to trounce NTFS, and it consistently leads the performance scores on tpc.org; until very recently, SQL Server on XFS/Linux was well ahead of Windows.
- Gravityloss 3y ago
  Oh my, that prompted me to read Wikipedia - and we have MUMPS also from VMS. A quite different programming paradigm. The last common ancestor with current mainstream programming is really far back and it looks very odd. New large scale multi billion dollar healthcare systems are still being written in it.
  - 3y ago
deleted 3y ago
[deleted]

by jmmv 3y ago

HankB99 3y ago

Is the book labelled as fiction? I understand that early versions of NT emitted error messages identifying itself as OS/2. Did the book explain when NT became "brand new OS from scratch". I think there's been a bit of revisionist history WRT the roots of NT. MS wants to make it something "we did without their help." I didn't read your entire blog so apologies if you addressed this.

jmmv 3y ago
My understanding is that the original collaboration between Microsoft and IBM was around certain parts of OS/2, not the whole system, so it is plausible that Microsoft worked simultaneously on OS/2 and NT as the book describes. Note that NT is heavily based on the work previously done at Digital on VMS (by people from the same team), so there are some pre-existing foundations there, at least in terms of design principles and ideas. https://en.wikipedia.org/wiki/OS/2 https://en.wikipedia.org/wiki/OS/2 provides a recap of the history but I cannot find details on exactly who did what in that page. And my memory from what I read in the book is fuzzy at this point...
dboreham 3y ago
It's been a few decades since I read the book, but note it was published just after NT was released, when everyone knew that it had been called OS/2-NT for a while. So probably no wool pulling going on there. Note also that this was a naming move. NT was never based on OS/2. otoh in my experience (I've worked for two companies that had books written about them), book authors tend to a) focus on drama and b) get their information from a small number of senior participants.
trealira 3y ago
The book doesn't shy away from describing the collaboration between Microsoft and IBM. Also, the book depicts Microsoft knowingly screwing over IBM in their collaboration. So Gates sided with Maritz. NT would assume the Windows “personality,” by assuming those of its attributes that were visible to the customer. NT also would be redesigned to run applications written specifically for Windows. But Gates added a wrinkle to his position. Given the depth of IBM’s attachment to OS/2, Gates reasoned, abandoning the program altogether would only trigger a nasty split with IBM. Why not maintain a pretense of cooperation by convincing IBM of the merits of an operating system that hosted both Windows and OS/2? NT could be designed to do this, Gates reasoned. If IBM accepted this, Microsoft could continue to describe Cutler’s program as “NT OS/2.” Even though it contradicted his rhetoric about the need for a single software standard, Gates was prepared to tell IBM: “There’s no reason to think two [program personalities] can’t be of equal importance” to customers. This was a bold stroke. By denying reality, Gates’s tactic would freeze IBM long enough for Microsoft to install Windows as the standard for computing. Gates would peddle this apparent compromise to IBM and rival software makers, while Maritz and Cutler’s team pursued the real objective internally. ... Cutler asked about Ballmer’s promise that NT would still sport an OS/2 personality. Not to worry, Ballmer assured him, someone else could take care of that. Indeed, OS/2 was now so unimportant to Microsoft that Ballmer had assigned a tiny team in Israel to create an OS/2 variant of NT. The Microsofties hated OS/2 so much they couldn’t even bear to have someone working on it in the same continent.
voidfunc 3y ago
Dave Cutler has elaborated on a lot of this stuff in some recent YT interviews that are excellent: https://www.youtube.com/watch?v=9K3eMzF6x28 https://www.youtube.com/watch?v=9K3eMzF6x28 Part of it is Cutler really really hated working with IBM... it starts around 3:00 minute mark.
- canucker2016 3y ago
  Several MSFT employees I talked to mentioned that they didn't like working with IBM - too much of a culture clash from what I could gather.
  - EvanAnderson 3y ago
    An earlier comment mentioned Gordon Letwin[0] (lead architect of OS/2 at MSFT), and that led me down the rabbit hole to an "infamous" USENET post he made re: OS/2[1]. It sounds like there was acrimony all around: Another problem that came up here was that IBM didn't want us to use the windows API for the graphical environment under OS/2. Many key folks inside IBM had always hated Windows. IBM had this crazy thing called TopView, it was a character oriented windowing scheme and not very good. Bill Gates, myself, and some other folks made several trips to Boca Raton to try to explain to those guys why a character oriented windowing scheme was obsolete before it was even written, but to no avail. One of IBM's most major problems is that although their top guys may be smart, they aren't techically savvy. [0] https://en.wikipedia.org/wiki/Gordon_Letwin https://en.wikipedia.org/wiki/Gordon_Letwin [1] http://gunkies.org/wiki/Gordon_Letwin_OS/2_usenet_post http://gunkies.org/wiki/Gordon_Letwin_OS/2_usenet_post
    - canucker2016 3y ago
      You cut off the best bits of Letwin's rant: "...although their top guys may be smart, they aren't techically savvy. And their low level guys are often neither. IBM doesn't promote on the basis of your skills and ability; they promote on the basis of seniority and other secondary factors. So the guy who makes these decisions often doesn't know what he's doing. And he doesn't know that he doesn't know, because his peers are equally butt-ignorant too. So these guys can never figure out how other folks, including but not limited to Microsoft, keep doing better! Must be dumb luck, they think. I always agreed that it was dumb luck. If you catch my drift... :-) "
chasil 3y ago
The NT kernel, a rewrite in C of the VMS kernel (which itself was written in assembler), presented three APIs to userland: Win32, POSIX, and OS/2. It seems reasonable that the OS/2 emulator/API/userland inside of early Windows would identify itself as truly OS/2, which it was likely seen to be internally. The VMS kernel itself would also emulate programs written for the earlier RSX-11 operating system that ran on the PDP-11, so there is a history of emulation between these operating systems. "Broad software compatibility was initially achieved with support for several API 'personalities', including Windows API, POSIX, and OS/2 APIs – the latter two were phased out starting with Windows XP... "Mark Russinovich wrote in 1998 that while creating Windows NT, Microsoft developers rewrote VMS in C. Although they added the Win32 API, NTFS file system, GUI, and backwards compatibility with DOS, OS/2, and Win16, DEC engineers almost immediately noticed the two operating systems' internal similarities; parts of VAX/VMS Internals and Data Structures, published by Digital Press, accurately describe Windows NT internals using VMS terms. Instead of a lawsuit, Microsoft agreed to pay DEC $65–100 million, help market VMS, train Digital personnel on Windows NT, and continue Windows NT support for the DEC Alpha." https://en.wikipedia.org/wiki/Windows_NT https://en.wikipedia.org/wiki/Windows_NT Information on the Windows POSIX subsystem: https://en.wikipedia.org/wiki/Microsoft_POSIX_subsystem https://en.wikipedia.org/wiki/Microsoft_POSIX_subsystem
- nrr 3y ago
  "The NT kernel, a rewrite in C of the VMS kernel (which itself was written in assembler), presented three APIs to userland: Win32, POSIX, and OS/2." Not quite. I'd rephrase this as "NT presented three userland personalities—the CSR (Win32), OS/2, and POSIX—through a composition of kernel primitives implemented as userspace subsystems." The core binary image ntoskrnl.exe only exposes the primitives needed to get the CSR, OS2, and PSX subsystems started, and those (mostly, save for some code that lives in win32k.sys) run in userspace. They more or less act as translators between the native NT ABI/API and the respective subsystems' ABIs/APIs. PSX in particular did a lot of its own PE image mapping and parsing (something that you'd perhaps think would be done in kernel space) so that it could defer to the other subsystems to run foreign code among other things.
  - 3y ago
mschaef 3y ago
> Is the book labelled as fiction? > I understand that early versions of NT emitted error messages identifying itself as OS/2. Did the book explain when NT became "brand new OS from scratch". Windows NT started out as Microsoft's effort to do OS/2 NT. There was a 32-bit Microsoft OS/2 2.0 (distinct from IBM OS/2 2.0), and OS/2 NT was to be OS/2 3.0. OS/2 NT wasn't rebranded Windows NT until Windows 3.0 eliminated OS/2 from relevance, and with it, the Microsoft/IBM partnership. That's when IBM introduced their own (very nice) OS/2 2.0 and started a years long effort to eliminate Microsoft code from the OS/2 codebase (and the Microsoft copyright message from all over the UI)

borissk 3y ago

The part about NTFS is slightly incorrect. It wasn't really a new file system. It was a copy of High Performance File System (HPFS) for OS/2, that was originally implemented in assembler by Microsoft's best programmer Gordon Letwin. NTFS and HPFS even used the same partition id.

jmmv 3y ago
Oh thanks. I must have forgotten about that; if you have read the book, do you know if this was explained? Will need to read more and try to update the text!
jvert 3y ago
NTFS was definitely a new filesystem and supported many capabilities not in HPFS. Windows NT also shipped with HPFS support which was a complete reimplementation of the HPFS filesystem which shipped with OS/2.
chasil 3y ago
The DEC people added some stuff. "The MFT structure supports algorithms which minimize disk fragmentation... While this strongly resembles the W_FID of Files-11, other NTFS structures radically differ." https://en.wikipedia.org/wiki/NTFS https://en.wikipedia.org/wiki/NTFS "ODS-2 is the standard VMS file system, and remains the most common file system for system disks (the disk on which the operating system is installed)... NTFS - Has many structural and metadata similarities with Files-11 and is almost certainly conceptually derived from it." https://en.wikipedia.org/wiki/Files-11 https://en.wikipedia.org/wiki/Files-11
AshamedCaptain 3y ago
They use the same partition id because that was the partition id used for IFSes in OS/2 (and therefore all non-DOS filesystems). This should not be taken as a sign of their similarities (or differences).
kimixa 3y ago
Also some of the features listed (Compression, ACLs, volume sizes) were incrementally added over years and not available at first release, which somewhat invalidates the comparisons with other filesystems "Of the era". The NTFS of today is rather different to the NTFS of 1993.

chasil 3y ago

I have just been dealing with problems of VMS this week, and I have Dave Cutler to thank. We have a temporary emulated VAX running VMS 7 and Rdb [a SQL database now owned by Oracle]. After several tries to move the Rdb backup files through our QA firewall with FTP, we finally had to use Multinet SFTP2 to push the backup instead (upping the transfer time from 3 to 40 minutes), which autodetected "VMS mode." There are some secret record blocking/delimiter capabilities in the ODS-2 filesystem used by VMS that an FTP binary transfer will not preserve, and which the Rdb backup/restore process absolutely needs. Thank heavens that UNIX thinks a file is a single stream of bytes. Windows continues the feature experimentation with NTFS "alternate data streams." My, how clever that is [not]. EDIT: from the article, the claim "The open-source BSDs and Linux had just seen the light of day." is not quite correct. Bill Joy was involved in the earliest BSD repacking of v6 Research UNIX, shipped in 1978. "In 1975, Ken Thompson took a sabbatical from Bell Labs and came to Berkeley as a visiting professor. He helped to install Version 6 Unix and started working on a Pascal implementation for the system. Graduate students Chuck Haley and Bill Joy improved Thompson's Pascal and implemented an improved text editor, ex. Other universities became interested in the software at Berkeley, and so in 1977 Joy started compiling the first Berkeley Software Distribution (1BSD), which was released on March 9, 1978." https://en.wikipedia.org/wiki/Berkeley_Software_Distribution https://en.wikipedia.org/wiki/Berkeley_Software_Distribution ...I have also read elsewhere that a conscious decision was made in Windows to never, ever support big-endian architectures (but I am not certain that this is correct). NetBSD and Linux can run big-endian. "Now, I know you will complain that NTFS is slow… but you know what? It almost-certainly is not... NTFS on its own is fine..." XFS appears to trounce NTFS, and it consistently leads the performance scores on tpc.org; until very recently, SQL Server on XFS/Linux was well ahead of Windows.

Gravityloss 3y ago
Oh my, that prompted me to read Wikipedia - and we have MUMPS also from VMS. A quite different programming paradigm. The last common ancestor with current mainstream programming is really far back and it looks very odd. New large scale multi billion dollar healthcare systems are still being written in it.
- 3y ago

deleted 3y ago

[deleted]