11 ms·

Windows: A software engineering odyssey (2000)

180 points by xo5vik 3y ago

xo5vik 3y ago
Transcript of slides at https://danluu.com/microsoft-culture/ https://danluu.com/microsoft-culture/
canucker2016 3y ago
MarkL's latest move - leaving Google, after changes in the VR/AR org at Google - https://arstechnica.com/gadgets/2023/07/googles-head-of-ar-software-quits-citing-unstable-commitment-and-vision/ https://arstechnica.com/gadgets/2023/07/googles-head-of-ar-s...
jamesfinlayson 3y ago
> Source control system (Windows 2000) > Branch capability sorely needed, tree copies used as substitutes, so merging is a nightmare Ouch, and it looks like they only had version control with branching for the last nine months of development.
- inglor 3y ago
  Branching is an overstatement. SourceDepot didn't _really_ do branching. You had patches you'd float with "changelists" on top of enlistments. Each part large enough in thhe org (for example Excel or Word) gets a "branch" and it gets "forward integrated" and "reverse integrated" to the main "branch". From your perspective the tool used to submit stuff (usubmit usually) you just push to the same branch as everyone else in your org and if your code breaks things it gets "backed out" by an automatic process. Using git now is so much nicer.
  - drewcoo 3y ago
    SD was basically P4 with syntactic sugar. You likely didn't have permissions to create branches. Windows sources 20 years ago used to have a ridiculously complicated branching strategy, driven by middle managers and made worse by having actual devs sneak around the edges to do "buddy builds" of changes with some godawful batch file that I heard may have originated with RaymondC (who was exactly the kind of person to make ridiculous MSFT somehow bearable for the rest of us). It was Conway's Law, somehow twisted and applied to version control. With permissions SNAFUs. I still see companies today trying to map their org chart into their branching strategy and just shake my head . . . and run away.
    - inglor 3y ago
      > You likely didn't have permissions to create branches I did, it was just a very long and complicated process. You had to set up a lot of tooling for it and you were strongly discouraged from doing it so in the year of SourceDepot (on Office) I saw this option being used exactly once.
    - Kwpolska 3y ago
      Did branching in SD work the Git way (where creating a branch is instantaneous and requires zero resources), or did it work the TFVC/old-style-VCS way, where a branch required creating a copy of all files and took many minutes (probably hours at NT scale)? If you are stuck with this kind of system (why would you be these days?), long lived team branches would be the only sustainable strategy.
LispSporks22 3y ago
Can’t imagine how these Windows engineers feel about the enshitification of their baby. So much time invested into it - must be hard to see it taking this current trajectory.
- muricula 3y ago
  A few years ago when I was there, there were still remnants of that same Dave Cutler NT culture, especially around the folks who worked on minkernel/. I agree there are definitely shitty chunks Windows, but there are still some very solid foundations there to this day.
  - ForOldHack 3y ago
    I switched from 98 to ME for a week before 2k, and it (2k) was rock solid for years.
  - sanxiyn 3y ago
    For example, Windows kernel's write watch feature is useful for writing a GC. Linux lacked (and as far as I know, still lacks) this feature, so Microsoft had to rewrite .NET runtime. https://devblogs.microsoft.com/dotnet/working-through-things-on-other-oss/ https://devblogs.microsoft.com/dotnet/working-through-things...
    - kouteiheika 3y ago
      > Linux lacked (and as far as I know, still lacks) this feature AFAIK it is possible to do this on Linux (either through mprotect + SIGSEGV or userfaultfd) but it's slow. But there's a work-in-progress patch that the Collabora folks (probably on a contract from Valve if I'd had to guess, as some games do use this) are working on which will add a new fast way of doing this.
      - sanxiyn 3y ago
        userfaultfd was ~2015 (https://lwn.net/Articles/636226/ https://lwn.net/Articles/636226/), so Microsoft couldn't use it at the time. It could be better, but yes, Linux is making progress. Even in current form, userfaultfd is useful for GC, so Linux's lack of the feature in 2015 was unfortunate. Android 13 added a new GC taking advantage of userfaultfd: https://android-developers.googleblog.com/2022/08/android-13-is-in-aosp.html https://android-developers.googleblog.com/2022/08/android-13.... > A new garbage collector based on the Linux kernel feature userfaultfd is coming to ART on Android 13... The new garbage collector... leading to as much as ~10% reduction in compiled code size.
gigel82 3y ago
Source Depot was still a thing in Office as of last year when they finally transitioned to Git. Windows started transitioning to Git 5 years ago. Also, they have about 10k people working on Windows (and devices) and about 10k people working on ads nowadays (that paints a good story of priorities). Source: 2nd hand from MS friends
- com2kid 3y ago
  Source Depot's largest disadvantage VS git how hard it is to share changes with others. But I miss the ability to only pull down a portion of a monorepo, and the ability to remap where folders are at, or to pull down a single folder into multiple locations. So much bullshit in with monorepos in Git land exists because Git doesn't support things that Source Depot (and Perforce I presume) supported decades ago. As an aside for those who don't know what I am talking about, when pulling down a repo in source depot you can specify which directories to pull down locally, and you can also remap directories to a different path. This is super useful for header files, or any other sort of shared dependency. Instead of making the build system get all funky and fancy, the source control system handled putting files into expected locations. So imagine a large monorepo for a company and you can have some shared CSS styles that exist and they always end up in every projects `styles` folder or what have you. Or the repo keeps all shared styles in a single place, and you can then import them into your project, but instead of build system bullshit you just go to your mappings and tell it to pull the proper files put them into a sub-directory of your project. It is a really damn nice to have feature. (That also got misused a ton...)
  - forrestthewoods 3y ago
    Ehhh can’t say I’m a fan of folder remapping. It gets a little too auto-magical and since all tools access the file system directly different users can see different things. That’s just begging for bugs and “works on my machine”. I’m moderately confident the correct path is monorepo + centralization + virtual filesystem. Not every tool plays nice with VFS but at this point most do. The D in DVCS is almost entirely a waste. Source control systems should, imho, trivially support petabytes of history and terabyte scale clones.
ForOldHack 3y ago
There are details here no one else would know. Why would there be only a single mention of Xenix, when Microsoft bet the farm on it, only to vapor it for 4+ (79 to 84) years and outsource it less than 3 years later. (87)
- osullivj 3y ago
  Remember it well. I wrote a custom graphics card device driver for some mech eng code that would never fit in the 640K limit imposed by DOS.
- bitwize 3y ago
  Vapor? Xenix existed, and worked well, on the TRS-80 Model 16. Many small businesses basically ran on a Model 16 with Xenix back in the day.
yakubin 3y ago
Slide 19 doesn’t load for me.
- zczc 3y ago
  Slide 19 content (from .ppt): Serialized Development The model from NT 3.1 -> Windows 2000 All developers on team check-in to a single main line branch Master build lab synchs to main branch and builds and releases from that branch Checked in defect affects everyone waiting for results Diagram: Developer Developer Developer Developer -> Single MainBranch -> Product Build Machine -> Product Release Server
jstanley 3y ago
What's crazy about this is that it sounds like the people writing the code haven't got the slightest hope of even compiling it, let alone running it. They're just writing code, crossing their fingers, and committing it. It's a wonder the thing ever worked at all.
- yokoprime 3y ago
  There’s a slide saying few developers could build a whole instance of win2000, but when working at this scale you rarely build everything from scratch locally. You work on your module and the build system will have to build everything nightly. I don’t read this as devs throwing completely unchecked code into source control.
  - Athas 3y ago
    Certainly building everything from scratch is time consuming, but why should it be difficult? Windows 2000 was developed before I became a programmer, but not too many years later (2003) I was a teenager running Gentoo, and while it was time consuming to build everything from scratch, it was easy and reliable. It was also quite easy to switch out or hack on individual parts. In particular I remember fiddling endlessly with the kernel. Windows 2000 was certainly complex, but was it really substantially more complex than a full Linux distribution (including compilers, desktop environment, office suite, etc)? Why was it so difficult to build from scratch?
    - avidphantasm 3y ago
      I have no information here, but I wouldn’t be surprised if in 2003 Windows had an order of magnitude more code in it than a Linux distribution. Don’t forget, back then, Windows NT had support for these runtimes: Win32, Win16, DOS, and POSIX. Also, Windows had drivers for a lot more hardware than Linux did. Add in all the management stuff (Active Directory came about around this time), and I think it almost had to have substantially more code than Linux.
      - 3y ago
osullivj 3y ago
Zachary's Show Stopper covers this ground in a very readable manner, and gives a lot of useful detail on Dave Cutler's design ideas, and DEC VAX VMS background.
- easton 3y ago
  Not sure why you were downvoted, but that book was excellent. Shows that people 30 years ago were having precisely the same problems on big software projects we have today.
  - andsoitis 3y ago
    > Shows that people 30 years ago were having precisely the same problems on big software projects we have today. An interesting question is: why are we still having the same problems today? why haven't they been solved yet?
    - oaiey 3y ago
      Because our industry barely left their baby shoes behind it. We are not even teenagers. And why our brilliance does not make a difference: it is a human problem :)
sylens 3y ago
This has certainly whetted my appetite for a deeper dive - what's a good book on the history of Windows (or even Microsoft), from the earliest days up until at least Windows 95?
- fudgy 3y ago
  G. Pascal Zachary Show Stopper! Cloth: THE BREAKNECK RACE TO CREATE WINDOWS NT AND THE NEXT GENERATION AT MICROSOFT https://www.amazon.de/Show-Stopper-Cloth-BREAKNECK-GENERATION/dp/0029356717/ https://www.amazon.de/Show-Stopper-Cloth-BREAKNECK-GENERATIO...
lasereyes136 3y ago
Another great source is Show Stopper, a book about creating Windows NT. I read it years ago, it was the first time I saw the term dog fooding to describe using what you are building as you are building it. https://www.goodreads.com/en/book/show/1416925 https://www.goodreads.com/en/book/show/1416925
hunterjrj 3y ago
Plug for Dave Plummer's youtube channel: https://www.youtube.com/@DavesGarage https://www.youtube.com/@DavesGarage Dave was an engineer on NT and creator of Task Manager and zip folders. Lots of interesting stories and anecdotes from that period on that channel.
- Kwpolska 3y ago
  The built in zip folders that Dave wrote aren't something to be proud of these days, they are super slow and basic.

180 points by xo5vik 3y ago

Transcript of slides at https://danluu.com/microsoft-culture/ https://danluu.com/microsoft-culture/

canucker2016 3y ago

MarkL's latest move - leaving Google, after changes in the VR/AR org at Google - https://arstechnica.com/gadgets/2023/07/googles-head-of-ar-software-quits-citing-unstable-commitment-and-vision/ https://arstechnica.com/gadgets/2023/07/googles-head-of-ar-s...

jamesfinlayson 3y ago

> Source control system (Windows 2000) > Branch capability sorely needed, tree copies used as substitutes, so merging is a nightmare Ouch, and it looks like they only had version control with branching for the last nine months of development.

inglor 3y ago
Branching is an overstatement. SourceDepot didn't _really_ do branching. You had patches you'd float with "changelists" on top of enlistments. Each part large enough in thhe org (for example Excel or Word) gets a "branch" and it gets "forward integrated" and "reverse integrated" to the main "branch". From your perspective the tool used to submit stuff (usubmit usually) you just push to the same branch as everyone else in your org and if your code breaks things it gets "backed out" by an automatic process. Using git now is so much nicer.
- drewcoo 3y ago
  SD was basically P4 with syntactic sugar. You likely didn't have permissions to create branches. Windows sources 20 years ago used to have a ridiculously complicated branching strategy, driven by middle managers and made worse by having actual devs sneak around the edges to do "buddy builds" of changes with some godawful batch file that I heard may have originated with RaymondC (who was exactly the kind of person to make ridiculous MSFT somehow bearable for the rest of us). It was Conway's Law, somehow twisted and applied to version control. With permissions SNAFUs. I still see companies today trying to map their org chart into their branching strategy and just shake my head . . . and run away.
  - inglor 3y ago
    > You likely didn't have permissions to create branches I did, it was just a very long and complicated process. You had to set up a lot of tooling for it and you were strongly discouraged from doing it so in the year of SourceDepot (on Office) I saw this option being used exactly once.
  - Kwpolska 3y ago
    Did branching in SD work the Git way (where creating a branch is instantaneous and requires zero resources), or did it work the TFVC/old-style-VCS way, where a branch required creating a copy of all files and took many minutes (probably hours at NT scale)? If you are stuck with this kind of system (why would you be these days?), long lived team branches would be the only sustainable strategy.

LispSporks22 3y ago

Can’t imagine how these Windows engineers feel about the enshitification of their baby. So much time invested into it - must be hard to see it taking this current trajectory.

muricula 3y ago
A few years ago when I was there, there were still remnants of that same Dave Cutler NT culture, especially around the folks who worked on minkernel/. I agree there are definitely shitty chunks Windows, but there are still some very solid foundations there to this day.
- ForOldHack 3y ago
  I switched from 98 to ME for a week before 2k, and it (2k) was rock solid for years.
- sanxiyn 3y ago
  For example, Windows kernel's write watch feature is useful for writing a GC. Linux lacked (and as far as I know, still lacks) this feature, so Microsoft had to rewrite .NET runtime. https://devblogs.microsoft.com/dotnet/working-through-things-on-other-oss/ https://devblogs.microsoft.com/dotnet/working-through-things...
  - kouteiheika 3y ago
    > Linux lacked (and as far as I know, still lacks) this feature AFAIK it is possible to do this on Linux (either through mprotect + SIGSEGV or userfaultfd) but it's slow. But there's a work-in-progress patch that the Collabora folks (probably on a contract from Valve if I'd had to guess, as some games do use this) are working on which will add a new fast way of doing this.
    - sanxiyn 3y ago
      userfaultfd was ~2015 (https://lwn.net/Articles/636226/ https://lwn.net/Articles/636226/), so Microsoft couldn't use it at the time. It could be better, but yes, Linux is making progress. Even in current form, userfaultfd is useful for GC, so Linux's lack of the feature in 2015 was unfortunate. Android 13 added a new GC taking advantage of userfaultfd: https://android-developers.googleblog.com/2022/08/android-13-is-in-aosp.html https://android-developers.googleblog.com/2022/08/android-13.... > A new garbage collector based on the Linux kernel feature userfaultfd is coming to ART on Android 13... The new garbage collector... leading to as much as ~10% reduction in compiled code size.

gigel82 3y ago

Source Depot was still a thing in Office as of last year when they finally transitioned to Git. Windows started transitioning to Git 5 years ago. Also, they have about 10k people working on Windows (and devices) and about 10k people working on ads nowadays (that paints a good story of priorities). Source: 2nd hand from MS friends

com2kid 3y ago
Source Depot's largest disadvantage VS git how hard it is to share changes with others. But I miss the ability to only pull down a portion of a monorepo, and the ability to remap where folders are at, or to pull down a single folder into multiple locations. So much bullshit in with monorepos in Git land exists because Git doesn't support things that Source Depot (and Perforce I presume) supported decades ago. As an aside for those who don't know what I am talking about, when pulling down a repo in source depot you can specify which directories to pull down locally, and you can also remap directories to a different path. This is super useful for header files, or any other sort of shared dependency. Instead of making the build system get all funky and fancy, the source control system handled putting files into expected locations. So imagine a large monorepo for a company and you can have some shared CSS styles that exist and they always end up in every projects `styles` folder or what have you. Or the repo keeps all shared styles in a single place, and you can then import them into your project, but instead of build system bullshit you just go to your mappings and tell it to pull the proper files put them into a sub-directory of your project. It is a really damn nice to have feature. (That also got misused a ton...)
- forrestthewoods 3y ago
  Ehhh can’t say I’m a fan of folder remapping. It gets a little too auto-magical and since all tools access the file system directly different users can see different things. That’s just begging for bugs and “works on my machine”. I’m moderately confident the correct path is monorepo + centralization + virtual filesystem. Not every tool plays nice with VFS but at this point most do. The D in DVCS is almost entirely a waste. Source control systems should, imho, trivially support petabytes of history and terabyte scale clones.

ForOldHack 3y ago

There are details here no one else would know. Why would there be only a single mention of Xenix, when Microsoft bet the farm on it, only to vapor it for 4+ (79 to 84) years and outsource it less than 3 years later. (87)

osullivj 3y ago
Remember it well. I wrote a custom graphics card device driver for some mech eng code that would never fit in the 640K limit imposed by DOS.
bitwize 3y ago
Vapor? Xenix existed, and worked well, on the TRS-80 Model 16. Many small businesses basically ran on a Model 16 with Xenix back in the day.

yakubin 3y ago

Slide 19 doesn’t load for me.

zczc 3y ago
Slide 19 content (from .ppt): Serialized Development The model from NT 3.1 -> Windows 2000 All developers on team check-in to a single main line branch Master build lab synchs to main branch and builds and releases from that branch Checked in defect affects everyone waiting for results Diagram: Developer Developer Developer Developer -> Single MainBranch -> Product Build Machine -> Product Release Server

jstanley 3y ago

What's crazy about this is that it sounds like the people writing the code haven't got the slightest hope of even compiling it, let alone running it. They're just writing code, crossing their fingers, and committing it. It's a wonder the thing ever worked at all.

yokoprime 3y ago
There’s a slide saying few developers could build a whole instance of win2000, but when working at this scale you rarely build everything from scratch locally. You work on your module and the build system will have to build everything nightly. I don’t read this as devs throwing completely unchecked code into source control.
- Athas 3y ago
  Certainly building everything from scratch is time consuming, but why should it be difficult? Windows 2000 was developed before I became a programmer, but not too many years later (2003) I was a teenager running Gentoo, and while it was time consuming to build everything from scratch, it was easy and reliable. It was also quite easy to switch out or hack on individual parts. In particular I remember fiddling endlessly with the kernel. Windows 2000 was certainly complex, but was it really substantially more complex than a full Linux distribution (including compilers, desktop environment, office suite, etc)? Why was it so difficult to build from scratch?
  - avidphantasm 3y ago
    I have no information here, but I wouldn’t be surprised if in 2003 Windows had an order of magnitude more code in it than a Linux distribution. Don’t forget, back then, Windows NT had support for these runtimes: Win32, Win16, DOS, and POSIX. Also, Windows had drivers for a lot more hardware than Linux did. Add in all the management stuff (Active Directory came about around this time), and I think it almost had to have substantially more code than Linux.
    - 3y ago

osullivj 3y ago

Zachary's Show Stopper covers this ground in a very readable manner, and gives a lot of useful detail on Dave Cutler's design ideas, and DEC VAX VMS background.

easton 3y ago
Not sure why you were downvoted, but that book was excellent. Shows that people 30 years ago were having precisely the same problems on big software projects we have today.
- andsoitis 3y ago
  > Shows that people 30 years ago were having precisely the same problems on big software projects we have today. An interesting question is: why are we still having the same problems today? why haven't they been solved yet?
  - oaiey 3y ago
    Because our industry barely left their baby shoes behind it. We are not even teenagers. And why our brilliance does not make a difference: it is a human problem :)

sylens 3y ago

This has certainly whetted my appetite for a deeper dive - what's a good book on the history of Windows (or even Microsoft), from the earliest days up until at least Windows 95?

fudgy 3y ago
G. Pascal Zachary Show Stopper! Cloth: THE BREAKNECK RACE TO CREATE WINDOWS NT AND THE NEXT GENERATION AT MICROSOFT https://www.amazon.de/Show-Stopper-Cloth-BREAKNECK-GENERATION/dp/0029356717/ https://www.amazon.de/Show-Stopper-Cloth-BREAKNECK-GENERATIO...

lasereyes136 3y ago

Another great source is Show Stopper, a book about creating Windows NT. I read it years ago, it was the first time I saw the term dog fooding to describe using what you are building as you are building it. https://www.goodreads.com/en/book/show/1416925 https://www.goodreads.com/en/book/show/1416925

hunterjrj 3y ago

Plug for Dave Plummer's youtube channel: https://www.youtube.com/@DavesGarage https://www.youtube.com/@DavesGarage Dave was an engineer on NT and creator of Task Manager and zip folders. Lots of interesting stories and anecdotes from that period on that channel.

Kwpolska 3y ago
The built in zip folders that Dave wrote aren't something to be proud of these days, they are super slow and basic.