13 ms·

Microsoft Build Accelerator – open-source build engine for large systems

154 points by shanselman 7y ago

mikece 7y ago
"BuildXL runs 30,000+ builds per day on monorepo codebases up to a half-terabyte in size with a half-million process executions per build... You may find our technology useful if you face similar issues of scale." I know this wasn't supposed to be a humorous announcement but I couldn't help laughing out loud at that! Kudos to the managers at Microsoft who now seem to be asking "Why not?" instead of "Why should we?" when the topic of releasing code to the community is raised.
- algorithmsRcool 7y ago
  I'm a bit confused. Are you critical of the size of the monorepo, or the number of builds?
  - detaro 7y ago
    I think the comment isn't critical at all.
  - nickpeterson 7y ago
    I believe the joke is that very few companies are the size of Microsoft, and as such very few would find this useful.
    - hermitdev 7y ago
      You'd be surprised at the volume of code a smaller company can produce. Former employer was a big C++ shop in finance. Of around 1000 employees, roughly 3/4 of those were developers. They definitely could take advantage of something like this. I dont know how many 100s of million of LOC they have between C++ and later C#, but I was responsible for around 3 million alone (largely generated). A full coordinated firm wide rebuild could take weeks.
      - nickpeterson 7y ago
        I'm always stunned by these sorts of stories. Was the opinion that the scale of code was, low, high, or appropriate given the problems being tackled?
        cryptonector 7y ago
        I've a similar story or three of finance companies that in just ten years produced enormous amounts of legacy code. It's really not that hard. Solaris was enormous too, with just 2k devs for all the time I was there, and about 30-40 years of history, depending on how you count it. If your 1k developers each write 10Kloc/year on average, then after a decade you can expect to have 10Mloc, but since a lot of code will be forked external open source (or even not forked, but just imported to freeze at a particular version, or for some other reason) you might find your devs building and looking after many tens more Mloc than that. If you hire lots of 5x and 10x engineers, that too leads to a sizeable increment. There are many many companies out there that have huge megarepos.
        bryanrasmussen 7y ago
        Ok but, what's the byte size of 10MLoc, and how many process executions per build - since these were actually the metrics used. My experience is that lines of code don't actually take up that much space.
        hermitdev 7y ago
        Depends largely on how the code is structured with C++. There's the number of compilation units within a lib vs overall. Typically you can parallelize within a module, but not externally unless you have some smarts. Edit: I use module in this sense as a producible result, not the future language concept of modules.
        hermitdev 7y ago
        In my experience, it felt appropriate. We (the developers) were tasked with enormous responsibilities. I was back office, but responsible for managing a client/server for all non security reference data. There were easily over 200 data objects modelled. No direct DB access was allowed except for the owning service. Although it ended up around 3M LOC, it wasnt as bad as it seems, because only about 10% was manually written code. A lot was generated C++ (and aside: I was able to write a C++ wrapper around a generic API that exposed some 300 types through a home grown reflection esque API amd the Python wrapper never had to be updated necause it could use this reflection and run time cose generation to generate a strongly typed API at runtime; to my knowledge, the Python code has remained unchanged since about 2006 despite the underlying C++ API changing constantly, when I first wrote it - I get occasional updates from former coworkers). The big problem was the dependency management and scale. At least at the time I was there, neither were done well. Scale was a problem because of tight coupling between libs. Upgrade a core lib? Everyone had to rebuild. Want to upgrade a 3rd party dependency? Firm wide rebuild that took a min 2 weeks. It was a mess. We were supposed to be client/server to minimize dependencies, but we so tightly coupled our clients to our servers, we just exacerbated the problem. A few us could handle multiple client versions with a single server, but most couldn't. Don't recommend.
    - onion2k 7y ago
      I might simplify my Webpack builds with it.
    - oblio 7y ago
      I agree with the other commenters. You'd be surprised how much code companies you've never heard of. I used to work for Axway, an enterprise middleware vendor. They had at least 3 products that I knew about with multi-million LOC codebases, in Java and C and various other languages. I wouldn't be surprised if they had at least 50 million LOC in total. Actually, now that I think of it, 100 million was more likely. And that was almost 10 years ago...
  - hermitdev 7y ago
    I think the GP is more taking a jab at MS's history of being anti-open source, which has taken an about face in the last 5 years or so. I mean we had Ballmer under whom you'd likely never have seen anything (or only trivial, non monetizable things) open sourced and likely no Linux support. Now under Satya, we have MS open sourcing lots of projects and embracing Linux as a first class citizen. Probably far a complete list of Linux support, off the top of my I can name: VS Code, building for Linux via VS proper, Cmake support, WSL, SQL Server on Linux, SQL server odbc drivers, .Net Core and Linux support in Azure. Even if projects are completely open, such as WSL and connhost, they have github projects at least for bug tracking that allows end-users to directly interact with the teams responsible for those projects. Personally, I've filed several issues against WSL and I've gotten fairly quick responses and resolutions usually appear in a few weeks time (I'm in the Fast Insider's ring at home).
    - mikepurvis 7y ago
      I'm pretty impressed with WSL. I've just moved back to Windows (Dell XPS) after ~13 years using Mac OS, and in many ways WSL is nicer than what I used to do before with juggling Homebrew packages and Parallels VMs depending on the demands of a particular task. Between that, PowerShell, and installing stuff via Chocolatey, I feel right at home. But without things like WSL and native OpenSSH, I doubt I would even have looked; I would have stuck with MacOS forever or just gone native Ubuntu.
      - phunky 7y ago
        You've still got access to brew via linuxbrew. https://docs.brew.sh/Homebrew-on-Linux https://docs.brew.sh/Homebrew-on-Linux I'm also in the process of switching back from Macos to Windows thanks to WSL.
      - Sangeppato 7y ago
        I personally prefer scoop as a package manager, give it a try!
        mikepurvis 7y ago
- m0zg 7y ago
  Microsoft has something like 30K devs. 30K builds a day seems really, really low.
  - yati 7y ago
    Maybe they have multiple big repos, and that number is per repo?
azhenley 7y ago
I interned with TSE in 2016 and had a blast. It is nice seeing one of their projects get open sourced.
mikerg87 7y ago
The why and how of this can be found here : https://github.com/Microsoft/BuildXL/blob/master/Documentation/Wiki/WhyBuildXL.md https://github.com/Microsoft/BuildXL/blob/master/Documentati... Seems what drove this was a 90 build times of 90 hours for and end to end build of Office. What I can gather this has a means of capturing all read/write operations for a build step and placing it in a cache store to determine if a change necessitates the rebuild of a component. Since it can hook in at a lower level, it isn't sensitive to time stamps for building. Actually quite interesting
pianoben 7y ago
Congratulations to the BuildXL team! Domino, incidentally, was/is the internal for BuildXL; there are a few papers published where the system is described under that name. I had the privilege to see it gradually rolled out in the Office codebase over the course of a year or two. It was a massive improvement, and the lengths to which the team went to see it through are really beyond description here. I _have_ to wonder why we didn't just pick up Bazel, which is Google's open-source distributed build engine for large systems, which also happens to have been stable for years. Perhaps its Windows support wasn't up to snuff at the time, but it feels like that would have been easier to fix that than to build a whole new build system. Regardless, congrats again! So cool to see this out in the open.
- jchw 7y ago
  Windows support in Bazel is still far from perfect; to this day, you need Msys2 for many things to work. Still, things have been moving forward quite a bit recently, I think in part due to Angular. Honestly, it's still really nice to use imo, but I can see why it would be hard for Microsoft to adopt. (I work at Google and have contributed a very tiny bit to improving Bazel on Windows, though it is not related to my actual work at Google.)
  - m0zg 7y ago
    You have to wonder though whether the effort spent building Build Accelerator would be better spent fixing Bazel bugs on Windows. Bazel really has no equal on the Linux side, and Google's internal version (+Forge and other systems) is far and away the best dev experience I've ever seen anywhere.
    - jchw 7y ago
      I love Bazel, but above all I hope open source can adopt Bazel more. It’s certainly much more usable for the case where you want to import external programs, versus systems like CMake, in my opinion.
    - pjmlp 7y ago
      I do Maven for Java and JVM languages, Gradle only because Google forces me to on Android, for better or worse CMake has won for C++ and MSBuild for .NET languages. I am yet to see what Bazel does better, beyond faster Android builds, which is quite easy given Gradle and how Android team implements their plugins.
pojntfx 7y ago
No Linux support, so who cares?
- acct1771 7y ago
  Anyone running build servers on Windows, or ReactOS.
- pjmlp 7y ago
  All of us that don't use Linux daily.
whalesalad 7y ago
“Its own internal scripting language, DScript, an experimental TypeScript based format used as an intermediate language by a small number of teams inside Microsoft” Anyone else notice this?
- daemin 7y ago
  Yes, but really is it any different to what every build system does by inventing its own DSL or making a new DSL in some other language? Ant did it in XML, Premake uses Lua, CMake has its own, etc
scanr 7y ago
Anyone know a bit more about DScript?

154 points by shanselman 7y ago

"BuildXL runs 30,000+ builds per day on monorepo codebases up to a half-terabyte in size with a half-million process executions per build... You may find our technology useful if you face similar issues of scale." I know this wasn't supposed to be a humorous announcement but I couldn't help laughing out loud at that! Kudos to the managers at Microsoft who now seem to be asking "Why not?" instead of "Why should we?" when the topic of releasing code to the community is raised.

algorithmsRcool 7y ago
I'm a bit confused. Are you critical of the size of the monorepo, or the number of builds?
- detaro 7y ago
  I think the comment isn't critical at all.
- nickpeterson 7y ago
  I believe the joke is that very few companies are the size of Microsoft, and as such very few would find this useful.
  - hermitdev 7y ago
    You'd be surprised at the volume of code a smaller company can produce. Former employer was a big C++ shop in finance. Of around 1000 employees, roughly 3/4 of those were developers. They definitely could take advantage of something like this. I dont know how many 100s of million of LOC they have between C++ and later C#, but I was responsible for around 3 million alone (largely generated). A full coordinated firm wide rebuild could take weeks.
    - nickpeterson 7y ago
      I'm always stunned by these sorts of stories. Was the opinion that the scale of code was, low, high, or appropriate given the problems being tackled?
      - cryptonector 7y ago
        I've a similar story or three of finance companies that in just ten years produced enormous amounts of legacy code. It's really not that hard. Solaris was enormous too, with just 2k devs for all the time I was there, and about 30-40 years of history, depending on how you count it. If your 1k developers each write 10Kloc/year on average, then after a decade you can expect to have 10Mloc, but since a lot of code will be forked external open source (or even not forked, but just imported to freeze at a particular version, or for some other reason) you might find your devs building and looking after many tens more Mloc than that. If you hire lots of 5x and 10x engineers, that too leads to a sizeable increment. There are many many companies out there that have huge megarepos.
        bryanrasmussen 7y ago
        Ok but, what's the byte size of 10MLoc, and how many process executions per build - since these were actually the metrics used. My experience is that lines of code don't actually take up that much space.
        hermitdev 7y ago
        Depends largely on how the code is structured with C++. There's the number of compilation units within a lib vs overall. Typically you can parallelize within a module, but not externally unless you have some smarts. Edit: I use module in this sense as a producible result, not the future language concept of modules.
      - hermitdev 7y ago
        In my experience, it felt appropriate. We (the developers) were tasked with enormous responsibilities. I was back office, but responsible for managing a client/server for all non security reference data. There were easily over 200 data objects modelled. No direct DB access was allowed except for the owning service. Although it ended up around 3M LOC, it wasnt as bad as it seems, because only about 10% was manually written code. A lot was generated C++ (and aside: I was able to write a C++ wrapper around a generic API that exposed some 300 types through a home grown reflection esque API amd the Python wrapper never had to be updated necause it could use this reflection and run time cose generation to generate a strongly typed API at runtime; to my knowledge, the Python code has remained unchanged since about 2006 despite the underlying C++ API changing constantly, when I first wrote it - I get occasional updates from former coworkers). The big problem was the dependency management and scale. At least at the time I was there, neither were done well. Scale was a problem because of tight coupling between libs. Upgrade a core lib? Everyone had to rebuild. Want to upgrade a 3rd party dependency? Firm wide rebuild that took a min 2 weeks. It was a mess. We were supposed to be client/server to minimize dependencies, but we so tightly coupled our clients to our servers, we just exacerbated the problem. A few us could handle multiple client versions with a single server, but most couldn't. Don't recommend.
  - onion2k 7y ago
    I might simplify my Webpack builds with it.
  - oblio 7y ago
    I agree with the other commenters. You'd be surprised how much code companies you've never heard of. I used to work for Axway, an enterprise middleware vendor. They had at least 3 products that I knew about with multi-million LOC codebases, in Java and C and various other languages. I wouldn't be surprised if they had at least 50 million LOC in total. Actually, now that I think of it, 100 million was more likely. And that was almost 10 years ago...
- hermitdev 7y ago
  I think the GP is more taking a jab at MS's history of being anti-open source, which has taken an about face in the last 5 years or so. I mean we had Ballmer under whom you'd likely never have seen anything (or only trivial, non monetizable things) open sourced and likely no Linux support. Now under Satya, we have MS open sourcing lots of projects and embracing Linux as a first class citizen. Probably far a complete list of Linux support, off the top of my I can name: VS Code, building for Linux via VS proper, Cmake support, WSL, SQL Server on Linux, SQL server odbc drivers, .Net Core and Linux support in Azure. Even if projects are completely open, such as WSL and connhost, they have github projects at least for bug tracking that allows end-users to directly interact with the teams responsible for those projects. Personally, I've filed several issues against WSL and I've gotten fairly quick responses and resolutions usually appear in a few weeks time (I'm in the Fast Insider's ring at home).
  - mikepurvis 7y ago
    I'm pretty impressed with WSL. I've just moved back to Windows (Dell XPS) after ~13 years using Mac OS, and in many ways WSL is nicer than what I used to do before with juggling Homebrew packages and Parallels VMs depending on the demands of a particular task. Between that, PowerShell, and installing stuff via Chocolatey, I feel right at home. But without things like WSL and native OpenSSH, I doubt I would even have looked; I would have stuck with MacOS forever or just gone native Ubuntu.
    - phunky 7y ago
      You've still got access to brew via linuxbrew. https://docs.brew.sh/Homebrew-on-Linux https://docs.brew.sh/Homebrew-on-Linux I'm also in the process of switching back from Macos to Windows thanks to WSL.
    - Sangeppato 7y ago
      I personally prefer scoop as a package manager, give it a try!
      - mikepurvis 7y ago
m0zg 7y ago
Microsoft has something like 30K devs. 30K builds a day seems really, really low.
- yati 7y ago
  Maybe they have multiple big repos, and that number is per repo?

azhenley 7y ago

I interned with TSE in 2016 and had a blast. It is nice seeing one of their projects get open sourced.

mikerg87 7y ago

The why and how of this can be found here : https://github.com/Microsoft/BuildXL/blob/master/Documentation/Wiki/WhyBuildXL.md https://github.com/Microsoft/BuildXL/blob/master/Documentati... Seems what drove this was a 90 build times of 90 hours for and end to end build of Office. What I can gather this has a means of capturing all read/write operations for a build step and placing it in a cache store to determine if a change necessitates the rebuild of a component. Since it can hook in at a lower level, it isn't sensitive to time stamps for building. Actually quite interesting

pianoben 7y ago

Congratulations to the BuildXL team! Domino, incidentally, was/is the internal for BuildXL; there are a few papers published where the system is described under that name. I had the privilege to see it gradually rolled out in the Office codebase over the course of a year or two. It was a massive improvement, and the lengths to which the team went to see it through are really beyond description here. I _have_ to wonder why we didn't just pick up Bazel, which is Google's open-source distributed build engine for large systems, which also happens to have been stable for years. Perhaps its Windows support wasn't up to snuff at the time, but it feels like that would have been easier to fix that than to build a whole new build system. Regardless, congrats again! So cool to see this out in the open.

jchw 7y ago
Windows support in Bazel is still far from perfect; to this day, you need Msys2 for many things to work. Still, things have been moving forward quite a bit recently, I think in part due to Angular. Honestly, it's still really nice to use imo, but I can see why it would be hard for Microsoft to adopt. (I work at Google and have contributed a very tiny bit to improving Bazel on Windows, though it is not related to my actual work at Google.)
- m0zg 7y ago
  You have to wonder though whether the effort spent building Build Accelerator would be better spent fixing Bazel bugs on Windows. Bazel really has no equal on the Linux side, and Google's internal version (+Forge and other systems) is far and away the best dev experience I've ever seen anywhere.
  - jchw 7y ago
    I love Bazel, but above all I hope open source can adopt Bazel more. It’s certainly much more usable for the case where you want to import external programs, versus systems like CMake, in my opinion.
  - pjmlp 7y ago
    I do Maven for Java and JVM languages, Gradle only because Google forces me to on Android, for better or worse CMake has won for C++ and MSBuild for .NET languages. I am yet to see what Bazel does better, beyond faster Android builds, which is quite easy given Gradle and how Android team implements their plugins.

pojntfx 7y ago

No Linux support, so who cares?

acct1771 7y ago
Anyone running build servers on Windows, or ReactOS.
pjmlp 7y ago
All of us that don't use Linux daily.

whalesalad 7y ago

“Its own internal scripting language, DScript, an experimental TypeScript based format used as an intermediate language by a small number of teams inside Microsoft” Anyone else notice this?

daemin 7y ago
Yes, but really is it any different to what every build system does by inventing its own DSL or making a new DSL in some other language? Ant did it in XML, Premake uses Lua, CMake has its own, etc

scanr 7y ago

Anyone know a bit more about DScript?