3 ms·

Build systems a la carte: theory and practice (2020)

102 points by transpute 3y ago

jahnu 3y ago
I have a fantasy where I win the lottery or inherit millions from a long lost relative and can live out my days developing whatever project I want. That project would be the ultimate build system. There are at least two problems of course. I'm probably not that smart and I don't buy lottery tickets.
- xyzzy_plugh 3y ago
  We've had `make` for north of forty years. I'm all for it being demoted to the penultimate build system but we mustn't ignore our Sisyphean tendencies. I personally hope Nix eats the world.
- Rochus 3y ago
  Maybe the long lost relative has already developed an ultimate build system nobody knows of.
- GuB-42 3y ago
  I am sure that some companies would spend billions (yes, with a "b") for that. And they already have some of the smartest people on Earth working for them. The article is more theory than practice. It practice, the big problem is not really about knowing what should be rebuilt. It is about tying the work of many people with different ideas on how thing should be done together. The part about getting the build order correct, which is what the article is about is still important. But that's typically not the reason why everyone hates build systems. People can live with the occasional inefficiency, or "make clean". But getting that binary-only library that is not in the standard location with that source-only other library and its weird versioning scheme, both needing a different version of the same third library, written in a different programming language and using a different packet manager...
fouronnes3 3y ago
This is a fantastic paper that I come back to over the years. I remember talking about it with a coworker when it came out, and we were amazed how elegantly it made the point that Excel is a build system, and how it helps understand how all build systems fit together in their design space.
- kryptiskt 3y ago
  An exercise for the esoteric programmer would be to make a sheet that turns Excel into an actual build system that can drive the build of a typical C program.
- ted_bunny 3y ago
  I'm great at excel and terrible at programming. Can you break down what's meant here?
  - klodolph 3y ago
    In Excel, every cell has some number of inputs (which are other cells), and a rule for calculating the cell’s value from its input. That is basically how software build systems work, except usually they use files instead of spreadsheet cells, and they run commands (like “compile this source code”, or “combine these seven files into one program”) instead of Excel formulas.
  - ufo 3y ago
    In a build system, when you update a source code file, it should recompile the binaries that depend on it. Similarly, in excel when you update a cell in the spreadsheet, it should recompute the cells that refer to it. The paper is about thinking about build systems in a more modular way. Are dependencies specified statically or dynamically? What algorithm is used to decide what should get rebuilt after an update? What order do we rebuild things? What happens if we pick different combinations of algorithm+order?
    - dayjaby 3y ago
      Is that actually a thing? What algorithms do you mean? I thought everything that depends on a change gets rebuilt.
gavinhoward 3y ago
One of my favorite papers, and it guided my build system design.
PartiallyTyped 3y ago
This is the first time I encountered this, and I wish I had seen it earlier. It's a great read!
hiAndrewQuinn 3y ago
Hey, it's Dr. Simon Peyton Jones of the Haskell community! Love seeing his name pop up.
justinlloyd 3y ago
I've worked on build systems quite a bit across a disparity of languages, frameworks, operating systems and engines. I've constructed some really big build systems spitting out 100's of SKUs across a dozen target platform, and I've built studio-wide build systems that again, spit out hundreds of SKU's across dozens of projects, with hundreds to thousands of programmer and artist hours per day of code and art and other binary assets pouring into the version control every hour of every day. And the universal constant in all build systems is always a problem of management. I've turned 7 hour builds into 7 minute builds, and management doesn't care. I've saved literally thousands of hours of programmer time per year on projects, and management doesn't care. Build systems are a cost center. Always neglected, never funded, and not seen as critical infrastructure. Build systems, at the end of the day, no matter the language or target platform, are never really that complicated, we just tend to over-complicate them. What I really want, if I had the time to implement, is a build file system. All that said, I'll never go back to working on a big build system without having a PM that insulates me from the unreasonable demands of being woken up at 4AM by a Producer absolutely frantic because "the build system isn't working at all!" but when checking on the problem, it's because a developer checked in some code late and night and then went home. I have some serious trauma from those years of my life.

102 points by transpute 3y ago

I have a fantasy where I win the lottery or inherit millions from a long lost relative and can live out my days developing whatever project I want. That project would be the ultimate build system. There are at least two problems of course. I'm probably not that smart and I don't buy lottery tickets.

xyzzy_plugh 3y ago
We've had `make` for north of forty years. I'm all for it being demoted to the penultimate build system but we mustn't ignore our Sisyphean tendencies. I personally hope Nix eats the world.
Rochus 3y ago
Maybe the long lost relative has already developed an ultimate build system nobody knows of.
GuB-42 3y ago
I am sure that some companies would spend billions (yes, with a "b") for that. And they already have some of the smartest people on Earth working for them. The article is more theory than practice. It practice, the big problem is not really about knowing what should be rebuilt. It is about tying the work of many people with different ideas on how thing should be done together. The part about getting the build order correct, which is what the article is about is still important. But that's typically not the reason why everyone hates build systems. People can live with the occasional inefficiency, or "make clean". But getting that binary-only library that is not in the standard location with that source-only other library and its weird versioning scheme, both needing a different version of the same third library, written in a different programming language and using a different packet manager...

fouronnes3 3y ago

This is a fantastic paper that I come back to over the years. I remember talking about it with a coworker when it came out, and we were amazed how elegantly it made the point that Excel is a build system, and how it helps understand how all build systems fit together in their design space.

kryptiskt 3y ago
An exercise for the esoteric programmer would be to make a sheet that turns Excel into an actual build system that can drive the build of a typical C program.
ted_bunny 3y ago
I'm great at excel and terrible at programming. Can you break down what's meant here?
- klodolph 3y ago
  In Excel, every cell has some number of inputs (which are other cells), and a rule for calculating the cell’s value from its input. That is basically how software build systems work, except usually they use files instead of spreadsheet cells, and they run commands (like “compile this source code”, or “combine these seven files into one program”) instead of Excel formulas.
- ufo 3y ago
  In a build system, when you update a source code file, it should recompile the binaries that depend on it. Similarly, in excel when you update a cell in the spreadsheet, it should recompute the cells that refer to it. The paper is about thinking about build systems in a more modular way. Are dependencies specified statically or dynamically? What algorithm is used to decide what should get rebuilt after an update? What order do we rebuild things? What happens if we pick different combinations of algorithm+order?
  - dayjaby 3y ago
    Is that actually a thing? What algorithms do you mean? I thought everything that depends on a change gets rebuilt.

gavinhoward 3y ago

One of my favorite papers, and it guided my build system design.

PartiallyTyped 3y ago

This is the first time I encountered this, and I wish I had seen it earlier. It's a great read!

hiAndrewQuinn 3y ago

Hey, it's Dr. Simon Peyton Jones of the Haskell community! Love seeing his name pop up.

justinlloyd 3y ago

I've worked on build systems quite a bit across a disparity of languages, frameworks, operating systems and engines. I've constructed some really big build systems spitting out 100's of SKUs across a dozen target platform, and I've built studio-wide build systems that again, spit out hundreds of SKU's across dozens of projects, with hundreds to thousands of programmer and artist hours per day of code and art and other binary assets pouring into the version control every hour of every day. And the universal constant in all build systems is always a problem of management. I've turned 7 hour builds into 7 minute builds, and management doesn't care. I've saved literally thousands of hours of programmer time per year on projects, and management doesn't care. Build systems are a cost center. Always neglected, never funded, and not seen as critical infrastructure. Build systems, at the end of the day, no matter the language or target platform, are never really that complicated, we just tend to over-complicate them. What I really want, if I had the time to implement, is a build file system. All that said, I'll never go back to working on a big build system without having a PM that insulates me from the unreasonable demands of being woken up at 4AM by a Producer absolutely frantic because "the build system isn't working at all!" but when checking on the problem, it's because a developer checked in some code late and night and then went home. I have some serious trauma from those years of my life.