6 ms·

Good post. Good analysis. One problem: If turing-completeness is necessary, why should there even exist a build system independent from a programming language

Good post. Good analysis. One problem: If turing-completeness is necessary, why should there even exist a build system independent from a programming language now? Isn't the obvious solution to script your language's default build system in precisely the language you're trying to build from? So when you build Rig, build it in C++ and make C++ your scripting ... Ok, that left a bad taste in my mouth. Seriously, though. Build systems as the author describes them are going to compete against the project managers that come with languages (Cargo, Go). The only real use case for Rig would be legacy languages that don't come with language-central packaging repositories, i.e., Fortran, C++, C. While I applaud the author for the attempt to improve this ecosystem, I really hope it will become less relevant some day.

gavinhoward 3y ago
> If turing-completeness is necessary, why should there even exist a build system independent from a programming language now? This is a great question that I will make sure to answer in my Turing-completeness post in a week. But to answer now, it's because of multi-project builds, where projects can be in multiple languages. If you have build systems that are not independent from their language, how do you get them to play nice? I designed Rig to be able to act as a babysitter and make them play nice. If I play my cards right, Rig will be the reason other build systems become irrelevant. :)
- codethatwerks 3y ago
  What about a single API for builds, implemented in multiple general purpose languages. End users get to pick a familiar language to them. Similar to how Pulumi is available in multiple languages. Then code both builds and meta build concerns as you would code anything else.
- aappleby 3y ago
  Not if Hancho beats you to it. ;) https://github.com/aappleby/Hancho https://github.com/aappleby/Hancho In all seriousness, I think both our projects are aiming in the right direction, mine is just more focused on minimalism at the cost of having extra-pointy corner cases.
- talideon 3y ago
  I've a similar question to GP, but from a slightly different angle. I can see the attraction of using a Turing complete language, but on the other hand, an important properly of a build system is that it ought to halt at some point. Something like, say, Starlark, which is used in Bazel, is primitive recursive, so it's not Turing complete, but gets you almost all the way to Turing complete while also being guaranteed to halt. It'd be interesting to see this aspect covered. EDIT: s/Skylark/Starlark/
  - gavinhoward 3y ago
    I'm afraid I have bad news. The thing about Starlark is that it is effectively Turing complete. You can simulate a while loop with a long enough foreach, an if statement, and a break statement. And by "long enough," I mean long enough to make anyone just stop the build, but hey, if I stack the foreach loops, I can make the inner loop exponentially long, even though ints are limited to 32 bits. By not taking out the break keyword, Starlark remains Turing-complete. And yes, I tested this.
    - talideon 3y ago
      That's not Turing complete. That's still primitive recursive. You're going out of your way make it loop for a really, really long time, but it's still guaranteed to terminate, and thus isn't Turing complete, even if it'd take a really long time to terminate. This isn't some theoretical thing: it's how you can introduce guarantees into the system that it can't get out of control by accident and it also makes it easier to reason about the system. A primitive recursive language gets you most of the benefits of a general recursive one _without_ needing to be _actually_ Turing complete. So this isn't "bad news" for me. I'm saying that I think you should avoid using a general recursive (i.e. Turing complete) language for a build system and instead have guarantees that the system will terminate by using a primitive recursive language of some kind because you literally, as you did, need to go out of your way to get code written in such a fashion to run for an inordinate amount of time. And yet, an inordinate amount of time isn't forever and unlike a program written in an actually Turing complete language, you can use static analysis to detect that kind of thing if only primitive recursion is supported.
JohnFen 3y ago
> why should there even exist a build system independent from a programming language now? I want my build system to be independent of the programming language for two reasons: 1) What gavinhoward said: I regularly use several different languages, sometimes in the same project, sometimes not. But in all cases, I want to use the same build system in order to minimize my environmental complexity. Much like I want to use the same IDE regardless of language. 2) I don't want to risk version dependencies. I want to be able to use the same build system even when I'm compiling something using an old version of a compiler, and I don't want to be forced into a build system upgrade just because I want to upgrade one of my compilers. For me, the separation of the two roles is very important. If they were tied together, it would mean that I'd have to add complexity elsewhere to cover my use cases. Languages that include their own package managers (like Rust, Go, etc.) give me headaches.
- eadmund 3y ago
  > I regularly use several different languages, sometimes in the same project, sometimes not. But in all cases, I want to use the same build system in order to minimize my environmental complexity. It’s not obvious to me that one can solve the problem of making N languages co-operate by adding another. Why not just use the most powerful language to orchestrate everything? > I don't want to risk version dependencies. I want to be able to use the same build system even when I'm compiling something using an old version of a compiler, and I don't want to be forced into a build system upgrade just because I want to upgrade one of my compilers. That begs the question of whether compiler upgrades should force build system upgrades. In a mature programming language, this should never be the case, although practically speaking it’s definitely possible (e.g. ASDF 3.3, released earlier this year but built on a language which has been static for thirty years, adds compatibility for Allegro Common Lisp!). Incidentally, ASDF is pretty cool: https://asdf.common-lisp.dev/ https://asdf.common-lisp.dev/
  - JohnFen 3y ago
    > Why not just use the most powerful language to orchestrate everything? I usually do everything in a single language, but sometimes that's not the best solution. It's not a matter of how powerful the language is in a general sense, but how good it is at a particular kind of task. But even ignoring my projects that involve multiple languages simultaneously, I still regularly use several languages in total, even if only one per project. So having a build system that's independent of the language is still highly desirable. > That begs the question of whether compiler upgrades should force build system upgrades. My answer to that is "no, never", of course. That leads to increased complexity for me, in exchange for no gain. The build system should have no special knowledge or expectation of what language or language tools are being used.
zdragnar 3y ago
I was once on a project that inadvertently used ruby in some way for everything. Rails Bundler Vagrant Chef Opsworks configuration JavaScript bundling Generating PDFs Probably other things The system was basically frozen in time because too many cross-dependencies couldn't update in sync to something newer (the opsworks cookbook in chef were the worst offenders iirc). It was so bad when I started that I couldn't even get the system up and running locally. I ended up tossing hundreds of lines of random ruby code and with a few lines of docker commands had it up and running. Since then I've preferred keeping things nicely not dependent on each other whenever possible.
eadmund 3y ago
> If turing-completeness is necessary, why should there even exist a build system independent from a programming language now? It shouldn’t. The only issue is that most folks aren’t using powerful enough languages. As you note, using C++ as a scripting language would be pretty crazy. So don’t do that!
teo_zero 3y ago
> Isn't the obvious solution to script your language's default build system in precisely the language you're trying to build from? Please no. I don't want to handle filename and other string manipulation in C.
InitEnabler 3y ago
>The only real use case for Rig would be legacy languages that don't come with language-central packaging repositories, i.e., Fortran, C++, C. What about packages for *nix based OS's? dnf, pacman, apt, etc. essential have there own "meta" build systems to create packages for there particular package flavor weather that be rpm, deb, tar.gz etc.
scrubs 3y ago
"The only real use case for Rig would be legacy languages that don't come with language-central packaging repositories, i.e., Fortran, C++, C." C/C++ are big time numerous and serious real. And they need all the help they can get to do builds more like golang
- shrimp_emoji 3y ago
  Yeah. Less relevant one day? When C and C++ become less relevant? I hope you're willing to wait a century or two famalamadingdong

gavinhoward 3y ago

> If turing-completeness is necessary, why should there even exist a build system independent from a programming language now? This is a great question that I will make sure to answer in my Turing-completeness post in a week. But to answer now, it's because of multi-project builds, where projects can be in multiple languages. If you have build systems that are not independent from their language, how do you get them to play nice? I designed Rig to be able to act as a babysitter and make them play nice. If I play my cards right, Rig will be the reason other build systems become irrelevant. :)

codethatwerks 3y ago
What about a single API for builds, implemented in multiple general purpose languages. End users get to pick a familiar language to them. Similar to how Pulumi is available in multiple languages. Then code both builds and meta build concerns as you would code anything else.
aappleby 3y ago
Not if Hancho beats you to it. ;) https://github.com/aappleby/Hancho https://github.com/aappleby/Hancho In all seriousness, I think both our projects are aiming in the right direction, mine is just more focused on minimalism at the cost of having extra-pointy corner cases.
talideon 3y ago
I've a similar question to GP, but from a slightly different angle. I can see the attraction of using a Turing complete language, but on the other hand, an important properly of a build system is that it ought to halt at some point. Something like, say, Starlark, which is used in Bazel, is primitive recursive, so it's not Turing complete, but gets you almost all the way to Turing complete while also being guaranteed to halt. It'd be interesting to see this aspect covered. EDIT: s/Skylark/Starlark/
- gavinhoward 3y ago
  I'm afraid I have bad news. The thing about Starlark is that it is effectively Turing complete. You can simulate a while loop with a long enough foreach, an if statement, and a break statement. And by "long enough," I mean long enough to make anyone just stop the build, but hey, if I stack the foreach loops, I can make the inner loop exponentially long, even though ints are limited to 32 bits. By not taking out the break keyword, Starlark remains Turing-complete. And yes, I tested this.
  - talideon 3y ago
    That's not Turing complete. That's still primitive recursive. You're going out of your way make it loop for a really, really long time, but it's still guaranteed to terminate, and thus isn't Turing complete, even if it'd take a really long time to terminate. This isn't some theoretical thing: it's how you can introduce guarantees into the system that it can't get out of control by accident and it also makes it easier to reason about the system. A primitive recursive language gets you most of the benefits of a general recursive one _without_ needing to be _actually_ Turing complete. So this isn't "bad news" for me. I'm saying that I think you should avoid using a general recursive (i.e. Turing complete) language for a build system and instead have guarantees that the system will terminate by using a primitive recursive language of some kind because you literally, as you did, need to go out of your way to get code written in such a fashion to run for an inordinate amount of time. And yet, an inordinate amount of time isn't forever and unlike a program written in an actually Turing complete language, you can use static analysis to detect that kind of thing if only primitive recursion is supported.

JohnFen 3y ago

> why should there even exist a build system independent from a programming language now? I want my build system to be independent of the programming language for two reasons: 1) What gavinhoward said: I regularly use several different languages, sometimes in the same project, sometimes not. But in all cases, I want to use the same build system in order to minimize my environmental complexity. Much like I want to use the same IDE regardless of language. 2) I don't want to risk version dependencies. I want to be able to use the same build system even when I'm compiling something using an old version of a compiler, and I don't want to be forced into a build system upgrade just because I want to upgrade one of my compilers. For me, the separation of the two roles is very important. If they were tied together, it would mean that I'd have to add complexity elsewhere to cover my use cases. Languages that include their own package managers (like Rust, Go, etc.) give me headaches.

eadmund 3y ago
> I regularly use several different languages, sometimes in the same project, sometimes not. But in all cases, I want to use the same build system in order to minimize my environmental complexity. It’s not obvious to me that one can solve the problem of making N languages co-operate by adding another. Why not just use the most powerful language to orchestrate everything? > I don't want to risk version dependencies. I want to be able to use the same build system even when I'm compiling something using an old version of a compiler, and I don't want to be forced into a build system upgrade just because I want to upgrade one of my compilers. That begs the question of whether compiler upgrades should force build system upgrades. In a mature programming language, this should never be the case, although practically speaking it’s definitely possible (e.g. ASDF 3.3, released earlier this year but built on a language which has been static for thirty years, adds compatibility for Allegro Common Lisp!). Incidentally, ASDF is pretty cool: https://asdf.common-lisp.dev/ https://asdf.common-lisp.dev/
- JohnFen 3y ago
  > Why not just use the most powerful language to orchestrate everything? I usually do everything in a single language, but sometimes that's not the best solution. It's not a matter of how powerful the language is in a general sense, but how good it is at a particular kind of task. But even ignoring my projects that involve multiple languages simultaneously, I still regularly use several languages in total, even if only one per project. So having a build system that's independent of the language is still highly desirable. > That begs the question of whether compiler upgrades should force build system upgrades. My answer to that is "no, never", of course. That leads to increased complexity for me, in exchange for no gain. The build system should have no special knowledge or expectation of what language or language tools are being used.

zdragnar 3y ago

I was once on a project that inadvertently used ruby in some way for everything. Rails Bundler Vagrant Chef Opsworks configuration JavaScript bundling Generating PDFs Probably other things The system was basically frozen in time because too many cross-dependencies couldn't update in sync to something newer (the opsworks cookbook in chef were the worst offenders iirc). It was so bad when I started that I couldn't even get the system up and running locally. I ended up tossing hundreds of lines of random ruby code and with a few lines of docker commands had it up and running. Since then I've preferred keeping things nicely not dependent on each other whenever possible.

eadmund 3y ago

> If turing-completeness is necessary, why should there even exist a build system independent from a programming language now? It shouldn’t. The only issue is that most folks aren’t using powerful enough languages. As you note, using C++ as a scripting language would be pretty crazy. So don’t do that!

teo_zero 3y ago

> Isn't the obvious solution to script your language's default build system in precisely the language you're trying to build from? Please no. I don't want to handle filename and other string manipulation in C.

InitEnabler 3y ago

>The only real use case for Rig would be legacy languages that don't come with language-central packaging repositories, i.e., Fortran, C++, C. What about packages for *nix based OS's? dnf, pacman, apt, etc. essential have there own "meta" build systems to create packages for there particular package flavor weather that be rpm, deb, tar.gz etc.

scrubs 3y ago

"The only real use case for Rig would be legacy languages that don't come with language-central packaging repositories, i.e., Fortran, C++, C." C/C++ are big time numerous and serious real. And they need all the help they can get to do builds more like golang

shrimp_emoji 3y ago
Yeah. Less relevant one day? When C and C++ become less relevant? I hope you're willing to wait a century or two famalamadingdong