7 ms·

I wish there was some attempt to standardize the build tools for different languages. Every language today has its own, it's crazy. At least makers of the make

by js8 9y ago

I wish there was some attempt to standardize the build tools for different languages. Every language today has its own, it's crazy. At least makers of the make understood this.

pathsjs 9y ago
Usually it is because as soon as the build gets a little complex, you need to define your own logic. In a minute, you are scripting your build tool with a Turing complete language. And of course users of language X want to script their build using X. That just makes sense - define a build DSL inside your own language. For some reasons, though, the de facto build DSL used by Scala suffers by all the issues mentioned by Li Haoyi in his blog post
- wink 9y ago
  Of course it will be hard, but most build tools often do the same thing - no matter the language or ecosystem. I haven't seen any real effort to try to standardize on APIs, so you could write your "plugins" in your language of choice and the "glue" code would be this language-independent build tool. I'd say I've mostly given up on this. I have a Makefile for every project, which in turn of course uses the language-specific build tool, but for example when I wanted to use webpack (migrating from a mess of gulp+bower+npm from angular.js to just webpack for angular) I wrote a simple PoC Makefile with hacky sed-/awk-/lines and a lot of cp and cat in just a few hours (massive build script). Then the result was working as intended and I could refactor all the stuff where there's a nice webpack plugin and get rid of my Makefile hacks. Now in the end the Makefile just provides a common starting point that has "executable docs" for every language specific tool. "make bootstrap/test/run/build", no matter what language the project is in - for the low cost of having to update the Makefile when you change something major or some arguments. Not advocating this for all orgs (please use what makes YOUR team happy) but I've heard nothing but positive things about this from my team.
  - Someone 9y ago
    https://github.com/apple/swift-llbuild https://github.com/apple/swift-llbuild may be too low-level for what you envision, but captures a lot of your ideas, I think. https://llvm.org/devmtg/2016-11/#abstracts https://llvm.org/devmtg/2016-11/#abstracts has a presentation. It focuses on compiling with clang, but the system is more flexible. Slides at https://llvm.org/devmtg/2016-11/Slides/Dunbar-NewArchitectureForBuildingSoftware.pdf https://llvm.org/devmtg/2016-11/Slides/Dunbar-NewArchitectur..., matching video is at https://youtu.be/b_T-eCToX1I https://youtu.be/b_T-eCToX1I.
- js8 9y ago
  I am not sure if build scripts really need to be Turing complete. Wouldn't something like Dhall be appropriate?
  - oblio 9y ago
    It would be glorious if anyone could understand it. After reading their readme (https://github.com/dhall-lang/dhall-lang https://github.com/dhall-lang/dhall-lang) for about 10 minutes, actually trying to understand it, I don't. That's a problem. A big one. Maybe you could explain for the feeble-minded such as myself: - how would you represent a simple key=value in dhall? - how yould you represent a dictionary in dhall (multiple key=value entries)? - how would you represent an array in dhall? - maybe a more complex structure, say, translate (by hand) a package.json file to dhall? (Nevermind, I found the tutorial. But the readme page is IMO a big fail if dhall wants wide adoption. If it doesn't then, ignore my criticism :) )
  - crdoconnor 9y ago
    I am. I need turing complete code to build stuff very regularly. I'd rather we aimed to make build scripts that are turing complete and as short and simple as possible rather than non-turing complete.
    - Someone 9y ago
      Can you explain what you do that requires a Turing complete build language and couldn’t be done, for example, in the ‘make equivalent’ of the dtrace scripting language?
      - crdoconnor 9y ago
        * Checking some condition before building something in a particular way. * Retrying something if it fails. * A hundred other different scenarios which crop up occasionally, require about 4 lines of turing complete code, and are an absolute bitch to handle in a language deliberately designed not to make it possible.
- deleted 9y ago
  [deleted]
pjmlp 9y ago
Which is why make sucks for any language that isn't C.
- imtringued 9y ago
  It also sucks for C and C++. You need to use a compiler flag to generate the header dependencies or risk subtle bugs happening during incremental compilation if you forgot to update them manually.
  - pjmlp 9y ago
    True, make deps is a common target on my makefiles.
dtech 9y ago
However, make adds little value in comparison to a "language-specific" build tool like SBT. In SBT, I can add `crossScalaVersions := Seq("2.12.0", "2.11")` and everything will be (more or less) automatically compiled and published for both Scala 2.11 and 2.12. That's not something a more generic build tool can ever really hope to achieve.
- gaius 9y ago
  everything will be (more or less) automatically compiled and published for both Scala 2.11 and 2.12 But, having to worry about minor version differences like this is a wholly unnecessary problem, and one that didn't really exist in the world that make comes from. These days we build tools on top of tools to solve problems of our own making that just didn't exist 10 or 20 years ago. All the build tools, packaging tools, deployment tools, containers, blah blah blah. None of that crap is needed to deliver working software in the form of 1 statically linked binary + 1 text configuration file, which will suffice for any software in the world.
  - virtualwhys 9y ago
    > But, having to worry about minor version differences like this is a wholly unnecessary problem 2.11 and 2.12 are major Scala versions. Current minor version is 2.12.4
    - simon_o 9y ago
      But having to worry about compatibility between minor versions is still bad. People have come to expect that minor versions are interchangeable (modulo fixes for clear regressions and changes made in reaction to important outside events like major releases of Java).
      - dtech 9y ago
        The minor versions are interchangeable. An increase to the second number is a new (binary incompatible) major version occurring every 1-3 years. It's unusual but not unique, see Python, Linux kernel, Postgres.
SOLAR_FIELDS 9y ago
You have this pretty much in Gradle for the JVM. It's become the de-facto standard for building mixed-JVM language projects in my company because it has pretty good plugin support for pretty much everything JVM
pacala 9y ago
Blaze [Google's internal tool], gets it right. Available as open source at https://bazel.build https://bazel.build, plus a clone at https://www.pantsbuild.org https://www.pantsbuild.org.
kodablah 9y ago
> At least makers of the make understood this. Once you need to do anything moderately complex, people drop into os-specific scripts for language-specific tasks. I think people are just gonna realize the best build conf/file is a real-yet-simple language (Go?) whereby features are built as libs to depend on. I have built a couple of more complex cross-platform build scripts this way. We're developers, why can't we treat our build scripts like the rest of our code? (granted cargo is a great middle ground, but language specific)
- stevedonovan 9y ago
  Agreed, except for one thing - it works best when a build-tool language is mostly used declaratively, except for those little cases when you need imperative as well. I had great fun building Lake (build tool in Lua)

by js8 9y ago

I wish there was some attempt to standardize the build tools for different languages. Every language today has its own, it's crazy. At least makers of the make understood this.

pathsjs 9y ago

Usually it is because as soon as the build gets a little complex, you need to define your own logic. In a minute, you are scripting your build tool with a Turing complete language. And of course users of language X want to script their build using X. That just makes sense - define a build DSL inside your own language. For some reasons, though, the de facto build DSL used by Scala suffers by all the issues mentioned by Li Haoyi in his blog post

wink 9y ago
Of course it will be hard, but most build tools often do the same thing - no matter the language or ecosystem. I haven't seen any real effort to try to standardize on APIs, so you could write your "plugins" in your language of choice and the "glue" code would be this language-independent build tool. I'd say I've mostly given up on this. I have a Makefile for every project, which in turn of course uses the language-specific build tool, but for example when I wanted to use webpack (migrating from a mess of gulp+bower+npm from angular.js to just webpack for angular) I wrote a simple PoC Makefile with hacky sed-/awk-/lines and a lot of cp and cat in just a few hours (massive build script). Then the result was working as intended and I could refactor all the stuff where there's a nice webpack plugin and get rid of my Makefile hacks. Now in the end the Makefile just provides a common starting point that has "executable docs" for every language specific tool. "make bootstrap/test/run/build", no matter what language the project is in - for the low cost of having to update the Makefile when you change something major or some arguments. Not advocating this for all orgs (please use what makes YOUR team happy) but I've heard nothing but positive things about this from my team.
- Someone 9y ago
  https://github.com/apple/swift-llbuild https://github.com/apple/swift-llbuild may be too low-level for what you envision, but captures a lot of your ideas, I think. https://llvm.org/devmtg/2016-11/#abstracts https://llvm.org/devmtg/2016-11/#abstracts has a presentation. It focuses on compiling with clang, but the system is more flexible. Slides at https://llvm.org/devmtg/2016-11/Slides/Dunbar-NewArchitectureForBuildingSoftware.pdf https://llvm.org/devmtg/2016-11/Slides/Dunbar-NewArchitectur..., matching video is at https://youtu.be/b_T-eCToX1I https://youtu.be/b_T-eCToX1I.
js8 9y ago
I am not sure if build scripts really need to be Turing complete. Wouldn't something like Dhall be appropriate?
- oblio 9y ago
  It would be glorious if anyone could understand it. After reading their readme (https://github.com/dhall-lang/dhall-lang https://github.com/dhall-lang/dhall-lang) for about 10 minutes, actually trying to understand it, I don't. That's a problem. A big one. Maybe you could explain for the feeble-minded such as myself: - how would you represent a simple key=value in dhall? - how yould you represent a dictionary in dhall (multiple key=value entries)? - how would you represent an array in dhall? - maybe a more complex structure, say, translate (by hand) a package.json file to dhall? (Nevermind, I found the tutorial. But the readme page is IMO a big fail if dhall wants wide adoption. If it doesn't then, ignore my criticism :) )
- crdoconnor 9y ago
  I am. I need turing complete code to build stuff very regularly. I'd rather we aimed to make build scripts that are turing complete and as short and simple as possible rather than non-turing complete.
  - Someone 9y ago
    Can you explain what you do that requires a Turing complete build language and couldn’t be done, for example, in the ‘make equivalent’ of the dtrace scripting language?
    - crdoconnor 9y ago
      * Checking some condition before building something in a particular way. * Retrying something if it fails. * A hundred other different scenarios which crop up occasionally, require about 4 lines of turing complete code, and are an absolute bitch to handle in a language deliberately designed not to make it possible.
deleted 9y ago
[deleted]

pjmlp 9y ago

Which is why make sucks for any language that isn't C.

imtringued 9y ago
It also sucks for C and C++. You need to use a compiler flag to generate the header dependencies or risk subtle bugs happening during incremental compilation if you forgot to update them manually.
- pjmlp 9y ago
  True, make deps is a common target on my makefiles.

dtech 9y ago

However, make adds little value in comparison to a "language-specific" build tool like SBT. In SBT, I can add `crossScalaVersions := Seq("2.12.0", "2.11")` and everything will be (more or less) automatically compiled and published for both Scala 2.11 and 2.12. That's not something a more generic build tool can ever really hope to achieve.

gaius 9y ago
everything will be (more or less) automatically compiled and published for both Scala 2.11 and 2.12 But, having to worry about minor version differences like this is a wholly unnecessary problem, and one that didn't really exist in the world that make comes from. These days we build tools on top of tools to solve problems of our own making that just didn't exist 10 or 20 years ago. All the build tools, packaging tools, deployment tools, containers, blah blah blah. None of that crap is needed to deliver working software in the form of 1 statically linked binary + 1 text configuration file, which will suffice for any software in the world.
- virtualwhys 9y ago
  > But, having to worry about minor version differences like this is a wholly unnecessary problem 2.11 and 2.12 are major Scala versions. Current minor version is 2.12.4
  - simon_o 9y ago
    But having to worry about compatibility between minor versions is still bad. People have come to expect that minor versions are interchangeable (modulo fixes for clear regressions and changes made in reaction to important outside events like major releases of Java).
    - dtech 9y ago
      The minor versions are interchangeable. An increase to the second number is a new (binary incompatible) major version occurring every 1-3 years. It's unusual but not unique, see Python, Linux kernel, Postgres.

SOLAR_FIELDS 9y ago

You have this pretty much in Gradle for the JVM. It's become the de-facto standard for building mixed-JVM language projects in my company because it has pretty good plugin support for pretty much everything JVM

pacala 9y ago

Blaze [Google's internal tool], gets it right. Available as open source at https://bazel.build https://bazel.build, plus a clone at https://www.pantsbuild.org https://www.pantsbuild.org.

kodablah 9y ago

> At least makers of the make understood this. Once you need to do anything moderately complex, people drop into os-specific scripts for language-specific tasks. I think people are just gonna realize the best build conf/file is a real-yet-simple language (Go?) whereby features are built as libs to depend on. I have built a couple of more complex cross-platform build scripts this way. We're developers, why can't we treat our build scripts like the rest of our code? (granted cargo is a great middle ground, but language specific)

stevedonovan 9y ago
Agreed, except for one thing - it works best when a build-tool language is mostly used declaratively, except for those little cases when you need imperative as well. I had great fun building Lake (build tool in Lua)