4 ms·

I'm having a hard time thinking of a use case for complex data structures in a build script. Can you give a real world example? Array is the only one that I ha

by adtac 9y ago

I'm having a hard time thinking of a use case for complex data structures in a build script. Can you give a real world example? Array is the only one that I have found lacking, in my experience.

vog 9y ago
If you aim for slightly larger projects and efficiency, you'd need at least a DAG (because that's the fundamental nature of a dependency graph). You can find a nice explanation in the "tup" build tool, although there are of course many others: http://gittup.org/tup/ http://gittup.org/tup/ And yes, you can express DAGs with text and/or filesystem operations (plus symlinks), but not efficiently and writing code that operates on that level is cumbersome.
- akkartik 9y ago
  Ah, thanks for the pointer to `tup`! I'd spent a while looking for it a few months ago but couldn't remember the name.
- chenglou 9y ago
  Tup is pretty darn amazing. I've never had such a high success rate while writing build rules under it. "Once it builds, it has no bug". Not to mention the great speed to begin with. Is there any similar build system? I know of https://github.com/droundy/fac https://github.com/droundy/fac, but it's not cross-platform
  - dom0 9y ago
    tup sounds similar to qbs. I found qbs to be a most amazing C++ build system, it's the only one I used that is 1) reasonably easy to use 2) very fast 3) very correct Most build systems fail 3), usually 2) and often also 1).
    - CJefferson 9y ago
      What makes tup amazing is it traces which files a program touches, so assuming your programs obey the rule "I need to rerun if I have different command line arguments or something I read has changed", you get perfect dependencies for any program. I use it for running experiments with custom programs and scripts all the time, and it just refund the parts I need.
- svckr 9y ago
  Regarding DAG: tsort (1) sorts input lines by interpreting them as a list of edges. I guess that's what you where referring to when you said "cumbersome" :) $ cat > f a c a d b c ^D $ tsort f a b d c
  - vog 9y ago
    The problem is not tsort, it is adding, removing, updating items in such a text file. (not to mention nasty corner cases like filenames containing whitespaces, or the security implications of filenames containing newlines)
cbcoutinho 9y ago
I've used a project in the past that was a number of shell scripts that were written to handle cross-platform and (at least on Linux) various shells (bash, zsh, etc.). It can be done, but it becomes a complete kludge very quickly.
johnt15 9y ago
Any large project needs a build system with complex data structures to express all the dependencies just due to sheer amount of them. Add various types of source generation, tooling, non-trivial build steps and simple build systems quickly become infeasible.
dom0 9y ago
> I'm having a hard time thinking of a use case for complex data structures in a build script. Can you give a real world example? Look at how a modern build system (like qbs) organizes the build. It builds a declarative object graph describing the project and maps that to a dependency graph where each node can be a relatively complex object ("compile C++ file X using flags Y" or "link LargeSetOfFiles") annotated with modification data (like input file hashes). (Then it compares the dependency graph to what is found on the system and creates a list of commands to execute in order to bring whatever is on the system to the desired state, i.e. create an up to date build).
chubot 9y ago
Hash tables are useful for expressing build variants -- some examples at the end of this comment: https://news.ycombinator.com/item?id=15043633 https://news.ycombinator.com/item?id=15043633 debug/release, internationalized builds, coverage builds, profile-directed feedback, running parameterized test suites, etc.

by adtac 9y ago

I'm having a hard time thinking of a use case for complex data structures in a build script. Can you give a real world example? Array is the only one that I have found lacking, in my experience.

vog 9y ago

If you aim for slightly larger projects and efficiency, you'd need at least a DAG (because that's the fundamental nature of a dependency graph). You can find a nice explanation in the "tup" build tool, although there are of course many others: http://gittup.org/tup/ http://gittup.org/tup/ And yes, you can express DAGs with text and/or filesystem operations (plus symlinks), but not efficiently and writing code that operates on that level is cumbersome.

akkartik 9y ago
Ah, thanks for the pointer to `tup`! I'd spent a while looking for it a few months ago but couldn't remember the name.
chenglou 9y ago
Tup is pretty darn amazing. I've never had such a high success rate while writing build rules under it. "Once it builds, it has no bug". Not to mention the great speed to begin with. Is there any similar build system? I know of https://github.com/droundy/fac https://github.com/droundy/fac, but it's not cross-platform
- dom0 9y ago
  tup sounds similar to qbs. I found qbs to be a most amazing C++ build system, it's the only one I used that is 1) reasonably easy to use 2) very fast 3) very correct Most build systems fail 3), usually 2) and often also 1).
  - CJefferson 9y ago
    What makes tup amazing is it traces which files a program touches, so assuming your programs obey the rule "I need to rerun if I have different command line arguments or something I read has changed", you get perfect dependencies for any program. I use it for running experiments with custom programs and scripts all the time, and it just refund the parts I need.
svckr 9y ago
Regarding DAG: tsort (1) sorts input lines by interpreting them as a list of edges. I guess that's what you where referring to when you said "cumbersome" :) $ cat > f a c a d b c ^D $ tsort f a b d c
- vog 9y ago
  The problem is not tsort, it is adding, removing, updating items in such a text file. (not to mention nasty corner cases like filenames containing whitespaces, or the security implications of filenames containing newlines)

cbcoutinho 9y ago

I've used a project in the past that was a number of shell scripts that were written to handle cross-platform and (at least on Linux) various shells (bash, zsh, etc.). It can be done, but it becomes a complete kludge very quickly.

johnt15 9y ago

Any large project needs a build system with complex data structures to express all the dependencies just due to sheer amount of them. Add various types of source generation, tooling, non-trivial build steps and simple build systems quickly become infeasible.

dom0 9y ago

> I'm having a hard time thinking of a use case for complex data structures in a build script. Can you give a real world example? Look at how a modern build system (like qbs) organizes the build. It builds a declarative object graph describing the project and maps that to a dependency graph where each node can be a relatively complex object ("compile C++ file X using flags Y" or "link LargeSetOfFiles") annotated with modification data (like input file hashes). (Then it compares the dependency graph to what is found on the system and creates a list of commands to execute in order to bring whatever is on the system to the desired state, i.e. create an up to date build).

chubot 9y ago

Hash tables are useful for expressing build variants -- some examples at the end of this comment: https://news.ycombinator.com/item?id=15043633 https://news.ycombinator.com/item?id=15043633 debug/release, internationalized builds, coverage builds, profile-directed feedback, running parameterized test suites, etc.