5 ms·

I applaud the effort, but is "pure /bin/sh" really a good feature in today's systems? To my knowledge, today every Unix system has Python, usually at least 2.7

by vog 9y ago

I applaud the effort, but is "pure /bin/sh" really a good feature in today's systems? To my knowledge, today every Unix system has Python, usually at least 2.7, if not Python 3. Some even replaced most of their system shell scripts with Python scripts. (Formerly, Perl had this status, but never really caught on in that area.) Python is a much cleaner language than any shell language, especially with regard to error handling. (yes, I'm aware of "set -eu") Also, if you occationally need some more elaborated data structures or algorithms, you can express these in a straight forward way, rather than a series of complicated (and usually very fragile) text processing steps.

akkartik 9y ago
Totally agreed; I'd use a different language if I needed even an array.
adtac 9y ago
I'm having a hard time thinking of a use case for complex data structures in a build script. Can you give a real world example? Array is the only one that I have found lacking, in my experience.
- vog 9y ago
  If you aim for slightly larger projects and efficiency, you'd need at least a DAG (because that's the fundamental nature of a dependency graph). You can find a nice explanation in the "tup" build tool, although there are of course many others: http://gittup.org/tup/ http://gittup.org/tup/ And yes, you can express DAGs with text and/or filesystem operations (plus symlinks), but not efficiently and writing code that operates on that level is cumbersome.
  - akkartik 9y ago
    Ah, thanks for the pointer to `tup`! I'd spent a while looking for it a few months ago but couldn't remember the name.
  - chenglou 9y ago
    Tup is pretty darn amazing. I've never had such a high success rate while writing build rules under it. "Once it builds, it has no bug". Not to mention the great speed to begin with. Is there any similar build system? I know of https://github.com/droundy/fac https://github.com/droundy/fac, but it's not cross-platform
    - dom0 9y ago
      tup sounds similar to qbs. I found qbs to be a most amazing C++ build system, it's the only one I used that is 1) reasonably easy to use 2) very fast 3) very correct Most build systems fail 3), usually 2) and often also 1).
      - CJefferson 9y ago
        What makes tup amazing is it traces which files a program touches, so assuming your programs obey the rule "I need to rerun if I have different command line arguments or something I read has changed", you get perfect dependencies for any program. I use it for running experiments with custom programs and scripts all the time, and it just refund the parts I need.
  - svckr 9y ago
    Regarding DAG: tsort (1) sorts input lines by interpreting them as a list of edges. I guess that's what you where referring to when you said "cumbersome" :) $ cat > f a c a d b c ^D $ tsort f a b d c
    - vog 9y ago
      The problem is not tsort, it is adding, removing, updating items in such a text file. (not to mention nasty corner cases like filenames containing whitespaces, or the security implications of filenames containing newlines)
- cbcoutinho 9y ago
  I've used a project in the past that was a number of shell scripts that were written to handle cross-platform and (at least on Linux) various shells (bash, zsh, etc.). It can be done, but it becomes a complete kludge very quickly.
- johnt15 9y ago
  Any large project needs a build system with complex data structures to express all the dependencies just due to sheer amount of them. Add various types of source generation, tooling, non-trivial build steps and simple build systems quickly become infeasible.
- dom0 9y ago
  > I'm having a hard time thinking of a use case for complex data structures in a build script. Can you give a real world example? Look at how a modern build system (like qbs) organizes the build. It builds a declarative object graph describing the project and maps that to a dependency graph where each node can be a relatively complex object ("compile C++ file X using flags Y" or "link LargeSetOfFiles") annotated with modification data (like input file hashes). (Then it compares the dependency graph to what is found on the system and creates a list of commands to execute in order to bring whatever is on the system to the desired state, i.e. create an up to date build).
- chubot 9y ago
  Hash tables are useful for expressing build variants -- some examples at the end of this comment: https://news.ycombinator.com/item?id=15043633 https://news.ycombinator.com/item?id=15043633 debug/release, internationalized builds, coverage builds, profile-directed feedback, running parameterized test suites, etc.
nailer 9y ago
Also most current Unixes have at least Bash. Bourne shell is surprisingly rare outside BSD.
- kingmanaz 9y ago
  Or Korn, which is better than Bash, IMHO.
majewsky 9y ago
I see you've not yet encountered Alpine-based Docker containers. $ docker run --rm -ti alpine sh / # bash sh: bash: not found / # python sh: python: not found / # perl sh: perl: not found / # ruby sh: ruby: not found Sure, you can install all of these with a single `apk add`, but it makes your container much fatter, i.e. every CI job that has to pull the image will be taking a longer time. Same for when you push the image into a far-away datacenter.
- user5994461 9y ago
  It's one of the reason that alpine should not be used outside of tests and toys projects. It cannot be substituted to an usual distribution.
- wrs 9y ago
  In a base Alpine container, it would be awfully hard to build anything interesting. You are probably going to need to install some build tools first anyway. (Thus the two-stage container pattern.)
swiley 9y ago
Python never works right for me, having to deal with yet another dependency just to use the build system is obnoxious. Just use make. It's not hard and it works even for fairly large projects.
edwinyzh 9y ago
Speaking of using Python or Node.js as a language for a build process, there were so many times it failed when I trying to run the build script included in the github open source project I found interesting and downloaded to my local file system, and usually it's a project not written in my familiar language, and since I'm NOT familiar with Python nor node.js, so when the build script emit something like a dependency error, after several tries without success, I usually just give up.

by vog 9y ago

akkartik 9y ago

Totally agreed; I'd use a different language if I needed even an array.

adtac 9y ago

I'm having a hard time thinking of a use case for complex data structures in a build script. Can you give a real world example? Array is the only one that I have found lacking, in my experience.

vog 9y ago
If you aim for slightly larger projects and efficiency, you'd need at least a DAG (because that's the fundamental nature of a dependency graph). You can find a nice explanation in the "tup" build tool, although there are of course many others: http://gittup.org/tup/ http://gittup.org/tup/ And yes, you can express DAGs with text and/or filesystem operations (plus symlinks), but not efficiently and writing code that operates on that level is cumbersome.
- akkartik 9y ago
  Ah, thanks for the pointer to `tup`! I'd spent a while looking for it a few months ago but couldn't remember the name.
- chenglou 9y ago
  Tup is pretty darn amazing. I've never had such a high success rate while writing build rules under it. "Once it builds, it has no bug". Not to mention the great speed to begin with. Is there any similar build system? I know of https://github.com/droundy/fac https://github.com/droundy/fac, but it's not cross-platform
  - dom0 9y ago
    tup sounds similar to qbs. I found qbs to be a most amazing C++ build system, it's the only one I used that is 1) reasonably easy to use 2) very fast 3) very correct Most build systems fail 3), usually 2) and often also 1).
    - CJefferson 9y ago
      What makes tup amazing is it traces which files a program touches, so assuming your programs obey the rule "I need to rerun if I have different command line arguments or something I read has changed", you get perfect dependencies for any program. I use it for running experiments with custom programs and scripts all the time, and it just refund the parts I need.
- svckr 9y ago
  Regarding DAG: tsort (1) sorts input lines by interpreting them as a list of edges. I guess that's what you where referring to when you said "cumbersome" :) $ cat > f a c a d b c ^D $ tsort f a b d c
  - vog 9y ago
    The problem is not tsort, it is adding, removing, updating items in such a text file. (not to mention nasty corner cases like filenames containing whitespaces, or the security implications of filenames containing newlines)
cbcoutinho 9y ago
I've used a project in the past that was a number of shell scripts that were written to handle cross-platform and (at least on Linux) various shells (bash, zsh, etc.). It can be done, but it becomes a complete kludge very quickly.
johnt15 9y ago
Any large project needs a build system with complex data structures to express all the dependencies just due to sheer amount of them. Add various types of source generation, tooling, non-trivial build steps and simple build systems quickly become infeasible.
dom0 9y ago
> I'm having a hard time thinking of a use case for complex data structures in a build script. Can you give a real world example? Look at how a modern build system (like qbs) organizes the build. It builds a declarative object graph describing the project and maps that to a dependency graph where each node can be a relatively complex object ("compile C++ file X using flags Y" or "link LargeSetOfFiles") annotated with modification data (like input file hashes). (Then it compares the dependency graph to what is found on the system and creates a list of commands to execute in order to bring whatever is on the system to the desired state, i.e. create an up to date build).
chubot 9y ago
Hash tables are useful for expressing build variants -- some examples at the end of this comment: https://news.ycombinator.com/item?id=15043633 https://news.ycombinator.com/item?id=15043633 debug/release, internationalized builds, coverage builds, profile-directed feedback, running parameterized test suites, etc.

nailer 9y ago

Also most current Unixes have at least Bash. Bourne shell is surprisingly rare outside BSD.

kingmanaz 9y ago
Or Korn, which is better than Bash, IMHO.

majewsky 9y ago

I see you've not yet encountered Alpine-based Docker containers. $ docker run --rm -ti alpine sh / # bash sh: bash: not found / # python sh: python: not found / # perl sh: perl: not found / # ruby sh: ruby: not found Sure, you can install all of these with a single `apk add`, but it makes your container much fatter, i.e. every CI job that has to pull the image will be taking a longer time. Same for when you push the image into a far-away datacenter.

user5994461 9y ago
It's one of the reason that alpine should not be used outside of tests and toys projects. It cannot be substituted to an usual distribution.
wrs 9y ago
In a base Alpine container, it would be awfully hard to build anything interesting. You are probably going to need to install some build tools first anyway. (Thus the two-stage container pattern.)

swiley 9y ago

Python never works right for me, having to deal with yet another dependency just to use the build system is obnoxious. Just use make. It's not hard and it works even for fairly large projects.

edwinyzh 9y ago

Speaking of using Python or Node.js as a language for a build process, there were so many times it failed when I trying to run the build script included in the github open source project I found interesting and downloaded to my local file system, and usually it's a project not written in my familiar language, and since I'm NOT familiar with Python nor node.js, so when the build script emit something like a dependency error, after several tries without success, I usually just give up.