5 ms·

Autoconf is one of the absolutely hilarious things about UNIX. On the one hand, we've got people optimizing kernels down to the individual instructions (often d

by titzer 3y ago

Autoconf is one of the absolutely hilarious things about UNIX. On the one hand, we've got people optimizing kernels down to the individual instructions (often doing very unsafe dirty C tricks underneath), sometimes with super-clunky and overly complex APIs as a result...and on the other hand you have all the shell script absolute nuttery like the behemoth heap of kludges that is autoconf. It's crazy to me the disconnect. And, oh by the way, underneath? Shells have some of the absolutely bonkersly dumb parsers and interpreters; absolutely embarrassingly dumb stuff like aliases that can override keywords and change the parsing of shell script. The fact that some (most) shells interpret a shell script one line at a time, and that "built-ins" like the syntax for conditions in ifs looks like syntax but might be little binaries underneath (look up how "[" and "]" is handled in some shells--it might not be!). What a wild irony that all the ickiest parts of UNIX--shells and scripts and autoconf and all that stringly-typed pipe stuff, ended up becoming (IMHO) the most reusable, pluggable programming environment yet devised...

nabla9 3y ago
Alan Perlis Epigram #9: "It is better to have 100 functions operate on one data structure than 10 functions on 10 data structures." Unix settled for string as a data structure and it's little too low level. All Lisp programmers know that s-expressions would have been the right choice. JSON works also.
- chubot 3y ago
  Yup, I call that the Perlis-Thompson Principle -- because Ken Thompson made a similar combinatorial argument about software composition: you should design it around "one thing". https://www.oilshell.org/blog/2021/07/blog-backlog-1.html#concepts https://www.oilshell.org/blog/2021/07/blog-backlog-1.html#co... Files had structure on pre-Unix OSes, but they don't on Unix, because it doesn't compose. The "Uniform Interface Constraint" of REST is the same thing, and resembles a file system -- everything is GET / POST on URLs. --- So the reason that shell/Unix and HTTP are so common is a MATHEMATICAL property of software growth. How do you make a Zig program talk to a Mojo program? Probably with a byte stream. What about a Clojure program and a Common Lisp program? Probably a byte stream. (Ironically, S-expressions have no commonly used "exterior" interchange format) Every time a new language is introduced, I think "well there's another reason you're going to need a shell script". --- The larger the system, the more heterogeneous it is. And software is larger now, which is why MORE GLUE is needed. This is why shell was the #6 fastest growing language on Github in 2022: https://octoverse.github.com/2022/top-programming-languages https://octoverse.github.com/2022/top-programming-languages And the #1 fastest growing language is HCL, which is a very closely related form of glue. It's basically "shell for distributed systems" --- https://www.oilshell.org/ https://www.oilshell.org/ has JSON as of a few months ago, and is now pure native code (no more Python) All the normal shell stuff works: osh$ ls */*.py | wc -l; whoami 327 andy But you can seamlessly create dicts and lists like JavaScript. They are fully GC'd data structures: osh$ var d = {name: 42} And then you can write them as JSON: osh$ json write (d) { "name": 42 } And you can also upgrade to ysh with one line: shopt --set ysh:upgrade (YSH isn't stable yet, but OSH is very stable) I think of the data languages as layered -- i.e. an entire json message literally is a string, and it can also contain strings: https://www.oilshell.org/blog/2023/06/ysh-design.html#exterior-designs-are-layered https://www.oilshell.org/blog/2023/06/ysh-design.html#exteri...
  - drekipus 3y ago
    What benefit is this? I mean, I love the look of it, but how can it help me from day dot?
    - chubot 3y ago
      There are many answers, but one situation is 1. you already have a bash script (e.g. in your CI) 2. you want to use an array or dict Bash arrays and dicts have all sorts of warts. You have to remember to use "${array[@]}" (exactly like that - https://www.oilshell.org/blog/2016/11/06.html https://www.oilshell.org/blog/2016/11/06.html ) You can upgrade to YSH, and start using our Python/JS like arrays and dicts instead. And then you can write them to JSON and upload them somewhere. --- Another situation is if you use ShellCheck, but you want even more strictness at RUNTIME, then you can do shopt --set strict:all Oils is very ambitious -- a complete overhaul of shell -- but that's a couple immediate things. https://www.oilshell.org/blog/2021/01/why-a-new-shell.html https://www.oilshell.org/blog/2021/01/why-a-new-shell.html
      - chubot 3y ago
        Idioms - https://www.oilshell.org/release/latest/doc/idioms.html https://www.oilshell.org/release/latest/doc/idioms.html
  - phoe-krk 3y ago
    > What about a Clojure program and a Common Lisp program? Probably a byte stream. (Ironically, S-expressions have no commonly used "exterior" interchange format) Probably because the details of S-expressions differ underneath the hood - a simple (foo ()) can mean a different thing on CL (where the second element is the symbol NIL) than on Scheme (where it is an empty list and not a symbol); also because Clojure introduces additional bracketing patterns, such as [] for arrays and {} for maps, which will confuse non-Clojures. And let's not talk about sexp variants of Lisp-Flavored Erlang. Also, if you squint hard enough, https://en.wikipedia.org/wiki/Canonical_S-expressions https://en.wikipedia.org/wiki/Canonical_S-expressions works. Also I have no idea if anyone ever cared enough to implement cross-language software transactional memory that would work across a process boundary, to close the gap between a Clojure process and a Lisp process. It could work in theory, but someone would need to do it - either get paid to do it, or have enough mojo of their own to write and bulletprove such a thing. And, since everyone serializes to JSON anyway, serializing to JSON is a decent enough choice for communicating across Lisps as well.
- Scarblac 3y ago
  IMO a big reason Python is popular is because so many big seemingly unrelated librarues (AI things, scipy, Pillow, pandas, GDAL rasters, matplotlib, etc) all work with the "Numpy array" datatype and can therefore do things with each other's data.
  - vineyardmike 3y ago
    Relatedly, this is why big companies (eg Google) all have internal standard interchange formats (eg Protos). It’s so every programmer and every service and every file stored can always be consumed or produced as needed. Even smaller companies I’ve seen end up with an “interfaces” or “types” repo that contains definitions of every external object.
    - deleted 3y ago
      [deleted]
  - chubot 3y ago
    Yup, I agree that Python composes like Unix does, which is a reason for its phenomenal growth: Narrow Waists Can Be Interior or Exterior: PyObject vs. Unix Files - https://www.oilshell.org/blog/2023/06/narrow-waist.html https://www.oilshell.org/blog/2023/06/narrow-waist.html The appendix links to: Why is Python the language of machine learning? https://news.ycombinator.com/item?id=29171519 https://news.ycombinator.com/item?id=29171519 In particular, I highly recommend this bit of history from NumPy creator Travis Oliphant: https://www.youtube.com/watch?v=gFEE3w7F0ww https://www.youtube.com/watch?v=gFEE3w7F0ww
- sevensor 3y ago
  What's worse, with autotools, is that while maybe only one programmer in 100 knows how to do ./configure; make; make install, only one in 10,000 knows m4. It's one of those odd corners of the ecosystem like troff. It would be very easy to slip by me an exploit written in m4, or troff for that matter, because I've never paid any attention to either.
k8svet 3y ago
> scripts and autoconf and all that stringly-typed pipe stuff, ended up becoming (IMHO) the most reusable, pluggable programming environment yet devised... I mean yes, but also only because the rest of computing is stuck in the same era. Meanwhile I'm over here popping nix-shells having my nice exact same nushell on every machine I could possibly want it on. IMO it's all symptoms of the same problems. Related, I don't know how people start new projects and use autotools. But then again I'd say the same about cmake. But then again I just yesterday read a nightmare Lobsters comment about the realities of meson at the edges. And then I just sigh, thankful that I'm a Rust hipster (and then deal with cmake at the edges all the same). Computers are fun!
- nyanpasu64 3y ago
  Which comment about Meson are you referring to?
  - k8svet 3y ago
    https://lobste.rs/s/zht9p5/strengths_weaknesses_opportunities#c_w7s2mg https://lobste.rs/s/zht9p5/strengths_weaknesses_opportunitie...
chubot 3y ago
Shells have some of the absolutely bonkersly dumb parsers and interpreters Oils fixes all this, it's a from-scratch reimplementation of shell with an upgrade path from OSH (compatible) to YSH (new). --- aliases that can override keywords and change the parsing of shell script You can control this with shopt -u expand_aliases in bash, and YSH turns it off by default. Oils is statically parsed: How To Parse Shell Like a Programming Language - https://www.oilshell.org/blog/2019/02/07.html https://www.oilshell.org/blog/2019/02/07.html --- look up how "[" and "]" is handled in some shells--it might not be! OSH runs POSIX [ as-is, for compatibility: osh$ [ x = x ] && echo yes yes But YSH nudges you toward a "saner" style: ysh$ [ x = x ] && echo yes [ x = x ] && echo yes ^ [ interactive ]:2: '[' should be invoked as 'test' (simple_test_builtin) It also gives you a more familiar syntax. It's for Python and JavaScript users who avoid shell: ysh$ if ('x' === 'x') { echo yes } yes --- A Tour of YSH - https://www.oilshell.org/release/latest/doc/ysh-tour.html https://www.oilshell.org/release/latest/doc/ysh-tour.html Why Create a New Shell? - https://www.oilshell.org/blog/2021/01/why-a-new-shell.html https://www.oilshell.org/blog/2021/01/why-a-new-shell.html (OSH is very stable and compatible, but YSH isn't stable yet. Getting there)
- justahuman74 3y ago
  I'm pretty excited about YSH As a heavy user of bash I'm fully aware of how unaware I am of everything that can go wrong.
  - chubot 3y ago
    Cool! If you use and know bash, right now OSH with shopt -s strict:all will prevent almost all of the "silent footguns". On the other hand, YSH has strict:all and then a bunch of NEW features. There's been good feedback recently, which has led to many concrete changes. So your experience can definitely influence the language! https://github.com/oilshell/oil/wiki/Where-To-Send-Feedback https://github.com/oilshell/oil/wiki/Where-To-Send-Feedback
- nindalf 3y ago
  I hope this can become the new standard shell. https://xkcd.com/927/ https://xkcd.com/927/
forrestthewoods 3y ago
> What a wild irony that all the ickiest parts of UNIX--shells and scripts and autoconf and all that stringly-typed pipe stuff, ended up becoming (IMHO) the most reusable, pluggable programming environment yet devised... I increasingly believe any shell script over ~128 “symbols” should be re-written in a “real” programming language. I’ll gladly take a slightly longer Python, Go, or Rust “script” over Bash hell every day of the week.
- calgoo 3y ago
  These days, apart from python, I end up rewriting a lot of bash in ansible as that’s what a lot of infrastructure engineers know and it’s somewhat decent at running the code.
pjmlp 3y ago
The UNIX haters book happened for a reason, and sadly many of the complaints have hardly changed. At least on cloud and mobile space, we have largely moved beyond that, in the age of serverless, managed containers and programming languages.
pjmlp 3y ago
The UNIX haters book happened for a reason, and sadly many of the complaints have hardly changed. At least on cloud and mobile space, we have largely moved beyond that, in the age of serverless, managed containers and programming languages. Smalltalk and Lisp based REPLs are just as programmable, even more so given the whole OS exposure, and is no wonder why Notebooks got adopted by the science community and are inspired by them.
matheusmoreira 3y ago
Note that quite a lot of these problems would just go away if people would just stop trying to conform to some "portable" POSIX nonsense. There's no reason why shell scripting needs to be this painful but people keep doing it to themselves by restricting themselves to the smallest possible subset of features, the most ancient ones at that. The so called "bashisms" for example make everything so much easier and less error prone but people act like it's wrong to use those features. When you read stuff about shell scripting there's usually an entire section on avoiding the evil bashisms and torturing the shell script until it runs in the mythical POSIX shell as if it was some kind of virtuous thing to do, like bash is pushing you into temptation or something. Just throw POSIX into the trash where it belongs. Then we can finally evolve past it. Also, shellcheck is a wonderful tool. The script isn't done until shellcheck stops complaining.

by titzer 3y ago

nabla9 3y ago

Alan Perlis Epigram #9: "It is better to have 100 functions operate on one data structure than 10 functions on 10 data structures." Unix settled for string as a data structure and it's little too low level. All Lisp programmers know that s-expressions would have been the right choice. JSON works also.

chubot 3y ago
Yup, I call that the Perlis-Thompson Principle -- because Ken Thompson made a similar combinatorial argument about software composition: you should design it around "one thing". https://www.oilshell.org/blog/2021/07/blog-backlog-1.html#concepts https://www.oilshell.org/blog/2021/07/blog-backlog-1.html#co... Files had structure on pre-Unix OSes, but they don't on Unix, because it doesn't compose. The "Uniform Interface Constraint" of REST is the same thing, and resembles a file system -- everything is GET / POST on URLs. --- So the reason that shell/Unix and HTTP are so common is a MATHEMATICAL property of software growth. How do you make a Zig program talk to a Mojo program? Probably with a byte stream. What about a Clojure program and a Common Lisp program? Probably a byte stream. (Ironically, S-expressions have no commonly used "exterior" interchange format) Every time a new language is introduced, I think "well there's another reason you're going to need a shell script". --- The larger the system, the more heterogeneous it is. And software is larger now, which is why MORE GLUE is needed. This is why shell was the #6 fastest growing language on Github in 2022: https://octoverse.github.com/2022/top-programming-languages https://octoverse.github.com/2022/top-programming-languages And the #1 fastest growing language is HCL, which is a very closely related form of glue. It's basically "shell for distributed systems" --- https://www.oilshell.org/ https://www.oilshell.org/ has JSON as of a few months ago, and is now pure native code (no more Python) All the normal shell stuff works: osh$ ls */*.py | wc -l; whoami 327 andy But you can seamlessly create dicts and lists like JavaScript. They are fully GC'd data structures: osh$ var d = {name: 42} And then you can write them as JSON: osh$ json write (d) { "name": 42 } And you can also upgrade to ysh with one line: shopt --set ysh:upgrade (YSH isn't stable yet, but OSH is very stable) I think of the data languages as layered -- i.e. an entire json message literally is a string, and it can also contain strings: https://www.oilshell.org/blog/2023/06/ysh-design.html#exterior-designs-are-layered https://www.oilshell.org/blog/2023/06/ysh-design.html#exteri...
- drekipus 3y ago
  What benefit is this? I mean, I love the look of it, but how can it help me from day dot?
  - chubot 3y ago
    There are many answers, but one situation is 1. you already have a bash script (e.g. in your CI) 2. you want to use an array or dict Bash arrays and dicts have all sorts of warts. You have to remember to use "${array[@]}" (exactly like that - https://www.oilshell.org/blog/2016/11/06.html https://www.oilshell.org/blog/2016/11/06.html ) You can upgrade to YSH, and start using our Python/JS like arrays and dicts instead. And then you can write them to JSON and upload them somewhere. --- Another situation is if you use ShellCheck, but you want even more strictness at RUNTIME, then you can do shopt --set strict:all Oils is very ambitious -- a complete overhaul of shell -- but that's a couple immediate things. https://www.oilshell.org/blog/2021/01/why-a-new-shell.html https://www.oilshell.org/blog/2021/01/why-a-new-shell.html
    - chubot 3y ago
      Idioms - https://www.oilshell.org/release/latest/doc/idioms.html https://www.oilshell.org/release/latest/doc/idioms.html
- phoe-krk 3y ago
  > What about a Clojure program and a Common Lisp program? Probably a byte stream. (Ironically, S-expressions have no commonly used "exterior" interchange format) Probably because the details of S-expressions differ underneath the hood - a simple (foo ()) can mean a different thing on CL (where the second element is the symbol NIL) than on Scheme (where it is an empty list and not a symbol); also because Clojure introduces additional bracketing patterns, such as [] for arrays and {} for maps, which will confuse non-Clojures. And let's not talk about sexp variants of Lisp-Flavored Erlang. Also, if you squint hard enough, https://en.wikipedia.org/wiki/Canonical_S-expressions https://en.wikipedia.org/wiki/Canonical_S-expressions works. Also I have no idea if anyone ever cared enough to implement cross-language software transactional memory that would work across a process boundary, to close the gap between a Clojure process and a Lisp process. It could work in theory, but someone would need to do it - either get paid to do it, or have enough mojo of their own to write and bulletprove such a thing. And, since everyone serializes to JSON anyway, serializing to JSON is a decent enough choice for communicating across Lisps as well.
Scarblac 3y ago
IMO a big reason Python is popular is because so many big seemingly unrelated librarues (AI things, scipy, Pillow, pandas, GDAL rasters, matplotlib, etc) all work with the "Numpy array" datatype and can therefore do things with each other's data.
- vineyardmike 3y ago
  Relatedly, this is why big companies (eg Google) all have internal standard interchange formats (eg Protos). It’s so every programmer and every service and every file stored can always be consumed or produced as needed. Even smaller companies I’ve seen end up with an “interfaces” or “types” repo that contains definitions of every external object.
  - deleted 3y ago
    [deleted]
- chubot 3y ago
  Yup, I agree that Python composes like Unix does, which is a reason for its phenomenal growth: Narrow Waists Can Be Interior or Exterior: PyObject vs. Unix Files - https://www.oilshell.org/blog/2023/06/narrow-waist.html https://www.oilshell.org/blog/2023/06/narrow-waist.html The appendix links to: Why is Python the language of machine learning? https://news.ycombinator.com/item?id=29171519 https://news.ycombinator.com/item?id=29171519 In particular, I highly recommend this bit of history from NumPy creator Travis Oliphant: https://www.youtube.com/watch?v=gFEE3w7F0ww https://www.youtube.com/watch?v=gFEE3w7F0ww
sevensor 3y ago
What's worse, with autotools, is that while maybe only one programmer in 100 knows how to do ./configure; make; make install, only one in 10,000 knows m4. It's one of those odd corners of the ecosystem like troff. It would be very easy to slip by me an exploit written in m4, or troff for that matter, because I've never paid any attention to either.

k8svet 3y ago

> scripts and autoconf and all that stringly-typed pipe stuff, ended up becoming (IMHO) the most reusable, pluggable programming environment yet devised... I mean yes, but also only because the rest of computing is stuck in the same era. Meanwhile I'm over here popping nix-shells having my nice exact same nushell on every machine I could possibly want it on. IMO it's all symptoms of the same problems. Related, I don't know how people start new projects and use autotools. But then again I'd say the same about cmake. But then again I just yesterday read a nightmare Lobsters comment about the realities of meson at the edges. And then I just sigh, thankful that I'm a Rust hipster (and then deal with cmake at the edges all the same). Computers are fun!

nyanpasu64 3y ago
Which comment about Meson are you referring to?
- k8svet 3y ago
  https://lobste.rs/s/zht9p5/strengths_weaknesses_opportunities#c_w7s2mg https://lobste.rs/s/zht9p5/strengths_weaknesses_opportunitie...

chubot 3y ago

Shells have some of the absolutely bonkersly dumb parsers and interpreters Oils fixes all this, it's a from-scratch reimplementation of shell with an upgrade path from OSH (compatible) to YSH (new). --- aliases that can override keywords and change the parsing of shell script You can control this with shopt -u expand_aliases in bash, and YSH turns it off by default. Oils is statically parsed: How To Parse Shell Like a Programming Language - https://www.oilshell.org/blog/2019/02/07.html https://www.oilshell.org/blog/2019/02/07.html --- look up how "[" and "]" is handled in some shells--it might not be! OSH runs POSIX [ as-is, for compatibility: osh$ [ x = x ] && echo yes yes But YSH nudges you toward a "saner" style: ysh$ [ x = x ] && echo yes [ x = x ] && echo yes ^ [ interactive ]:2: '[' should be invoked as 'test' (simple_test_builtin) It also gives you a more familiar syntax. It's for Python and JavaScript users who avoid shell: ysh$ if ('x' === 'x') { echo yes } yes --- A Tour of YSH - https://www.oilshell.org/release/latest/doc/ysh-tour.html https://www.oilshell.org/release/latest/doc/ysh-tour.html Why Create a New Shell? - https://www.oilshell.org/blog/2021/01/why-a-new-shell.html https://www.oilshell.org/blog/2021/01/why-a-new-shell.html (OSH is very stable and compatible, but YSH isn't stable yet. Getting there)

justahuman74 3y ago
I'm pretty excited about YSH As a heavy user of bash I'm fully aware of how unaware I am of everything that can go wrong.
- chubot 3y ago
  Cool! If you use and know bash, right now OSH with shopt -s strict:all will prevent almost all of the "silent footguns". On the other hand, YSH has strict:all and then a bunch of NEW features. There's been good feedback recently, which has led to many concrete changes. So your experience can definitely influence the language! https://github.com/oilshell/oil/wiki/Where-To-Send-Feedback https://github.com/oilshell/oil/wiki/Where-To-Send-Feedback
nindalf 3y ago
I hope this can become the new standard shell. https://xkcd.com/927/ https://xkcd.com/927/

forrestthewoods 3y ago

> What a wild irony that all the ickiest parts of UNIX--shells and scripts and autoconf and all that stringly-typed pipe stuff, ended up becoming (IMHO) the most reusable, pluggable programming environment yet devised... I increasingly believe any shell script over ~128 “symbols” should be re-written in a “real” programming language. I’ll gladly take a slightly longer Python, Go, or Rust “script” over Bash hell every day of the week.

calgoo 3y ago
These days, apart from python, I end up rewriting a lot of bash in ansible as that’s what a lot of infrastructure engineers know and it’s somewhat decent at running the code.

pjmlp 3y ago

The UNIX haters book happened for a reason, and sadly many of the complaints have hardly changed. At least on cloud and mobile space, we have largely moved beyond that, in the age of serverless, managed containers and programming languages. Smalltalk and Lisp based REPLs are just as programmable, even more so given the whole OS exposure, and is no wonder why Notebooks got adopted by the science community and are inspired by them.

matheusmoreira 3y ago

Note that quite a lot of these problems would just go away if people would just stop trying to conform to some "portable" POSIX nonsense. There's no reason why shell scripting needs to be this painful but people keep doing it to themselves by restricting themselves to the smallest possible subset of features, the most ancient ones at that. The so called "bashisms" for example make everything so much easier and less error prone but people act like it's wrong to use those features. When you read stuff about shell scripting there's usually an entire section on avoiding the evil bashisms and torturing the shell script until it runs in the mythical POSIX shell as if it was some kind of virtuous thing to do, like bash is pushing you into temptation or something. Just throw POSIX into the trash where it belongs. Then we can finally evolve past it. Also, shellcheck is a wonderful tool. The script isn't done until shellcheck stops complaining.