4 ms·

Alan Perlis Epigram #9: "It is better to have 100 functions operate on one data structure than 10 functions on 10 data structures." Unix settled for string as

by nabla9 3y ago

Alan Perlis Epigram #9: "It is better to have 100 functions operate on one data structure than 10 functions on 10 data structures." Unix settled for string as a data structure and it's little too low level. All Lisp programmers know that s-expressions would have been the right choice. JSON works also.

chubot 3y ago
Yup, I call that the Perlis-Thompson Principle -- because Ken Thompson made a similar combinatorial argument about software composition: you should design it around "one thing". https://www.oilshell.org/blog/2021/07/blog-backlog-1.html#concepts https://www.oilshell.org/blog/2021/07/blog-backlog-1.html#co... Files had structure on pre-Unix OSes, but they don't on Unix, because it doesn't compose. The "Uniform Interface Constraint" of REST is the same thing, and resembles a file system -- everything is GET / POST on URLs. --- So the reason that shell/Unix and HTTP are so common is a MATHEMATICAL property of software growth. How do you make a Zig program talk to a Mojo program? Probably with a byte stream. What about a Clojure program and a Common Lisp program? Probably a byte stream. (Ironically, S-expressions have no commonly used "exterior" interchange format) Every time a new language is introduced, I think "well there's another reason you're going to need a shell script". --- The larger the system, the more heterogeneous it is. And software is larger now, which is why MORE GLUE is needed. This is why shell was the #6 fastest growing language on Github in 2022: https://octoverse.github.com/2022/top-programming-languages https://octoverse.github.com/2022/top-programming-languages And the #1 fastest growing language is HCL, which is a very closely related form of glue. It's basically "shell for distributed systems" --- https://www.oilshell.org/ https://www.oilshell.org/ has JSON as of a few months ago, and is now pure native code (no more Python) All the normal shell stuff works: osh$ ls */*.py | wc -l; whoami 327 andy But you can seamlessly create dicts and lists like JavaScript. They are fully GC'd data structures: osh$ var d = {name: 42} And then you can write them as JSON: osh$ json write (d) { "name": 42 } And you can also upgrade to ysh with one line: shopt --set ysh:upgrade (YSH isn't stable yet, but OSH is very stable) I think of the data languages as layered -- i.e. an entire json message literally is a string, and it can also contain strings: https://www.oilshell.org/blog/2023/06/ysh-design.html#exterior-designs-are-layered https://www.oilshell.org/blog/2023/06/ysh-design.html#exteri...
- drekipus 3y ago
  What benefit is this? I mean, I love the look of it, but how can it help me from day dot?
  - chubot 3y ago
    There are many answers, but one situation is 1. you already have a bash script (e.g. in your CI) 2. you want to use an array or dict Bash arrays and dicts have all sorts of warts. You have to remember to use "${array[@]}" (exactly like that - https://www.oilshell.org/blog/2016/11/06.html https://www.oilshell.org/blog/2016/11/06.html ) You can upgrade to YSH, and start using our Python/JS like arrays and dicts instead. And then you can write them to JSON and upload them somewhere. --- Another situation is if you use ShellCheck, but you want even more strictness at RUNTIME, then you can do shopt --set strict:all Oils is very ambitious -- a complete overhaul of shell -- but that's a couple immediate things. https://www.oilshell.org/blog/2021/01/why-a-new-shell.html https://www.oilshell.org/blog/2021/01/why-a-new-shell.html
    - chubot 3y ago
      Idioms - https://www.oilshell.org/release/latest/doc/idioms.html https://www.oilshell.org/release/latest/doc/idioms.html
- phoe-krk 3y ago
  > What about a Clojure program and a Common Lisp program? Probably a byte stream. (Ironically, S-expressions have no commonly used "exterior" interchange format) Probably because the details of S-expressions differ underneath the hood - a simple (foo ()) can mean a different thing on CL (where the second element is the symbol NIL) than on Scheme (where it is an empty list and not a symbol); also because Clojure introduces additional bracketing patterns, such as [] for arrays and {} for maps, which will confuse non-Clojures. And let's not talk about sexp variants of Lisp-Flavored Erlang. Also, if you squint hard enough, https://en.wikipedia.org/wiki/Canonical_S-expressions https://en.wikipedia.org/wiki/Canonical_S-expressions works. Also I have no idea if anyone ever cared enough to implement cross-language software transactional memory that would work across a process boundary, to close the gap between a Clojure process and a Lisp process. It could work in theory, but someone would need to do it - either get paid to do it, or have enough mojo of their own to write and bulletprove such a thing. And, since everyone serializes to JSON anyway, serializing to JSON is a decent enough choice for communicating across Lisps as well.
Scarblac 3y ago
IMO a big reason Python is popular is because so many big seemingly unrelated librarues (AI things, scipy, Pillow, pandas, GDAL rasters, matplotlib, etc) all work with the "Numpy array" datatype and can therefore do things with each other's data.
- vineyardmike 3y ago
  Relatedly, this is why big companies (eg Google) all have internal standard interchange formats (eg Protos). It’s so every programmer and every service and every file stored can always be consumed or produced as needed. Even smaller companies I’ve seen end up with an “interfaces” or “types” repo that contains definitions of every external object.
  - deleted 3y ago
    [deleted]
- chubot 3y ago
  Yup, I agree that Python composes like Unix does, which is a reason for its phenomenal growth: Narrow Waists Can Be Interior or Exterior: PyObject vs. Unix Files - https://www.oilshell.org/blog/2023/06/narrow-waist.html https://www.oilshell.org/blog/2023/06/narrow-waist.html The appendix links to: Why is Python the language of machine learning? https://news.ycombinator.com/item?id=29171519 https://news.ycombinator.com/item?id=29171519 In particular, I highly recommend this bit of history from NumPy creator Travis Oliphant: https://www.youtube.com/watch?v=gFEE3w7F0ww https://www.youtube.com/watch?v=gFEE3w7F0ww
sevensor 3y ago
What's worse, with autotools, is that while maybe only one programmer in 100 knows how to do ./configure; make; make install, only one in 10,000 knows m4. It's one of those odd corners of the ecosystem like troff. It would be very easy to slip by me an exploit written in m4, or troff for that matter, because I've never paid any attention to either.

by nabla9 3y ago

chubot 3y ago

Yup, I call that the Perlis-Thompson Principle -- because Ken Thompson made a similar combinatorial argument about software composition: you should design it around "one thing". https://www.oilshell.org/blog/2021/07/blog-backlog-1.html#concepts https://www.oilshell.org/blog/2021/07/blog-backlog-1.html#co... Files had structure on pre-Unix OSes, but they don't on Unix, because it doesn't compose. The "Uniform Interface Constraint" of REST is the same thing, and resembles a file system -- everything is GET / POST on URLs. --- So the reason that shell/Unix and HTTP are so common is a MATHEMATICAL property of software growth. How do you make a Zig program talk to a Mojo program? Probably with a byte stream. What about a Clojure program and a Common Lisp program? Probably a byte stream. (Ironically, S-expressions have no commonly used "exterior" interchange format) Every time a new language is introduced, I think "well there's another reason you're going to need a shell script". --- The larger the system, the more heterogeneous it is. And software is larger now, which is why MORE GLUE is needed. This is why shell was the #6 fastest growing language on Github in 2022: https://octoverse.github.com/2022/top-programming-languages https://octoverse.github.com/2022/top-programming-languages And the #1 fastest growing language is HCL, which is a very closely related form of glue. It's basically "shell for distributed systems" --- https://www.oilshell.org/ https://www.oilshell.org/ has JSON as of a few months ago, and is now pure native code (no more Python) All the normal shell stuff works: osh$ ls */*.py | wc -l; whoami 327 andy But you can seamlessly create dicts and lists like JavaScript. They are fully GC'd data structures: osh$ var d = {name: 42} And then you can write them as JSON: osh$ json write (d) { "name": 42 } And you can also upgrade to ysh with one line: shopt --set ysh:upgrade (YSH isn't stable yet, but OSH is very stable) I think of the data languages as layered -- i.e. an entire json message literally is a string, and it can also contain strings: https://www.oilshell.org/blog/2023/06/ysh-design.html#exterior-designs-are-layered https://www.oilshell.org/blog/2023/06/ysh-design.html#exteri...

drekipus 3y ago
What benefit is this? I mean, I love the look of it, but how can it help me from day dot?
- chubot 3y ago
  There are many answers, but one situation is 1. you already have a bash script (e.g. in your CI) 2. you want to use an array or dict Bash arrays and dicts have all sorts of warts. You have to remember to use "${array[@]}" (exactly like that - https://www.oilshell.org/blog/2016/11/06.html https://www.oilshell.org/blog/2016/11/06.html ) You can upgrade to YSH, and start using our Python/JS like arrays and dicts instead. And then you can write them to JSON and upload them somewhere. --- Another situation is if you use ShellCheck, but you want even more strictness at RUNTIME, then you can do shopt --set strict:all Oils is very ambitious -- a complete overhaul of shell -- but that's a couple immediate things. https://www.oilshell.org/blog/2021/01/why-a-new-shell.html https://www.oilshell.org/blog/2021/01/why-a-new-shell.html
  - chubot 3y ago
    Idioms - https://www.oilshell.org/release/latest/doc/idioms.html https://www.oilshell.org/release/latest/doc/idioms.html
phoe-krk 3y ago
> What about a Clojure program and a Common Lisp program? Probably a byte stream. (Ironically, S-expressions have no commonly used "exterior" interchange format) Probably because the details of S-expressions differ underneath the hood - a simple (foo ()) can mean a different thing on CL (where the second element is the symbol NIL) than on Scheme (where it is an empty list and not a symbol); also because Clojure introduces additional bracketing patterns, such as [] for arrays and {} for maps, which will confuse non-Clojures. And let's not talk about sexp variants of Lisp-Flavored Erlang. Also, if you squint hard enough, https://en.wikipedia.org/wiki/Canonical_S-expressions https://en.wikipedia.org/wiki/Canonical_S-expressions works. Also I have no idea if anyone ever cared enough to implement cross-language software transactional memory that would work across a process boundary, to close the gap between a Clojure process and a Lisp process. It could work in theory, but someone would need to do it - either get paid to do it, or have enough mojo of their own to write and bulletprove such a thing. And, since everyone serializes to JSON anyway, serializing to JSON is a decent enough choice for communicating across Lisps as well.

Scarblac 3y ago

IMO a big reason Python is popular is because so many big seemingly unrelated librarues (AI things, scipy, Pillow, pandas, GDAL rasters, matplotlib, etc) all work with the "Numpy array" datatype and can therefore do things with each other's data.

vineyardmike 3y ago
Relatedly, this is why big companies (eg Google) all have internal standard interchange formats (eg Protos). It’s so every programmer and every service and every file stored can always be consumed or produced as needed. Even smaller companies I’ve seen end up with an “interfaces” or “types” repo that contains definitions of every external object.
- deleted 3y ago
  [deleted]
chubot 3y ago
Yup, I agree that Python composes like Unix does, which is a reason for its phenomenal growth: Narrow Waists Can Be Interior or Exterior: PyObject vs. Unix Files - https://www.oilshell.org/blog/2023/06/narrow-waist.html https://www.oilshell.org/blog/2023/06/narrow-waist.html The appendix links to: Why is Python the language of machine learning? https://news.ycombinator.com/item?id=29171519 https://news.ycombinator.com/item?id=29171519 In particular, I highly recommend this bit of history from NumPy creator Travis Oliphant: https://www.youtube.com/watch?v=gFEE3w7F0ww https://www.youtube.com/watch?v=gFEE3w7F0ww

sevensor 3y ago

What's worse, with autotools, is that while maybe only one programmer in 100 knows how to do ./configure; make; make install, only one in 10,000 knows m4. It's one of those odd corners of the ecosystem like troff. It would be very easy to slip by me an exploit written in m4, or troff for that matter, because I've never paid any attention to either.