6 ms·

I've recently started porting some bash scripts over to Racket, using https://docs.racket-lang.org/shell-pipeline https://docs.racket-lang.org/shell-pipeline an

I've recently started porting some bash scripts over to Racket, using https://docs.racket-lang.org/shell-pipeline https://docs.racket-lang.org/shell-pipeline and it's been pretty straightforward and painless so far!

jonathonf 10y ago
Are there (m)any advantages to doing this? I've seen a number of shell-interop modules for various languages but they all appear to add an extra layer that reduces portability or ease of maintenance.
- jlg23 10y ago
  > Are there (m)any advantages to doing this? [trigger warning: cynicism distilled from 15+ years of bitter tears] More fun and better job security for the current maintainer. The former because one hacks away in the preferred language, the latter because one cannot be replaced easily by some unix geek unless s/he happens to speak the same language fluently. One can chose from various implementations in various languages (eg: psh and scsh) but hacking your own is easy: * Just re-implement the basic unix tools and shell semantics and invent some nifty convenience syntax and features along the way. * Either don't document at all or document everything with at least 2 pages aimed at people who have never used a computer before. Both ways will prevent the only ones who could replace you from even looking at your "shell in X". * Never ever do this in a typed language, write a ton of unit-, functional- and integration tests to make up for that - of course all of them completely undocumented. * On top of your creation, invent a fragile "shell syntax like" layer to make adoption easier. * Under no circumstances create some new abstraction layer that goes beyond shell semantics that brings something substantial to the table and is well documented. * Enjoy the time saved because you don't have to "man $tool" anymore, it's all your code anway. * Under no circumstances ever write in pure posix sh again - the realization that with all the stuff you've learned along the way posix sh covers 99% of the use-cases in a more expressive way will be crushing.
  - SwellJoe 10y ago
    Remind me never to hire you for anything ever. You've truly internalized the Tao of the BOFH.
    - jlg23 10y ago
      Thanks for the compliment. I'm usually hired for that attitude (and my ability to hide it except for the few meetings where it really counts) ;)
  - qwertyuiop924 10y ago
    That actually doesn't sound much like SCSH to me. SCSH was really just a library for posix integration, as well as some common shell idioms that came with it. And a damned good one, too. It was originally designed for situations in which you might write (notoriously unmaintainable) shell scripts, but it's so good that some of its APIs have become almost de-facto standard. In particular, SRE and the AWK macro really caught on.
- chriswarbo 10y ago
  > Are there (m)any advantages to doing this? In the general case, as always, "it depends". I'm of the opinion that shell scripts are very powerful for quickly prototyping something, but as soon as you need to do real software engineering (e.g. test suites, etc.) then it's probably worth porting over to a saner language (yes, there are test frameworks for bash; no, that doesn't make bash's semantics any more sane ;) ) Piping data between subprocesses is almost universally painful in anything other than a shell, so these sorts of libraries are great for that sort of glue; I wouldn't use them for anything that's not a bog-standard "foo | bar | baz | ..." though, since that's what the rest of the language is for! In this case it was a no-brainer: the project requires a lot of s-expression manipulation, which is implemented as small racket scripts; these are called from bash scripts which contain the main logic, data flow, etc. I wrote it this way since I'd never used racket before, but figured it would be better suited to this s-expr manipulation than bash, python, haskell, etc. (which it is!); the remaining parts were simple enough to do with bash. This worked fine for a while, but a recent change in requirements has invalidated a lot of the code's assumptions, so I need to ensure I'm making changes in the right place, that test cases are updated to reflect the changes, etc. which motivates porting to something more sophisticated, and racket is the clear choice here. I wanted to use a shell-interop library after previously finding Scala's "process" package to be very pleasant to use. It's certainly made the porting job much more manageable, as it just becomes a case of recursively refactoring each part: bash: echo "foo" | ./bar.sh | ./baz.sh | grep "quux" racket: ---- Convert shell scripts to racket functions bash: echo "foo" | ./bar.rkt | ./baz.rkt | grep "quux" racket: (define (bar) ...) (define (baz) ...) ---- Move pipeline over to racket bash: racket: (define (bar) ...) (define (baz) ...) (run-pipeline '(echo "foo") '(./bar.rkt) '(./baz.rkt) '(grep "quux")) ---- Call racket functions directly, rather than invoking scripts bash: racket: (define (bar) ...) (define (baz) ...) (run-pipeline '(echo "foo") `(,bar) `(,baz) '(grep "quux")) ---- Encapsulate stdio bash: racket: (define (bar) ...) (define (baz) ...) (with-input-from-string (with-output-to-string (lambda () (run-pipeline '(,bar) '(,baz) '(grep "quux"))) "foo") ---- Replace stdio with arguments and return values bash: racket: (define (bar x) ...) (define (baz x) ...) (string-join (filter (lambda (line) (string-contains? line "quux")) (string-split (baz (bar "foo")) "\n"))) ---- Replace strings with more useful datastructures bash: racket: (define (bar x) ...) (define (baz x) ...) (filter (lambda (line) (string-contains? line "quux")) (baz (bar "foo")))
616c 10y ago
Link to your code? I would be very interested to see this thus far. I have looked at avesh (CL) and xonsh (Python) because I think the future of fun *nix tooling is like libraries that implement a bash layer to like gradually move code away from shell languages. This is the sweet spot I would want in my life.
- chriswarbo 10y ago
  Not the cleanest, but it's at http://chriswarbo.net/projects/repos/theory-exploration-benchmarks.html http://chriswarbo.net/projects/repos/theory-exploration-benc... Note that the static HTML views are truncated to 10 commits, to save space (useful on larger repos), but the repo itself contains the full history.
  - 616c 10y ago
    Just got back around to looking at this. Thanks, I will check it out.

jonathonf 10y ago

Are there (m)any advantages to doing this? I've seen a number of shell-interop modules for various languages but they all appear to add an extra layer that reduces portability or ease of maintenance.

jlg23 10y ago
> Are there (m)any advantages to doing this? [trigger warning: cynicism distilled from 15+ years of bitter tears] More fun and better job security for the current maintainer. The former because one hacks away in the preferred language, the latter because one cannot be replaced easily by some unix geek unless s/he happens to speak the same language fluently. One can chose from various implementations in various languages (eg: psh and scsh) but hacking your own is easy: * Just re-implement the basic unix tools and shell semantics and invent some nifty convenience syntax and features along the way. * Either don't document at all or document everything with at least 2 pages aimed at people who have never used a computer before. Both ways will prevent the only ones who could replace you from even looking at your "shell in X". * Never ever do this in a typed language, write a ton of unit-, functional- and integration tests to make up for that - of course all of them completely undocumented. * On top of your creation, invent a fragile "shell syntax like" layer to make adoption easier. * Under no circumstances create some new abstraction layer that goes beyond shell semantics that brings something substantial to the table and is well documented. * Enjoy the time saved because you don't have to "man $tool" anymore, it's all your code anway. * Under no circumstances ever write in pure posix sh again - the realization that with all the stuff you've learned along the way posix sh covers 99% of the use-cases in a more expressive way will be crushing.
- SwellJoe 10y ago
  Remind me never to hire you for anything ever. You've truly internalized the Tao of the BOFH.
  - jlg23 10y ago
    Thanks for the compliment. I'm usually hired for that attitude (and my ability to hide it except for the few meetings where it really counts) ;)
- qwertyuiop924 10y ago
  That actually doesn't sound much like SCSH to me. SCSH was really just a library for posix integration, as well as some common shell idioms that came with it. And a damned good one, too. It was originally designed for situations in which you might write (notoriously unmaintainable) shell scripts, but it's so good that some of its APIs have become almost de-facto standard. In particular, SRE and the AWK macro really caught on.
chriswarbo 10y ago
> Are there (m)any advantages to doing this? In the general case, as always, "it depends". I'm of the opinion that shell scripts are very powerful for quickly prototyping something, but as soon as you need to do real software engineering (e.g. test suites, etc.) then it's probably worth porting over to a saner language (yes, there are test frameworks for bash; no, that doesn't make bash's semantics any more sane ;) ) Piping data between subprocesses is almost universally painful in anything other than a shell, so these sorts of libraries are great for that sort of glue; I wouldn't use them for anything that's not a bog-standard "foo | bar | baz | ..." though, since that's what the rest of the language is for! In this case it was a no-brainer: the project requires a lot of s-expression manipulation, which is implemented as small racket scripts; these are called from bash scripts which contain the main logic, data flow, etc. I wrote it this way since I'd never used racket before, but figured it would be better suited to this s-expr manipulation than bash, python, haskell, etc. (which it is!); the remaining parts were simple enough to do with bash. This worked fine for a while, but a recent change in requirements has invalidated a lot of the code's assumptions, so I need to ensure I'm making changes in the right place, that test cases are updated to reflect the changes, etc. which motivates porting to something more sophisticated, and racket is the clear choice here. I wanted to use a shell-interop library after previously finding Scala's "process" package to be very pleasant to use. It's certainly made the porting job much more manageable, as it just becomes a case of recursively refactoring each part: bash: echo "foo" | ./bar.sh | ./baz.sh | grep "quux" racket: ---- Convert shell scripts to racket functions bash: echo "foo" | ./bar.rkt | ./baz.rkt | grep "quux" racket: (define (bar) ...) (define (baz) ...) ---- Move pipeline over to racket bash: racket: (define (bar) ...) (define (baz) ...) (run-pipeline '(echo "foo") '(./bar.rkt) '(./baz.rkt) '(grep "quux")) ---- Call racket functions directly, rather than invoking scripts bash: racket: (define (bar) ...) (define (baz) ...) (run-pipeline '(echo "foo") `(,bar) `(,baz) '(grep "quux")) ---- Encapsulate stdio bash: racket: (define (bar) ...) (define (baz) ...) (with-input-from-string (with-output-to-string (lambda () (run-pipeline '(,bar) '(,baz) '(grep "quux"))) "foo") ---- Replace stdio with arguments and return values bash: racket: (define (bar x) ...) (define (baz x) ...) (string-join (filter (lambda (line) (string-contains? line "quux")) (string-split (baz (bar "foo")) "\n"))) ---- Replace strings with more useful datastructures bash: racket: (define (bar x) ...) (define (baz x) ...) (filter (lambda (line) (string-contains? line "quux")) (baz (bar "foo")))

616c 10y ago

Link to your code? I would be very interested to see this thus far. I have looked at avesh (CL) and xonsh (Python) because I think the future of fun *nix tooling is like libraries that implement a bash layer to like gradually move code away from shell languages. This is the sweet spot I would want in my life.

chriswarbo 10y ago
Not the cleanest, but it's at http://chriswarbo.net/projects/repos/theory-exploration-benchmarks.html http://chriswarbo.net/projects/repos/theory-exploration-benc... Note that the static HTML views are truncated to 10 commits, to save space (useful on larger repos), but the repo itself contains the full history.
- 616c 10y ago
  Just got back around to looking at this. Thanks, I will check it out.