3 ms·
> Are there (m)any advantages to doing this? In the general case, as always, "it depends". I'm of the opinion that shell scripts are very powerful for quickly
by chriswarbo 10y ago
> Are there (m)any advantages to doing this?
In the general case, as always, "it depends". I'm of the opinion that shell scripts are very powerful for quickly prototyping something, but as soon as you need to do real software engineering (e.g. test suites, etc.) then it's probably worth porting over to a saner language (yes, there are test frameworks for bash; no, that doesn't make bash's semantics any more sane ;) )
Piping data between subprocesses is almost universally painful in anything other than a shell, so these sorts of libraries are great for that sort of glue; I wouldn't use them for anything that's not a bog-standard "foo | bar | baz | ..." though, since that's what the rest of the language is for!
In this case it was a no-brainer: the project requires a lot of s-expression manipulation, which is implemented as small racket scripts; these are called from bash scripts which contain the main logic, data flow, etc. I wrote it this way since I'd never used racket before, but figured it would be better suited to this s-expr manipulation than bash, python, haskell, etc. (which it is!); the remaining parts were simple enough to do with bash.
This worked fine for a while, but a recent change in requirements has invalidated a lot of the code's assumptions, so I need to ensure I'm making changes in the right place, that test cases are updated to reflect the changes, etc. which motivates porting to something more sophisticated, and racket is the clear choice here.
I wanted to use a shell-interop library after previously finding Scala's "process" package to be very pleasant to use. It's certainly made the porting job much more manageable, as it just becomes a case of recursively refactoring each part:
bash:
echo "foo" | ./bar.sh | ./baz.sh | grep "quux"
racket:
---- Convert shell scripts to racket functions
bash:
echo "foo" | ./bar.rkt | ./baz.rkt | grep "quux"
racket:
(define (bar) ...)
(define (baz) ...)
---- Move pipeline over to racket
bash:
racket:
(define (bar) ...)
(define (baz) ...)
(run-pipeline '(echo "foo") '(./bar.rkt) '(./baz.rkt) '(grep "quux"))
---- Call racket functions directly, rather than invoking scripts
bash:
racket:
(define (bar) ...)
(define (baz) ...)
(run-pipeline '(echo "foo") `(,bar) `(,baz) '(grep "quux"))
---- Encapsulate stdio
bash:
racket:
(define (bar) ...)
(define (baz) ...)
(with-input-from-string
(with-output-to-string (lambda ()
(run-pipeline '(,bar) '(,baz) '(grep "quux")))
"foo")
---- Replace stdio with arguments and return values
bash:
racket:
(define (bar x) ...)
(define (baz x) ...)
(string-join (filter (lambda (line)
(string-contains? line "quux"))
(string-split (baz (bar "foo"))
"\n")))
---- Replace strings with more useful datastructures
bash:
racket:
(define (bar x) ...)
(define (baz x) ...)
(filter (lambda (line)
(string-contains? line "quux"))
(baz (bar "foo")))
- jonathonf 10y ago> bog-standard "foo | bar | baz | ..." Surely if you're doing that then shell is the way to go? What cases are there for performing that sort of piping inside a "real software engineering" program? (And if you're doing it inside a single program, why aren't you making several independent programs that pipe to each other?)
- chriswarbo 10y agoWhen there are a few pipes, surrounded by hundreds of lines of logic, parsing, pretty-printing, error reporting, etc. then it makes sense to use a non-shell language. The problem is that invoking subprocesses can be quite verbose. If you do the naive thing and pass strings in/out of each step, it increases verbosity, gives you temporary variables to abstract away and can eat up a lot of memory with temporary data. If you do the clever thing and set up pipes between the subprocesses, it removes some of the variables, but takes more code and hits deadlocks when the buffers fill up. If you do the right thing and spawn separate threads to read/write data to the pipeline then it takes even more code, and now you have to worry about multithreading. Hiding that sort of boilerplate is why I like shells and shell-like libraries. Both are rather unsuited to anything else, but at least with a shell-like library you can fall back to a decent language.
- qwertyuiop924 10y ago>The problem is that invoking subprocesses can be quite verbose. Which is why SCSH is kickass.