3 ms·

The second I start wanting a bash pipeline, probably around the third pipe, I scrap it immediately and move to using a text editor to write a script. Because if

The second I start wanting a bash pipeline, probably around the third pipe, I scrap it immediately and move to using a text editor to write a script. Because if I'm wanting a pipeline, I'm also going to want to store it, and transmit it over the network, and manage it, and put it down and come back to it later. All things perfectly manageable inside a PORO. Bundler even has an inline mode, so you can put everything in one file, close to your data, Bundler just handles dep management under the hood like a boss. Check out bundler-inline. Sure, you can do all that with bash. But you can also make system() calls with Ruby, with full string interpolation power tools. If you're a Rubyist, and you want to do data analysis, this is the workflow you want.

bluehatbrit 8y ago
I hadn't seen bundler-inline before, thanks for posting this. That'll be nice for small scripts that don't warrant a whole directory themselves, or useful for a sort of "tools" / "utils" repo with small utility scripts.
barrkel 8y ago
xargs -P and other ad-hoc parallelism tools usually stop me from leaving the command line too quickly (e.g. <(), fork / join). The other thing that keeps me back is scale. If everything fits in memory, cool, but sometimes I need to sort or uniq something that won't.
- psadauskas 8y ago
  I agree. I love Ruby, but I'll pipe 'httpie', 'jq' and 'xargs' for quite awhile before deciding to write a Ruby script to do something similar.
- vinceguidry 8y ago
  Yeah I suppose my attitude will change should I ever start needing to do heavy lifting. But I'll definitely explore instrumenting bash through ruby first, just to see where that line lies. There's also the option of pushing the workload off onto AWS that I may look into.
- derefr 8y ago
  sort(1) is a merge sort (specifically, an "external R-way merge")—and, as a merge sort, it actually isn't constrained by memory. It spills its sorted intermediates to disk as temp files! (And, in fact, it even compresses each intermediate, for the same reason a column store like Cassandra does.) It's pretty optimal! If you're wondering how this is at-all fast (and it is), in Linux at least /tmp is a tmpfs, meaning that these tempfiles are actually ending up as memory after all. They get "spilled to disk" in the sense that the memory associated with the files spills to swap; but, importantly, tmpfs pages know that they're from files, so they have much better swapping semantics than regular memory for the case where you're writing the file as a stream, and then reading it as a stream. (Basically, juggling streams under a tmpfs is equivalent in overhead to managing memory when you've got it, and managing files when you don't, but without any of the dev-time overhead of having to write code for both cases.)
Lio 8y ago
Don't you loose the easy threading that's inherent in a pipeline of processes though? I love ruby but for throw away jobs I'd be more likely to do it on the command line than a ruby script simply because it would be quicker for me to write and run in a shell. Unless I'm wrong at the very least you wouldn't have to deal with the GIL.
- somebehemoth 8y ago
  This is a good point. Using Rb with subshell and pipeline would be the best of both worlds?
derefr 8y ago
> Because if I'm wanting a pipeline, I'm also going to want to store it, and transmit it over the network, and manage it, and put it down and come back to it later. Not my experience. More often than not, if I'm writing a pipeline (in a shell or in a REPL), it's because someone handed me a file they made in a one-off effort that has nonsensical nonstandard formatting, and I'm trying to normalize it. If they made another file, it'd have different nonsensical nonstandard formatting, so there's no reuse here. Meanwhile, writing a pipeline at a REPL (or shell) enables me to just work on it expression by expression, keeping a variable-assignment at the beginning of the expression so that I can then work on the intermediate created data to figure out how to munge it just a little bit more, and then add that back to the pipeline once it's right. If there was a way to edit a Ruby script such that, each time I save it, my existing REPL session re-runs the script and loads the bindings from it into my existing session—without destroying any other bindings I've made—then I'd be happy to use that. But that's less like "using files", and much closer to using something like a Jupyter notebook.

bluehatbrit 8y ago

I hadn't seen bundler-inline before, thanks for posting this. That'll be nice for small scripts that don't warrant a whole directory themselves, or useful for a sort of "tools" / "utils" repo with small utility scripts.

barrkel 8y ago

xargs -P and other ad-hoc parallelism tools usually stop me from leaving the command line too quickly (e.g. <(), fork / join). The other thing that keeps me back is scale. If everything fits in memory, cool, but sometimes I need to sort or uniq something that won't.

psadauskas 8y ago
I agree. I love Ruby, but I'll pipe 'httpie', 'jq' and 'xargs' for quite awhile before deciding to write a Ruby script to do something similar.
vinceguidry 8y ago
Yeah I suppose my attitude will change should I ever start needing to do heavy lifting. But I'll definitely explore instrumenting bash through ruby first, just to see where that line lies. There's also the option of pushing the workload off onto AWS that I may look into.
derefr 8y ago
sort(1) is a merge sort (specifically, an "external R-way merge")—and, as a merge sort, it actually isn't constrained by memory. It spills its sorted intermediates to disk as temp files! (And, in fact, it even compresses each intermediate, for the same reason a column store like Cassandra does.) It's pretty optimal! If you're wondering how this is at-all fast (and it is), in Linux at least /tmp is a tmpfs, meaning that these tempfiles are actually ending up as memory after all. They get "spilled to disk" in the sense that the memory associated with the files spills to swap; but, importantly, tmpfs pages know that they're from files, so they have much better swapping semantics than regular memory for the case where you're writing the file as a stream, and then reading it as a stream. (Basically, juggling streams under a tmpfs is equivalent in overhead to managing memory when you've got it, and managing files when you don't, but without any of the dev-time overhead of having to write code for both cases.)

Lio 8y ago

Don't you loose the easy threading that's inherent in a pipeline of processes though? I love ruby but for throw away jobs I'd be more likely to do it on the command line than a ruby script simply because it would be quicker for me to write and run in a shell. Unless I'm wrong at the very least you wouldn't have to deal with the GIL.

somebehemoth 8y ago
This is a good point. Using Rb with subshell and pipeline would be the best of both worlds?

derefr 8y ago

> Because if I'm wanting a pipeline, I'm also going to want to store it, and transmit it over the network, and manage it, and put it down and come back to it later. Not my experience. More often than not, if I'm writing a pipeline (in a shell or in a REPL), it's because someone handed me a file they made in a one-off effort that has nonsensical nonstandard formatting, and I'm trying to normalize it. If they made another file, it'd have different nonsensical nonstandard formatting, so there's no reuse here. Meanwhile, writing a pipeline at a REPL (or shell) enables me to just work on it expression by expression, keeping a variable-assignment at the beginning of the expression so that I can then work on the intermediate created data to figure out how to munge it just a little bit more, and then add that back to the pipeline once it's right. If there was a way to edit a Ruby script such that, each time I save it, my existing REPL session re-runs the script and loads the bindings from it into my existing session—without destroying any other bindings I've made—then I'd be happy to use that. But that's less like "using files", and much closer to using something like a Jupyter notebook.