3 ms·

xargs -P and other ad-hoc parallelism tools usually stop me from leaving the command line too quickly (e.g. <(), fork / join). The other thing that keeps me ba

• psadauskas 8y ago
  I agree. I love Ruby, but I'll pipe 'httpie', 'jq' and 'xargs' for quite awhile before deciding to write a Ruby script to do something similar.
• vinceguidry 8y ago
  Yeah I suppose my attitude will change should I ever start needing to do heavy lifting. But I'll definitely explore instrumenting bash through ruby first, just to see where that line lies. There's also the option of pushing the workload off onto AWS that I may look into.
• derefr 8y ago
  sort(1) is a merge sort (specifically, an "external R-way merge")—and, as a merge sort, it actually isn't constrained by memory. It spills its sorted intermediates to disk as temp files! (And, in fact, it even compresses each intermediate, for the same reason a column store like Cassandra does.) It's pretty optimal! If you're wondering how this is at-all fast (and it is), in Linux at least /tmp is a tmpfs, meaning that these tempfiles are actually ending up as memory after all. They get "spilled to disk" in the sense that the memory associated with the files spills to swap; but, importantly, tmpfs pages know that they're from files, so they have much better swapping semantics than regular memory for the case where you're writing the file as a stream, and then reading it as a stream. (Basically, juggling streams under a tmpfs is equivalent in overhead to managing memory when you've got it, and managing files when you don't, but without any of the dev-time overhead of having to write code for both cases.)