4 ms·

It's not for everyone, but the Unix/BSD/Linux shell is still my favorite command invocation environment. Despite their simplicity, pipelines are a really handy

by rbc 8y ago

It's not for everyone, but the Unix/BSD/Linux shell is still my favorite command invocation environment. Despite their simplicity, pipelines are a really handy way to pass data between programs when solving certain kinds problems. It also provides a cheap way of providing some parallel processing.

zrobotics 8y ago
The other thing I absolutely love about pipes is the ease of writing custom utilities. I'd refactored a filetype for use with a game last week, and needed to insert several lines into ~1.5k ASCII files. 1/2 hour to make a hacked c++ program to make the changes, and pass the outputs of grep to that. Although I'm unfamiliar with using pipes to provide parallel processing, could you elaborate? If you can pass to another thread I will be kicking myself for missing that.
- flukus 8y ago
  > Although I'm unfamiliar with using pipes to provide parallel processing, could you elaborate? If you can pass to another thread I will be kicking myself for missing that. You didn't miss it, each stage in the pipeline runs in it's own process so you're already using it. With say "grep -r foo ./ | do-something | do-omething-else" the second and third stages will be operating on the data from grep while grep is still running. In addition to that you can get more parallelism pretty simply with xargs (http://offbytwo.com/2011/06/26/things-you-didnt-know-about-xargs.html http://offbytwo.com/2011/06/26/things-you-didnt-know-about-x...). If that's not enough then check out gnu parallel (https://www.gnu.org/software/parallel/ https://www.gnu.org/software/parallel/). While we're talking about unix as an IDE I want to put in a plug for inotifywait (https://linux.die.net/man/1/inotifywait https://linux.die.net/man/1/inotifywait). Combining this with a makefile you can get some very fast builds with near instant feedback everytime you save a file. I have one that executes units tests and valgrind every time for instant memory leak detection. Another great use case is getting WYSIWYG like features with latex files.
  - harry8 8y ago
    using make -j $num is another way to go to get parallelism that is sometimes convenient.
    - flukus 8y ago
      Just make sure $num is set, otherwise it tries to do everything it can at once. I found out the hard way compiling gcc last week, system was unresponsive so I reset, but I'm curious if it would have worked eventually.
      - btschaegg 8y ago
        This is a pet peeve I have had for a while now. I still have a definition for MAKEFLAGS (with -j [# of cpus-1]) in my profile and rc files and had to learn that if you allow make to start too many things at once, the whole machine would hang. Even switching to a different TTY took ages. To me, it sounds like a tremendous scheduling issue if a build job can drown a whole system like that. Shouldn't there be some emphasis on non-waiting threads/processes that haven't done much over those who have hogged the CPU in the past without me having to manually adjust the niceness of every build task?
        __blockcipher__ 8y ago
        I can’t offer specifics, but it’s probably at the point that you’d compile a different scheduler?
      - jcelerier 8y ago
        IIRC make -j is equivalent to make -j256...
        benchaney 8y ago
        From the man page: > If the -j option is given without an argument, make will not limit the number of jobs that can run simultaneously. Although other implementations might be different of course.
pjmlp 8y ago
You mean function composition on Lisp Machines REPL and Smalltalk Transcript, with the benefit of structured data.
- rbc 8y ago
  I've used guile to do pipeline like transformations of data by using multiple function calls. It has the advantage of a more consistent api to do the transformations on the stream of data and surely better type checking. The syntax is different but I think about it as being similar. Despite its messiness, doing ad-hoc integration of command line tools to transform text is really handy. It also spreads the cpu utilization and memory usage across the commands that you use in the pipeline.
  - rbc 8y ago
    I'll add that I enjoy using Common Lisp and Scheme REPL environments. I also really like Squeak as an interactive programming environment. Neither has replaced the old school Unix shell as my preferred command invocation environment, at least in my professional life.
    - madhadron 8y ago
      I have the same experience, and I suspect it's because we haven't replicated the utilities in these environments. No one uses them as a replacement for shell because no one has done the work to use them as a replacement for shell. Personally, I think Prolog is the right way to go, but I have this idea that I don't want to spend the rest of my life typing into a teletype emulator.
      - pjmlp 8y ago
        We kind of do on Windows, thanks PowerShell and PowerShell IDE.
  - pjmlp 8y ago
    > It also spreads the cpu utilization and memory usage across the commands that you use in the pipeline. For example with Clojure you can use pmap and pcall to achieve that.
coatmatter 8y ago
Which is your favourite shell? bash, oh-my-zsh, fish?

by rbc 8y ago

zrobotics 8y ago

The other thing I absolutely love about pipes is the ease of writing custom utilities. I'd refactored a filetype for use with a game last week, and needed to insert several lines into ~1.5k ASCII files. 1/2 hour to make a hacked c++ program to make the changes, and pass the outputs of grep to that. Although I'm unfamiliar with using pipes to provide parallel processing, could you elaborate? If you can pass to another thread I will be kicking myself for missing that.

flukus 8y ago
> Although I'm unfamiliar with using pipes to provide parallel processing, could you elaborate? If you can pass to another thread I will be kicking myself for missing that. You didn't miss it, each stage in the pipeline runs in it's own process so you're already using it. With say "grep -r foo ./ | do-something | do-omething-else" the second and third stages will be operating on the data from grep while grep is still running. In addition to that you can get more parallelism pretty simply with xargs (http://offbytwo.com/2011/06/26/things-you-didnt-know-about-xargs.html http://offbytwo.com/2011/06/26/things-you-didnt-know-about-x...). If that's not enough then check out gnu parallel (https://www.gnu.org/software/parallel/ https://www.gnu.org/software/parallel/). While we're talking about unix as an IDE I want to put in a plug for inotifywait (https://linux.die.net/man/1/inotifywait https://linux.die.net/man/1/inotifywait). Combining this with a makefile you can get some very fast builds with near instant feedback everytime you save a file. I have one that executes units tests and valgrind every time for instant memory leak detection. Another great use case is getting WYSIWYG like features with latex files.
- harry8 8y ago
  using make -j $num is another way to go to get parallelism that is sometimes convenient.
  - flukus 8y ago
    Just make sure $num is set, otherwise it tries to do everything it can at once. I found out the hard way compiling gcc last week, system was unresponsive so I reset, but I'm curious if it would have worked eventually.
    - btschaegg 8y ago
      This is a pet peeve I have had for a while now. I still have a definition for MAKEFLAGS (with -j [# of cpus-1]) in my profile and rc files and had to learn that if you allow make to start too many things at once, the whole machine would hang. Even switching to a different TTY took ages. To me, it sounds like a tremendous scheduling issue if a build job can drown a whole system like that. Shouldn't there be some emphasis on non-waiting threads/processes that haven't done much over those who have hogged the CPU in the past without me having to manually adjust the niceness of every build task?
      - __blockcipher__ 8y ago
        I can’t offer specifics, but it’s probably at the point that you’d compile a different scheduler?
    - jcelerier 8y ago
      IIRC make -j is equivalent to make -j256...
      - benchaney 8y ago
        From the man page: > If the -j option is given without an argument, make will not limit the number of jobs that can run simultaneously. Although other implementations might be different of course.

pjmlp 8y ago

You mean function composition on Lisp Machines REPL and Smalltalk Transcript, with the benefit of structured data.

rbc 8y ago
I've used guile to do pipeline like transformations of data by using multiple function calls. It has the advantage of a more consistent api to do the transformations on the stream of data and surely better type checking. The syntax is different but I think about it as being similar. Despite its messiness, doing ad-hoc integration of command line tools to transform text is really handy. It also spreads the cpu utilization and memory usage across the commands that you use in the pipeline.
- rbc 8y ago
  I'll add that I enjoy using Common Lisp and Scheme REPL environments. I also really like Squeak as an interactive programming environment. Neither has replaced the old school Unix shell as my preferred command invocation environment, at least in my professional life.
  - madhadron 8y ago
    I have the same experience, and I suspect it's because we haven't replicated the utilities in these environments. No one uses them as a replacement for shell because no one has done the work to use them as a replacement for shell. Personally, I think Prolog is the right way to go, but I have this idea that I don't want to spend the rest of my life typing into a teletype emulator.
    - pjmlp 8y ago
      We kind of do on Windows, thanks PowerShell and PowerShell IDE.
- pjmlp 8y ago
  > It also spreads the cpu utilization and memory usage across the commands that you use in the pipeline. For example with Clojure you can use pmap and pcall to achieve that.

coatmatter 8y ago

Which is your favourite shell? bash, oh-my-zsh, fish?