4 ms·

The other thing I absolutely love about pipes is the ease of writing custom utilities. I'd refactored a filetype for use with a game last week, and needed to in

The other thing I absolutely love about pipes is the ease of writing custom utilities. I'd refactored a filetype for use with a game last week, and needed to insert several lines into ~1.5k ASCII files. 1/2 hour to make a hacked c++ program to make the changes, and pass the outputs of grep to that. Although I'm unfamiliar with using pipes to provide parallel processing, could you elaborate? If you can pass to another thread I will be kicking myself for missing that.

flukus 8y ago
> Although I'm unfamiliar with using pipes to provide parallel processing, could you elaborate? If you can pass to another thread I will be kicking myself for missing that. You didn't miss it, each stage in the pipeline runs in it's own process so you're already using it. With say "grep -r foo ./ | do-something | do-omething-else" the second and third stages will be operating on the data from grep while grep is still running. In addition to that you can get more parallelism pretty simply with xargs (http://offbytwo.com/2011/06/26/things-you-didnt-know-about-xargs.html http://offbytwo.com/2011/06/26/things-you-didnt-know-about-x...). If that's not enough then check out gnu parallel (https://www.gnu.org/software/parallel/ https://www.gnu.org/software/parallel/). While we're talking about unix as an IDE I want to put in a plug for inotifywait (https://linux.die.net/man/1/inotifywait https://linux.die.net/man/1/inotifywait). Combining this with a makefile you can get some very fast builds with near instant feedback everytime you save a file. I have one that executes units tests and valgrind every time for instant memory leak detection. Another great use case is getting WYSIWYG like features with latex files.
- harry8 8y ago
  using make -j $num is another way to go to get parallelism that is sometimes convenient.
  - flukus 8y ago
    Just make sure $num is set, otherwise it tries to do everything it can at once. I found out the hard way compiling gcc last week, system was unresponsive so I reset, but I'm curious if it would have worked eventually.
    - btschaegg 8y ago
      This is a pet peeve I have had for a while now. I still have a definition for MAKEFLAGS (with -j [# of cpus-1]) in my profile and rc files and had to learn that if you allow make to start too many things at once, the whole machine would hang. Even switching to a different TTY took ages. To me, it sounds like a tremendous scheduling issue if a build job can drown a whole system like that. Shouldn't there be some emphasis on non-waiting threads/processes that haven't done much over those who have hogged the CPU in the past without me having to manually adjust the niceness of every build task?
      - __blockcipher__ 8y ago
        I can’t offer specifics, but it’s probably at the point that you’d compile a different scheduler?
    - jcelerier 8y ago
      IIRC make -j is equivalent to make -j256...
      - benchaney 8y ago
        From the man page: > If the -j option is given without an argument, make will not limit the number of jobs that can run simultaneously. Although other implementations might be different of course.
        JdeBP 8y ago
        It's not a notion that I carried into my implementation of redo, for example. % redo --jobs redo: ERROR: jobs: missing option argument %
        flukus 8y ago
        What made you go with that rather than trying to pick a sensible default like the number of cores available? Or was it just a general goal to require explicitness?

flukus 8y ago

> Although I'm unfamiliar with using pipes to provide parallel processing, could you elaborate? If you can pass to another thread I will be kicking myself for missing that. You didn't miss it, each stage in the pipeline runs in it's own process so you're already using it. With say "grep -r foo ./ | do-something | do-omething-else" the second and third stages will be operating on the data from grep while grep is still running. In addition to that you can get more parallelism pretty simply with xargs (http://offbytwo.com/2011/06/26/things-you-didnt-know-about-xargs.html http://offbytwo.com/2011/06/26/things-you-didnt-know-about-x...). If that's not enough then check out gnu parallel (https://www.gnu.org/software/parallel/ https://www.gnu.org/software/parallel/). While we're talking about unix as an IDE I want to put in a plug for inotifywait (https://linux.die.net/man/1/inotifywait https://linux.die.net/man/1/inotifywait). Combining this with a makefile you can get some very fast builds with near instant feedback everytime you save a file. I have one that executes units tests and valgrind every time for instant memory leak detection. Another great use case is getting WYSIWYG like features with latex files.

harry8 8y ago
using make -j $num is another way to go to get parallelism that is sometimes convenient.
- flukus 8y ago
  Just make sure $num is set, otherwise it tries to do everything it can at once. I found out the hard way compiling gcc last week, system was unresponsive so I reset, but I'm curious if it would have worked eventually.
  - btschaegg 8y ago
    This is a pet peeve I have had for a while now. I still have a definition for MAKEFLAGS (with -j [# of cpus-1]) in my profile and rc files and had to learn that if you allow make to start too many things at once, the whole machine would hang. Even switching to a different TTY took ages. To me, it sounds like a tremendous scheduling issue if a build job can drown a whole system like that. Shouldn't there be some emphasis on non-waiting threads/processes that haven't done much over those who have hogged the CPU in the past without me having to manually adjust the niceness of every build task?
    - __blockcipher__ 8y ago
      I can’t offer specifics, but it’s probably at the point that you’d compile a different scheduler?
  - jcelerier 8y ago
    IIRC make -j is equivalent to make -j256...
    - benchaney 8y ago
      From the man page: > If the -j option is given without an argument, make will not limit the number of jobs that can run simultaneously. Although other implementations might be different of course.
      - JdeBP 8y ago
        It's not a notion that I carried into my implementation of redo, for example. % redo --jobs redo: ERROR: jobs: missing option argument %
        flukus 8y ago
        What made you go with that rather than trying to pick a sensible default like the number of cores available? Or was it just a general goal to require explicitness?