5 ms·

> Although I'm unfamiliar with using pipes to provide parallel processing, could you elaborate? If you can pass to another thread I will be kicking myself for m

by flukus 8y ago

> Although I'm unfamiliar with using pipes to provide parallel processing, could you elaborate? If you can pass to another thread I will be kicking myself for missing that. You didn't miss it, each stage in the pipeline runs in it's own process so you're already using it. With say "grep -r foo ./ | do-something | do-omething-else" the second and third stages will be operating on the data from grep while grep is still running. In addition to that you can get more parallelism pretty simply with xargs (http://offbytwo.com/2011/06/26/things-you-didnt-know-about-xargs.html http://offbytwo.com/2011/06/26/things-you-didnt-know-about-x...). If that's not enough then check out gnu parallel (https://www.gnu.org/software/parallel/ https://www.gnu.org/software/parallel/). While we're talking about unix as an IDE I want to put in a plug for inotifywait (https://linux.die.net/man/1/inotifywait https://linux.die.net/man/1/inotifywait). Combining this with a makefile you can get some very fast builds with near instant feedback everytime you save a file. I have one that executes units tests and valgrind every time for instant memory leak detection. Another great use case is getting WYSIWYG like features with latex files.

harry8 8y ago
using make -j $num is another way to go to get parallelism that is sometimes convenient.
- flukus 8y ago
  Just make sure $num is set, otherwise it tries to do everything it can at once. I found out the hard way compiling gcc last week, system was unresponsive so I reset, but I'm curious if it would have worked eventually.
  - btschaegg 8y ago
    This is a pet peeve I have had for a while now. I still have a definition for MAKEFLAGS (with -j [# of cpus-1]) in my profile and rc files and had to learn that if you allow make to start too many things at once, the whole machine would hang. Even switching to a different TTY took ages. To me, it sounds like a tremendous scheduling issue if a build job can drown a whole system like that. Shouldn't there be some emphasis on non-waiting threads/processes that haven't done much over those who have hogged the CPU in the past without me having to manually adjust the niceness of every build task?
    - __blockcipher__ 8y ago
      I can’t offer specifics, but it’s probably at the point that you’d compile a different scheduler?
  - jcelerier 8y ago
    IIRC make -j is equivalent to make -j256...
    - benchaney 8y ago
      From the man page: > If the -j option is given without an argument, make will not limit the number of jobs that can run simultaneously. Although other implementations might be different of course.
      - JdeBP 8y ago
        It's not a notion that I carried into my implementation of redo, for example. % redo --jobs redo: ERROR: jobs: missing option argument %
        flukus 8y ago
        What made you go with that rather than trying to pick a sensible default like the number of cores available? Or was it just a general goal to require explicitness?
        JdeBP 8y ago
        It was simplicity, of design, implementation, use, and indeed explanation. There's only one default, the default if the --jobs option is not used; rather than there being two defaults, one for not having the option and another for having the option but not the option argument. More generally, I try to avoid optional arguments to command options, in line with the guideline in the Single Unix Specification. (I actually picked such ideas up many years ago, before the first POSIX specification, from a book by Eric Foxley titled Unix for Super Users, where there was an appendix on command line option syntax.) * http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap12.html#tag_12_02 http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_... Of course, if the user wants to use the number of cores available, then xe can work that number out and pass it as an argument; and indeed I have done that very thing in some of the package/make scripts that I have published.

by flukus 8y ago

harry8 8y ago

using make -j $num is another way to go to get parallelism that is sometimes convenient.

flukus 8y ago
Just make sure $num is set, otherwise it tries to do everything it can at once. I found out the hard way compiling gcc last week, system was unresponsive so I reset, but I'm curious if it would have worked eventually.
- btschaegg 8y ago
  This is a pet peeve I have had for a while now. I still have a definition for MAKEFLAGS (with -j [# of cpus-1]) in my profile and rc files and had to learn that if you allow make to start too many things at once, the whole machine would hang. Even switching to a different TTY took ages. To me, it sounds like a tremendous scheduling issue if a build job can drown a whole system like that. Shouldn't there be some emphasis on non-waiting threads/processes that haven't done much over those who have hogged the CPU in the past without me having to manually adjust the niceness of every build task?
  - __blockcipher__ 8y ago
    I can’t offer specifics, but it’s probably at the point that you’d compile a different scheduler?
- jcelerier 8y ago
  IIRC make -j is equivalent to make -j256...
  - benchaney 8y ago
    From the man page: > If the -j option is given without an argument, make will not limit the number of jobs that can run simultaneously. Although other implementations might be different of course.
    - JdeBP 8y ago
      It's not a notion that I carried into my implementation of redo, for example. % redo --jobs redo: ERROR: jobs: missing option argument %
      - flukus 8y ago
        What made you go with that rather than trying to pick a sensible default like the number of cores available? Or was it just a general goal to require explicitness?
        JdeBP 8y ago
        It was simplicity, of design, implementation, use, and indeed explanation. There's only one default, the default if the --jobs option is not used; rather than there being two defaults, one for not having the option and another for having the option but not the option argument. More generally, I try to avoid optional arguments to command options, in line with the guideline in the Single Unix Specification. (I actually picked such ideas up many years ago, before the first POSIX specification, from a book by Eric Foxley titled Unix for Super Users, where there was an appendix on command line option syntax.) * http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap12.html#tag_12_02 http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_... Of course, if the user wants to use the number of cores available, then xe can work that number out and pass it as an argument; and indeed I have done that very thing in some of the package/make scripts that I have published.