8 ms·

Show HN: Cr – Runs your tasks at maximum concurrency

83 points by cirowrc 9y ago

deleted 9y ago
[deleted]
znpy 9y ago
That's what make has been doing for the last like 40 years.
- blablabla123 9y ago
  I wish the make documentation was better though. I've also read oftentimes that people say that turning on make concurrency with `-j` is on your own risk.
  - Coding_Cat 9y ago
    Make's concurrency is pretty good, as long as your dependency graph is correct. Problems arise when one of your build steps creates temporary files for each invocation and they end up overwritting each other (race condition). This would have the same issue
    - jschwartzi 9y ago
      Or when people create phony targets instead of actually declaring an intermediate file and its prerequisites.
  - xyzzy_plugh 9y ago
    That's because people write shitty makefiles. Most makefiles I see are full of improperly used double colon rules, phony targets not marked .PHONY and implicit ordering assumptions. If `-j' was enabled by default, then it would be much harder for writers to develop these issues. But I suspect most of these authors, who are already unwilling to learn make properly would simply give up and use a different tool.
    - willejs 9y ago
      Here here.
      - IgorPartola 9y ago
        It’s hear hear.
    - vvanders 9y ago
      The problem is that most people don't set out to be masters of make. They just want to get their shit built so they can make forward progress on the thing they want to create. Having to learn all of these subtleties is what gets you into these problems in the first place. I'll happily take a more opinionated build tool with proper escape hatches over something that expects you to know it in detail for even the most basic tasks.
      - Sir_Cmpwn 9y ago
        I really hate this attitude. The same people who don't grok Make also don't grok git or bourne shell or who knows what else. Don't take shortcuts in understanding the fundamental tools of your stack. Your build system is just as important to your project as the code it's building!
        vvanders 9y ago
        Make is only a fundamental tool if that's what your library/binary uses. You may have a mix of CMake, MSVC and language specific build tools like Cargo and others. By your logic I should be an expert in all of them. Make's support outside of the posix world isn't that great(as opposed to CMake) so calling it fundamental seems that you are making assumptions about the environments that development happens in.
        Sir_Cmpwn 9y ago
        >You may have a mix of CMake, MSVC and language specific build tools like Cargo and others. By your logic I should be an expert in all of them. Correct. >Make's support outside of the posix world The world outside of posix doesn't really matter.
  - AFNobody 9y ago
    Documentation of open source projects is a great example of the tragedy of the commons.
- peterwwillis 9y ago
  BUT THIS IS WRITTEN IN GO!
  - sctb 9y ago
    Please don't. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
    - peterwwillis 9y ago
      But this is written in Go!
- pweissbrod 9y ago
  Make is possibly the most popular thing people re-invent for specific platforms. Touch of irony in this project, with that Makefile sitting in the root directory. EDIT: My apologies if the above comment was cynical. There's a salient need for workflow systems which provide features above/beyond the basics of make. Built-in things like a central broker, adjustable worker processes, scheduling and history. Python/Luigi is an example of such a workflow system, yet it keeps the workflow definitions very similar to Make for conciseness
- haldean 9y ago
  And tup for the last 10. Lots of people in this thread complaining about Make's syntax might want to check it out; it has a nice, simple "inputs |> command |> outputs" syntax, a similar "output the DAG" mode to Cr, and defaults to parallelism. http://gittup.org/tup/ http://gittup.org/tup/
  - aidenn0 9y ago
    That's funny because I would list tup's syntax as a negative for an otherwise very useful tool. Also, Tup is much more single-purpose than Make or (from a quick readthrough) Cr. It is very strongly focused on the "N files in, 1 file out" for compilation, and specializes at that to the point of not doing other things. As far as a minimal "better make" I find redo to be the best. Make is a weird combination of the shell and makefile languages, but redo boils it down to just a single language and manages to handle dependency of build rules in an elegant manner.
    - dilap 9y ago
      curious, which variant of redo do you actually use?
  - CJefferson 9y ago
    I love tup, but I find too many places where it doesn't work to use it for things I distribute (sometimes fuse fails, doesn't work on Linux subsystem for Windows, flakey on full windows).
stephenr 9y ago
So, `make -j`?
- deleted 9y ago
  [deleted]
jchw 9y ago
Everyone is going to compare it to Make, but it's not like Make syntax is ideal or consistent; I can see this being especially useful when using it in part of a pipeline, where maybe the YAML is generated elsewhere.
- garmaine 9y ago
  > but it's not like Make syntax is ideal or consistent It's standardized in POSIX. "Ideal" is subjective.
  - jchw 9y ago
    Fine. But most sane people would agree that Makefile generation is not a fun business. Autotools, anyone? OTOH, generating YAML documents is easy. I already write code that generates YAML documents. P.S.: POSIX make doesn't support concurrency. So, it's not fair to compare this purely to POSIX make.
    - sirn 9y ago
      > OTOH, generating YAML documents is easy. On the topic of syntax, I'll say YAML is not the best format for configuration either. At a glance, it looks simple and easy to write, but YAML can bite its user in an unexpected ways due to its implicit typing[1]. For example this YAML: on: yes …will be parsed to: {true => true} …which cannot be serialized to JSON in a lossless way (since JSON spec specify that a key must be a string): {"true":true} …and translating this back to YAML would result in: 'true': true This is without considering other features such as anchor, aliases and tags[2], which further make the YAML spec incredibly complex. [1]: https://github.com/crdoconnor/strictyaml/blob/master/FAQ.rst#what-is-wrong-with-implicit-typing https://github.com/crdoconnor/strictyaml/blob/master/FAQ.rst... [2]: http://www.yaml.org/spec/1.2/spec.html http://www.yaml.org/spec/1.2/spec.html
      - icebraining 9y ago
        I agree that the implicit typing is dangerous, but developers shouldn't expect to convert YAML to JSON losslessly. It's a superset, after all.
redshirt 9y ago
The nice clean dependency graph is pretty cool. Not having tabs and spaces as an integral part of syntax, priceless.
EdSchouten 9y ago
What about Ninja? https://ninja-build.org/ https://ninja-build.org/
gyoubu 9y ago
Looks nice! Will inspire my project as well. Made something recently as well which is based on the same principle. A simple parallel/sequential task runner which has built in retries and allows steps to be continued even if they return errors. https://github.com/abelmokadem/scriptz https://github.com/abelmokadem/scriptz
JhGhnbgHJ 9y ago
Been looking for something like this forever. Thanks! I’m running batch jobs of long running single threaded tasks that depend on each other in a somewhat complicated manner.
jdright 9y ago
I thought it was related to this: https://github.com/fungos/cr https://github.com/fungos/cr It was recently over Twitter.
jlebrech 9y ago
I miss read this, I thought it was an implementation of workers with a dependency graph. for similar projects look at: Bazel https://bazel.build/ https://bazel.build/ Tup http://gittup.org/tup/ http://gittup.org/tup/ Make https://www.gnu.org/software/make/ https://www.gnu.org/software/make/
peterwwillis 9y ago
I'm sure somebody has a use for this. But with no scheduler tuning, no prioritization, no dynamic configuration, no multiprocess logic, etc it's effectively just "xargs" with dependency tracking, written in Go. And I'm pretty sure you can do this with Ansible.
- dullgiulio 9y ago
  Indeed it's almost trivial. I wrote a similar tool that is also able to split chunks of work between different machines: https://github.com/dullgiulio/empa/tree/master/cmd/ep https://github.com/dullgiulio/empa/tree/master/cmd/ep But again, no IPC, just a simplified "GNU Parallel."
gabrielcsapo 9y ago
This is awesome. I have been working on something similar, but added a UI to visualize the tasks and the order they ran in. https://www.gabrielcsapo.com/build.sh/ https://www.gabrielcsapo.com/build.sh/
jitl 9y ago
A co-worker of mine recently implemented something very similar. Our major feature over Make was emitting events as newline separated JSON, so that a downstream process can visualize them in a pretty way, track progress, etc. We use ours run multiple Chef recipes in parallel.
yellowapple 9y ago
License?

83 points by cirowrc 9y ago

deleted 9y ago

[deleted]

znpy 9y ago

That's what make has been doing for the last like 40 years.

blablabla123 9y ago
I wish the make documentation was better though. I've also read oftentimes that people say that turning on make concurrency with `-j` is on your own risk.
- Coding_Cat 9y ago
  Make's concurrency is pretty good, as long as your dependency graph is correct. Problems arise when one of your build steps creates temporary files for each invocation and they end up overwritting each other (race condition). This would have the same issue
  - jschwartzi 9y ago
    Or when people create phony targets instead of actually declaring an intermediate file and its prerequisites.
- xyzzy_plugh 9y ago
  That's because people write shitty makefiles. Most makefiles I see are full of improperly used double colon rules, phony targets not marked .PHONY and implicit ordering assumptions. If `-j' was enabled by default, then it would be much harder for writers to develop these issues. But I suspect most of these authors, who are already unwilling to learn make properly would simply give up and use a different tool.
  - willejs 9y ago
    Here here.
    - IgorPartola 9y ago
      It’s hear hear.
  - vvanders 9y ago
    The problem is that most people don't set out to be masters of make. They just want to get their shit built so they can make forward progress on the thing they want to create. Having to learn all of these subtleties is what gets you into these problems in the first place. I'll happily take a more opinionated build tool with proper escape hatches over something that expects you to know it in detail for even the most basic tasks.
    - Sir_Cmpwn 9y ago
      I really hate this attitude. The same people who don't grok Make also don't grok git or bourne shell or who knows what else. Don't take shortcuts in understanding the fundamental tools of your stack. Your build system is just as important to your project as the code it's building!
      - vvanders 9y ago
        Make is only a fundamental tool if that's what your library/binary uses. You may have a mix of CMake, MSVC and language specific build tools like Cargo and others. By your logic I should be an expert in all of them. Make's support outside of the posix world isn't that great(as opposed to CMake) so calling it fundamental seems that you are making assumptions about the environments that development happens in.
        Sir_Cmpwn 9y ago
        >You may have a mix of CMake, MSVC and language specific build tools like Cargo and others. By your logic I should be an expert in all of them. Correct. >Make's support outside of the posix world The world outside of posix doesn't really matter.
- AFNobody 9y ago
  Documentation of open source projects is a great example of the tragedy of the commons.
peterwwillis 9y ago
BUT THIS IS WRITTEN IN GO!
- sctb 9y ago
  Please don't. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
  - peterwwillis 9y ago
    But this is written in Go!
pweissbrod 9y ago
Make is possibly the most popular thing people re-invent for specific platforms. Touch of irony in this project, with that Makefile sitting in the root directory. EDIT: My apologies if the above comment was cynical. There's a salient need for workflow systems which provide features above/beyond the basics of make. Built-in things like a central broker, adjustable worker processes, scheduling and history. Python/Luigi is an example of such a workflow system, yet it keeps the workflow definitions very similar to Make for conciseness
haldean 9y ago
And tup for the last 10. Lots of people in this thread complaining about Make's syntax might want to check it out; it has a nice, simple "inputs |> command |> outputs" syntax, a similar "output the DAG" mode to Cr, and defaults to parallelism. http://gittup.org/tup/ http://gittup.org/tup/
- aidenn0 9y ago
  That's funny because I would list tup's syntax as a negative for an otherwise very useful tool. Also, Tup is much more single-purpose than Make or (from a quick readthrough) Cr. It is very strongly focused on the "N files in, 1 file out" for compilation, and specializes at that to the point of not doing other things. As far as a minimal "better make" I find redo to be the best. Make is a weird combination of the shell and makefile languages, but redo boils it down to just a single language and manages to handle dependency of build rules in an elegant manner.
  - dilap 9y ago
    curious, which variant of redo do you actually use?
- CJefferson 9y ago
  I love tup, but I find too many places where it doesn't work to use it for things I distribute (sometimes fuse fails, doesn't work on Linux subsystem for Windows, flakey on full windows).

stephenr 9y ago

So, `make -j`?

deleted 9y ago
[deleted]

jchw 9y ago

Everyone is going to compare it to Make, but it's not like Make syntax is ideal or consistent; I can see this being especially useful when using it in part of a pipeline, where maybe the YAML is generated elsewhere.

garmaine 9y ago
> but it's not like Make syntax is ideal or consistent It's standardized in POSIX. "Ideal" is subjective.
- jchw 9y ago
  Fine. But most sane people would agree that Makefile generation is not a fun business. Autotools, anyone? OTOH, generating YAML documents is easy. I already write code that generates YAML documents. P.S.: POSIX make doesn't support concurrency. So, it's not fair to compare this purely to POSIX make.
  - sirn 9y ago
    > OTOH, generating YAML documents is easy. On the topic of syntax, I'll say YAML is not the best format for configuration either. At a glance, it looks simple and easy to write, but YAML can bite its user in an unexpected ways due to its implicit typing[1]. For example this YAML: on: yes …will be parsed to: {true => true} …which cannot be serialized to JSON in a lossless way (since JSON spec specify that a key must be a string): {"true":true} …and translating this back to YAML would result in: 'true': true This is without considering other features such as anchor, aliases and tags[2], which further make the YAML spec incredibly complex. [1]: https://github.com/crdoconnor/strictyaml/blob/master/FAQ.rst#what-is-wrong-with-implicit-typing https://github.com/crdoconnor/strictyaml/blob/master/FAQ.rst... [2]: http://www.yaml.org/spec/1.2/spec.html http://www.yaml.org/spec/1.2/spec.html
    - icebraining 9y ago
      I agree that the implicit typing is dangerous, but developers shouldn't expect to convert YAML to JSON losslessly. It's a superset, after all.

redshirt 9y ago

The nice clean dependency graph is pretty cool. Not having tabs and spaces as an integral part of syntax, priceless.

EdSchouten 9y ago

What about Ninja? https://ninja-build.org/ https://ninja-build.org/

gyoubu 9y ago

Looks nice! Will inspire my project as well. Made something recently as well which is based on the same principle. A simple parallel/sequential task runner which has built in retries and allows steps to be continued even if they return errors. https://github.com/abelmokadem/scriptz https://github.com/abelmokadem/scriptz

JhGhnbgHJ 9y ago

Been looking for something like this forever. Thanks! I’m running batch jobs of long running single threaded tasks that depend on each other in a somewhat complicated manner.

jdright 9y ago

I thought it was related to this: https://github.com/fungos/cr https://github.com/fungos/cr It was recently over Twitter.

jlebrech 9y ago

I miss read this, I thought it was an implementation of workers with a dependency graph. for similar projects look at: Bazel https://bazel.build/ https://bazel.build/ Tup http://gittup.org/tup/ http://gittup.org/tup/ Make https://www.gnu.org/software/make/ https://www.gnu.org/software/make/

peterwwillis 9y ago

I'm sure somebody has a use for this. But with no scheduler tuning, no prioritization, no dynamic configuration, no multiprocess logic, etc it's effectively just "xargs" with dependency tracking, written in Go. And I'm pretty sure you can do this with Ansible.

dullgiulio 9y ago
Indeed it's almost trivial. I wrote a similar tool that is also able to split chunks of work between different machines: https://github.com/dullgiulio/empa/tree/master/cmd/ep https://github.com/dullgiulio/empa/tree/master/cmd/ep But again, no IPC, just a simplified "GNU Parallel."

gabrielcsapo 9y ago

This is awesome. I have been working on something similar, but added a UI to visualize the tasks and the order they ran in. https://www.gabrielcsapo.com/build.sh/ https://www.gabrielcsapo.com/build.sh/

jitl 9y ago

A co-worker of mine recently implemented something very similar. Our major feature over Make was emitting events as newline separated JSON, so that a downstream process can visualize them in a pretty way, track progress, etc. We use ours run multiple Chef recipes in parallel.

yellowapple 9y ago

License?