3 ms·

Comparing Norc to batch is similar to comparing it to cron. Many of the limitations of batch processing come with retrying or auditing complex trees of depende

Comparing Norc to batch is similar to comparing it to cron. Many of the limitations of batch processing come with retrying or auditing complex trees of dependencies (bigger than a handful of tasks). It's like wrapper scripts. They work well until they don't: managing the logs, getting efficient execution, understanding what happens when all become quite unwieldy, whereas a Norc-like approach proves more self-documenting and easy to manage. I believe resource management is limited to overall system load. Batch isn't designed for things like managing available licenses. I'm not a huge expert at batch, so if I've missed something let me know.

noste 17y ago
Comparing Norc to batch is similar to comparing it to cron. Many of the limitations of batch processing come with retrying or auditing complex trees of dependencies (bigger than a handful of tasks). Condor has DAGMan (http://www.cs.wisc.edu/condor/dagman/ http://www.cs.wisc.edu/condor/dagman/) for managing DAGs of jobs, and I suspect that it can scale higher than just a handful of tasks. I believe resource management is limited to overall system load. Batch isn't designed for things like managing available licenses. Condor acts as match maker for jobs and cluster nodes using "classified advertisements" (http://www.cs.wisc.edu/condor/classad/ http://www.cs.wisc.edu/condor/classad/). Classads allow you to describe the jobs and the cluster nodes, and to express arbitrary requirements and preferences for both. This allows jobs to say "I need a node with X", but it also allows nodes to express requirements (e.g. "I won't any run jobs coming from the psych department"). I don't see why you couldn't use this system for expressing requirements regarding licenses. However, Condor appears to be very complex. The PDF version of the Condor 7.3 manual has 991 pages in it. OTOH, Red Hat MRG Grid is based on Condor (http://www.redhat.com/mrg/grid/ http://www.redhat.com/mrg/grid/), so commercial support should be available.
- timf 17y ago
  > don't see why you couldn't use this system for expressing requirements regarding licenses People do just that: http://www.cs.wisc.edu/condor/techpaper/licenses.html http://www.cs.wisc.edu/condor/techpaper/licenses.html I would not say it is "complex" but ultra flexible. You will not have things working in just one day or anything, but I've set it up without a significant hassle. People have had a lot of ongoing success with Condor in large (and very large) computing environments.
  - cabacon 17y ago
    I'd agree that "length of manual" is a poor heuristic for complexity. Condor is super-customizable, and they document those options pretty extensively. To the direct point, they've even started highlighting the use of condor as "condor cron" in the Open Science Grid to run monitoring applications: http://vdt.cs.wisc.edu/releases/1.10.1/notes/Condor-Cron.html http://vdt.cs.wisc.edu/releases/1.10.1/notes/Condor-Cron.htm... Without that, I'd say it would be a slightly masochistic exercise to take a big system like condor and try to make it a cron replacement; while it clearly has that capability, if that's not one of its use cases, it's going to be a real bear to force it into that configuration. I've been a fringe user of condor before, and it can result in some unfriendly "rejected your job for unknown reasons" situations.

Comparing Norc to batch is similar to comparing it to cron. Many of the limitations of batch processing come with retrying or auditing complex trees of dependencies (bigger than a handful of tasks). It's like wrapper scripts. They work well until they don't: managing the logs, getting efficient execution, understanding what happens when all become quite unwieldy, whereas a Norc-like approach proves more self-documenting and easy to manage. I believe resource management is limited to overall system load. Batch isn't designed for things like managing available licenses. I'm not a huge expert at batch, so if I've missed something let me know.

noste 17y ago

Comparing Norc to batch is similar to comparing it to cron. Many of the limitations of batch processing come with retrying or auditing complex trees of dependencies (bigger than a handful of tasks). Condor has DAGMan (http://www.cs.wisc.edu/condor/dagman/ http://www.cs.wisc.edu/condor/dagman/) for managing DAGs of jobs, and I suspect that it can scale higher than just a handful of tasks. I believe resource management is limited to overall system load. Batch isn't designed for things like managing available licenses. Condor acts as match maker for jobs and cluster nodes using "classified advertisements" (http://www.cs.wisc.edu/condor/classad/ http://www.cs.wisc.edu/condor/classad/). Classads allow you to describe the jobs and the cluster nodes, and to express arbitrary requirements and preferences for both. This allows jobs to say "I need a node with X", but it also allows nodes to express requirements (e.g. "I won't any run jobs coming from the psych department"). I don't see why you couldn't use this system for expressing requirements regarding licenses. However, Condor appears to be very complex. The PDF version of the Condor 7.3 manual has 991 pages in it. OTOH, Red Hat MRG Grid is based on Condor (http://www.redhat.com/mrg/grid/ http://www.redhat.com/mrg/grid/), so commercial support should be available.

timf 17y ago
> don't see why you couldn't use this system for expressing requirements regarding licenses People do just that: http://www.cs.wisc.edu/condor/techpaper/licenses.html http://www.cs.wisc.edu/condor/techpaper/licenses.html I would not say it is "complex" but ultra flexible. You will not have things working in just one day or anything, but I've set it up without a significant hassle. People have had a lot of ongoing success with Condor in large (and very large) computing environments.
- cabacon 17y ago
  I'd agree that "length of manual" is a poor heuristic for complexity. Condor is super-customizable, and they document those options pretty extensively. To the direct point, they've even started highlighting the use of condor as "condor cron" in the Open Science Grid to run monitoring applications: http://vdt.cs.wisc.edu/releases/1.10.1/notes/Condor-Cron.html http://vdt.cs.wisc.edu/releases/1.10.1/notes/Condor-Cron.htm... Without that, I'd say it would be a slightly masochistic exercise to take a big system like condor and try to make it a cron replacement; while it clearly has that capability, if that's not one of its use cases, it's going to be a real bear to force it into that configuration. I've been a fringe user of condor before, and it can result in some unfriendly "rejected your job for unknown reasons" situations.