6 ms·

Innovating Cron: Announcing Norc

51 points by jeremymims 17y ago

jcapote 17y ago
"AutoSys is a closed, proprietary system, and SQS / RabbitMQ are queueing systems, without support for scheduling." Right, why would a message queue support scheduling?
- darrellsilver 17y ago
  Norc isn't meant as a replacement for any queuing system, rather a replacement for Cron. We use Norc in conjunction with SQS, and find the former is good for scheduling, management, logging, while SQS is better for lots and lots of repetitive tasks that we want processed ASAP. Perpetually.com crawls, archives and versions any web site on demand, with any repeating schedule. One of the challenges is managing bursts of activity, so what we do is use Norc to manage the timing of archive requests, then SQS to manage farming out the actual archiving to available hosts. Since we hook SQS into Norc we can easily monitor/audit any system delay or outage. We also use it to handle system backups, sending alerts and other system administration tasks.
  - hapless 17y ago
    So it's a hybrid between a batch job processor and cron? Batch cron?
  - deleted 17y ago
    [deleted]
    - deleted 17y ago
      [deleted]
- sophacles 17y ago
  The answer is in the article: While cron is great, it’s not geared toward solving this problem: Tasks are tied to a single computer, and they’re managed independently for each host and user from the command line.
DannoHung 17y ago
Is it possible to start a task inside a job and have it's dependencies start up after it finishes without starting the whole job? Like, say I have a sequence of operations to one, in the middle, something screws up (like a network connection is down for some reason). Can I rerun from the step that screwed up and get the rest to follow through?
- deleted 17y ago
  [deleted]
- darrellsilver 17y ago
  If I understand the question the answer is yes. I think you're asking if you can retry portions of a job, like if a task breaks but doesn't exit with an error, thus causing the rest of the job to continue. In this case, yes, you can retry a task in the middle of a tree of tasks, and retry all the children tasks as well. This will leave some tasks untouched (the parents and peers) and retry the children in the same order as the first time. If a task in the middle of the tree fails children tasks will not run until it's been skipped or successfully run. Retrying children tasks would be a great job for a GUI for Norc because then you could choose which children tasks to retry interactively from a web app or somesuch...
  - DannoHung 17y ago
    Yep. That's what I wanted to know. Sorry for the poor phrasing. I was little confused because it seemed like norc would only start jobs at the top level.
    - darrellsilver 17y ago
      Yeah, the terminology isn't well-defined. Jobs start with any tasks that return True when asked due_to_run(), which can be schedule, parents' status, etc. Glad I could clear it up!
- silentbicycle 17y ago
  Provided the events are run locally (not distributed over multiple servers), this could be done pretty easily by running make targets out of cron.
pinko 17y ago
While cron is clearly pushed way past its capabilities by many people (without always realizing it), this strikes me as a reinvention of something that's been around forever: the batch system. If the goals are improved reliability, management, fault-tolerance, etc., you'd be better off using mature software that was designed to solve this problem, and already has good community support, like Condor.
- darrellsilver 17y ago
  Comparing Norc to batch is similar to comparing it to cron. Many of the limitations of batch processing come with retrying or auditing complex trees of dependencies (bigger than a handful of tasks). It's like wrapper scripts. They work well until they don't: managing the logs, getting efficient execution, understanding what happens when all become quite unwieldy, whereas a Norc-like approach proves more self-documenting and easy to manage. I believe resource management is limited to overall system load. Batch isn't designed for things like managing available licenses. I'm not a huge expert at batch, so if I've missed something let me know.
  - noste 17y ago
    Comparing Norc to batch is similar to comparing it to cron. Many of the limitations of batch processing come with retrying or auditing complex trees of dependencies (bigger than a handful of tasks). Condor has DAGMan (http://www.cs.wisc.edu/condor/dagman/ http://www.cs.wisc.edu/condor/dagman/) for managing DAGs of jobs, and I suspect that it can scale higher than just a handful of tasks. I believe resource management is limited to overall system load. Batch isn't designed for things like managing available licenses. Condor acts as match maker for jobs and cluster nodes using "classified advertisements" (http://www.cs.wisc.edu/condor/classad/ http://www.cs.wisc.edu/condor/classad/). Classads allow you to describe the jobs and the cluster nodes, and to express arbitrary requirements and preferences for both. This allows jobs to say "I need a node with X", but it also allows nodes to express requirements (e.g. "I won't any run jobs coming from the psych department"). I don't see why you couldn't use this system for expressing requirements regarding licenses. However, Condor appears to be very complex. The PDF version of the Condor 7.3 manual has 991 pages in it. OTOH, Red Hat MRG Grid is based on Condor (http://www.redhat.com/mrg/grid/ http://www.redhat.com/mrg/grid/), so commercial support should be available.
timf 17y ago
Celery supports scheduled tasks (including recurring ones like cron): http://ask.github.com/celery/introduction.html http://ask.github.com/celery/introduction.html It was originally written in the context of Django but it is a general purpose Python library now.
- darrellsilver 17y ago
  I've heard some great things about Celery, and I think it may be better at handling massive amounts of tasks (10s of thousands) without using SQS like Norc. How does it handle logs, resources and changing trees? Gonna have to look into it more!
  - timf 17y ago
    Instead of SQS it uses RabbitMQ. I don't know about it's limits but since a serious RabbitMQ installation can handle something like millions of messages per second I assume the limits you will hit are in the persistence solution on celeryd. It can use a Tokyo Tyrant or MongoDB backend instead of a RDBMS, as well as memcached support. Those seem like they would help in that department. I don't think you can change trees after they are sent (you mean subtasks, right?). You can see logging example at "defining and executing tasks" section here: http://ask.github.com/celery/introduction.html#usage http://ask.github.com/celery/introduction.html#usage Sorry, I am only a light user at this point in time.
    - darrellsilver 17y ago
      I've been hearing really good things about RabbitMQ, and haven't been particularly impressed with SQS. It works at scale (we're currently at 10s of thousands a day, but it's probably the same performance at 100 or 1000x that...) but tasks take many seconds (like 10-30) to show up in the queue, which absolutely kills us. We'll be adding a RabbitMQ plugin to Norc, just like SQS, when we can.
    - vegai 17y ago
      http://www.rabbitmq.com/faq.html#performance http://www.rabbitmq.com/faq.html#performance "From our testing, we expect easily-achievable throughputs of 4000 persistent, non-transacted one-kilobyte messages per second (Intel Pentium D, 2.8GHz, dual core, gigabit ethernet) from a single RabbitMQ broker node writing to a single spindle." Or what do you mean by serious?
Hoff 17y ago
Apple has supplanted cron with launchd within Mac OS X and Mac OS X Server: http://developer.apple.com/macosx/launchd.html http://developer.apple.com/macosx/launchd.html
- blasdel 17y ago
  It also replaces /bin/init, init scripts, xinetd scripts, and more! Almost everything that executes something automatically is now done through launchd (except LoginWindow still does its own instead of providing input to launchd). The only annoyance is Apple's worst-of-both-worlds XML plist format, which replaced something fairly close to JSON with an awful pair-wise angle-bracket shit-pile.
- wtallis 17y ago
  More importantly, http://launchd.macosforge.org/ http://launchd.macosforge.org/
NikkiA 17y ago
fcron has always been my 'go to' for a 'better cron' (http://fcron.free.fr/ http://fcron.free.fr/) and thankfully it has been available as an 'alternative' to cron in most distros for a while now. That said, norc appears interesting, and I'll certainly be keeping an eye on it (I doubt I'll switch until it's a bit more established - and available in my favourite distros :)
djb_hackernews 17y ago
I know I'm abusing github.com, but does anyone get a redirection loop error in firefox when trying to view http://github.com/darrellsilver/norc/blob/master/core/models.py http://github.com/darrellsilver/norc/blob/master/core/models...?
amcfague 17y ago
Darrell, I know this is a bad place to ask for it, but we actually have just started a story at my company to look for a better scheduling tool--mostly, just a distributed cron. Do you have any preference for Q&A? Should we use HN? If not, feel free to send me an email so I could bounce some questions off you (amcfague at wgen dot net) EDIT: Considering these questions are not answered in the current README, I'd also like to be able to propose additions to the documentation as a result! :)

51 points by jeremymims 17y ago

"AutoSys is a closed, proprietary system, and SQS / RabbitMQ are queueing systems, without support for scheduling." Right, why would a message queue support scheduling?

darrellsilver 17y ago
Norc isn't meant as a replacement for any queuing system, rather a replacement for Cron. We use Norc in conjunction with SQS, and find the former is good for scheduling, management, logging, while SQS is better for lots and lots of repetitive tasks that we want processed ASAP. Perpetually.com crawls, archives and versions any web site on demand, with any repeating schedule. One of the challenges is managing bursts of activity, so what we do is use Norc to manage the timing of archive requests, then SQS to manage farming out the actual archiving to available hosts. Since we hook SQS into Norc we can easily monitor/audit any system delay or outage. We also use it to handle system backups, sending alerts and other system administration tasks.
- hapless 17y ago
  So it's a hybrid between a batch job processor and cron? Batch cron?
- deleted 17y ago
  [deleted]
  - deleted 17y ago
    [deleted]
sophacles 17y ago
The answer is in the article: While cron is great, it’s not geared toward solving this problem: Tasks are tied to a single computer, and they’re managed independently for each host and user from the command line.

DannoHung 17y ago

Is it possible to start a task inside a job and have it's dependencies start up after it finishes without starting the whole job? Like, say I have a sequence of operations to one, in the middle, something screws up (like a network connection is down for some reason). Can I rerun from the step that screwed up and get the rest to follow through?

deleted 17y ago
[deleted]
darrellsilver 17y ago
If I understand the question the answer is yes. I think you're asking if you can retry portions of a job, like if a task breaks but doesn't exit with an error, thus causing the rest of the job to continue. In this case, yes, you can retry a task in the middle of a tree of tasks, and retry all the children tasks as well. This will leave some tasks untouched (the parents and peers) and retry the children in the same order as the first time. If a task in the middle of the tree fails children tasks will not run until it's been skipped or successfully run. Retrying children tasks would be a great job for a GUI for Norc because then you could choose which children tasks to retry interactively from a web app or somesuch...
- DannoHung 17y ago
  Yep. That's what I wanted to know. Sorry for the poor phrasing. I was little confused because it seemed like norc would only start jobs at the top level.
  - darrellsilver 17y ago
    Yeah, the terminology isn't well-defined. Jobs start with any tasks that return True when asked due_to_run(), which can be schedule, parents' status, etc. Glad I could clear it up!
silentbicycle 17y ago
Provided the events are run locally (not distributed over multiple servers), this could be done pretty easily by running make targets out of cron.

pinko 17y ago

While cron is clearly pushed way past its capabilities by many people (without always realizing it), this strikes me as a reinvention of something that's been around forever: the batch system. If the goals are improved reliability, management, fault-tolerance, etc., you'd be better off using mature software that was designed to solve this problem, and already has good community support, like Condor.

darrellsilver 17y ago
Comparing Norc to batch is similar to comparing it to cron. Many of the limitations of batch processing come with retrying or auditing complex trees of dependencies (bigger than a handful of tasks). It's like wrapper scripts. They work well until they don't: managing the logs, getting efficient execution, understanding what happens when all become quite unwieldy, whereas a Norc-like approach proves more self-documenting and easy to manage. I believe resource management is limited to overall system load. Batch isn't designed for things like managing available licenses. I'm not a huge expert at batch, so if I've missed something let me know.
- noste 17y ago
  Comparing Norc to batch is similar to comparing it to cron. Many of the limitations of batch processing come with retrying or auditing complex trees of dependencies (bigger than a handful of tasks). Condor has DAGMan (http://www.cs.wisc.edu/condor/dagman/ http://www.cs.wisc.edu/condor/dagman/) for managing DAGs of jobs, and I suspect that it can scale higher than just a handful of tasks. I believe resource management is limited to overall system load. Batch isn't designed for things like managing available licenses. Condor acts as match maker for jobs and cluster nodes using "classified advertisements" (http://www.cs.wisc.edu/condor/classad/ http://www.cs.wisc.edu/condor/classad/). Classads allow you to describe the jobs and the cluster nodes, and to express arbitrary requirements and preferences for both. This allows jobs to say "I need a node with X", but it also allows nodes to express requirements (e.g. "I won't any run jobs coming from the psych department"). I don't see why you couldn't use this system for expressing requirements regarding licenses. However, Condor appears to be very complex. The PDF version of the Condor 7.3 manual has 991 pages in it. OTOH, Red Hat MRG Grid is based on Condor (http://www.redhat.com/mrg/grid/ http://www.redhat.com/mrg/grid/), so commercial support should be available.

timf 17y ago

Celery supports scheduled tasks (including recurring ones like cron): http://ask.github.com/celery/introduction.html http://ask.github.com/celery/introduction.html It was originally written in the context of Django but it is a general purpose Python library now.

darrellsilver 17y ago
I've heard some great things about Celery, and I think it may be better at handling massive amounts of tasks (10s of thousands) without using SQS like Norc. How does it handle logs, resources and changing trees? Gonna have to look into it more!
- timf 17y ago
  Instead of SQS it uses RabbitMQ. I don't know about it's limits but since a serious RabbitMQ installation can handle something like millions of messages per second I assume the limits you will hit are in the persistence solution on celeryd. It can use a Tokyo Tyrant or MongoDB backend instead of a RDBMS, as well as memcached support. Those seem like they would help in that department. I don't think you can change trees after they are sent (you mean subtasks, right?). You can see logging example at "defining and executing tasks" section here: http://ask.github.com/celery/introduction.html#usage http://ask.github.com/celery/introduction.html#usage Sorry, I am only a light user at this point in time.
  - darrellsilver 17y ago
    I've been hearing really good things about RabbitMQ, and haven't been particularly impressed with SQS. It works at scale (we're currently at 10s of thousands a day, but it's probably the same performance at 100 or 1000x that...) but tasks take many seconds (like 10-30) to show up in the queue, which absolutely kills us. We'll be adding a RabbitMQ plugin to Norc, just like SQS, when we can.
  - vegai 17y ago
    http://www.rabbitmq.com/faq.html#performance http://www.rabbitmq.com/faq.html#performance "From our testing, we expect easily-achievable throughputs of 4000 persistent, non-transacted one-kilobyte messages per second (Intel Pentium D, 2.8GHz, dual core, gigabit ethernet) from a single RabbitMQ broker node writing to a single spindle." Or what do you mean by serious?

Hoff 17y ago

Apple has supplanted cron with launchd within Mac OS X and Mac OS X Server: http://developer.apple.com/macosx/launchd.html http://developer.apple.com/macosx/launchd.html

blasdel 17y ago
It also replaces /bin/init, init scripts, xinetd scripts, and more! Almost everything that executes something automatically is now done through launchd (except LoginWindow still does its own instead of providing input to launchd). The only annoyance is Apple's worst-of-both-worlds XML plist format, which replaced something fairly close to JSON with an awful pair-wise angle-bracket shit-pile.
wtallis 17y ago
More importantly, http://launchd.macosforge.org/ http://launchd.macosforge.org/

NikkiA 17y ago

fcron has always been my 'go to' for a 'better cron' (http://fcron.free.fr/ http://fcron.free.fr/) and thankfully it has been available as an 'alternative' to cron in most distros for a while now. That said, norc appears interesting, and I'll certainly be keeping an eye on it (I doubt I'll switch until it's a bit more established - and available in my favourite distros :)

djb_hackernews 17y ago

I know I'm abusing github.com, but does anyone get a redirection loop error in firefox when trying to view http://github.com/darrellsilver/norc/blob/master/core/models.py http://github.com/darrellsilver/norc/blob/master/core/models...?

amcfague 17y ago

Darrell, I know this is a bad place to ask for it, but we actually have just started a story at my company to look for a better scheduling tool--mostly, just a distributed cron. Do you have any preference for Q&A? Should we use HN? If not, feel free to send me an email so I could bounce some questions off you (amcfague at wgen dot net) EDIT: Considering these questions are not answered in the current README, I'd also like to be able to propose additions to the documentation as a result! :)