6 ms·
A Scheduler for the Internet
- blowski 7y agoI don't really understand. What could I do with this that I couldn't do with cron? Or is it that somebody who can't use cron could use this? Or that this is more reliable than cron?
- deleted 7y ago[deleted]
- slap_shot 7y agoWhile cron is a nice tool, people should not be using cron for any production grade automation. Tools have been created to that orchestrate automation with more powerful capabilities, and Airflow has become a defacto standard in the industry. Prefect is an open automation tool, comparable in a lot of ways to Airflow. The author of the post is an early committer to Airflow.
- ahbyb 7y ago>While cron is a nice tool, people should not be using cron for any production grade automation. ?????????????????????????
- jlv2 7y agoBecause many decades of stability based upon simplicity are not good enough for today's world.
- slap_shot 7y agoI don't really get into this argument anymore - I'm convinced opinions like this are held tighter than religion or politics. Here are a few thoughts, written by others, that I would probably bring up if I wanted to get into this argument: https://medium.com/eshares-blog/why-you-shouldnt-use-cron-jobs-5d76001e18d5 https://medium.com/eshares-blog/why-you-shouldnt-use-cron-jo... https://engblog.nextdoor.com/we-don-t-run-cron-jobs-at-nextdoor-6f7f9cc62040 https://engblog.nextdoor.com/we-don-t-run-cron-jobs-at-nextd... https://medium.com/videoamp/what-we-learned-migrating-off-cron-to-airflow-b391841a0da4 https://medium.com/videoamp/what-we-learned-migrating-off-cr... https://medium.com/@rbahaguejr/airflow-a-beautiful-cron-alternative-or-replacement-for-data-pipelines-b6fb6d0cddef https://medium.com/@rbahaguejr/airflow-a-beautiful-cron-alte... As a point of reference, I run a company that brokers terabytes of mission critical data each month for our customers. I've advised dozens of companies on how to properly ensure scheduled jobs are maintainable, instrumented, logged, and reliable. The defenses I've heard for "professional" software and systems engineers using cron in production software are appalling and I find the behavior negligent and to be a terminable offense.
- deleted 7y ago[deleted]
- jkaye2012 7y agoYour view is exceedingly narrow and uninformed if you really believe running Cron in production to be a blanket "terminable offense". A tool not being sufficient for your use case does not make that tool worthless or poor. You should reflect a bit before making statements like that if you actually run a business.
- rovr138 7y agoI’ll only touch on the first link because it's late. Haven't looked at the others. > Smallest resolution is 1 minute — If a task needs to run every 30 seconds, you can’t do it with cron. This is the only one I can see that could be an issue. But engineering wise, I don't think this is the best solution (OP's). You still have overhead if you're launching from an external tool. I'd probably run this inside the application (like they do here). > Error handling — If a job fails, what should happen? Solutions have been built to solve this single problem. Developers love adding more band-aids rather than admitting there is a better way. Deja Vu? If it fails to launch or what was supposed to run failed? Going to assume whatever was supposed to run failed since any application can fail/crash. Even celery beat which is what they propose here. You handle the condition. The same thing you'd do anywhere, true && echo “True” || echo “False” false && echo “True” || echo “False” > Logging — Crons don’t log, unless you tell them too. It is rare to see people log output of their cron jobs. Cron by default mails stdout and stderr. This is a log. Regarding the second point, I'm assuming it means redirects. If so, that's an issue of misconfiguration and not understanding your tool. Same with any tool/program. > Working with cron pulls you out of the application — cron is a system level process. Not an application process. This suggests another process to run jobs (`celery -A proj beat`). It’s another dependency to manage and requires you to launch it. So it still handles an external process. > It’s challenging to give application developers access to anything at a system level. Developers shouldn’t care where their application runs… A good example of this is timezones. If a system person changes the timezone of a server, the cron may run at a different time than expected. The less the app developers have to worry about what they run on, the better. So the author is saying this applications doesn’t know about time zones? Celery Beat does know about them. It works on UTC by default but it can be configured. Regarding giving developers access to system settings... why? This article says they devs can't log cron correctly or deal with timezones, allowing all devs to be able to manipulate this is crazy. If the server runs more than one thing, the mess would be horrible. Timezone misconfiguration on the server, or any kind of misconfiguration, could be an issue. But so is an applications being misconfigured and running on a wrong timezone. If teams can do things like changing timezones without coordinating and talking, that's an internal issue. > The defenses I've heard for "professional" software and systems engineers using cron in production software are appalling and I find the behavior negligent and to be a terminable offense. Except for the 30s resolution, I haven't seen a good reason on this article that doesn't boil down to misconfiguration, not knowing how something works or communication. 30s resolution can be done easy enough, #!/usr/bin/env bash while [ true ]; do sleep 30; #run whatever you need. # Append & if you don't need to wait for it to finish /path/to/program done And then just create a service for it on system.d with `Restart=always`.
- unilynx 7y agocron stable? I remember having to add a cronjob to touch a file every 5 minutes and having Nagios monitor that because occasionally it would just stop processing new jobs and have to be restarted That may have been a Centos5/6 specific thing and it was probably caused by too frequent -HUP ing it, but it didn't feel like the decades made it that stable..
- wuliwong 7y agoMy understanding of why using cron for background tasks with web apps isn't great in production is keeping track of things when you have distributed servers. I think cron is pretty limited in scope to whatever machine it is running on.
- Spivak 7y agoLook, don't get me wrong, cron is a super useful tool for doing system-level tasks on a schedule but "grep a timestamp and run a program if it matches" is practically the definition of "doesn't scale." Unless you code it yourself in your program cron doesn't... - understand dependencies - run jobs across multiple hosts - handle errors (e.g. if err run this) - do any branching whatsoever - handle timespecs with more granularity than 1m - do any reporting (mail counts-ish but if you're telling me you have an email based automation flow I won't believe you) - do any metrics collection - run pools of jobs - understand priorities - allow you to queue jobs The value prop of things like airflow is that they factor all this logic out of your applications. At the point where you need these things "run a thing at a specific time" is the most trivial part.
- cat199 7y ago> - understand priorities > - allow you to queue jobs technically this can be done oldschool with cron+at/batch.
- blowski 7y agoI think you should write their homepage as your comment is much more informative.
- ahbyb 7y agoThen it's the wrong comparison, because cron simply schedules other commands, it's not a programming language.
- dragonwriter 7y agoYes, chron is just a very simple, single-node, low-time-resolution, system-level scheduler. While it may be good at that, that is very often (always would be an exaggeration, probably) not a great tool to be used (at least as an exposed component) in automation of modern production systems.
- dredmorbius 7y agoCron is a scheduler. If you need other features, your cron script(s) themselves can check those. Addressing your specific items: Dependencies. Depending on which of these you're referencing, the package manager (which installs a given cron script), configuration manager, or script itself, can and should test and check these. Multiple hosts. There are numerous multi-host utilities, ranging from SSH extensions / utilities to full multi-host configuration tools (Ansible, Salt, Chef, cfengine, Puppet, etc.) At heart, this isn't cron's problem. It's a schedular, not an orchestrator. Error handling. Your script's issue, not crons. Cron is a schedular, not an error handler. Branching. Not sure what you're referencing, but again, the task of the script, generally. Sub-minute granularity. Cron is a schedular, not a daemon substitute. If you need a daemon operating at sub-minute granularity, write a daemon. Reporting. This is the job of your logging facility, generally. Metrics. This is the job of your accounting facility: systat, syslog, etc., etc. Pools of jobs. This is the job of your script(s), generally. If you have a set or sequence of tasks, trigger the master with cron, and have it manage the set. Again, cron is a scheduler, not an orchestrator. Priorities. Nice. Queue jobs. You want 'at' or 'batch', not cron. Cron is a scheuler, not a job queue.
- untreasure 7y agoUse systemd timers
- oarabbus_ 7y agoPretty sure a lot of production systems use cron
- cat199 7y agothis appears to handle interdependencies / multi machine / in-application hooks you could create some cron framework + glue code that accomplishes the same thing, but that would be alot of code.
- brycesbeard 7y agoWhat’s the context for this? Seems like it’s a marketing piece to get people to sign up for their fake scarcity early access program. Very little interesting in the article.
- wuliwong 7y agoYou can't possibly have read the article.
- brycesbeard 7y agoYour comment is in violation of HN guidelines. All I saw is a marketing piece designed to tell me about the product line of a company I’d never heard of. No context as to things I could relate to, mostly just information and excitement about this prefect platform. Outside of prefects products and features, what is the information you took away from this article that made it worth your time?
- bpt3 7y agoI read it and came to a similar conclusion as the parent poster.
- glial 7y agoHow does this compare to Airflow? It's unclear from the website - can you create DAGs (task sequences)?
- slap_shot 7y agoThis post is focusing on Scheduler, but the parent project, Prefect, has the concept of Flows, which are similar to DAGs. You can convert an Airflow project into a Prefect project. One of their employees wrote about the comparison of the two: https://medium.com/the-prefect-blog/why-not-airflow-4cfa423299c4 https://medium.com/the-prefect-blog/why-not-airflow-4cfa4232...
- sergiomattei 7y agoYeah, this introduction/launch needs a lot of work. I've read over the first paragraph or two and all I see is buzzwords: "turn-key", "orchestration", "cloud", etc... If you can't explain your value proposition simply and concisely so anyone that is not familiar with your product ecosystem can understand, you have a big problem in your hands. This product seems like a simple scheduling tool, but your marketing jargon fluff piece says otherwise.
- mrpotato 7y agoI would think that Quartz is more of a competitor than cron or Windows Scheduler to Prefect Scheduler. How does Prefect Scheduler compare to something like Quartz?