3 ms·

I don't really get into this argument anymore - I'm convinced opinions like this are held tighter than religion or politics. Here are a few thoughts, written b

I don't really get into this argument anymore - I'm convinced opinions like this are held tighter than religion or politics. Here are a few thoughts, written by others, that I would probably bring up if I wanted to get into this argument: https://medium.com/eshares-blog/why-you-shouldnt-use-cron-jobs-5d76001e18d5 https://medium.com/eshares-blog/why-you-shouldnt-use-cron-jo... https://engblog.nextdoor.com/we-don-t-run-cron-jobs-at-nextdoor-6f7f9cc62040 https://engblog.nextdoor.com/we-don-t-run-cron-jobs-at-nextd... https://medium.com/videoamp/what-we-learned-migrating-off-cron-to-airflow-b391841a0da4 https://medium.com/videoamp/what-we-learned-migrating-off-cr... https://medium.com/@rbahaguejr/airflow-a-beautiful-cron-alternative-or-replacement-for-data-pipelines-b6fb6d0cddef https://medium.com/@rbahaguejr/airflow-a-beautiful-cron-alte... As a point of reference, I run a company that brokers terabytes of mission critical data each month for our customers. I've advised dozens of companies on how to properly ensure scheduled jobs are maintainable, instrumented, logged, and reliable. The defenses I've heard for "professional" software and systems engineers using cron in production software are appalling and I find the behavior negligent and to be a terminable offense.

deleted 7y ago
[deleted]
jkaye2012 7y ago
Your view is exceedingly narrow and uninformed if you really believe running Cron in production to be a blanket "terminable offense". A tool not being sufficient for your use case does not make that tool worthless or poor. You should reflect a bit before making statements like that if you actually run a business.
rovr138 7y ago
I’ll only touch on the first link because it's late. Haven't looked at the others. > Smallest resolution is 1 minute — If a task needs to run every 30 seconds, you can’t do it with cron. This is the only one I can see that could be an issue. But engineering wise, I don't think this is the best solution (OP's). You still have overhead if you're launching from an external tool. I'd probably run this inside the application (like they do here). > Error handling — If a job fails, what should happen? Solutions have been built to solve this single problem. Developers love adding more band-aids rather than admitting there is a better way. Deja Vu? If it fails to launch or what was supposed to run failed? Going to assume whatever was supposed to run failed since any application can fail/crash. Even celery beat which is what they propose here. You handle the condition. The same thing you'd do anywhere, true && echo “True” || echo “False” false && echo “True” || echo “False” > Logging — Crons don’t log, unless you tell them too. It is rare to see people log output of their cron jobs. Cron by default mails stdout and stderr. This is a log. Regarding the second point, I'm assuming it means redirects. If so, that's an issue of misconfiguration and not understanding your tool. Same with any tool/program. > Working with cron pulls you out of the application — cron is a system level process. Not an application process. This suggests another process to run jobs (`celery -A proj beat`). It’s another dependency to manage and requires you to launch it. So it still handles an external process. > It’s challenging to give application developers access to anything at a system level. Developers shouldn’t care where their application runs… A good example of this is timezones. If a system person changes the timezone of a server, the cron may run at a different time than expected. The less the app developers have to worry about what they run on, the better. So the author is saying this applications doesn’t know about time zones? Celery Beat does know about them. It works on UTC by default but it can be configured. Regarding giving developers access to system settings... why? This article says they devs can't log cron correctly or deal with timezones, allowing all devs to be able to manipulate this is crazy. If the server runs more than one thing, the mess would be horrible. Timezone misconfiguration on the server, or any kind of misconfiguration, could be an issue. But so is an applications being misconfigured and running on a wrong timezone. If teams can do things like changing timezones without coordinating and talking, that's an internal issue. > The defenses I've heard for "professional" software and systems engineers using cron in production software are appalling and I find the behavior negligent and to be a terminable offense. Except for the 30s resolution, I haven't seen a good reason on this article that doesn't boil down to misconfiguration, not knowing how something works or communication. 30s resolution can be done easy enough, #!/usr/bin/env bash while [ true ]; do sleep 30; #run whatever you need. # Append & if you don't need to wait for it to finish /path/to/program done And then just create a service for it on system.d with `Restart=always`.
dredmorbius 7y ago
Addressing one of those articles, "What we learned migrating off Cron to Airflow (https://medium.com/videoamp/what-we-learned-migrating-off-cron-to-airflow-b391841a0da4 https://medium.com/videoamp/what-we-learned-migrating-off-cr...): ...changes to the crontab file were not easily traceable... Don't edit the crontab. Use cron.d entries, atomic files dedicated to specific packages, services, and/or tasks. In a multi-host environment, manage those under your configuration management tool. Mind: a major gripe I've had with Chef in the past was that it failed to use individual cron.d files and instead edited the crontab file directly under its cron management interface. That is a critical design error. Cron keeps a log of job outputs on the server where the jobs were run That'a fault of your logging system, as all system services log locally by default. Set up a log server if you need centralised logging. ...bash profile settings have been stripped from the Cron environment... Sounds like someone unfamiliar with cron, or sane script development and testing procedures. If you need environments set, set or source them explicitly (I recommend the latter, FWIW). These are PEBKAC, not cron, errors.

deleted 7y ago

[deleted]

jkaye2012 7y ago

Your view is exceedingly narrow and uninformed if you really believe running Cron in production to be a blanket "terminable offense". A tool not being sufficient for your use case does not make that tool worthless or poor. You should reflect a bit before making statements like that if you actually run a business.

rovr138 7y ago

I’ll only touch on the first link because it's late. Haven't looked at the others. > Smallest resolution is 1 minute — If a task needs to run every 30 seconds, you can’t do it with cron. This is the only one I can see that could be an issue. But engineering wise, I don't think this is the best solution (OP's). You still have overhead if you're launching from an external tool. I'd probably run this inside the application (like they do here). > Error handling — If a job fails, what should happen? Solutions have been built to solve this single problem. Developers love adding more band-aids rather than admitting there is a better way. Deja Vu? If it fails to launch or what was supposed to run failed? Going to assume whatever was supposed to run failed since any application can fail/crash. Even celery beat which is what they propose here. You handle the condition. The same thing you'd do anywhere, true && echo “True” || echo “False” false && echo “True” || echo “False” > Logging — Crons don’t log, unless you tell them too. It is rare to see people log output of their cron jobs. Cron by default mails stdout and stderr. This is a log. Regarding the second point, I'm assuming it means redirects. If so, that's an issue of misconfiguration and not understanding your tool. Same with any tool/program. > Working with cron pulls you out of the application — cron is a system level process. Not an application process. This suggests another process to run jobs (`celery -A proj beat`). It’s another dependency to manage and requires you to launch it. So it still handles an external process. > It’s challenging to give application developers access to anything at a system level. Developers shouldn’t care where their application runs… A good example of this is timezones. If a system person changes the timezone of a server, the cron may run at a different time than expected. The less the app developers have to worry about what they run on, the better. So the author is saying this applications doesn’t know about time zones? Celery Beat does know about them. It works on UTC by default but it can be configured. Regarding giving developers access to system settings... why? This article says they devs can't log cron correctly or deal with timezones, allowing all devs to be able to manipulate this is crazy. If the server runs more than one thing, the mess would be horrible. Timezone misconfiguration on the server, or any kind of misconfiguration, could be an issue. But so is an applications being misconfigured and running on a wrong timezone. If teams can do things like changing timezones without coordinating and talking, that's an internal issue. > The defenses I've heard for "professional" software and systems engineers using cron in production software are appalling and I find the behavior negligent and to be a terminable offense. Except for the 30s resolution, I haven't seen a good reason on this article that doesn't boil down to misconfiguration, not knowing how something works or communication. 30s resolution can be done easy enough, #!/usr/bin/env bash while [ true ]; do sleep 30; #run whatever you need. # Append & if you don't need to wait for it to finish /path/to/program done And then just create a service for it on system.d with `Restart=always`.

dredmorbius 7y ago

Addressing one of those articles, "What we learned migrating off Cron to Airflow (https://medium.com/videoamp/what-we-learned-migrating-off-cron-to-airflow-b391841a0da4 https://medium.com/videoamp/what-we-learned-migrating-off-cr...): ...changes to the crontab file were not easily traceable... Don't edit the crontab. Use cron.d entries, atomic files dedicated to specific packages, services, and/or tasks. In a multi-host environment, manage those under your configuration management tool. Mind: a major gripe I've had with Chef in the past was that it failed to use individual cron.d files and instead edited the crontab file directly under its cron management interface. That is a critical design error. Cron keeps a log of job outputs on the server where the jobs were run That'a fault of your logging system, as all system services log locally by default. Set up a log server if you need centralised logging. ...bash profile settings have been stripped from the Cron environment... Sounds like someone unfamiliar with cron, or sane script development and testing procedures. If you need environments set, set or source them explicitly (I recommend the latter, FWIW). These are PEBKAC, not cron, errors.