4 ms·
I can’t believe an org of Slack’s size relied on cron scripts for anything critical. It is the worst possible way to schedule jobs at scale. Serious problems wi
by Scubabear68 3y ago
I can’t believe an org of Slack’s size relied on cron scripts for anything critical. It is the worst possible way to schedule jobs at scale. Serious problems with discoverability, single points of failure, and failure mode options.
Also surprised they didn’t just use an open source scheduler or product. There are a gazillion of them.
- dimgl 3y agoWhy is this so hard to believe? This is proof that simple software can scale very well. Your takeaway, IMO, should be that a lot of solutions are over-engineered relative to the businesses they serve. Not every company handles Slack's scale.
- Scubabear68 3y agoHard to believe because I have seen the cron dumpster fire at so many companies. Random cronjobs running on random boxes ends up with mystery jobs whose existence often goes unnoticed as people leave the org and docs are out of date. Random accounts with different cron jobs, with no visibility from one account to another. Box failure leads to mystery jobs failing with no way to figure out what happened. No “advanced” features like retry and retry with exponential back off. No database with job history. No parallelism. Every place I have worked cron turned into a dumpster fire.
- teaearlgraycold 3y agoOr you could have a canonical cron box.
- doubloon 3y agoi will see your rando cron box and raise you Excel VBA macros run on Windows Task Scheduler on a spare machine in a cubicle.
- mdekkers 3y agoSure, but those are organisational failures, not technical failures. If you need better technology because the org sucks at planning, the tech isn’t the issue and you are just sweeping issues under the carpet.
- reactordev 3y agoIt’s no longer simple when you have platform code to prevent nodes from disappearing or dying on the minute on a kubernetes cluster the size of slack. A triggered pulse event stream would have done the trick to invoke a lambda or call code for every “thing” that needed a beat. Kubernetes comes with a scheduler…
- zerbinxx 3y agoThis is my take as well, just because you can get away with something doesn’t necessarily make it desirable to maintain or extend, and I honestly cannot imagine the effort in terms of labor hours that you’d have to go through to develop something like this compared to just plugging in an off-the-shelf scheduler to something slightly more sophisticated like k8s or even just a worker-and-queue system. When you’re talking about platform engineering to solve a problem that a relatively extensible Celery service could do (and have tests and such), I have no idea how the former could be “less work” or cheaper in the long haul.
- Scubabear68 3y agoCron scaling? Seriously? By definition cron does not scale, it is account by account per VM/machine with no rhyme or reason.
- Supermancho 3y agoAlternately, if you define cron as a starting/startup point, an expectation when scaling, then simple format for a job queue, the context of the post is understood.
- baq 3y agoYeah that’s the point. If managed properly, turns out it can handle a $20B tech company.
- ohnoesjmr 3y agoCan you name some? I've been looking for software in that space, a thing that runs cronjobs, with ability to kick off adhoc runs from some UI, see what is running, and if possible parameterise the custom runs (i.e., run an ad-hoc report for a different client than usual cron does)
- djbusby 3y agoBuild the UI or API for the report, so you can ad-hoc from there. Cron uses that interface. You don't custom the cron, you cron the custom.
- sciurus 3y agoI almost hate to say it, but Jenkins.
- magarnicle 3y agoThat's what I did. I know it was the right solution because I happily keep adding more and more scripts to run. I didn't do that when I was using cron jobs. Plus, Jenkins has a few nice extensions to the crontab, including setting a timezone and using "H" to spread job execution load.
- fukawi2 3y agoRundeck fits this space.
- 8organicbits 3y agoThere's a lot to expore here. First, I'd suggest that cron and shell scripts may not be what you want. Cron has a complex format for scheduling and can lack features like sub-minute level scheduling, maintenance windows, and other task cadences. Shell scripts are OK for small things, but often I find I move to a different language if I keep something around long enough. Do the people writing jobs know shell? For jobs run by data analysts, airflow and python work great. For devops jobs, begrudgingly, Jenkins or GitHub Actions. But there's so many varieties.
- baq 3y ago
- otabdeveloper4 3y ago> There are a gazillion of them. Such as?
- mdekkers 3y ago> It is the worst possible way to schedule jobs at scale. The literal evidence proves that this approach is absolutely workable at scale, so “the worst possible way” clearly doesn’t apply. The interesting question here is “how did they make this work?” The answer to which is immensely valuable to the technical community at large. “What are they building next?”, whilst interesting, is less immediately valuable as they will be building something Slack-scale, and most orgs are not Slack or Slack-scale
- krisoft 3y ago> The literal evidence proves that this approach is absolutely workable at scale, so “the worst possible way” clearly doesn’t apply. The “worst possible” to me implies that it is possible. Think about trying to solve the same problem by hurling engineers into an erupting vulcano. It is expensive, hurts the morale, causes staff retention issues, but also fundamentally does not solve the task of scheduled task running. I would not describe that as “worst possible” because it lacks the second factor by not being a possible solution.
- blitzar 3y agoSurvivorship bias. Organisations that cobble together cron scripts for critical apps become an "an org of Slack’s size". Companies that deploy solutions that handle all the problems with discoverability, single points of failure, failure mode options and interplanetary multi species scale when they have 0 coustomers never become "an org of Slack’s size".