7 ms·
Executing Cron Scripts Reliably at Scale
- throwawaaarrgh 3y ago[flagged]
- Bjartr 3y agoWhat does a more sane sausage factory look like wrt cron or the like?
- SteveNuts 3y agoIt may not necessarily be "more sane" but we define any batch jobs like this in Nomad. The benefits are: - We can collect metrics and logs of the jobs to see how long they take start to finish, and whether they started/died early - More robust scheduling to tolerate node failures - Easier to deploy as IAC
- bomewish 3y agoThis is unhelpful without a more explicit and detailed account of what is deficient in their approach. It’s the internet so not holding mu breath but fyi.
- djbusby 3y agoI'm still over here loving cron, not at this large of a scale for sure but I love getting all these tools to email me key information, in plain text, at the right times.
- swyx 3y agothis is a pretty simple cron system. curious if the authors investigated temporal and other similar workflow engines for the advanced cron feature set (https://docs.temporal.io/workflows#spec https://docs.temporal.io/workflows#spec disclaimer: i used to work there)
- jaydeegee 3y ago100% agree this is a perfect use case for Cadence or Temporal.
- nerdponx 3y agoComing from the "data world", how does a tool like Airflow/Dagster/Prefect differ from these?
- mianos 3y agoIt doesn't. They didn't look for it. This is exactly what DAG based workflow systems to, in a modern professional manner. Crons don't do a DAG.
- jtmarmon 3y agoI recently evaluated Dagster, Prefect, and Flyte for a data pipeliney workflow and ended up going with Temporal. The shared feature between Temporal and those three is the workflow orchestration piece. All 3 can manage a dependency graph of jobs, handle retries, start from checkpoints, etc. At a high level the big reason they’re different is Temporal is entirely focused on the orchestration piece, and the others are much more focused on the data piece, which comes out in a lot of the different features. Temporal has SDKs in most languages, and has a queuing system that allows you to run different workflows or even activities (tasks within a workflow) in different workers, manage concurrency, etc. You can write a parent workflow that orchestrates sub-workflows that could live in 5 other services. It’s just really composable and fits much more nicely into the critical path of your app. Prefect is probably the closest of your list to temporal, in that it’s less opinionated than others about the workflows being “data oriented”, but it’s still only in python, and it deosn't have queueing. In short this means that your workflows are kinda supposed to run in one box running python somewhere. Temporal will let you define a 10 part workflow where two parts run on a python service running with a GPU, and the remaining parts are running in the same node.js process as your main server. Dagster’s feature set is even more focused on data-workflows, as your workflows are meant to produce data “assets” which can be materialized/cached, etc. They’re pretty much all designed for a data engineering team to manage many individual pipelines that are external from your application code, whereas temporal is designed to be a system that manages workflow complexity for code that (more often) runs in your application.
- jaydeegee 3y agoInteresting to see how different teams approaches to the same issue. Often when reading this sort of thing I'm in PR mode and question why they did certain things the way they did. Then I realise that has probably been asked and answered a dozen times before it got released.
- WatchDog 3y agoWe have a slack reminder that executes on the hour, I’ve noticed that it take about 30 seconds to a minute past the hour for the reminder to actually fire.
- AtlasBarfed 3y agojitter on top of the hour for massively scaled notifications is typical. OF COURSE everyone wants reminders at the even points, but what, do you scale up your system for every half-hour/hourly peak? Or just put in the TOS that the activation will jitter by a bit.
- wodenokoto 3y agoDoesn't the client actually ring the bell for the notification? They client could be told "ring on the top of the oncoming minute".
- xyzzy123 3y agoInteresting they went straight from "1 box running shell scripts with flock" to "mega custom thing" w/out going thru something like kube crob jobs or an off-the-shelf scheduler in between. I find jumps like this hint at "political stiction", sometimes it's hard to get permission to do incremental updates to things, you have to wait until the smoke from the burning tires is unmissable, then get big political consensus and a "visible project" to allocate budget and time for what would otherwise be unsexy maintenance work.
- lijok 3y agoIs that not what HN at large keeps on talking about doing more - stretch the thing you use to its absolute limits before you upgrade
- xyzzy123 3y agoFair, tho a load bearing cronjob pet at the beating heart of a $20bn company with 30m+ users is further than I would dare take this advice if I had to carry the pager. It's very common though and I guess the blog is more evidence that simple things (greased by some tears and toil) can take you much further than you might think.
- JohnMakin 3y agoThis is because simplicity in a complex system allows easy problem solving and tooling built around it. Knowledge silos and haunted forests are much easier to avoid this way. Any complex distributed system to be comprehensible to a team let alone a single person has to have individually simple components with obvious design. There’s way too many rube goldberg machines out there.
- noptd 3y agoYeah, either that or "architecting something new looks better on my resume than using an existing solution."
- 3y ago
- deleted 3y ago[deleted]
- djboz 3y agoWhat's the workflow for job registration look like? Does an engineer upload a single script/binary to some storage service(e.g., EFS/S3/whatever) or builds and uploads a container with script/binary, registers the rate in the conductor UI and then the job queue just pulls the script or container and runs it? Is registration through an IaC of sorts, or just through some UI/CLI?
- whorleater 3y agoI would be curious why kube cron jobs didn't seem to fit the bill, my favorite part of these posts are when they have a section hinting that they explored other options picked specific tradeoffs
- gtirloni 3y agoYeah, the article raises more questions than it answers them. > When designing this new, more reliable service, we decided to leverage many existing services to decrease the amount we had to build This might explain building from scratch. Maybe the existing solutions had dependencies they didn't want to maintain and they opted for using the existing internal systems. It feels like that influenced all the rest.
- paxys 3y agoSpinning up a new Kubernetes pod for every single job run is a very expensive and wasteful operation, starting at least in the order of seconds (usually more) vs just milliseconds for a new process in an already hot environment.
- zerbinxx 3y agoSure, but if you need that thing to run every hour for a few seconds, then seconds aren’t really the limiting factor. I don’t doubt that the resource management side of k8s would make it dicey at a certain volume of these things running, though, especially if they eat a lot of compute.
- nickjj 3y agoKubernetes cron jobs are pretty good I must say. I definitely don't come anywhere near Slack's scale but I've managed systems where over 3,000 cron jobs ran per day, half of which came from a cron job running every minute which usually finished in a few seconds. Some of these jobs run for X minutes too. It's nice because there's properties you can configure for each cron job around retries and if it should be uniquely run or not. Maybe certain cron jobs should be re-tried if they fail, for others maybe it's ok to be picked up on the next interval if it fails. Overall it's been super stable for almost 2 years which is when I started using them. Only a handful of jobs failed over this period of time and they weren't the result of Kubernetes, it was because the HTTP endpoint that was being hit from the cron job failed to respond and the cron job failure threshold was reached. It's a good reminder that important jobs run on a schedule should be resilient to failure (saving progress, idempotent, etc.).
- Scubabear68 3y agoI can’t believe an org of Slack’s size relied on cron scripts for anything critical. It is the worst possible way to schedule jobs at scale. Serious problems with discoverability, single points of failure, and failure mode options. Also surprised they didn’t just use an open source scheduler or product. There are a gazillion of them.
- dimgl 3y agoWhy is this so hard to believe? This is proof that simple software can scale very well. Your takeaway, IMO, should be that a lot of solutions are over-engineered relative to the businesses they serve. Not every company handles Slack's scale.
- Scubabear68 3y agoHard to believe because I have seen the cron dumpster fire at so many companies. Random cronjobs running on random boxes ends up with mystery jobs whose existence often goes unnoticed as people leave the org and docs are out of date. Random accounts with different cron jobs, with no visibility from one account to another. Box failure leads to mystery jobs failing with no way to figure out what happened. No “advanced” features like retry and retry with exponential back off. No database with job history. No parallelism. Every place I have worked cron turned into a dumpster fire.
- teaearlgraycold 3y agoOr you could have a canonical cron box.
- thr0 3y ago[dead]
- progbits 3y ago> To make this transition between pods seamless, we implemented logic to prevent the node from going down at the top of a minute when possible since — given the nature of cron — that is when it is likely that scripts will need to be scheduled to run. Crons with precisely specified time where everyone just uses whole minutes/hours are not great practice. Very unlikely you actually need such precision in a cron job and you get spiky load. Usual approach is to set the minute to a hash of the cron config name or something, modulo 60. Hourly jobs still run hourly, but each one on random minute. (Setting aside how fragile that setup of avoiding pod downtime sounds)
- brown9-2 3y ago> we implemented logic to prevent the node from going down at the top of a minute when possible since — given the nature of cron — that is when it is likely that scripts will need to be scheduled to run Why not smear the start time of the jobs across seconds of that minute to avoid any thundering herd problems? How much functionality relies on a script being invoked at exactly the :00 mark? And if the functionality depends on that exact timing, doesn’t it suggest something is fragile and could be redesigned to be more resilient?
- wutwutwat 3y agoAt their scale, staggering script start times over a 60 second window likely wouldn’t have much of an impact if they are experiencing a thundering herd, imo. If it did help, it would be a bandaid and ticking time bomb before someone has to actually solve the load problem that staggering start times kicked down the road
- paxys 3y agoSlack running all their scheduled tasks on a single box using linux cron all the way up till now is an amazing point in favor of every "you don't need overly complex system architecture at your company" argument anyone has ever made.
- ilrwbwrkhv 3y ago[flagged]
- latchkey 3y agoProbably the simplest scheduling service I've ever used was Google Cloud Scheduler. You give it a schedule and an endpoint (HTTP or a Cloud Task) and then it hits that endpoint on a schedule. I go with Scheduler->Cloud Tasks->Cloud Functions, which gives you reliability and near infinite scalability. Very easy to reason about and full monitoring of the whole stack.
- TruthWillHurt 3y agoo rly? what happens if they fails? can you control the backoff period? can you schedule another job in the event it fails? what if it fails before starting the function, so your in-code error handling isn't triggered? is there a "dead-letter" queue? will the next scheduled job run if the previous run failed? should it? can you define if it should or not? can you view history of executed jobs? what if your logging function that saves that history fails? can anyone add jobs or just you/superadmin? should they? and more and more. And these are just ones I personally ran into using GCP schedulers, pub/sub and functions. See, what you're doing is re-inventing the wheel. No matter how cool your tool is - there's always work in the edge cases beyond just running the job.
- 1a527dd5 3y agoWow, you sound a little bitter. We use AWS CloudWatch Events at work, and it's fine.
- latchkey 3y agoAs the person below states, you sound bitter. There is nothing about your questions that wouldn't apply to any system. Literally all of your questions are answered in the rather well written documentation. I could go through them and answer them for you, but I don't think you'd really appreciate that.
- fdefilippo 3y agoWasn't it simpler to use Cronicle (https://github.com/jhuckaby/Cronicle https://github.com/jhuckaby/Cronicle)?
- roydivision 3y agoReading this I realise an unconscious rule I've followed for years - Cron is for running jobs pertaining to that box, and that box alone. It _can_ be used to schedule anything, but that doesn't mean it should, especially business logic. "I wrote a CRM system in ksh script, it works great!"
- reacharavindh 3y agoIsnt cron inherently easy to horizontally scale? As in, if you have 500 cron scripts and you think you're reaching capacity of that box, just distribute the 500 scripts in one cron tab file to two boxes with 250 each? If one cares more about the reliability of things, you can keep tab on the cron scripts starting at their times, and if they dont, then bring the box down and start a new box with the same cron tab?
- ryze20245 3y agoI feel like a more effective way to use cron is just to dispatch jobs into a queue that will perform the actual processing. And not to do the processing within the cron scripts themselves. That way the load on the cron is light and the heavy lifting is done by your queue/worker system.
- getoffmyyawn 3y agoThis works great in my experience. A lifetime ago I scaled up cron jobs for a client with Gearman. Using cron to trigger jobs on the Gearman server and the pool of runners to do all the work. This proved to be so reliable they still use the system today, over 10 years later.
- djboz 3y agoReally cool! For the Gearman workers, did you load jobs dynamically? Or, would you have to re-deploy for new jobs/updated jobs?
- getoffmyyawn 3y agoAs I recall, all the jobs are checked into a repo that is deployed to all the runners, which each start gearman workers for their assigned role.
- getoffmyyawn 3y agoThat there are numerous mature battle tested open source solutions to distributed and/or centrally managed job queues that it really makes me wonder how they justified building something from scratch.
- chronid 3y agoI think there's a bit of "they could" but also something that is considered very little in many contexts unless you have experienced the contrary: integration is costly and integrating properly sometimes is more work than doing something from "scratch", so you don't do it and then you have a mess that hurts you in the long run.
- getoffmyyawn 3y agoI'm sure it's indeed something like that. I think it also comes down to, at least partly, having a culture that is more about building components than systems. I suspect it could also be the "buzz" factor. The press release about building a new system always seems more exciting than one about solving a familiar problem with boring old existing software.