7 ms·
Interesting they went straight from "1 box running shell scripts with flock" to "mega custom thing" w/out going thru something like kube crob jobs or an off-the
by xyzzy123 3y ago
Interesting they went straight from "1 box running shell scripts with flock" to "mega custom thing" w/out going thru something like kube crob jobs or an off-the-shelf scheduler in between.
I find jumps like this hint at "political stiction", sometimes it's hard to get permission to do incremental updates to things, you have to wait until the smoke from the burning tires is unmissable, then get big political consensus and a "visible project" to allocate budget and time for what would otherwise be unsexy maintenance work.
- lijok 3y agoIs that not what HN at large keeps on talking about doing more - stretch the thing you use to its absolute limits before you upgrade
- xyzzy123 3y agoFair, tho a load bearing cronjob pet at the beating heart of a $20bn company with 30m+ users is further than I would dare take this advice if I had to carry the pager. It's very common though and I guess the blog is more evidence that simple things (greased by some tears and toil) can take you much further than you might think.
- JohnMakin 3y agoThis is because simplicity in a complex system allows easy problem solving and tooling built around it. Knowledge silos and haunted forests are much easier to avoid this way. Any complex distributed system to be comprehensible to a team let alone a single person has to have individually simple components with obvious design. There’s way too many rube goldberg machines out there.
- noptd 3y agoYeah, either that or "architecting something new looks better on my resume than using an existing solution."
- hparadiz 3y agoThe value of total control and flexibility sometimes has no price
- marcosdumay 3y agoDo you have "implemented a task scheduler" on your resume?
- noptd 3y agoPersonally I'd see it as a negative vs using an industry standard solution. However, I'm sure some folks would be tempted to add something like "designed and implemented a distributed task scheduler and execution engine for generalized asynchronous jobs utilized by X number of devs across Y teams" to their resumes.
- marcosdumay 3y agoMy question is on the line of do you think it's relevant enough so that it would deserve being added to a resume? Even if it's something you personally don't like. Because I can't imagine why it would award that relevance. It's right there with "implemented function to reverse a list because the stdlib had a bug".
- teeray 3y ago“Why are we doing this refactor in this PR along with the feature? Let’s do the feature first and then do the refactor later (read: never)”
- Spivak 3y agoThe bet your making is you never need to do the refactor. The amount of absolutely garbage code that has never needed to change ever is easily worth a wait-and-see attitude.
- VWWHFSfQ 3y ago> "1 box running shell scripts with flock" to "mega custom thing" Literally went from a working "tidy little house" to building a massive sky-scraper. Obviously Slack operates at a much larger scale than I do...but holy moly. hey, good luck to them with that!
- spiffytech 3y agoThis line from the k8s CronJob docs has always made me nervous about adopting them: > The scheduling is approximate because there are certain circumstances where two Jobs might be created, or no Job might be created. Kubernetes tries to avoid those situations, but does not completely prevent them.
- ants_everywhere 3y agoThis Stack Overflow [0] answer makes it sound like that's just stating a triviality about distributed systems. For example, if the task that runs your cron is down when your cron is supposed to run, then it won't run. The slack blog says they did some tinkering like preventing nodes from going down at the top of a minute because that's when they think cron jobs are most likely to run. But at scale things are going to break when they break, and you have to weigh the pros and cons of designing the jobs to be robust to failure vs trying to organize failures to correspond to the needs of your jobs. So I think there is space for solutions that make different tradeoffs. But it does seem vastly easier to tune an existing solution that someone else is maintaining than to build your own solution on top of Kafka. [0] https://stackoverflow.com/questions/47691278/why-in-kubernetes-cron-job-two-jobs-might-be-created-or-no-job-might-be-created https://stackoverflow.com/questions/47691278/why-in-kubernet...
- josephg 3y agoMost distributed systems at least promise they do something at least once or at most once. You can often achieve exactly once in practice with a combination of idempotent APIs and a shared database. For a task runner, there are a lot of different behaviours you might want if the system crashes. Maybe the runner should “catch up” after coming back online. That’s easy enough to achieve if you move away from cron and track which tasks have been run in a small data store somewhere.
- marcosdumay 3y agoLooks like they made a cron on top of an eventual-consistent database. Yeah, I'd avoid that too.
- josephg 3y ago> Interesting they went straight from "1 box running shell scripts with flock" to "mega custom thing" I’ve ended up building a custom job scheduler at a couple companies I’ve worked at. It’s a fun little problem. I don’t think it’s fair to characterise the problem as a “mega” thing at all - you can build a custom task scheduler on top of a database or Kafka that’ll be way more reliable than cron in a few hundred lines of code, in just about any language. Are those lines of code wasted? Maybe. But the trade off is that you’re the world expert in that little thing you made. It’ll be easy to connect it to your dashboards for observability, and add the exact set of features you need in your org. I don’t think everyone should roll their own task runner. But it’s not the mega engineering problem you’re imagining. A week or two of one engineer’s time is peanuts at a company like slack.