4 ms·
I agree with most of this, but was surprised by your comment: > Oh, also, the implementation of Kubernetes cron jobs is also complete garbage (spawning a new p
by avita1 6y ago
I agree with most of this, but was surprised by your comment:
> Oh, also, the implementation of Kubernetes cron jobs is also complete garbage (spawning a new pod every job is insanely wasteful).
How often/how many cron jobs are you running that spawning a new pod per job is a problem ?
- dvt 6y agoThe first iteration (I actually wasn't around for that) was trying to run a cron for every "data ingestion job" -- at some points, we were doing about 50k+ API requests daily (FB/Instagram/Twitter/etc.) and that was absolutely not tenable using k8s cronjobs.
- deleted 6y ago[deleted]
- snuxoll 6y agoWhy use cronjobs at all for this? This is a classic work queue problem.
- dvt 6y agoI wasn't there for this decision, but I assume cronjobs were being treated as "cloud functions" -- and to be fair, the k8s documentation kind of makes it seem that you could technically do that, but fails miserably if you try to do so in practice.
- dilyevsky 6y ago50k/day is less than 1 qps. This is nothing. This is either not the full story or your cluster was setup completely wrong
- regularfry 6y agoDepends how spiky the distribution was. 20k a second at 2pm? Gonna have a bad time.