5 ms·

I find the best comments here to be ones where people use their knowledge and experience to discuss the relative strengths and weaknesses of the technology in t

by gnat 1y ago

I find the best comments here to be ones where people use their knowledge and experience to discuss the relative strengths and weaknesses of the technology in the post. I see a bunch of short single-sentence comments here that add no value. For my part, I see this pattern repeatedly at different places. The raw tools in the platforms are too codey and the third-party frameworks like Temporal seem overkill, so you build a scheduler and need to solve the problems OP did: only run once, know if it errored, etc. But it's amazing how "it's firing off a basic action!" becomes a script, then becomes a script composed of reusable actions that can pick up where they left off in case of errors ... Over time your "it's just enough for us!" feature creeps towards the framework's functionality. I'd be curious to know how long the OP's solution stays simple before it submits to the feature creep demands. (Long may complexity be fought off, though! Every day you can live without the complexity of full workflows is a blessing)

ants_everywhere 1y ago
Cloud companies also provide globe-scale cronjobs that work a lot like a Unix cronjob. Arguably less mental overhead than adopting a separate framework. And such a service provides reliability guarantees. If I have to do a reliable periodic service, my go-to is a kubernetes cronjob, which is like a baby version of a cloud cronjob. I'd be reluctant to adopt some sort of task queue framework because of the complexity of the mental model plus the complexity of keeping one more thing running reliably. K8s is already running reliably, I might as well use that.
aloha2436 1y ago
Maybe I'm just lucky to work at a place with good tools, but in my experience Temporal isn't super heavyweight to use compared to building your own even-very-simple scheduler. And it's worth it because now you have Temporal, which is the bees knees as far as I'm concerned. I will gladly sing praises of any tool that saves me getting paged, and Temporal has that in spades.
- booi 1y ago
  second temporal. plus it gives you more freedom to write jobs in different languages... not that you would or should in most cases but there's definitely good reasons
  - cyberpunk 1y ago
    Don’t do it onprem unless you want to spend six figures monthly on cassandra database nodes for pretty shit performance and face constant saas upselling and then discover how hard it is to migrate off of. Write your own scheduler. Oracle is cheaper in the long run.
- tkiolp4 1y ago
  Temporal is awful. Difficult to test, difficult to decouple from your domain code. At least that’s what I have seen in organizations. OP’s solution is rather understandable: with a couple of interfaces, you make the code easily testable.
vasco 1y ago
The pragmatic answer is Jenkins. Always has been.
- deleted 1y ago
  [deleted]
- flakes 1y ago
  Jenkins is a place where you can be safe for a long time, however, it starts to break down at scale. I see it time after time for these batch workflow jobs. At the start, jobs run in seconds and everyone is happy. Over time, jobs start taking long enough to the point where you need to split them. Separate jobs are assigned slices of the original batch. Eventually, there are so many slices that you make a Jenkins job where the sole responsibility is firing off these individual jobs. Then you start hitting the real painpoints in Jenkins. Poor allocation of jobs across your nodes/agents, often overloading CPU/Mem on machines, and you struggle to manage the ungodly interface that is the Jenkins REST endpoint. You install many Jenkins addons to try and address the scheduling problems, and end up with a team dedicated to managing this Jenkins infrastructure. The scaling struggles continue to amass and you end up needing separate Jenkins instances to battle the load. Any attempt at replacing the Jenkins infrastructure goes on standstill, as the amount of random scripts found in Jenkinsfiles has created an insurmountable vendor lock-in. You read a post about a select-for-update job scheduler and reflect on simpler times. You cry as you refactor your Jenkins Groovy DSL.
  - dijit 1y ago
    it’s actually much more common than you think for people to reuse CI systems for cron tasking. It’s always a mistake, but it’s easy in the moment and sticks around longer than I’d like.
    - theshrike79 1y ago
      CI systems like Jenkins are there and they're corp-approved. Getting a weird 3rd party scheduling system with access to internal stuff approved is HARD in big corps. So we (ab)use the CI system we have. It has scheduling and it already accesses internal resources.

by gnat 1y ago

ants_everywhere 1y ago

Cloud companies also provide globe-scale cronjobs that work a lot like a Unix cronjob. Arguably less mental overhead than adopting a separate framework. And such a service provides reliability guarantees. If I have to do a reliable periodic service, my go-to is a kubernetes cronjob, which is like a baby version of a cloud cronjob. I'd be reluctant to adopt some sort of task queue framework because of the complexity of the mental model plus the complexity of keeping one more thing running reliably. K8s is already running reliably, I might as well use that.

aloha2436 1y ago

Maybe I'm just lucky to work at a place with good tools, but in my experience Temporal isn't super heavyweight to use compared to building your own even-very-simple scheduler. And it's worth it because now you have Temporal, which is the bees knees as far as I'm concerned. I will gladly sing praises of any tool that saves me getting paged, and Temporal has that in spades.

booi 1y ago
second temporal. plus it gives you more freedom to write jobs in different languages... not that you would or should in most cases but there's definitely good reasons
- cyberpunk 1y ago
  Don’t do it onprem unless you want to spend six figures monthly on cassandra database nodes for pretty shit performance and face constant saas upselling and then discover how hard it is to migrate off of. Write your own scheduler. Oracle is cheaper in the long run.
tkiolp4 1y ago
Temporal is awful. Difficult to test, difficult to decouple from your domain code. At least that’s what I have seen in organizations. OP’s solution is rather understandable: with a couple of interfaces, you make the code easily testable.

vasco 1y ago

The pragmatic answer is Jenkins. Always has been.

deleted 1y ago
[deleted]
flakes 1y ago
Jenkins is a place where you can be safe for a long time, however, it starts to break down at scale. I see it time after time for these batch workflow jobs. At the start, jobs run in seconds and everyone is happy. Over time, jobs start taking long enough to the point where you need to split them. Separate jobs are assigned slices of the original batch. Eventually, there are so many slices that you make a Jenkins job where the sole responsibility is firing off these individual jobs. Then you start hitting the real painpoints in Jenkins. Poor allocation of jobs across your nodes/agents, often overloading CPU/Mem on machines, and you struggle to manage the ungodly interface that is the Jenkins REST endpoint. You install many Jenkins addons to try and address the scheduling problems, and end up with a team dedicated to managing this Jenkins infrastructure. The scaling struggles continue to amass and you end up needing separate Jenkins instances to battle the load. Any attempt at replacing the Jenkins infrastructure goes on standstill, as the amount of random scripts found in Jenkinsfiles has created an insurmountable vendor lock-in. You read a post about a select-for-update job scheduler and reflect on simpler times. You cry as you refactor your Jenkins Groovy DSL.
- dijit 1y ago
  it’s actually much more common than you think for people to reuse CI systems for cron tasking. It’s always a mistake, but it’s easy in the moment and sticks around longer than I’d like.
  - theshrike79 1y ago
    CI systems like Jenkins are there and they're corp-approved. Getting a weird 3rd party scheduling system with access to internal stuff approved is HARD in big corps. So we (ab)use the CI system we have. It has scheduling and it already accesses internal resources.