4 ms·

> 1. Services like this cron with Dyno Deploy or Vercel where the cloud things are abstracted away for you. I agree with the sentiment, but to me it’s a shame

by klabb3 3y ago

> 1. Services like this cron with Dyno Deploy or Vercel where the cloud things are abstracted away for you. I agree with the sentiment, but to me it’s a shame this even qualifies as abstraction in the first place. Periodic execution of a function is extremely basic and already exists in almost every programming environment. If this needs to be a service (instead of a library call), there is something deeply wrong elsewhere, imo. We all laughed about leftpad, but at least that was native code and not “as-a-service”, dependent on networking and proprietary code on someone else’s machine. I have nothing against Deno in particular, and I’m also not saying this is easy from an implementation perspective (due to complexity elsewhere). I’m just surprised that trivial features – from an end-user perspective – justify such a ruckus.

notpachet 3y ago
I wonder why more people don't just set up a cheap VM and run this sort of thing via crontab.
- Swizec 3y ago
  Does crontab on a self-managed VM guarantee at least once delivery? For many, if not most, use-cases that guarantee is critical.
  - notpachet 3y ago
    No. But I don't think there's any environment out there that is going to 100% guarantee that your cron succeeds. Even if it is reenqueued on failure, it could just keep hitting the same problem and crashlooping. You still need some kind of failure reporting, and a human to jump in to fix whatever went wrong.
- fesc 3y ago
  The same reason why most people stopped manually editing some random files via FTP to do deployments: to get a proper reproducible, automated and monitored production environment.
  - codedokode 3y ago
    Cloud setups are not reproducible, because you cannot reproduce the same env locally or in the other cloud.
  - notpachet 3y ago
    I think there's a threshold below which this is just unnecessary infrastructure overhead, and I'd posit that most cron use cases fall below this threshold. If yours is above it, well and good.
- turtlebits 3y ago
  Because managing servers adds a lot of tech overhead (OS and security updates/access control/disk/HA/monitoring) + provisioning/config + deployment. That said, there will be a time when your SaaS costs more than managing infra.
- antod 3y ago
  crontab is just the schedule, there's a bunch of job handling missing. systemd timers get a bit closer and are worth learning for those that still touch actual Linux machines.
everforward 3y ago
> Periodic execution of a function is extremely basic and already exists in almost every programming environment. Periodic execution is surprisingly complicated. What happens when your node is down when it was supposed to execute and then comes back up? What happens around timezone changes and leap years? What does this thing do if a run takes so long that the previous run is still going? How do you get logs/notifications if it fails? "Best effort" periodic execution is super easy, where you try to make a thing happen every period but it's okay if it doesn't happen as long as it does happen at some point in the near future (i.e. periodic clean up jobs, or maybe batch emails). There's also a salient argument that you shouldn't run things inside the same process/VM if you can avoid it. It'd be a real shame if the production app went down because the cleanup cronjob went haywire and choked out the CPU.
- twic 3y ago
  I am a big fan of avoiding service proliferation and cloud complexity, and just making really good use of the facilities provided by your language and OS of choice. But for exactly the reasons you outline, scheduling jobs is one of the few things where i would be eager to use some reliable, observable, carefully-engineered service.
- klabb3 3y ago
  > What happens when your node is down when it was supposed to execute and then comes back up? If you want at-least-once scheduling you’d probably use a library that writes a last-completed time to your db. > What happens around timezone changes and leap years? For wall-time scheduling you’d need to handle it, yes. Why would a service be better than a library though? > What does this thing do if a run takes so long that the previous run is still going? I think we’d need either another db entry or simply prevent overlap directly in the library. > How do you get logs/notifications if it fails? Same way as other things in your app that fails I guess. Perhaps logging a critical event? If you’re talking about external health monitoring then yes, you’d need to have a separate service by definition, preferably with a completely different provider/host.
AgentME 3y ago
Scheduled actions on a scalable deployment aren't straight forward, unless you just want all your live instances to run the scheduled actions at the same time in parallel. You'd need the instances to coordinate with each other to make sure they aren't redundantly running the same actions, and you might need to scale the number of instances based on the scheduled actions. (Consider the case where the service is scaled down to 0 live instances, or where all the instances are busy with load and you want more instances to come up to deal with the scheduled actions.) In this case, you want cron handled by something aware of all your instances and can scale them (or you want it running outside of your scalable deployment and have it hit your load balancer which is aware of all your instances and can scale them, etc).

by klabb3 3y ago

notpachet 3y ago

I wonder why more people don't just set up a cheap VM and run this sort of thing via crontab.

Swizec 3y ago
Does crontab on a self-managed VM guarantee at least once delivery? For many, if not most, use-cases that guarantee is critical.
- notpachet 3y ago
  No. But I don't think there's any environment out there that is going to 100% guarantee that your cron succeeds. Even if it is reenqueued on failure, it could just keep hitting the same problem and crashlooping. You still need some kind of failure reporting, and a human to jump in to fix whatever went wrong.
fesc 3y ago
The same reason why most people stopped manually editing some random files via FTP to do deployments: to get a proper reproducible, automated and monitored production environment.
- codedokode 3y ago
  Cloud setups are not reproducible, because you cannot reproduce the same env locally or in the other cloud.
- notpachet 3y ago
  I think there's a threshold below which this is just unnecessary infrastructure overhead, and I'd posit that most cron use cases fall below this threshold. If yours is above it, well and good.
turtlebits 3y ago
Because managing servers adds a lot of tech overhead (OS and security updates/access control/disk/HA/monitoring) + provisioning/config + deployment. That said, there will be a time when your SaaS costs more than managing infra.
antod 3y ago
crontab is just the schedule, there's a bunch of job handling missing. systemd timers get a bit closer and are worth learning for those that still touch actual Linux machines.

everforward 3y ago

> Periodic execution of a function is extremely basic and already exists in almost every programming environment. Periodic execution is surprisingly complicated. What happens when your node is down when it was supposed to execute and then comes back up? What happens around timezone changes and leap years? What does this thing do if a run takes so long that the previous run is still going? How do you get logs/notifications if it fails? "Best effort" periodic execution is super easy, where you try to make a thing happen every period but it's okay if it doesn't happen as long as it does happen at some point in the near future (i.e. periodic clean up jobs, or maybe batch emails). There's also a salient argument that you shouldn't run things inside the same process/VM if you can avoid it. It'd be a real shame if the production app went down because the cleanup cronjob went haywire and choked out the CPU.

twic 3y ago
I am a big fan of avoiding service proliferation and cloud complexity, and just making really good use of the facilities provided by your language and OS of choice. But for exactly the reasons you outline, scheduling jobs is one of the few things where i would be eager to use some reliable, observable, carefully-engineered service.
klabb3 3y ago
> What happens when your node is down when it was supposed to execute and then comes back up? If you want at-least-once scheduling you’d probably use a library that writes a last-completed time to your db. > What happens around timezone changes and leap years? For wall-time scheduling you’d need to handle it, yes. Why would a service be better than a library though? > What does this thing do if a run takes so long that the previous run is still going? I think we’d need either another db entry or simply prevent overlap directly in the library. > How do you get logs/notifications if it fails? Same way as other things in your app that fails I guess. Perhaps logging a critical event? If you’re talking about external health monitoring then yes, you’d need to have a separate service by definition, preferably with a completely different provider/host.

AgentME 3y ago

Scheduled actions on a scalable deployment aren't straight forward, unless you just want all your live instances to run the scheduled actions at the same time in parallel. You'd need the instances to coordinate with each other to make sure they aren't redundantly running the same actions, and you might need to scale the number of instances based on the scheduled actions. (Consider the case where the service is scaled down to 0 live instances, or where all the instances are busy with load and you want more instances to come up to deal with the scheduled actions.) In this case, you want cron handled by something aware of all your instances and can scale them (or you want it running outside of your scalable deployment and have it hit your load balancer which is aware of all your instances and can scale them, etc).