4 ms·
Crontab for scheduling + BASH (or NodeJS/Python/Lua) scripts for map-reduce or whatever kind of computation you want to do + NFS v4.1 (something like AWS EFS)
by truth_seeker 5y ago
Crontab for scheduling
+ BASH (or NodeJS/Python/Lua) scripts for map-reduce or whatever kind of computation you want to do
+ NFS v4.1 (something like AWS EFS) will do the job.
You get immense flexibility, however the team of developer who is maintaining this must be more skilful.
- klysm 5y agoI hate the failure semantics of cron. Unreasonably hard to get right or to not run duplicate jobs. I’ve replaced it with systemd timers and never looked back.
- NavinF 5y agoNFS is such a pain in the ass for this use case. If the client crashes, everything’s corrupted. If the server crashes, everything’s corrupted. You can’t even atomically write 4k with O_APPEND. Why anyone would choose to build distributed systems your way is beyond me