4 ms·
The system I’m currently on is currently moving a lot of work into queues. Some operations, like “change the criteria of this rank” could be anywhere between 5
by charrondev 4y ago
The system I’m currently on is currently moving a lot of work into queues. Some operations, like “change the criteria of this rank” could be anywhere between 5 seconds (if the number of users of the criteria to evaluate are small) or 10+ hours if we need re-evaluate the rules against 10m+ users.
In this case we write our jobs as generators that can be paused, serialized and picked up again later. We give the job 5 seconds synchronously, then if it passes that time, queue and job and let the client know a job has been registered.
The users account holds the IDs of the jobs as well as some basic information about the the tasks they have queued. There is a rest endpoint to return the current status of the jobs and information about them (what are they doing, what’s their progress, how much work remains).
The client will negotiate a web socket connection with a different service to be notified whenever progress is made on the job and the client can then check the endpoint for the latest status.
- latchkey 4y agoThat 5 seconds is going to bite you. There is going to be some sort of stall in the future that causes all of your jobs to hit that 5 seconds and everything is going to start to back up and cause other problems up the line that are really hard to test for in advance. You're better off designing a system that doesn't rely on some arbitrary number of seconds (why not 4 or 6 seconds?) to begin with.
- naasking 4y agoYes, non-determinism is the bane of distributed systems. It should be minimized whenever possible.
- convolvatron 4y agoin general I've been a lot more successful making fewer assumptions than trying to enforce them
- charrondev 4y agoThe system doesn’t rely on the 5s FWIW worth and is capable of running entirely through the distributed queue. The 5s check is just there to improve UX on small user actions.
- latchkey 4y agoLet's say that ALL of the jobs suddenly take up 6s. They ALL get queued up. Now you have a queue problem and you have more to debug. You have to figure out why the queue is filling up and why things are taking longer than 5s. You also have an issue where the execution might die (machine crashes) before the job makes it into a queue. You're better off just going straight to queue with the jobs and remove the added complexity of a 5s rule.