3 ms·
Really really bursty loads. You have a customer upload a data file and you have to process it. If you crud it, you have a worker chopping it apart and making sy
by Raidion 6y ago
Really really bursty loads. You have a customer upload a data file and you have to process it. If you crud it, you have a worker chopping it apart and making sync API calls. If something fails in the middle, it has to retry, but what happens if your container/database goes down when you're halfway through? Now you have to reprocess that file again, etc.
You move this to a queue, and have a worker chop that data file up into individual records, those records go onto a queue, and you can process them however you want, no worries about something crashing and not being able to be retried. If the database goes down, everything just pauses until it can go again. You can limit the queue throughput to whatever you want to avoid having to scale your API/Database.
Can you handle stuff via all CRUD sync APIs? Sure, just like you could handle running a restaurant where you have one person who takes the order and cooks it and delivers it to a table. However, it's more efficient to have a waiter (API) take requests and give them to a cook (queue based async worker) to handle stuff that's not as time sensitive. This saves you a lot of money in certain situations.