3 ms·
I'm missing the one thing that's actually interesting (read hard) How did they collect the generated files? Did they take any precautions in case a worker died
by MartinMond 17y ago
I'm missing the one thing that's actually interesting (read hard)
How did they collect the generated files?
Did they take any precautions in case a worker died while creating a pdf? (Does AMPQ have some sort of transactional semantics? If so, how would one apply them)
- almost 17y agoAmazons own SQS service has a simple solution to this problem. When you get an item from the queue it is temporarily hidden then once you've processed it successfully you explicitly delete the item from the queue. If the item is not deleted within the set timeout period then it goes back into the queue ready for another worker to have a go at. I assume RabbitMQ has something similar. Actually I'm not sure why they used RanbitMQ rather than SQS for this task. SQS seems to me to be a perfect fit and there's no extra effort involved to set it up.
- seancribbs 17y agoWe uploaded the generated files to S3 at the end of each individual job. We could have used AMQP's 'ack' for each message, but it turned out not to be necessary.