As said people using multiple ec2s to process the data but the biggest reason is so I don't have to run it all in one go. Rabbit is used by a few of our service

As said people using multiple ec2s to process the data but the biggest reason is so I don't have to run it all in one go. Rabbit is used by a few of our services so is always up, means I can run for a few hours then come back to it another day. Also error handling, if one fails I just don't ack the message, after everything is processed I only have failed ones. Can slowly process the messages to see what is wrong