4 ms·
Pros and Cons of Redis-Resque and SQS
- PaulHoule 14y agoI looked pretty hard at systems to do parallel batch jobs. One thing I looked at was to store intermediate results in Amazon S3 and control the execution of job steps with SQS. Although the system couldn't do everything Hadoop does, it worked for a wide range of problems and had the desirable quality that it could transparently parallelize multiple processing phases when it was possible. I went with Hadoop, however, because I can run a Hadoop cluster in the office or in the cloud, and that's a big advantage.