I have experienced this dilemma in a different domain: machine learning. You can either go the Hadoop way and commit to simpler algorithms that can be run in di

I have experienced this dilemma in a different domain: machine learning. You can either go the Hadoop way and commit to simpler algorithms that can be run in distributed manner or you can keep pushing the single machine more and more by use of ever more clever algorithms. Unfortunately, it is hardly ever possible to follow the route of push-single-machine-to-max -> thorw-more-machines-in. The distributed way and shared memory way of doing things often differ in fundamental ways. Going with big data tool chains from the start is often a overkill for small experiments. But once you outgrow one machine, the pain of undoing all the nice (algorithmic) tricks is also quite severe. Perhaps it is time to accept that we now produce data at a rate that distributed is going to be the way to process it. But this also means that some of the techniques available for scaling to larger data sets may need to be given up.