Recently a big engineering firm (let's not harm them by revealing their name :) ), called on me. A query on our system takes 7 days they said, when it succeeds.

Recently a big engineering firm (let's not harm them by revealing their name :) ), called on me. A query on our system takes 7 days they said, when it succeeds... These guys deal with up to 4Tb a day. With proper multi-core map-reduce and open source search, I brought this down to 3 sec on a single 48 core machine. They used this without revealing it to their (big) customers (who pay a lot for the crappy system). Then they've got a new version of their original system, relying on hadoop. Still, it was slower than the single machine design. Sometimes the bottleneck isn't in the hardware resources... Distributed computing has many advantages, but I personally use it as a last resort (or I prepare the architecture for scalability, trying to never hit the two machines and beyond case!).