2 ms·

If you need 5 machines to get the same performance as Sector/Sphere on 1 machine, then that is a problem. The question is, at which point does Hadoop start to o

by jules 17y ago

If you need 5 machines to get the same performance as Sector/Sphere on 1 machine, then that is a problem. The question is, at which point does Hadoop start to outperform Sector/Sphere (assuming that Hadoop scales better). If the answer is if you have 10000 machines then Sector/Sphere is better for almost everyone. This is the same problem with Haskell/Parallelism with functional data structures. Maybe you can write a linearly scaling parallel quicksort on linked lists Haskell, but I can write an array based C sort that's 1000x faster. You could say that that's unfair and use arrays in Haskell, but now parallelism isn't so easy anymore.

jacquesm 17y ago
Yes, that's exactly the question. But knowing that there might be a cross over point is not the same as knowing there is none (which is very well possible). I'm wondering whether there even is a scenario under which Hadoop would outperform Sector/Sphere. The only time that I tried Hadoop I was so underwhelmed with the performance that I immediately decided that it wasn't for us. For a while glusterfs looked like it might be a winner but I'm getting more and more skeptical over time. Pohmelfs (in the linux kernel) is an absolute pain to get working at all. So anything that will give us that cluster file system with good failover will get a look, if performance is acceptable compared to what we do today. (custom built CDN).