3 ms·
A bit unfair, esp. when Microsoft budgeted $2000+ on SSDs alone. I think the paper has a good point: scale-up should be definitely considered. I see systems th
by dragontamer 14y ago
A bit unfair, esp. when Microsoft budgeted $2000+ on SSDs alone.
I think the paper has a good point: scale-up should be definitely considered. I see systems that support 1.5 TB of RAM now... do you really expect your dataset to grow beyond that?
And as long as the dataset fits within 1.5 TB of RAM, scale up will be more scalable than scale out.
- marshray 14y agoYeah, ignore my numbers and I agree that scale-up should be considered and it's worthwhile to read a paper where somebody reality-checks the conventional wisdom. > I see systems that support 1.5 TB of RAM now... do you really expect your dataset to grow beyond that? According to Figure 1, about 20% of their jobs are already beyond that. If you spend your entire hardware budget on a single server and all your analytic processing jobs end up being coded with the (gasp) single threaded assumption, what are you going to do next? Most of us have been there before because it's where you end up by accident ("Do you think if we max out the RAM on the database server it will return to acceptable performance?") This is why people love parallel map-reduce based systems.
- dragontamer 14y agoCertainly, I don't mean to contradict the conventional wisdom :-p. Scaling out with ~$1500 2U servers makes sense a lot of the times. I'm more concerned about another problem. There are people out there who think that they can scale out with Hadoop cluster built on top of Atoms (or more recently... ARMs). And while yes... there are some applications where that is a decent scaling strategy, I doubt it works in most cases. But yeah, I know that wasn't your argument. I'm just musing a bit on random stuff.
- marshray 14y agoI admit to being ARM/Atom M/R curious. Sometimes I wonder about Hadoop for a few seconds until I remember it's heavily JVM-based. Poking around on the web you find folks who sound serious about porting it to Android.
- dragontamer 14y agoCurious is good. Things definitely move quickly in the tech industry, and I'm willing to bet that within 5 years things can change. And I'd support building up the skills / infrastructure needed to hedge against the uprising of ARM. But at the moment, the evidence is solidly in favor of the typical 2U Xeon. (by nature of ARM servers barely exist right now. Maybe next year ARM will have something ready) Atom looks mostly dead however, with Intel beginning to advertize 16W Xeons. Can two 8W Centrino Atoms compete against a 16W Xeon? I doubt it. Intel only has the resources to choose one "hero" chipset in server-space, and it looks like it will be the Xeon again.
- marshray 14y agoSomeday somebody is going to realize they are in possession of a trash barge full of 5 million discarded smartphones and break into the top supercomputer list with them.
- seunosewa 14y agoThat idea is a non-starter. Consider the cost of transportation, the cost of connecting all those ridiculously slow CPUs and incompatible CPUs together, powering them, programming for them. Building a supercomputer using standard x86_64 CPUs will probably be cheaper.
- marshray 14y agoHere's an example: http://www.androidauthority.com/samsung-galaxy-s3-sales-20-million-112790/ http://www.androidauthority.com/samsung-galaxy-s3-sales-20-m... Within a few years there will be 20 million Samsung Galaxy S III's with dead batteries and/or cracked screens. That's a dual-core 1 GHz ARM with 1 GB of RAM, 8 GB of SSD, and a hardware GPU. Right now, there are professional 1st-world developers developing apps for all those "incompatible CPUs". So it's possible and even cost effective under the right circumstances. Yet someone is currently being paid to haul away this stuff as electronic recycling.
- sliverstorm 14y agoIf 20% of your jobs are beyond 1.5TB of RAM, ideally you want a heterogeneous environment. Some scale-up machines , and some scale-out machines. Mix and match as appropriate, and don't give in to the siren song of scale-out where it isn't appropriate.