6 ms·

> a single “big memory” (192 GB) server we are using has the performance capability of approximately 14 standard (12 GB) servers. That's 168 GB RAM total, with

> a single “big memory” (192 GB) server we are using has the performance capability of approximately 14 standard (12 GB) servers. That's 168 GB RAM total, within a single upgrade unit of their 192 GB single server, suggesting the problem is dominated by RAM. If I had $6638 + 2640 = $9278 to spend on computing hardware from NewEgg, how about: 1 at $1000 of HP ProLiant DL360e Gen8 Rack Server System Intel Xeon E5-2403 1.8GHz 4C/4T 4GB http://www.newegg.com/Product/Product.aspx?Item=N82E16859107943 http://h10010.www1.hp.com/wwpc/us/en/sm/WF06a/15351-15351-3328412-241644-241475-5249570.html?dnr=1 (12 DIMM slots) 4 at $70 of Kingston 8GB 240-Pin DDR3 SDRAM ECC Registered DDR3 1333 Server Memory Model KVR13LR9S4/8 http://www.newegg.com/Product/Product.aspx?Item=N82E16820239540 1 at $54 of Seagate Barracuda ST250DM000 250GB 7200 RPM 16MB Cache SATA 6.0Gb/s 3.5" Internal Hard Drive http://www.newegg.com/Product/Product.aspx?Item=N82E16822148765 $1334 ea server, 7 servers = $9338 So we could get 7 of these low-end name-brand 16 GB servers for the same money to give us 224 GB RAM. > MR++ runs on 27 servers whereas the standalone conﬁgurations are a single server running a single-threaded implementation Sure, nothing will beat a single system at message-passing algorithms when the entire graph fits in main memory. But when the dataset outgrows that (and it will), we can triple the RAM in the empty slots, or add more servers in units of $1334 instead of having to rewrite your whole analysis.

wmf 14y ago
You can't compare old pricing against new. Now you can get 384 GB for under $9K and 768 GB would be maybe $15K.
- marshray 14y ago
  Oh, the paper is dated January 2013 but I see now something about "prices converted to US$ as of 8 August 2012". I'm confused. I think the link changed to a different version of the paper. The numbers I went by aren't even in there any more!
  - wmf 14y ago
    Yeah, I convinced the moderators to update the link to a newer version of the paper. But keep in mind that a paper published in 2013 probably describes research performed in 2012 on hardware purchased in 2011 (if they're lucky). Getting back to hardware, the premium for 4-socket servers appears to be much smaller than it used to be, so now you can get a ton of DIMM slots for a reasonable price.
dragontamer 14y ago
A bit unfair, esp. when Microsoft budgeted $2000+ on SSDs alone. I think the paper has a good point: scale-up should be definitely considered. I see systems that support 1.5 TB of RAM now... do you really expect your dataset to grow beyond that? And as long as the dataset fits within 1.5 TB of RAM, scale up will be more scalable than scale out.
- marshray 14y ago
  Yeah, ignore my numbers and I agree that scale-up should be considered and it's worthwhile to read a paper where somebody reality-checks the conventional wisdom. > I see systems that support 1.5 TB of RAM now... do you really expect your dataset to grow beyond that? According to Figure 1, about 20% of their jobs are already beyond that. If you spend your entire hardware budget on a single server and all your analytic processing jobs end up being coded with the (gasp) single threaded assumption, what are you going to do next? Most of us have been there before because it's where you end up by accident ("Do you think if we max out the RAM on the database server it will return to acceptable performance?") This is why people love parallel map-reduce based systems.
  - dragontamer 14y ago
    Certainly, I don't mean to contradict the conventional wisdom :-p. Scaling out with ~$1500 2U servers makes sense a lot of the times. I'm more concerned about another problem. There are people out there who think that they can scale out with Hadoop cluster built on top of Atoms (or more recently... ARMs). And while yes... there are some applications where that is a decent scaling strategy, I doubt it works in most cases. But yeah, I know that wasn't your argument. I'm just musing a bit on random stuff.
    - marshray 14y ago
      I admit to being ARM/Atom M/R curious. Sometimes I wonder about Hadoop for a few seconds until I remember it's heavily JVM-based. Poking around on the web you find folks who sound serious about porting it to Android.

wmf 14y ago

You can't compare old pricing against new. Now you can get 384 GB for under $9K and 768 GB would be maybe $15K.

marshray 14y ago
Oh, the paper is dated January 2013 but I see now something about "prices converted to US$ as of 8 August 2012". I'm confused. I think the link changed to a different version of the paper. The numbers I went by aren't even in there any more!
- wmf 14y ago
  Yeah, I convinced the moderators to update the link to a newer version of the paper. But keep in mind that a paper published in 2013 probably describes research performed in 2012 on hardware purchased in 2011 (if they're lucky). Getting back to hardware, the premium for 4-socket servers appears to be much smaller than it used to be, so now you can get a ton of DIMM slots for a reasonable price.

dragontamer 14y ago

A bit unfair, esp. when Microsoft budgeted $2000+ on SSDs alone. I think the paper has a good point: scale-up should be definitely considered. I see systems that support 1.5 TB of RAM now... do you really expect your dataset to grow beyond that? And as long as the dataset fits within 1.5 TB of RAM, scale up will be more scalable than scale out.

marshray 14y ago
Yeah, ignore my numbers and I agree that scale-up should be considered and it's worthwhile to read a paper where somebody reality-checks the conventional wisdom. > I see systems that support 1.5 TB of RAM now... do you really expect your dataset to grow beyond that? According to Figure 1, about 20% of their jobs are already beyond that. If you spend your entire hardware budget on a single server and all your analytic processing jobs end up being coded with the (gasp) single threaded assumption, what are you going to do next? Most of us have been there before because it's where you end up by accident ("Do you think if we max out the RAM on the database server it will return to acceptable performance?") This is why people love parallel map-reduce based systems.
- dragontamer 14y ago
  Certainly, I don't mean to contradict the conventional wisdom :-p. Scaling out with ~$1500 2U servers makes sense a lot of the times. I'm more concerned about another problem. There are people out there who think that they can scale out with Hadoop cluster built on top of Atoms (or more recently... ARMs). And while yes... there are some applications where that is a decent scaling strategy, I doubt it works in most cases. But yeah, I know that wasn't your argument. I'm just musing a bit on random stuff.
  - marshray 14y ago
    I admit to being ARM/Atom M/R curious. Sometimes I wonder about Hadoop for a few seconds until I remember it's heavily JVM-based. Poking around on the web you find folks who sound serious about porting it to Android.