4 ms·

What do you mean by "clusters"? High-end, low-latency compute clusters (using e.g., Myrinet) or just a rack of machines? Are you doing lots of calculation or l

by johnm 20y ago

What do you mean by "clusters"? High-end, low-latency compute clusters (using e.g., Myrinet) or just a rack of machines? Are you doing lots of calculation or lots of data or both? If you have lots of data, what's your storage model? If you're going to have lots of machines, what's your needs for connectivity (NICs, intra-cluster, 'net drops, etc.)?

gyro_robo 20y ago
Isn't a Myrinet cluster just a rack of machines with Myrinet cards? I am under the impression that 10gigE with TCP/UDP offload is pretty similar to 10gb Myrinet. Lots of calculation and data, hence the request for lots of CPU and RAM. Storage is RAM binary-dumped to flat files (snapshots), so standard hard drives are fine. Intra-cluster: preferably 10gigE but multiple 1gigE might work. Net connectivity would probably be okay with 1gigE (total). I can probably do some lower-end testing on Sun's compute grid, but real-world testing needs real-world hardware. It might seem like overkill at first, but every successful service seems to hit scaling issues sooner than they think!
- johnm 20y ago
  If you need Myrinet for your clusters then you already know why. :-? The main reason for Myrinet, IMHO, is the low latency -- otherwise, it's not worth the money. 8GB RAM seems to be the sweet spot at the moment, price-wise. Similarly, the 500GB SATA drives are a much better deal than the 750+GB drives. Be careful when you spec. the boxes because if you don't specify that you're going to e.g., load them up with additional drives later, the builders will spec them with the smaller/cheapest controller they can. If all of the data can be handled locally then you definitely don't have a lot of data. :-). To calibrate, we're (krugle.com) pushing around terabytes of data. If you're pushing a lot of data between nodes, don't underestimate the importance of your network infrastructure and architecture. We're using multiple 1GigE NICs per node into Foundry SuperX's (IIRC, it has a 36Gbps fabric) and 10G crosses. We've got multiple 1Gbps backbone drops into our load balancers / firewalls. If you're going relatively mainstream on the CPU side, the dual-core Intel Xeons are definitely the choice at the moment. Watch out for the different FSB speeds. Re: Sun's Grid. They were very aggressively trying to get our business but they aren't really geared for big data and definitely not for big, relatively non-transient data. Hope this helps, John
  - gyro_robo 20y ago
    I've read about Myrinet but never used it; I wasn't sure about the distinction you were making about a compute cluster vs. a rack. As for latency, isn't that only on an empty connection? E.g., if you're sending half the max per-second traffic down the pipe, isn't any new message going to take half a second to arrive? (In which case 5 usec vs. 15 usec on an empty pipe is lost in the noise.) Sun's grid would just be for automated testing, as a step up from uniprocessor EC2 nodes. I agree on the network infrastructure using multiple 1gig cards or a 10gig. I'm not sure what you mean by "handled locally" -- each node handles part of the data, so collectively it's larger than what any single node can manage.
    - johnm 20y ago
      That's why these kinds of discussions are difficult. Not just the usual "it depends" but multiple levels of "it depends". As you say, if you're job mix is keeping a pipe saturated then raw latency is probably not your critical problem. Though, of course, if, for example, you have one pipe for bulk transfer and one for control/meta-data.... :-) The comment about handling data locally was trying to get to the issue of how much data needs to be going back and for across the cluster rather than just living on the individual nodes. For example, we crawl millions of sites so there's an initial lump of intake data to each of the crawlers. On each crawler, that lump of data will get processed, redacted, expanded, and what not and only the resulting indexes and segments are ever transfered off the crawlers.
      - gyro_robo 19y ago
        > Though, of course, if, for example, you have one pipe for bulk transfer and one for control/meta-data.... :-) I do, but 15 usec is fast enough for that. :) I generally don't like paying a premium for slightly better performance. Even these rackmount systems are very pricey compared to putting together your own vanilla boxes. Data bounces from node to node; the main reason to go 10gigE instead of 1gigE is actually for lower latency from the extra room. 100 Mbit of traffic on fast ethernet = 1 sec latency vs 0.1 sec on 1gigE vs. 0.01 sec on 10gigE (not counting overhead of course).

by johnm 20y ago

gyro_robo 20y ago

Isn't a Myrinet cluster just a rack of machines with Myrinet cards? I am under the impression that 10gigE with TCP/UDP offload is pretty similar to 10gb Myrinet. Lots of calculation and data, hence the request for lots of CPU and RAM. Storage is RAM binary-dumped to flat files (snapshots), so standard hard drives are fine. Intra-cluster: preferably 10gigE but multiple 1gigE might work. Net connectivity would probably be okay with 1gigE (total). I can probably do some lower-end testing on Sun's compute grid, but real-world testing needs real-world hardware. It might seem like overkill at first, but every successful service seems to hit scaling issues sooner than they think!

johnm 20y ago
If you need Myrinet for your clusters then you already know why. :-? The main reason for Myrinet, IMHO, is the low latency -- otherwise, it's not worth the money. 8GB RAM seems to be the sweet spot at the moment, price-wise. Similarly, the 500GB SATA drives are a much better deal than the 750+GB drives. Be careful when you spec. the boxes because if you don't specify that you're going to e.g., load them up with additional drives later, the builders will spec them with the smaller/cheapest controller they can. If all of the data can be handled locally then you definitely don't have a lot of data. :-). To calibrate, we're (krugle.com) pushing around terabytes of data. If you're pushing a lot of data between nodes, don't underestimate the importance of your network infrastructure and architecture. We're using multiple 1GigE NICs per node into Foundry SuperX's (IIRC, it has a 36Gbps fabric) and 10G crosses. We've got multiple 1Gbps backbone drops into our load balancers / firewalls. If you're going relatively mainstream on the CPU side, the dual-core Intel Xeons are definitely the choice at the moment. Watch out for the different FSB speeds. Re: Sun's Grid. They were very aggressively trying to get our business but they aren't really geared for big data and definitely not for big, relatively non-transient data. Hope this helps, John
- gyro_robo 20y ago
  I've read about Myrinet but never used it; I wasn't sure about the distinction you were making about a compute cluster vs. a rack. As for latency, isn't that only on an empty connection? E.g., if you're sending half the max per-second traffic down the pipe, isn't any new message going to take half a second to arrive? (In which case 5 usec vs. 15 usec on an empty pipe is lost in the noise.) Sun's grid would just be for automated testing, as a step up from uniprocessor EC2 nodes. I agree on the network infrastructure using multiple 1gig cards or a 10gig. I'm not sure what you mean by "handled locally" -- each node handles part of the data, so collectively it's larger than what any single node can manage.
  - johnm 20y ago
    That's why these kinds of discussions are difficult. Not just the usual "it depends" but multiple levels of "it depends". As you say, if you're job mix is keeping a pipe saturated then raw latency is probably not your critical problem. Though, of course, if, for example, you have one pipe for bulk transfer and one for control/meta-data.... :-) The comment about handling data locally was trying to get to the issue of how much data needs to be going back and for across the cluster rather than just living on the individual nodes. For example, we crawl millions of sites so there's an initial lump of intake data to each of the crawlers. On each crawler, that lump of data will get processed, redacted, expanded, and what not and only the resulting indexes and segments are ever transfered off the crawlers.
    - gyro_robo 19y ago
      > Though, of course, if, for example, you have one pipe for bulk transfer and one for control/meta-data.... :-) I do, but 15 usec is fast enough for that. :) I generally don't like paying a premium for slightly better performance. Even these rackmount systems are very pricey compared to putting together your own vanilla boxes. Data bounces from node to node; the main reason to go 10gigE instead of 1gigE is actually for lower latency from the extra room. 100 Mbit of traffic on fast ethernet = 1 sec latency vs 0.1 sec on 1gigE vs. 0.01 sec on 10gigE (not counting overhead of course).