3 ms·

Really the network is the least expensive part. To get IO that can saturate a 10G costs around 20k, while the infiniband card sets you back less than 4k. Now yo

Really the network is the least expensive part. To get IO that can saturate a 10G costs around 20k, while the infiniband card sets you back less than 4k. Now you're talking about 100G, which will go faster, you could easily be looking at a 500k box of ssds.

davrosthedalek 8y ago
Wait, what? 10G is 1.2 GByte/s, you can get that from a single SSD easily. 100G is 12 GByte/s, so 5 consumer-level ssds. Squeezing 12 GByte/s over the buses twice might be tricky, but certainly not a 500k problem.
- dragontamer 8y ago
  >To get IO that can saturate a 10G costs around 20k Assuming 10G Ethernet is 8b/10b like a lot of other protocols, that's 1GB/s over 10G Ethernet. Here's a $130 SSD with 500GB of storage: https://www.amazon.com/Mushkin-PILOT-500GB-Internal-MKNSSDPL500GB-D8/dp/B07CYJ4GS3 https://www.amazon.com/Mushkin-PILOT-500GB-Internal-MKNSSDPL... That's 2600 MB/s read speeds. Or more than double your 10G Ethernet. ------------ RAID0 8 of them together with ASRock's M2. Quad Ultra, and you've got $1040 of SSDs + $200 for the 2x Quad Ultra cards, or just $1200 for 20GB/s read/write speeds. More than enough to saturate any network I'm aware of. In fact, someone has already done this: http://www.guru3d.com/news-story/eight-nvme-m2-ssds-in-raid-on-x399-threadripper-reach-28-gbs.html http://www.guru3d.com/news-story/eight-nvme-m2-ssds-in-raid-... They used a higher-end NVM.e SSD and measured 28GB/s (that's capital B, gigaBYTES) on the Threadripper + x399 motherboard.
  - greglindahl 8y ago
    10 gigabit Ethernet is 10 gigabits of data. It's mainly Infinband that used a horrible marketing tactic of saying the signal rate instead of the data rate.
    - dragontamer 8y ago
      PCIe 2.0, USB 2.0, SATA / AHCI, and more protocols are 8b/10b. So all of these protocols were 10-bits per byte. Modern protocols tend to be 64b/66b or better. So that's why I listed "assuming 8b/10b", its hard to memorize which protocols are which. Apparently I'm wrong. 10G Ethernet seems to be a more modern 64b/66b in any case.
      - greglindahl 8y ago
        10 gigabit ethernet is not 1 GB/s. It's 10 gigabits of data per second, or 1.25 gigabytes per second. The encoding is not an issue with these data rate numbers because Ethernet quotes their data rate as a data rate.
  - davrosthedalek 8y ago
    10G is 64b/66b, so somewhat more efficient -> 1.2 GB/s Too fast for single SATA, but any nvme/pcie ssd should give that sustained.

davrosthedalek 8y ago

Wait, what? 10G is 1.2 GByte/s, you can get that from a single SSD easily. 100G is 12 GByte/s, so 5 consumer-level ssds. Squeezing 12 GByte/s over the buses twice might be tricky, but certainly not a 500k problem.

dragontamer 8y ago
>To get IO that can saturate a 10G costs around 20k Assuming 10G Ethernet is 8b/10b like a lot of other protocols, that's 1GB/s over 10G Ethernet. Here's a $130 SSD with 500GB of storage: https://www.amazon.com/Mushkin-PILOT-500GB-Internal-MKNSSDPL500GB-D8/dp/B07CYJ4GS3 https://www.amazon.com/Mushkin-PILOT-500GB-Internal-MKNSSDPL... That's 2600 MB/s read speeds. Or more than double your 10G Ethernet. ------------ RAID0 8 of them together with ASRock's M2. Quad Ultra, and you've got $1040 of SSDs + $200 for the 2x Quad Ultra cards, or just $1200 for 20GB/s read/write speeds. More than enough to saturate any network I'm aware of. In fact, someone has already done this: http://www.guru3d.com/news-story/eight-nvme-m2-ssds-in-raid-on-x399-threadripper-reach-28-gbs.html http://www.guru3d.com/news-story/eight-nvme-m2-ssds-in-raid-... They used a higher-end NVM.e SSD and measured 28GB/s (that's capital B, gigaBYTES) on the Threadripper + x399 motherboard.
- greglindahl 8y ago
  10 gigabit Ethernet is 10 gigabits of data. It's mainly Infinband that used a horrible marketing tactic of saying the signal rate instead of the data rate.
  - dragontamer 8y ago
    PCIe 2.0, USB 2.0, SATA / AHCI, and more protocols are 8b/10b. So all of these protocols were 10-bits per byte. Modern protocols tend to be 64b/66b or better. So that's why I listed "assuming 8b/10b", its hard to memorize which protocols are which. Apparently I'm wrong. 10G Ethernet seems to be a more modern 64b/66b in any case.
    - greglindahl 8y ago
      10 gigabit ethernet is not 1 GB/s. It's 10 gigabits of data per second, or 1.25 gigabytes per second. The encoding is not an issue with these data rate numbers because Ethernet quotes their data rate as a data rate.
- davrosthedalek 8y ago
  10G is 64b/66b, so somewhat more efficient -> 1.2 GB/s Too fast for single SATA, but any nvme/pcie ssd should give that sustained.