5 ms·

Will We Ever See InfiniBand in Desktop Computers?

25 points by kungfudoi 12y ago

sahyee 12y ago
Interesting. I think it's possible that Infiniband will one day work its way to desktop computers, but there are a lot of players in the game that have to agree to make it work and make it popular.
reportingsjr 12y ago
Why switch to infiniband? Ethernet can go just as fast (not as good of latency) and the cables are cheap and ubiquitous. The only thing stopping higher speeds is cost. I have been watching 10GbE PCI-E cards for the last year or two and they have dramatically dropped in price ($200 from newegg). You can get server motherboards with 2 integrated 10GbE ports for $500 as well. 10GbE does run over cat6 by the way.
- mattzito 12y ago
  I think for general-purpose workloads, it's not relevant. However, some of the IB-specific capabilities like RDMA are extremely interesting for clustering use cases. You can dramatically reduce the overhead of moving data around between nodes by bypassing the kernel entirely.
  - reportingsjr 12y ago
    I understand that sort of use case, but this article is asking about desktop computers. It almost seems like the author only sees it used in high performance situations so clearly it is better for all situations.
  - spitfire 12y ago
    thunderbolt and the predecessor firewire both have rdma.
sp332 12y ago
How would Infiniband be so much faster than Ethernet over the same cables? And would a 5x faster Inifiniband be less than 5x as expensive?
- zymhan 12y ago
  Going off of the [Wikipedia page](http://en.wikipedia.org/wiki/Infiniband http://en.wikipedia.org/wiki/Infiniband), I think it's due to the "switched fabric topology" of InfiniBand.
  - sp332 12y ago
    Ethernet is mostly switched now (when was the last time you used a hub?), and the topology doesn't change the line speed.
varelse 12y ago
I don't mean to be cynical, but other than for niche clusters in the home, wouldn't such technology be hobbled instantly by incoming bandwidth? Even in Germany, where the author lives, the top bandwidth appears to be about 50 Mb/s: http://www.pcmag.com/article2/0,2817,2457284,00.asp http://www.pcmag.com/article2/0,2817,2457284,00.asp Which although it is still much better than the states: http://ispspeedindex.netflix.com/results/usa/graph http://ispspeedindex.netflix.com/results/usa/graph They're both well below 1 Gb/s, no?
- zymhan 12y ago
  I don't think the use case envisioned here is for home computers with InfiniBand, it's for work stations in corporate environments. Transferring large data sets to local servers is quite common, and can take a (relatively) long time even over Gigabit Ethernet.
  - varelse 12y ago
    Corporations can afford to buy pricy network hardware for the people who need it (or they can skimp on engineering resources and congratulate themselves over and over again over how thrifty they're being). I read desktop to mostly mean consumer, perhaps erroneously?
- jedberg 12y ago
  It's mostly for moving things around your internal network. I would love to have 10Gb to my NAS, because then my backups would take seconds instead of minutes. (First world problems, I know).
- deleted 12y ago
  [deleted]
bhouston 12y ago
In visual effects, there is need to transfer around massive data sets. WE have found that 1 Gbps Ethernet just doens't cut it, but right now 10 Gbps Ethernet solution are just too expensive for now, or require specialized cables (fiber optics). http://en.wikipedia.org/wiki/10-gigabit_Ethernet http://en.wikipedia.org/wiki/10-gigabit_Ethernet I would really like those prices to come down and also to use just regular CAT cables. Right now it seems that the switch to faster than 1 Gbps networking is just too costly and not easily incremental (because of the switch to optical generally.) PS. There is 10GBASE-T but I think there is zero adoption or promotion of this substandard: http://en.wikipedia.org/wiki/10-gigabit_Ethernet#10GBASE-T http://en.wikipedia.org/wiki/10-gigabit_Ethernet#10GBASE-T
- caw 12y ago
  10GBASE-T isn't popular because your cable runs are cut short. Costs have come down a lot on fiber. A short range SFP is like $120 or so, and short range cables are like $3 per meter. What a lot of people do as a stopgap is binding ports together, using something like LACP. There's trade-offs of course. A 4 port 1000BASE-T card could be used to boost a few important servers.
  - josh-wrale 12y ago
    10GBase-T also entails more power consumption per port and significant latency over other 10GbE standards.
- skorgu 12y ago
  10GBASE-T is pretty well established for top-of-rack datacenter switches. Intel's X520 chipset is pretty common and any major switch vendor will happily sell you a 24x or 48x 10-gig switch (HP's JG336A for example). Twinax is arguably better if you're building a rack from scratch, they're lower power, but if you're upgrading something existing it's hard to justify rewiring everything. Also twinax cables are expensive and annoying to work with.
myrandomcomment 12y ago
10GBaseT is cheap. Look at Arista networks. This idea is just not well reasoned. Ethernet had all the "stuff" needed to make it work when you plug it in. IB requires you to do a lot of configuration on the IB switch side, etc.
walshemj 12y ago
Would you want every workstation to have direct access to the cluster - ok I get for loading data it might be useful but IN that case surely you would have special purpose nodes for that
skynetv2 12y ago
There is no practical or commercial motivation for IB on consumer devices. Sure one may make a case that a home server or a media center can use IB to stream photos and movies to other devices. At this point, we have to stop and think like an average consumer and not an IT professional. And an average consumer prefers no wires forget running copper or fiber. And what benefit would it offer over wireless even if they were to run the cable? None from the consumer perspective. You can stream HD over wireless without any problems. Sure, we all prefer wired over wireless when available. But not in home, and not by an average consumer. Intel will integrate IB with their future processors, but they will not be consumer based SKUs.
- Spooky23 12y ago
  There are niches -- some people (interested parties for sure) even advocate for fiber to the desktop as a cost savings: http://www.thefoa.org/tech/allfiber.htm http://www.thefoa.org/tech/allfiber.htm But for regular consumers, wireless is the driver. Power users will care about cat 6. The rest is just niche.
blutgens 12y ago
No.
Nexxxeh 12y ago
You can do InfiniBand on the desktop, but it necessarily doesn't give the performance you might be expecting. I think I read about this on HN originally... http://www.davidhunt.ie/infiniband-at-home-10gb-networking-on-the-cheap/ http://www.davidhunt.ie/infiniband-at-home-10gb-networking-o... Edit: From the edit on the follow-up post about getting it working on Ubuntu: >Note: iperf maxed out at 1.2 gbps, and on the current linux install, I couldnt get netperf client working at all. netserver would work, but only showed a throughput of 25mbps from a Win7 client. HOWEVER, when I set up the raid with 6 old 160G drives, the “hdparm -t /dev/md0p1″ showed 250MB/sec reads, and I got the same from the Win7 machine using samba across the infiniband fabric. This seems to indicate that iperf and netperf are completely unreliable for testing this type of connection. Bear in mind though that I did have netperf running on the previous ubuntu install, but that installation was so messy I don’t know what drivers and user-space software was running. I reckon it’s the kind of think that may be fixed in the stock Ubuntu install in the near future. For the moment, just go with real-world testing, i.e. copying large files from ramdisk to ramdisk, for example.

25 points by kungfudoi 12y ago

Interesting. I think it's possible that Infiniband will one day work its way to desktop computers, but there are a lot of players in the game that have to agree to make it work and make it popular.

reportingsjr 12y ago

Why switch to infiniband? Ethernet can go just as fast (not as good of latency) and the cables are cheap and ubiquitous. The only thing stopping higher speeds is cost. I have been watching 10GbE PCI-E cards for the last year or two and they have dramatically dropped in price ($200 from newegg). You can get server motherboards with 2 integrated 10GbE ports for $500 as well. 10GbE does run over cat6 by the way.

mattzito 12y ago
I think for general-purpose workloads, it's not relevant. However, some of the IB-specific capabilities like RDMA are extremely interesting for clustering use cases. You can dramatically reduce the overhead of moving data around between nodes by bypassing the kernel entirely.
- reportingsjr 12y ago
  I understand that sort of use case, but this article is asking about desktop computers. It almost seems like the author only sees it used in high performance situations so clearly it is better for all situations.
- spitfire 12y ago
  thunderbolt and the predecessor firewire both have rdma.

sp332 12y ago

How would Infiniband be so much faster than Ethernet over the same cables? And would a 5x faster Inifiniband be less than 5x as expensive?

zymhan 12y ago
Going off of the [Wikipedia page](http://en.wikipedia.org/wiki/Infiniband http://en.wikipedia.org/wiki/Infiniband), I think it's due to the "switched fabric topology" of InfiniBand.
- sp332 12y ago
  Ethernet is mostly switched now (when was the last time you used a hub?), and the topology doesn't change the line speed.

varelse 12y ago

I don't mean to be cynical, but other than for niche clusters in the home, wouldn't such technology be hobbled instantly by incoming bandwidth? Even in Germany, where the author lives, the top bandwidth appears to be about 50 Mb/s: http://www.pcmag.com/article2/0,2817,2457284,00.asp http://www.pcmag.com/article2/0,2817,2457284,00.asp Which although it is still much better than the states: http://ispspeedindex.netflix.com/results/usa/graph http://ispspeedindex.netflix.com/results/usa/graph They're both well below 1 Gb/s, no?

zymhan 12y ago
I don't think the use case envisioned here is for home computers with InfiniBand, it's for work stations in corporate environments. Transferring large data sets to local servers is quite common, and can take a (relatively) long time even over Gigabit Ethernet.
- varelse 12y ago
  Corporations can afford to buy pricy network hardware for the people who need it (or they can skimp on engineering resources and congratulate themselves over and over again over how thrifty they're being). I read desktop to mostly mean consumer, perhaps erroneously?
jedberg 12y ago
It's mostly for moving things around your internal network. I would love to have 10Gb to my NAS, because then my backups would take seconds instead of minutes. (First world problems, I know).
deleted 12y ago
[deleted]

bhouston 12y ago

In visual effects, there is need to transfer around massive data sets. WE have found that 1 Gbps Ethernet just doens't cut it, but right now 10 Gbps Ethernet solution are just too expensive for now, or require specialized cables (fiber optics). http://en.wikipedia.org/wiki/10-gigabit_Ethernet http://en.wikipedia.org/wiki/10-gigabit_Ethernet I would really like those prices to come down and also to use just regular CAT cables. Right now it seems that the switch to faster than 1 Gbps networking is just too costly and not easily incremental (because of the switch to optical generally.) PS. There is 10GBASE-T but I think there is zero adoption or promotion of this substandard: http://en.wikipedia.org/wiki/10-gigabit_Ethernet#10GBASE-T http://en.wikipedia.org/wiki/10-gigabit_Ethernet#10GBASE-T

caw 12y ago
10GBASE-T isn't popular because your cable runs are cut short. Costs have come down a lot on fiber. A short range SFP is like $120 or so, and short range cables are like $3 per meter. What a lot of people do as a stopgap is binding ports together, using something like LACP. There's trade-offs of course. A 4 port 1000BASE-T card could be used to boost a few important servers.
- josh-wrale 12y ago
  10GBase-T also entails more power consumption per port and significant latency over other 10GbE standards.
skorgu 12y ago
10GBASE-T is pretty well established for top-of-rack datacenter switches. Intel's X520 chipset is pretty common and any major switch vendor will happily sell you a 24x or 48x 10-gig switch (HP's JG336A for example). Twinax is arguably better if you're building a rack from scratch, they're lower power, but if you're upgrading something existing it's hard to justify rewiring everything. Also twinax cables are expensive and annoying to work with.

myrandomcomment 12y ago

10GBaseT is cheap. Look at Arista networks. This idea is just not well reasoned. Ethernet had all the "stuff" needed to make it work when you plug it in. IB requires you to do a lot of configuration on the IB switch side, etc.

walshemj 12y ago

Would you want every workstation to have direct access to the cluster - ok I get for loading data it might be useful but IN that case surely you would have special purpose nodes for that

skynetv2 12y ago

There is no practical or commercial motivation for IB on consumer devices. Sure one may make a case that a home server or a media center can use IB to stream photos and movies to other devices. At this point, we have to stop and think like an average consumer and not an IT professional. And an average consumer prefers no wires forget running copper or fiber. And what benefit would it offer over wireless even if they were to run the cable? None from the consumer perspective. You can stream HD over wireless without any problems. Sure, we all prefer wired over wireless when available. But not in home, and not by an average consumer. Intel will integrate IB with their future processors, but they will not be consumer based SKUs.

Spooky23 12y ago
There are niches -- some people (interested parties for sure) even advocate for fiber to the desktop as a cost savings: http://www.thefoa.org/tech/allfiber.htm http://www.thefoa.org/tech/allfiber.htm But for regular consumers, wireless is the driver. Power users will care about cat 6. The rest is just niche.

blutgens 12y ago

No.

Nexxxeh 12y ago

You can do InfiniBand on the desktop, but it necessarily doesn't give the performance you might be expecting. I think I read about this on HN originally... http://www.davidhunt.ie/infiniband-at-home-10gb-networking-on-the-cheap/ http://www.davidhunt.ie/infiniband-at-home-10gb-networking-o... Edit: From the edit on the follow-up post about getting it working on Ubuntu: >Note: iperf maxed out at 1.2 gbps, and on the current linux install, I couldnt get netperf client working at all. netserver would work, but only showed a throughput of 25mbps from a Win7 client. HOWEVER, when I set up the raid with 6 old 160G drives, the “hdparm -t /dev/md0p1″ showed 250MB/sec reads, and I got the same from the Win7 machine using samba across the infiniband fabric. This seems to indicate that iperf and netperf are completely unreliable for testing this type of connection. Bear in mind though that I did have netperf running on the previous ubuntu install, but that installation was so messy I don’t know what drivers and user-space software was running. I reckon it’s the kind of think that may be fixed in the stock Ubuntu install in the near future. For the moment, just go with real-world testing, i.e. copying large files from ramdisk to ramdisk, for example.