3 ms·

If there are any reasonably-priced ways of doing this especially with more PCIe lanes please let me know. In my experience it's often easier to just buy more mo

by kierank 10y ago

If there are any reasonably-priced ways of doing this especially with more PCIe lanes please let me know. In my experience it's often easier to just buy more motherboards and CPUs than invest in PCIe expansion.

anoother 10y ago
A used Tesla s1075 from eBay looks like a good bet. That's my plan, anyway. Would require some modding to use more up-to-date GPUs (there are some mounting studs on the chassis floor that need trimming) but the forced airflow means that passive cards like the original Xeon Phis are also a possibility.
angry_octet 10y ago
Depends on your bisection bandwidth requirement and software cost. If you want to do MPI it is going to be cheaper to buy some InfiniBand cards. But if you have a small cluster and tightly coupled code, or large I/O requirements, then a PCIe switch might be the go. These get used on some video on demand systems and signal processing architectures: https://www.microsemi.com/products/drivers-interfaces-and-pcie-switches/pcie-switches/pcie-fanout-switches/pm8536-pfx-96xg3 https://www.microsemi.com/products/drivers-interfaces-and-pc... The Dolphin systems are tuned better for computation: http://www.dolphinics.com/products/IXS600.html http://www.dolphinics.com/products/IXS600.html I expect small PCIe NVMe external storage systems to become quite common in the near future, because enterprise systems need multipath storage for reliability; 8GB/s FC is too slow for SSDs, let alone NVMe, same with SAS bus expanders. https://events.linuxfoundation.org/sites/events/files/slides/LinuxVault2015_KeithBusch_PCIeMPath.pdf https://events.linuxfoundation.org/sites/events/files/slides...
- kierank 10y ago
  The Dolphin systems seem more expensive than just buying 4U servers.
  - angry_octet 10y ago
    You only buy them when you can't fit your problem into one server (i.e. not enough memory or I/O) or you need redundancy (multipath I/O, synchronised memory). In those cases they can be a lot quicker/cheaper/more reliable than trying to solve it with more computers, protocols and ethernet.

by kierank 10y ago

anoother 10y ago

A used Tesla s1075 from eBay looks like a good bet. That's my plan, anyway. Would require some modding to use more up-to-date GPUs (there are some mounting studs on the chassis floor that need trimming) but the forced airflow means that passive cards like the original Xeon Phis are also a possibility.

angry_octet 10y ago

Depends on your bisection bandwidth requirement and software cost. If you want to do MPI it is going to be cheaper to buy some InfiniBand cards. But if you have a small cluster and tightly coupled code, or large I/O requirements, then a PCIe switch might be the go. These get used on some video on demand systems and signal processing architectures: https://www.microsemi.com/products/drivers-interfaces-and-pcie-switches/pcie-switches/pcie-fanout-switches/pm8536-pfx-96xg3 https://www.microsemi.com/products/drivers-interfaces-and-pc... The Dolphin systems are tuned better for computation: http://www.dolphinics.com/products/IXS600.html http://www.dolphinics.com/products/IXS600.html I expect small PCIe NVMe external storage systems to become quite common in the near future, because enterprise systems need multipath storage for reliability; 8GB/s FC is too slow for SSDs, let alone NVMe, same with SAS bus expanders. https://events.linuxfoundation.org/sites/events/files/slides/LinuxVault2015_KeithBusch_PCIeMPath.pdf https://events.linuxfoundation.org/sites/events/files/slides...

kierank 10y ago
The Dolphin systems seem more expensive than just buying 4U servers.
- angry_octet 10y ago
  You only buy them when you can't fit your problem into one server (i.e. not enough memory or I/O) or you need redundancy (multipath I/O, synchronised memory). In those cases they can be a lot quicker/cheaper/more reliable than trying to solve it with more computers, protocols and ethernet.