3 ms·
Found these for CEPH that indicates there's fast deployments for Infiniband. Don't have more data as I've been out of HPC a while. https://www.mellanox.com/rel
by nickpsecurity 10y ago
Found these for CEPH that indicates there's fast deployments for Infiniband. Don't have more data as I've been out of HPC a while.
https://www.mellanox.com/related-docs/solutions/ppt_ceph_mellanox_ceph_day.pdf https://www.mellanox.com/related-docs/solutions/ppt_ceph_mel...
http://www.snia.org/sites/default/files/JohnKim_CephWithHighPerformanceNetworks_V2.pdf http://www.snia.org/sites/default/files/JohnKim_CephWithHigh...
Also, look at Sector/Sphere which was made by UDT designer for distributed, parallel workloads for supercomputing. It has significant advantages over Hadoop. It's used with high-performance links to share data between supercomputing centers.
http://sector.sourceforge.net/index.html http://sector.sourceforge.net/index.html
http://sector.sourceforge.net/pub/Sector%20vs%20Hadoop%20-%20v1.pdf http://sector.sourceforge.net/pub/Sector%20vs%20Hadoop%20-%2...
- ymse 10y agoNative RDMA support for Ceph is still a ways off. The current implementation requires disabling Cephx authentication, which is a no-go in any environment where you can't completely trust every client (e.g. "cloud", where most current deployments/users live). It also hasn't seen much development since the initial proof-of-concept (still highly experimental). That said, IPoIB should work just fine, and the main bottleneck currently is (Ethernet) latency. I'm running a couple of 1,6PB clusters (432 * 4TB) and can only get 20-60MBps on a single client with a 4kB block size, but got bored of benchmarking after saturating 5 concurrent 10Gb clients with a 4MB block size. I do expect the RDMA situation to improve substantially over the next year or so, even if authentication will still be unsupported. The latter generally isn't a problem in HPC where stuff like GPFS lives (where you also have to trust every client). And they clearly want that market now that CephFS is finally deemed production ready.