4 ms·

> Noone using this stuff cares about that I do. I expect the Mellanox ConnectX 3s in the Azure machines I develop on will be obsolete by the time my software i

> Noone using this stuff cares about that I do. I expect the Mellanox ConnectX 3s in the Azure machines I develop on will be obsolete by the time my software is deployed. I don't even know what they will be replaced with, let alone have access to such exotica. I think my choices today are POSIX sockets or DPDK. I hope for a day when I can try out this new fangled RDMA thing :-) I do accept your point that these things were possible a long time ago but I think it takes time for them to mature enough for mainstream adoption (ie for use in software you might expect to keep working for decades on a variety of hardware). I think Lemire's question of, "Building software on top of that requires some tooling and it is not clear to me how good it is right now", remains open.

cat199 8y ago
> I think my choices today are POSIX sockets or DPDK. I hope for a day when I can try out this new fangled RDMA thing :-) It's fairly straightforward, you use MPI, or code it yourself (so basically, you use MPI). https://wiki.mpich.org/mpich/index.php/MPI-3_RMA_Implementation_Timeline https://wiki.mpich.org/mpich/index.php/MPI-3_RMA_Implementat... https://www.open-mpi.org/faq/?category=openfabrics#roce-support https://www.open-mpi.org/faq/?category=openfabrics#roce-supp... maybe you use PVM, I think there are RDMA extenstions. But really, you probably use MPI.
- abainbridge 8y ago
  Interesting. I've never heard of MPI before. It looks like it is supported on Azure (https://docs.microsoft.com/en-us/azure/virtual-machines/linux/classic/rdma-cluster https://docs.microsoft.com/en-us/azure/virtual-machines/linu...). I'll look into it. Thanks.
  - abainbridge 8y ago
    The "clusters" that support this have to be co-located in the data center - I guess they have to be on the same ToR switch or something. That gives less flexibility for deployment, meaning there's less chance of being able to spawn 50 new VM instances when you need to. Also I expect they cost more than VMs without RDMA support, whereas the DPDK support seems to be free. I'll hopefully get chance to do some experiments soon.

cat199 8y ago

> I think my choices today are POSIX sockets or DPDK. I hope for a day when I can try out this new fangled RDMA thing :-) It's fairly straightforward, you use MPI, or code it yourself (so basically, you use MPI). https://wiki.mpich.org/mpich/index.php/MPI-3_RMA_Implementation_Timeline https://wiki.mpich.org/mpich/index.php/MPI-3_RMA_Implementat... https://www.open-mpi.org/faq/?category=openfabrics#roce-support https://www.open-mpi.org/faq/?category=openfabrics#roce-supp... maybe you use PVM, I think there are RDMA extenstions. But really, you probably use MPI.

abainbridge 8y ago
Interesting. I've never heard of MPI before. It looks like it is supported on Azure (https://docs.microsoft.com/en-us/azure/virtual-machines/linux/classic/rdma-cluster https://docs.microsoft.com/en-us/azure/virtual-machines/linu...). I'll look into it. Thanks.
- abainbridge 8y ago
  The "clusters" that support this have to be co-located in the data center - I guess they have to be on the same ToR switch or something. That gives less flexibility for deployment, meaning there's less chance of being able to spawn 50 new VM instances when you need to. Also I expect they cost more than VMs without RDMA support, whereas the DPDK support seems to be free. I'll hopefully get chance to do some experiments soon.