4 ms·

I have to admit, the technical aspects of this are way beyond me. Would this technology allow (in theory) secure distributed computation via RDMA? I've long su

by spangry 11y ago

I have to admit, the technical aspects of this are way beyond me. Would this technology allow (in theory) secure distributed computation via RDMA? I've long suspected that Intel understands the implications of mass adoption of cheap, RDMA capable network adapters (iWarp, ROCE and Infiniband): it will cannibalise future CPU sales revenue. Imagine a standard corporate working environment with 1000 workstations on a LAN. At any given time, average CPU utilisation is probably 5-10 per cent tops. It's a similar story with storage and I/O capacity. If you add RDMA (and ultra-low latency networking) to the equation, there is now no need to buy additional computational power for the next 5 years, as there are a tonne of idle resources that can now be efficiently utilised (even for non-parallelisable computation). From what I can surmise from recent Intel actions, they've opted to not take the 'Microsoft' approach (i.e. hold back the tide), and have instead decided that if CPU markets are going to be cannibalised, they may as well be the ones doing the cannibalising. Well, that's my theory anyway. Am I crazy?

wmf 11y ago
SGX has nothing to do with RDMA, and I suspect they don't play well together; all data entering/leaving an enclave probably has to be copied. Also, RDMA is only for servers and may not be as powerful as you think.
- spangry 11y ago
  Thanks for helping me understand this. Please bear with me here, as I'm not as technically skilled as the average HN user. On your first sentence, is the issue that some fundamental aspect of the SGX security model requires copying data in/out of enclaves, which would make direct computation on data stored in remote memory impossible? And breaking your second sentence into two parts: (1) That does seem to be the present state of affairs. From what I can gather, RDMA and low-latency networking is expensive due to the high cost of interconnect/cabling and switching infrastructure. So atm RDMA is exclusively used in HPC clusters and as backplane interconnect between server blades/racks. But I wonder if this will always be the case. Take cabling for example. Retail SFP+ optical interconnect is crazy expensive for even very short runs. If this is because production costs are high by nature, then I'd agree that LL networking and RDMA will remain confined to the server room. But if there are significant unrealised production economies of scale, or there are achievable advances in production techniques that will reduce costs, then deployment at the network edge might be economically feasible once we pass some level of demand/adoption. 2) On low-latency RDMA not being as powerful as I think: This might be because of my limited understanding. From what I understand, LL RDMA would allow a whole bunch of computers to be abstracted as a single 'super computer': the inter-memory-processor latency is so low that it makes this abstraction possible. Have I misunderstood the technology? (genuine question)
  - wmf 11y ago
    The whole point of SGX is that the memory of an enclave is ultra-protected so that nothing can get in or out without the enclave's permission. DMA goes completely against that concept. Much of the market segmentation between desktop and server is artificial, but there's still nothing customers can do about it. RDMA doesn't exist for 1G Ethernet because CPUs can easily keep up with copies. 10G Ethernet has been around for over 10 years and there's no evidence that it will ever trickle down to the desktop. Almost no networks support RDMA since it requires special network configuration, special NICs, and special libraries. No clouds support it. So any software that requires RDMA cannot be used by hardly anyone. (Example: http://blog.acolyer.org/2016/01/14/no-compromises/ http://blog.acolyer.org/2016/01/14/no-compromises/ Of course, the economics of hyperscale cloud providers are different.) Software can be written with an RDMA fast path and a TCP/IP slow path, but then 99% of users will use the slow path and so it's better to optimize around the characteristics of normal networking.

by spangry 11y ago

wmf 11y ago

SGX has nothing to do with RDMA, and I suspect they don't play well together; all data entering/leaving an enclave probably has to be copied. Also, RDMA is only for servers and may not be as powerful as you think.

spangry 11y ago
Thanks for helping me understand this. Please bear with me here, as I'm not as technically skilled as the average HN user. On your first sentence, is the issue that some fundamental aspect of the SGX security model requires copying data in/out of enclaves, which would make direct computation on data stored in remote memory impossible? And breaking your second sentence into two parts: (1) That does seem to be the present state of affairs. From what I can gather, RDMA and low-latency networking is expensive due to the high cost of interconnect/cabling and switching infrastructure. So atm RDMA is exclusively used in HPC clusters and as backplane interconnect between server blades/racks. But I wonder if this will always be the case. Take cabling for example. Retail SFP+ optical interconnect is crazy expensive for even very short runs. If this is because production costs are high by nature, then I'd agree that LL networking and RDMA will remain confined to the server room. But if there are significant unrealised production economies of scale, or there are achievable advances in production techniques that will reduce costs, then deployment at the network edge might be economically feasible once we pass some level of demand/adoption. 2) On low-latency RDMA not being as powerful as I think: This might be because of my limited understanding. From what I understand, LL RDMA would allow a whole bunch of computers to be abstracted as a single 'super computer': the inter-memory-processor latency is so low that it makes this abstraction possible. Have I misunderstood the technology? (genuine question)
- wmf 11y ago
  The whole point of SGX is that the memory of an enclave is ultra-protected so that nothing can get in or out without the enclave's permission. DMA goes completely against that concept. Much of the market segmentation between desktop and server is artificial, but there's still nothing customers can do about it. RDMA doesn't exist for 1G Ethernet because CPUs can easily keep up with copies. 10G Ethernet has been around for over 10 years and there's no evidence that it will ever trickle down to the desktop. Almost no networks support RDMA since it requires special network configuration, special NICs, and special libraries. No clouds support it. So any software that requires RDMA cannot be used by hardly anyone. (Example: http://blog.acolyer.org/2016/01/14/no-compromises/ http://blog.acolyer.org/2016/01/14/no-compromises/ Of course, the economics of hyperscale cloud providers are different.) Software can be written with an RDMA fast path and a TCP/IP slow path, but then 99% of users will use the slow path and so it's better to optimize around the characteristics of normal networking.