3 ms·

There was a great blog post I read a while back about constructing a caching layer across network by Dan Luu: https://danluu.com/infinite-disk/ https://danluu.c

There was a great blog post I read a while back about constructing a caching layer across network by Dan Luu: https://danluu.com/infinite-disk/ https://danluu.com/infinite-disk/ I asked a friend who works in a quant firm and he was like yes it’s true, and it is pretty insane. I think there’s research Microsoft and Google are doing for RDMA over 100G Ethernet for intra data center communication as well. Pretty neat.

HillaryBriss 7y ago
neat blog post! i like the table at the top comparing latencies. at one point the post says: I'm paying $60/month for 100Mb, and if the trend of the last two decades continues, we should see another 50x increase in bandwidth per dollar over the next decade. he seems to have more confidence in his bandwidth provider than I have in mine!
baybal2 7y ago
I worked on something similar like that 3 years ago as a sub-sub-subcontractor for a company making DCs for Alibaba. It took them almost 2.5 years since me signing off on work to roll it out in a limited commercial trial in their alicloud hosting. The original idea was to let purpose made hardware be distributed across DC rather than every server having to carry it: video codecs on FPGAs, hardware wire speed crypto/compression, databases and k/v stores exposed over RDMA, and remote block storage on SSDs. I was invited to the opening ceremony for the DC. When company's bosses were showed sfx infused 3D graphs allegedly representing their AI things running on it, I was unable to restrain myself from ruining the atmosphere by asking how it is running when all servers in the DC were shut down :D
theincredulousk 7y ago
Yes! Was surprised the paper didn't specifically mention RDMA, or to a lesser degreee SRIOV, with all their focus on NICs. Also, there may be ongoing research, but it isn't theory at all. HPCs, HFTs, and the could providers have been leveraging RDMA for a long time - e.g. Infiniband. Doing it over Ethernet (RRoCE) is relatively new, and it isn't necessarily any big leap that is happens over 100G instead of 40 or 1. However, an interesting point as network links go to 100G+ (esp. for RDMA) is again on the storage/processing side. E.g. a wireshark capture on a 100G connection? ~12.5 GB/second, near max bandwidth for DDR3, and can fill 64GB of RAM in about 5 seconds at full fire-hose. So again the hot-potato of bottleneck will be passed, at least for maximum sustained performance situations. Side note, AFAIK RoCE exists mostly due to non-technical arguments, particularly the inertia created by existing familiarity and deployment of Ethernet in data centers. I think Microsoft was the one flexing on a standards-body to push it through. It is somewhat of a kludge as Ethernet wasn't designed with RDMA in mind - no guaranteed predictable latency, frames can and will disappear if switch buffers overflow, etc. So IMO "research" into the topic isn't super profound - akin to studying how your sedan might be heavily modified to go off-roading almost (but not quite) as well as a pick-up truck. Even now many that have the luxury are just going Infiniband from the get-go if RDMA/latency are the key priorities rather than tacked on later.
- Veedrac 7y ago
  The paper does mention RDMA.
AtlasBarfed 7y ago
all that huge scale distributed storage is going to run into CAP concerns with imperfect network links, nodes, storage units. I get a lot of that is consumer/social network crapdata with low guarantees to the end user. Dropbox would probably like a little more of a guarantee, and enterprise even more so.

HillaryBriss 7y ago

neat blog post! i like the table at the top comparing latencies. at one point the post says: I'm paying $60/month for 100Mb, and if the trend of the last two decades continues, we should see another 50x increase in bandwidth per dollar over the next decade. he seems to have more confidence in his bandwidth provider than I have in mine!

baybal2 7y ago

I worked on something similar like that 3 years ago as a sub-sub-subcontractor for a company making DCs for Alibaba. It took them almost 2.5 years since me signing off on work to roll it out in a limited commercial trial in their alicloud hosting. The original idea was to let purpose made hardware be distributed across DC rather than every server having to carry it: video codecs on FPGAs, hardware wire speed crypto/compression, databases and k/v stores exposed over RDMA, and remote block storage on SSDs. I was invited to the opening ceremony for the DC. When company's bosses were showed sfx infused 3D graphs allegedly representing their AI things running on it, I was unable to restrain myself from ruining the atmosphere by asking how it is running when all servers in the DC were shut down :D

theincredulousk 7y ago

Yes! Was surprised the paper didn't specifically mention RDMA, or to a lesser degreee SRIOV, with all their focus on NICs. Also, there may be ongoing research, but it isn't theory at all. HPCs, HFTs, and the could providers have been leveraging RDMA for a long time - e.g. Infiniband. Doing it over Ethernet (RRoCE) is relatively new, and it isn't necessarily any big leap that is happens over 100G instead of 40 or 1. However, an interesting point as network links go to 100G+ (esp. for RDMA) is again on the storage/processing side. E.g. a wireshark capture on a 100G connection? ~12.5 GB/second, near max bandwidth for DDR3, and can fill 64GB of RAM in about 5 seconds at full fire-hose. So again the hot-potato of bottleneck will be passed, at least for maximum sustained performance situations. Side note, AFAIK RoCE exists mostly due to non-technical arguments, particularly the inertia created by existing familiarity and deployment of Ethernet in data centers. I think Microsoft was the one flexing on a standards-body to push it through. It is somewhat of a kludge as Ethernet wasn't designed with RDMA in mind - no guaranteed predictable latency, frames can and will disappear if switch buffers overflow, etc. So IMO "research" into the topic isn't super profound - akin to studying how your sedan might be heavily modified to go off-roading almost (but not quite) as well as a pick-up truck. Even now many that have the luxury are just going Infiniband from the get-go if RDMA/latency are the key priorities rather than tacked on later.

Veedrac 7y ago
The paper does mention RDMA.

AtlasBarfed 7y ago

all that huge scale distributed storage is going to run into CAP concerns with imperfect network links, nodes, storage units. I get a lot of that is consumer/social network crapdata with low guarantees to the end user. Dropbox would probably like a little more of a guarantee, and enterprise even more so.