6 ms·

This is why I'm really hoping RDMA [1] will catch on soon. It would be great if there was a cloud provider that would enable this feature on some of their offer

by en4bz 10y ago

This is why I'm really hoping RDMA [1] will catch on soon. It would be great if there was a cloud provider that would enable this feature on some of their offerings. Amazon has done something similar by allowing kernel bypass via DPDK [2] with their ENA offering but Kernel bypass is inferior to RDMA in so many ways IMO. At this point we have 200Gbit/s NICs being provided by Mellanox [3]. CPUs aren't getting any faster and the scale out approach is extremely difficult to get right without going across NUMA domains [4]. Based on the progression of CPUs lately there just isn't going to be enough time to process all these packets AND have time left over to actually run your application. There's a lot of work focusing on data locality at the moment but at this point it's still not fool proof and the work that has been done is woefully under documented. As the article mentioned we've already added a bunch of hardware offloads. RDMA is just a continuation of these offloads but unfortunately it requires some minor changes on the application side to take advantage of which is why it's probably been slow to be adopted. RDMA has so many great applications for data-transfer for backend services. Whether it's queries between a web-server and a DB, replication/clustering of DBs, or micro-service fabric with micro seconds latency. Overall there's a lot of low hanging fruit that could be optimized with RDMA. [1] https://en.wikipedia.org/wiki/Remote_direct_memory_access https://en.wikipedia.org/wiki/Remote_direct_memory_access [2] http://dpdk.org/doc/nics http://dpdk.org/doc/nics [3] http://www.mellanox.com/page/products_dyn?product_family=266&mtag=connectx_6_en_card http://www.mellanox.com/page/products_dyn?product_family=266... [4] http://rhelblog.redhat.com/2015/09/29/pushing-the-limits-of-kernel-networking/ http://rhelblog.redhat.com/2015/09/29/pushing-the-limits-of-...

kev009 10y ago
This has been ongoing debate since the beginning. RDMA means running a large amount of protocol in silicon and firmware. Can you imagine what might go wrong? In the FreeBSD community it is somewhat common knowledge that Chelsio is the best Ethernet vendor, mainly because their driver isn't a trainwreck like the dominant Intel (which share BSD HAL code on BSD/Linux/Illumos and are quite similar) If you are alright with the trade offs of a firmware stack, Chelsio do have some pretty slick TCP full offloads that require no substantial application changes: http://www.chelsio.com/wp-content/uploads/resources/T6-100G-DDP-FreeBSD.pdf http://www.chelsio.com/wp-content/uploads/resources/T6-100G-...
- en4bz 10y ago
  At the end of the day I think we will end up with RDMA being the norm. Additional offloads can only go so far and are going to require firmware anyway so why not just cut to the chase. Last time I checked Chelsio backed iWarp which has pretty much been defeated in the RDMA wars by RoCE since iWarp had terrible performance because it has to run over TCP. Intel has even removed from it their new cards and have pushed DPDK as a replacement. I'm also not surprised Chelsio has added additional offloads to compensate for their lose. That being said RoCE does have that Converged Ethernet part (CE) which is still not mainstream in many datacenters. I get people will be disappointed that their previously Open Source drivers will be replaced with proprietary firmware but as we see FPGAs take off I think that you may see VHDL code going open source on new fully programmable cards. I for one welcome our new hardware based overlords but perhaps I'm being far to optimistic.
  - kev009 10y ago
    But why? We are fairly confident we can make BSD pump several hundred Gbps doing real world long haul TCP for content serving in the next couple years on something like Naples or POWER9. At the other end, Isilon converted from Infiniband to OS TCP for the latest product: https://www.nextplatform.com/2016/05/06/emc-shoots-explosive-performance-isilon-nitro/ https://www.nextplatform.com/2016/05/06/emc-shoots-explosive.... That is pretty amazing because of low latency timing and incast. To your point, Intel's Altera acquisition may eventually bear fruit but I'm not holding my breath and don't really know how to reason about it until an offload/accelerator ecosystem is built up.
    - en4bz 10y ago
      Now that I've reviewed the Chelsio doc more carefully I've noticed that DDP is actually part of iWarp so the TCP offload you mentioned is just one part of iWarp that happens to be transparent to user space which is quite interesting. That being said iWarp has lost out to RoCE mainly because of latency and not throughput. I guess that I'm just more biased towards lower latency (at the same bandwidth) because it is empirically better, even though for most applications 10us vs 1us is negligible. That and because I work in low latency trading.
kev009 10y ago
One other point, these cloud companies like Amazon and Google do a great deal of self congratulations for some pretty embarrassing architectures. Necessitating million server datacenters is as WTF to me as being proud of millions of lines of code when someone else is doing it with magnitudes less. 40G networking has been commercially viable since the top of the decade and 100G (or partial variants) for a couple years now while ENA tops out at 20G. They are basically asleep at the wheel because their architectures are predicated on poor assumptions, shirking understanding of computer architecture for 10s of thousands of SREs and "full stack developers". Dovetailing a bit, back in the commercial UNIX and mainframe markets it is pretty common to have 80%-90% percent system utilization. In the Linux world, it's usually single digits. For some reasons (I guess it's more inviting to think holistically due to the base OS model), we are getting those 90% figures in the BSD community. See drewg's comment, WhatsApp, Isilon, LLNW: I led the creation of an OS team, to look from first principles, where we were spending CPU/bus/IO bandwidth, and focusing on north-south delivery instead of horizontal scale. A team of 5, we are able to deliver significant shareholder value http://investors.limelight.com/file/Index?KeyFile=38751761 http://investors.limelight.com/file/Index?KeyFile=38751761.
- en4bz 10y ago
  Yeah I suppose what I really want is physical procurement of hardware with the ease of cloud.
  - kev009 10y ago
    I haven't used them, but it looks compelling: https://www.packet.net/ https://www.packet.net/
- bogomipz 10y ago
  >"One other point, these cloud companies like Amazon and Google do a great deal of self congratulations for some pretty embarrassing architectures. Necessitating million server datacenters is as WTF to me as being proud of millions of lines of code when someone else is doing it with magnitudes less" Can you explain what these "embarrassing architectures"? Also two of the largest tech companies on the planet using millions of servers is a "WTF" to you? Care to explain? Also I don't think either Amazon or Google has ever released actual numbers for server counts. Please provide a citation for this. >"as being proud of millions of lines of code when someone else is doing it with magnitudes less" This is also a pretty vague statement. Lines of code for what? And who is doing that same "what" with less code? >"They are basically asleep at the wheel because their architectures are predicated on poor assumptions, shirking understanding of computer architecture for 10s of thousands of SREs and "full stack developers". Do you really believe that Google and Amazon don't understand computer architecture? Please provide citations for how you arrived at the at the number of SREs and "Full Stack Developers" at Google and Amazon. I don't think Google/Amazon even have "Full Stack Developer" as a job title. >"I led the creation of an OS team, to look from first principles, where we were spending CPU/bus/IO bandwidth, and focusing on north-south delivery instead of horizontal scale" You did this at Limelight Networks then I presume according to the link you provided? You realize that a CDN is predominantly North-South traffic right? CDNs are a very different business than being general purpose cloud providers which in addition to north-south traffic have substantial east-west traffic. You fault Google and Amazon for "congratulating" themselves yet the link you provide presumably to substantiate your own grand achievements is fluffy press release. A press release with zero detail around the claims being made and no link to a white paper or any technical specifics are given. Limelight is not exactly what I would call a market leader or a name associated with being a great technical innovator. Limelight lost 73 million dollars last years and has a 270 million market cap. They were also sued by CDN provider Akami and agreed to pay Akami licensing fees as part of the settlement for that infringement.[1] >"A team of 5, we are able to deliver significant shareholder value" Your company's stock trades at $2.56 a share while Amazon and Alphabets trade at well over $800 a share. I will gladly take the share holder value that Google's and Amazon's "10s of thousands of SREs and "full stack developers" provide over your team of 5. [1] http://www.bizjournals.com/phoenix/news/2017/02/08/limelight-networks-ends-year-with-73-9m-loss.html http://www.bizjournals.com/phoenix/news/2017/02/08/limelight...
drewg123 10y ago
RDMA can be rather challenging for a cloud provider, as it typically requires a lossless fabric. Cloud providers don't want to enable flow control on switches (to prevent packet drops) as that has the potential to deadlock the entire network if a buggy or malicious VM stops consuming traffic. Heck, I took our corp. net down a decade back by causing a test machine to drop into a kernel debugger with flow control enabled on the switches and the test machine. All of a sudden, my remote access to the serial console kernel debugger froze, and I was cursing the flaky VPN. Then a second later, I had the 'Oh Crap, that was me!' realization.
- ra1n85 10y ago
  Flow-control as in ECN or Ethernet/pause? Loss is inevitable in most cloud network fabrics due to the use of shallow buffer commodity ASICs, the amount of fan-in, and the reliance on tenants to use the network in a typical manner (don't fragment, don't encap, preserve entropy). RDMA is going to be tough - microseconds count, you can't wait for TCP to retrans, and UDP doesn't provide the reliability needed. A new transport protocol is likely the answer.
jsolson 10y ago
Note that DPDK works just fine with virtio-net on GCE as well, although we have not declared official support for it. In terms of RDMA and bypass messaging, I agree there can be substantial advantages to offloading transport layer message processing from VMs. Regrettably, I can't say anything more on the subject today. (I lead the team that owns the virtual NICs exposed to guests in GCE)
- en4bz 10y ago
  Do you not use SR-IOV for GCE?
  - jsolson 10y ago
    We do not, although in practice as a guest you wouldn't necessarily know one way or another. Speaking from (decently informed) personal opinion here-- I see SR-IOV as a means to achieving certain networking characteristics, but one that historically didn't play particularly nicely with live migration (among other things), and one that locks you into particular silicon vendors, etc. It's a bit unfortunate that SR-IOV is often assumed as a prerequisite for high throughput, low latency, low jitter networking. On its own it doesn't guarantee any of those things (you're still beholden to the VCPU scheduler, if nothing else, and there's a lot of 'else'), and it's also not the only way to achieve those things. In particular there are a lot of ways to dedicate hardware to I/O service they aren't SR-IOV
    - i336_ 10y ago
      What alternatives would you recommend to people leaning toward SR-IOV? I say this abstractly as I'm curious what your answer is and I think it might be very helpful to others considering your experience. (I just had to google "sr-iov" as I was typing this :) )
      - jsolson 10y ago
        In open source vhost is a pretty good option that abstracts hardware away from VMs and generally performs well. That said, SR-IOV isn't a BAD option if you don't care about live migration of VMs over a lifespan potentially longer than the underlying hardware platform. It's a good option if you can live with host-hardware-specific drivers in your guests, you're willing to standardize on a specific NIC as host hardware for the fleet where you'll be running VMs, and you're willing to deal with the resource commitment it requires to deliver on its performance promises.
shaklee3 10y ago
All good points, but keep in mind the 200gbps is somewhat of a gimmick since pcie3 will top out at 128gbps. It's useful for a redundant port, but you won't be pushing 200gbps for a while.
- en4bz 10y ago
  Apparently they have PCIe4 x16 and PCIe3 x32 models. Not sure if either of these exist at the moment but they so satisfy the PCIe bandwidth constraints.
  - kev009 10y ago
    They have one that is two PCIe3 x16 connections from one ASIC right now, it uses a ribbon cable :). That's janky, but it does work on currently available platforms. I've heard Mellanox will do a CAPI NIC, which is quite tantalizing.
    - shaklee3 10y ago
      Thanks, I didn't see that. But yes, that's weird. In regards to pcie4,I believe the power9 will be the first to have it, but I haven't seen a solid release date for that. Naples didn't look like it had it -- just more pcie3 lanes.
bogomipz 10y ago
RDMA has been around for years though. It's big in the HPC world Oracle Rac had used it for some time. In Solaris 11, the default transport for NFS uses RDMA: https://docs.oracle.com/cd/E23824_01/html/821-1454/rfsrefer-154.html https://docs.oracle.com/cd/E23824_01/html/821-1454/rfsrefer-...
- en4bz 10y ago
  Infiniband RDMA is quite old yes but the Ethernet protocols are still rather new and they (iWarp & RoCE) are still competing for supremacy which doesn't really help adoption. At this point it looks like RoCE will win because Intel dropped iWarp but at this point there is basically only one major manufacturer backing each protocol.
  - bogomipz 10y ago
    Oh I misunderstood the context, yes my examples were in the context of Infiniband cards, thanks. I admit I haven't kept up with RoCE/iWarp. If you have an links to share I would appreciate it. Cheers.

by en4bz 10y ago

kev009 10y ago

This has been ongoing debate since the beginning. RDMA means running a large amount of protocol in silicon and firmware. Can you imagine what might go wrong? In the FreeBSD community it is somewhat common knowledge that Chelsio is the best Ethernet vendor, mainly because their driver isn't a trainwreck like the dominant Intel (which share BSD HAL code on BSD/Linux/Illumos and are quite similar) If you are alright with the trade offs of a firmware stack, Chelsio do have some pretty slick TCP full offloads that require no substantial application changes: http://www.chelsio.com/wp-content/uploads/resources/T6-100G-DDP-FreeBSD.pdf http://www.chelsio.com/wp-content/uploads/resources/T6-100G-...

en4bz 10y ago
At the end of the day I think we will end up with RDMA being the norm. Additional offloads can only go so far and are going to require firmware anyway so why not just cut to the chase. Last time I checked Chelsio backed iWarp which has pretty much been defeated in the RDMA wars by RoCE since iWarp had terrible performance because it has to run over TCP. Intel has even removed from it their new cards and have pushed DPDK as a replacement. I'm also not surprised Chelsio has added additional offloads to compensate for their lose. That being said RoCE does have that Converged Ethernet part (CE) which is still not mainstream in many datacenters. I get people will be disappointed that their previously Open Source drivers will be replaced with proprietary firmware but as we see FPGAs take off I think that you may see VHDL code going open source on new fully programmable cards. I for one welcome our new hardware based overlords but perhaps I'm being far to optimistic.
- kev009 10y ago
  But why? We are fairly confident we can make BSD pump several hundred Gbps doing real world long haul TCP for content serving in the next couple years on something like Naples or POWER9. At the other end, Isilon converted from Infiniband to OS TCP for the latest product: https://www.nextplatform.com/2016/05/06/emc-shoots-explosive-performance-isilon-nitro/ https://www.nextplatform.com/2016/05/06/emc-shoots-explosive.... That is pretty amazing because of low latency timing and incast. To your point, Intel's Altera acquisition may eventually bear fruit but I'm not holding my breath and don't really know how to reason about it until an offload/accelerator ecosystem is built up.
  - en4bz 10y ago
    Now that I've reviewed the Chelsio doc more carefully I've noticed that DDP is actually part of iWarp so the TCP offload you mentioned is just one part of iWarp that happens to be transparent to user space which is quite interesting. That being said iWarp has lost out to RoCE mainly because of latency and not throughput. I guess that I'm just more biased towards lower latency (at the same bandwidth) because it is empirically better, even though for most applications 10us vs 1us is negligible. That and because I work in low latency trading.

kev009 10y ago

One other point, these cloud companies like Amazon and Google do a great deal of self congratulations for some pretty embarrassing architectures. Necessitating million server datacenters is as WTF to me as being proud of millions of lines of code when someone else is doing it with magnitudes less. 40G networking has been commercially viable since the top of the decade and 100G (or partial variants) for a couple years now while ENA tops out at 20G. They are basically asleep at the wheel because their architectures are predicated on poor assumptions, shirking understanding of computer architecture for 10s of thousands of SREs and "full stack developers". Dovetailing a bit, back in the commercial UNIX and mainframe markets it is pretty common to have 80%-90% percent system utilization. In the Linux world, it's usually single digits. For some reasons (I guess it's more inviting to think holistically due to the base OS model), we are getting those 90% figures in the BSD community. See drewg's comment, WhatsApp, Isilon, LLNW: I led the creation of an OS team, to look from first principles, where we were spending CPU/bus/IO bandwidth, and focusing on north-south delivery instead of horizontal scale. A team of 5, we are able to deliver significant shareholder value http://investors.limelight.com/file/Index?KeyFile=38751761 http://investors.limelight.com/file/Index?KeyFile=38751761.

en4bz 10y ago
Yeah I suppose what I really want is physical procurement of hardware with the ease of cloud.
- kev009 10y ago
  I haven't used them, but it looks compelling: https://www.packet.net/ https://www.packet.net/
bogomipz 10y ago
>"One other point, these cloud companies like Amazon and Google do a great deal of self congratulations for some pretty embarrassing architectures. Necessitating million server datacenters is as WTF to me as being proud of millions of lines of code when someone else is doing it with magnitudes less" Can you explain what these "embarrassing architectures"? Also two of the largest tech companies on the planet using millions of servers is a "WTF" to you? Care to explain? Also I don't think either Amazon or Google has ever released actual numbers for server counts. Please provide a citation for this. >"as being proud of millions of lines of code when someone else is doing it with magnitudes less" This is also a pretty vague statement. Lines of code for what? And who is doing that same "what" with less code? >"They are basically asleep at the wheel because their architectures are predicated on poor assumptions, shirking understanding of computer architecture for 10s of thousands of SREs and "full stack developers". Do you really believe that Google and Amazon don't understand computer architecture? Please provide citations for how you arrived at the at the number of SREs and "Full Stack Developers" at Google and Amazon. I don't think Google/Amazon even have "Full Stack Developer" as a job title. >"I led the creation of an OS team, to look from first principles, where we were spending CPU/bus/IO bandwidth, and focusing on north-south delivery instead of horizontal scale" You did this at Limelight Networks then I presume according to the link you provided? You realize that a CDN is predominantly North-South traffic right? CDNs are a very different business than being general purpose cloud providers which in addition to north-south traffic have substantial east-west traffic. You fault Google and Amazon for "congratulating" themselves yet the link you provide presumably to substantiate your own grand achievements is fluffy press release. A press release with zero detail around the claims being made and no link to a white paper or any technical specifics are given. Limelight is not exactly what I would call a market leader or a name associated with being a great technical innovator. Limelight lost 73 million dollars last years and has a 270 million market cap. They were also sued by CDN provider Akami and agreed to pay Akami licensing fees as part of the settlement for that infringement.[1] >"A team of 5, we are able to deliver significant shareholder value" Your company's stock trades at $2.56 a share while Amazon and Alphabets trade at well over $800 a share. I will gladly take the share holder value that Google's and Amazon's "10s of thousands of SREs and "full stack developers" provide over your team of 5. [1] http://www.bizjournals.com/phoenix/news/2017/02/08/limelight-networks-ends-year-with-73-9m-loss.html http://www.bizjournals.com/phoenix/news/2017/02/08/limelight...

drewg123 10y ago

RDMA can be rather challenging for a cloud provider, as it typically requires a lossless fabric. Cloud providers don't want to enable flow control on switches (to prevent packet drops) as that has the potential to deadlock the entire network if a buggy or malicious VM stops consuming traffic. Heck, I took our corp. net down a decade back by causing a test machine to drop into a kernel debugger with flow control enabled on the switches and the test machine. All of a sudden, my remote access to the serial console kernel debugger froze, and I was cursing the flaky VPN. Then a second later, I had the 'Oh Crap, that was me!' realization.

ra1n85 10y ago
Flow-control as in ECN or Ethernet/pause? Loss is inevitable in most cloud network fabrics due to the use of shallow buffer commodity ASICs, the amount of fan-in, and the reliance on tenants to use the network in a typical manner (don't fragment, don't encap, preserve entropy). RDMA is going to be tough - microseconds count, you can't wait for TCP to retrans, and UDP doesn't provide the reliability needed. A new transport protocol is likely the answer.

jsolson 10y ago

Note that DPDK works just fine with virtio-net on GCE as well, although we have not declared official support for it. In terms of RDMA and bypass messaging, I agree there can be substantial advantages to offloading transport layer message processing from VMs. Regrettably, I can't say anything more on the subject today. (I lead the team that owns the virtual NICs exposed to guests in GCE)

en4bz 10y ago
Do you not use SR-IOV for GCE?
- jsolson 10y ago
  We do not, although in practice as a guest you wouldn't necessarily know one way or another. Speaking from (decently informed) personal opinion here-- I see SR-IOV as a means to achieving certain networking characteristics, but one that historically didn't play particularly nicely with live migration (among other things), and one that locks you into particular silicon vendors, etc. It's a bit unfortunate that SR-IOV is often assumed as a prerequisite for high throughput, low latency, low jitter networking. On its own it doesn't guarantee any of those things (you're still beholden to the VCPU scheduler, if nothing else, and there's a lot of 'else'), and it's also not the only way to achieve those things. In particular there are a lot of ways to dedicate hardware to I/O service they aren't SR-IOV
  - i336_ 10y ago
    What alternatives would you recommend to people leaning toward SR-IOV? I say this abstractly as I'm curious what your answer is and I think it might be very helpful to others considering your experience. (I just had to google "sr-iov" as I was typing this :) )
    - jsolson 10y ago
      In open source vhost is a pretty good option that abstracts hardware away from VMs and generally performs well. That said, SR-IOV isn't a BAD option if you don't care about live migration of VMs over a lifespan potentially longer than the underlying hardware platform. It's a good option if you can live with host-hardware-specific drivers in your guests, you're willing to standardize on a specific NIC as host hardware for the fleet where you'll be running VMs, and you're willing to deal with the resource commitment it requires to deliver on its performance promises.

shaklee3 10y ago

All good points, but keep in mind the 200gbps is somewhat of a gimmick since pcie3 will top out at 128gbps. It's useful for a redundant port, but you won't be pushing 200gbps for a while.

en4bz 10y ago
Apparently they have PCIe4 x16 and PCIe3 x32 models. Not sure if either of these exist at the moment but they so satisfy the PCIe bandwidth constraints.
- kev009 10y ago
  They have one that is two PCIe3 x16 connections from one ASIC right now, it uses a ribbon cable :). That's janky, but it does work on currently available platforms. I've heard Mellanox will do a CAPI NIC, which is quite tantalizing.
  - shaklee3 10y ago
    Thanks, I didn't see that. But yes, that's weird. In regards to pcie4,I believe the power9 will be the first to have it, but I haven't seen a solid release date for that. Naples didn't look like it had it -- just more pcie3 lanes.

bogomipz 10y ago

RDMA has been around for years though. It's big in the HPC world Oracle Rac had used it for some time. In Solaris 11, the default transport for NFS uses RDMA: https://docs.oracle.com/cd/E23824_01/html/821-1454/rfsrefer-154.html https://docs.oracle.com/cd/E23824_01/html/821-1454/rfsrefer-...

en4bz 10y ago
Infiniband RDMA is quite old yes but the Ethernet protocols are still rather new and they (iWarp & RoCE) are still competing for supremacy which doesn't really help adoption. At this point it looks like RoCE will win because Intel dropped iWarp but at this point there is basically only one major manufacturer backing each protocol.
- bogomipz 10y ago
  Oh I misunderstood the context, yes my examples were in the context of Infiniband cards, thanks. I admit I haven't kept up with RoCE/iWarp. If you have an links to share I would appreciate it. Cheers.