4 ms·
I was a big Xen guy back in the day. The big takeaway from Hamilton's talk was actually something that dawned on me when they released the c3/m3/i2 instances w
by jpgvm 12y ago
I was a big Xen guy back in the day.
The big takeaway from Hamilton's talk was actually something that dawned on me when they released the c3/m3/i2 instances which is what they have done with SR-IOV.
The software stack that implements the AWS VPC fast path must be running on the NIC itself. This is a big deal.
Being able to implement the require isolation/tunneling/encapsulation logic and routing in the NIC is a huge performance boost and drastically simplifies the hypervisor. Well, the software portion of the hypervisor atleast.
If you listened closely to the talk he laid out the differences in latency impact of each layer in the networking stack, from the fibre (nanoseconds) up to the software stack at some number of milliseconds, several orders of magnitude worse than anything else in the stack.
The reason why the quoted latency is so large is how para-virtualized networking is implemented. In order for it to be (bandwith) performant it needs to use large ring buffers with many segments. This is very throughput optimised and latency suffers a ton as a result.
By moving all of the queuing to the NIC you get a bunch of benefits, namely the dom0 (basically the host in Xen terminology) is no longer involved in pushing packets and you are not incuring the cost of the Linux networking stack 2x. In the paravirt model the skbs are transferred across the circular buffer into the host OS where they are injected into a virtual interface, thus traversing the full net stack again.
In the SR-IOV model the address space of the virtual NIC is mapped into the guest OS using Intel's IO-MMU extensions and the guest is then able to communicate directly with the NIC, thus 100% bare metal performance.
If SR-IOV was the only improvement it would be impressive, however it's the consequence of it's existence which makes the biggest difference. If the guest is talking directly to the NIC, then all of the encap/decap is HW accelerated too and in theory this means the full networking stack is end-to-end in HW.
- Andys 12y agoNote: standard out-of-the-box SR-IOV allows VLAN tagging/stripping outside of the guest control, maybe they are simply using this in conjunction with a layer 3 switch to handle the VPC stuff?
- electrum 12y agoAmazon VPC is far more complex: https://www.youtube.com/watch?v=Zd5hsL-JNY4 https://www.youtube.com/watch?v=Zd5hsL-JNY4 (excellent talk from one of the creators of VPC)
- virtuallynathan 12y agoI'd guess they are using VXLAN or another similar technology.
- jpgvm 12y agoProbably something closer to MPLS all things considered. If you think about it they effectively are a carrier network with many virtual customer networks on top. MPLS has a bunch of advantages over VXLAN and is decidedly L3, which is more suited to AWS VPC. The real question however is what the control plane looks like. They may just go pure carrier style and just do tagging and resource control on the NIC but it's entirely possible for the NIC to be doing much more, especially if they have a dynamic control plane similar to OpenFlow (though probably not exactly OpenFlow.. it's less useful than it sounds). I would love to see a talk on AWS VPC internals rather than sitting around taking pot shots. :)
- Rapzid 12y agoI just watched that youtube video.. I had thought that VPC's were a clever combination of xvlan or grsec tunnels; I had no idea how custom it was. That's crazy in combination with sr-iov as you pointed out. I gather they are either running the VPC stuff on the card or what it's connected to.