10 ms·

AWS EC2 Virtualization 2017: Including Nitro

155 points by brendangregg 9y ago

saurik 9y ago
This article talks about the underlying technical differences between various classes of EC2 computer... is this documented by Amazon anywhere? I would totally care more about replacing stuff if I knew "ok, there is some new virtualization difference in this new set of machines" that I could analyze against my use cases instead of "I have no clue what if anything is different, other than the pricing is a little different".
- TheDong 9y ago
  Some of this is fairly well known if you happen to be quite familiar with Xen. If your use case is "I don't want to understand Xen", then fine, just use AWS and pick the newest instances, they're usually the best. AWS does have some words about PV vs HVM here: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/virtualization_types.html http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/virtualiz... But really, they don't say much and that's fine. If you just want to run stuff, you probably don't need to understand the deep details of Xen or PV drivers or whatever; the cpu speed, memory, and "disk io"/"cpu credits" they document quite well will be what matters significantly more. If you are at the scale of netflix, then it does make sense to figure out an extra several percent disk performance. > I would totally care more about replacing stuff if I knew ... instead of "I have no clue what if anything is different They almost always have blog posts accompanying each new instance launch which roughly outline the performance improvements and reasons that instance type is cool vs previous ones, even if they don't go into the depth of virtualization differences so often (which again, usually doesn't matter! for nitro, the C5 blog did talk about the visualization stuff in depth https://aws.amazon.com/blogs/aws/now-available-compute-intensive-c5-instances-for-amazon-ec2/ https://aws.amazon.com/blogs/aws/now-available-compute-inten...)
  - saurik 9y ago
    I knew about PV vs. HVM, as that is documented. However, there is a ton of nuance in this article about every instance type and specific changes to what parts of the system are virtualized (and in particular stuff with EBS, which I stress heavily). I guess you are saying that people need to essentially find all of the old blog posts from AWS to figure out any of this, but why isn't it just documented? I clearly am someone who cares a lot about these things, or I wouldn't have posted the question I did: but Amazon's instances are essentially just black boxes. I am hoping that there is somewhere in the documentation that I don't know about that just shows the differences between all of these instance types in a few centralized pages.
- aliguori 9y ago
  I tried to cover this in my re:Invent session and walked through how we have been evolving the Nitro System over the last 5 years. Videos and slides will be posted eventually.
  - _msw_ 9y ago
    The video is available here: https://www.youtube.com/watch?v=LabltEXk0VQ https://www.youtube.com/watch?v=LabltEXk0VQ
andrewstuart 9y ago
Does this allow nested virtualization? ie the ability to run VMs within an EC2 instance?
- crazysim 9y ago
  Definitely on the pure-hardware one. VMWare's running their stuff on that.
- aliguori 9y ago
  C5 does not support nested virtualization but i3.metal allows using virtualization technology without nested virtualization. Both i3.metal and c5 use the same underlying Nitro technology.
  - bkeroack 9y ago
    Wait, so even the "bare metal" i3.metal sits under a hypervisor? It would seem inaccurate to call it "bare metal" in that case.
    - gregdunn 9y ago
      No, Nitro refers to the whole technology stack as well as the hypervisor. *.metal instances utilize most of the Nitro technology stack, but not the Nitro hypervisor.
      - bkeroack 9y ago
        Ah, ok. That makes sense. Thank you for the clarification.
        gregdunn 9y ago
        You're welcome! https://www.youtube.com/watch?v=o9_4uGvbvnk https://www.youtube.com/watch?v=o9_4uGvbvnk might be of interest - Matt Wilson talks a lot about the underlying magic of the bare metal instances.
jhgg 9y ago
Am super curious to see some benches on AWS Nitro compared with GCP's Skylake VMs.
ksec 9y ago
Turns out Nitro is not only an improved KVM but also work exclusively with their Nitro Custom Silicon. [1] And we are finally here, the true cloud, elastic computing with little to no performance overhead. [1]: https://www.theregister.co.uk/2017/11/29/aws_reveals_nitro_architecture_bare_metal_ec2_guard_duty_security_tool/ https://www.theregister.co.uk/2017/11/29/aws_reveals_nitro_a...
- Annatar 9y ago
  We were here back in 2005 with the introduction of Solaris zones, which now live on in SmartOS. Took GNU/Linux “only“ 12 years to begin to catch up. This might be new to you, but it’s nothing new to those of us who built private clouds back in 2005/2006. Just don’t think it’s new because it isn’t.
  - mpe 9y ago
    Zones are not VMs. If you need to vent your Linux hate at least save it for a post about Docker.
    - Annatar 9y ago
      I don’t need to be patronized on the difference between zones and VM’s as I’m also one of those VMware users from 1999 in Brendan’s article. Zones however are superior to VM’s because they don’t require hard pre-allocation of resources like virtual machines do, so they are a superior choice in environments where Windows isn’t desired or required. As for my Linux hate, if Linux were so great and well made, I wouldn’t hate it. I like well made things.
      - aquadrop 9y ago
        > Zones however are superior to VM’s because they don’t require hard pre-allocation of resources like virtual machines do, so they are a superior choice in environments where Windows isn’t desired or required. In public cloud that's not a problem, that's a requirement, so one VM wouldn't trash all its neighbors.
zimbatm 9y ago
I wonder what measures are in place in the bare-metal edition to protect themselves from firmware attacks. What protects them from possible attacks against BIOS flash or potentially infect the CPU and HDD chips?
- _msw_ 9y ago
  There is a Nitro security chip that is integrated into the motherboard. SPI and I2C buses are routed through the security chip. During system reset, the security chip holds motherboard components in reset while firmware is inspected out of band. So all contents of flash are verified. Local NVMe storage is also handled through our Nitro hardware, and the underlying physical flash devices are protected from firmware modifications. https://www.youtube.com/watch?v=o9_4uGvbvnk https://www.youtube.com/watch?v=o9_4uGvbvnk provides an overview.
SteveNuts 9y ago
Can you use a regular AMI on the bare metal instances?
- gregdunn 9y ago
  AMIs that work on c5 instances should work fine on bare metal instances. NVMe and ENA support will be the big two things you need.
aedocw 9y ago
How much of this went back upstream to improve KVM itself? Or is all of the nitro work just special sauce on top that they're not required to share?
- _msw_ 9y ago
  https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/log/?qt=author&q=amazon https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

155 points by brendangregg 9y ago

This article talks about the underlying technical differences between various classes of EC2 computer... is this documented by Amazon anywhere? I would totally care more about replacing stuff if I knew "ok, there is some new virtualization difference in this new set of machines" that I could analyze against my use cases instead of "I have no clue what if anything is different, other than the pricing is a little different".

TheDong 9y ago
Some of this is fairly well known if you happen to be quite familiar with Xen. If your use case is "I don't want to understand Xen", then fine, just use AWS and pick the newest instances, they're usually the best. AWS does have some words about PV vs HVM here: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/virtualization_types.html http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/virtualiz... But really, they don't say much and that's fine. If you just want to run stuff, you probably don't need to understand the deep details of Xen or PV drivers or whatever; the cpu speed, memory, and "disk io"/"cpu credits" they document quite well will be what matters significantly more. If you are at the scale of netflix, then it does make sense to figure out an extra several percent disk performance. > I would totally care more about replacing stuff if I knew ... instead of "I have no clue what if anything is different They almost always have blog posts accompanying each new instance launch which roughly outline the performance improvements and reasons that instance type is cool vs previous ones, even if they don't go into the depth of virtualization differences so often (which again, usually doesn't matter! for nitro, the C5 blog did talk about the visualization stuff in depth https://aws.amazon.com/blogs/aws/now-available-compute-intensive-c5-instances-for-amazon-ec2/ https://aws.amazon.com/blogs/aws/now-available-compute-inten...)
- saurik 9y ago
  I knew about PV vs. HVM, as that is documented. However, there is a ton of nuance in this article about every instance type and specific changes to what parts of the system are virtualized (and in particular stuff with EBS, which I stress heavily). I guess you are saying that people need to essentially find all of the old blog posts from AWS to figure out any of this, but why isn't it just documented? I clearly am someone who cares a lot about these things, or I wouldn't have posted the question I did: but Amazon's instances are essentially just black boxes. I am hoping that there is somewhere in the documentation that I don't know about that just shows the differences between all of these instance types in a few centralized pages.
aliguori 9y ago
I tried to cover this in my re:Invent session and walked through how we have been evolving the Nitro System over the last 5 years. Videos and slides will be posted eventually.
- _msw_ 9y ago
  The video is available here: https://www.youtube.com/watch?v=LabltEXk0VQ https://www.youtube.com/watch?v=LabltEXk0VQ

andrewstuart 9y ago

Does this allow nested virtualization? ie the ability to run VMs within an EC2 instance?

crazysim 9y ago
Definitely on the pure-hardware one. VMWare's running their stuff on that.
aliguori 9y ago
C5 does not support nested virtualization but i3.metal allows using virtualization technology without nested virtualization. Both i3.metal and c5 use the same underlying Nitro technology.
- bkeroack 9y ago
  Wait, so even the "bare metal" i3.metal sits under a hypervisor? It would seem inaccurate to call it "bare metal" in that case.
  - gregdunn 9y ago
    No, Nitro refers to the whole technology stack as well as the hypervisor. *.metal instances utilize most of the Nitro technology stack, but not the Nitro hypervisor.
    - bkeroack 9y ago
      Ah, ok. That makes sense. Thank you for the clarification.
      - gregdunn 9y ago
        You're welcome! https://www.youtube.com/watch?v=o9_4uGvbvnk https://www.youtube.com/watch?v=o9_4uGvbvnk might be of interest - Matt Wilson talks a lot about the underlying magic of the bare metal instances.

jhgg 9y ago

Am super curious to see some benches on AWS Nitro compared with GCP's Skylake VMs.

ksec 9y ago

Turns out Nitro is not only an improved KVM but also work exclusively with their Nitro Custom Silicon. [1] And we are finally here, the true cloud, elastic computing with little to no performance overhead. [1]: https://www.theregister.co.uk/2017/11/29/aws_reveals_nitro_architecture_bare_metal_ec2_guard_duty_security_tool/ https://www.theregister.co.uk/2017/11/29/aws_reveals_nitro_a...

Annatar 9y ago
We were here back in 2005 with the introduction of Solaris zones, which now live on in SmartOS. Took GNU/Linux “only“ 12 years to begin to catch up. This might be new to you, but it’s nothing new to those of us who built private clouds back in 2005/2006. Just don’t think it’s new because it isn’t.
- mpe 9y ago
  Zones are not VMs. If you need to vent your Linux hate at least save it for a post about Docker.
  - Annatar 9y ago
    I don’t need to be patronized on the difference between zones and VM’s as I’m also one of those VMware users from 1999 in Brendan’s article. Zones however are superior to VM’s because they don’t require hard pre-allocation of resources like virtual machines do, so they are a superior choice in environments where Windows isn’t desired or required. As for my Linux hate, if Linux were so great and well made, I wouldn’t hate it. I like well made things.
    - aquadrop 9y ago
      > Zones however are superior to VM’s because they don’t require hard pre-allocation of resources like virtual machines do, so they are a superior choice in environments where Windows isn’t desired or required. In public cloud that's not a problem, that's a requirement, so one VM wouldn't trash all its neighbors.

zimbatm 9y ago

I wonder what measures are in place in the bare-metal edition to protect themselves from firmware attacks. What protects them from possible attacks against BIOS flash or potentially infect the CPU and HDD chips?

_msw_ 9y ago
There is a Nitro security chip that is integrated into the motherboard. SPI and I2C buses are routed through the security chip. During system reset, the security chip holds motherboard components in reset while firmware is inspected out of band. So all contents of flash are verified. Local NVMe storage is also handled through our Nitro hardware, and the underlying physical flash devices are protected from firmware modifications. https://www.youtube.com/watch?v=o9_4uGvbvnk https://www.youtube.com/watch?v=o9_4uGvbvnk provides an overview.

SteveNuts 9y ago

Can you use a regular AMI on the bare metal instances?

gregdunn 9y ago
AMIs that work on c5 instances should work fine on bare metal instances. NVMe and ENA support will be the big two things you need.

aedocw 9y ago

How much of this went back upstream to improve KVM itself? Or is all of the nitro work just special sauce on top that they're not required to share?

_msw_ 9y ago
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/log/?qt=author&q=amazon https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...