7 ms·

This seems to be such an obvious feature that I'm surprised they are only adding this now. I'm not an expert in hypervisors or anything like that and so I'm wo

This seems to be such an obvious feature that I'm surprised they are only adding this now. I'm not an expert in hypervisors or anything like that and so I'm wondering what was stopping them from adding it in the past?

cperciva 6y ago
The biggest reason is probably security. This is not something you want to take chances with.
- syncsynchalt 6y ago
  That's my take too. They've gotten so far without this functionality that I have to wonder what finally tipped the balance into their offering it.
  - cperciva 6y ago
    My guess is that an important vendor is shipping AMIs without sshd, and they need an "emergency back door".
    - bigiain 6y ago
      <cynical hat> s/vendor is shipping AMIs without sshd, and they/TLA/
pwarner 6y ago
I get this sense they are very conservative in their feature set there? Nested VMs for example are supposed in GCP, Azure and Oracle clouds I think, but not AWS. VM migration too I think exists in GCP at least, but not AWS. It's interesting.
- oblio 6y ago
  AWS is super slow to deprecate or remove features and services. A few people were noting that SimpleDB has been deprecated, it's not listed in the AWS web console, you can't find docs for it anymore, but if you have a running instance, your service API calls still work. And I think there have been many deprecation warnings since, plus migration messages. But they don't want to break existing clients. I'm guessing this is a similar case where they want to be really, really sure that it's worth offering the service.
  - ignoramous 6y ago
    > ...you can't find docs for it anymore Here are the docs: https://aws.amazon.com/simpledb/ https://aws.amazon.com/simpledb/
    - oblio 6y ago
      Ah, my bad. But the service is no longer listed in the AWS Console menus.
      - Dunedan 6y ago
        Because it never was, as it predated the AWS Management Console.
        oblio 6y ago
        S3 also predates the console, what kind of argument is that?
        detaro 6y ago
        "no longer" is weird phrasing if it never was there.
        oblio 6y ago
        You know those moments in sitcoms when some people discuss about something important and someone intervenes with some boring minutiae?
- p0rkbelly 6y ago
  Who needs nested virtualization when you have bare metal? VM Migration is only for maintenance on GCP -- and customers can't control it, just Google. AWS can hot patch live systems in place without any downtime, so, that's better than a migration (which has a brown out / maintenance period)
  - boulos 6y ago
    Disclosure: I worked on nested virtualization for GCE. Even when you offer bare metal, it’s actually still nice to have nested virt! Otherwise, every node has to be a full sized host. So when you have a K8s cluster or similar with a pile of nodes and want to allow some teams to use it (e.g., Android emulator, firecracker, whatever), it’s really nice not to have to say “okay, this group requires full bare metal hosts that they manage themselves”. tl;dr: nested virt is still a nice to have so that all your infrastructure looks the same. Edit: Also, you can trigger migration yourself if you want (gcloud compute instances simulate-maintenance-event), but that's mostly to convince yourself that nothing bad will happen.
  - JoshTriplett 6y ago
    > Who needs nested virtualization when you have bare metal? AWS's non-bare-metal systems can boot in ~10s with enough tuning. Their bare-metal systems take tens of minutes to boot. Nested virtualization would allow scaling up and starting new nodes much faster.
    - p0rkbelly 6y ago
      If you have a use case for spinning up nodes fast -- what is the use for nested virtualization? Sounds like that should be containers on any underlying single-virtualized layer?
      - JoshTriplett 6y ago
        Sometimes you want the safer isolation of virtualization, and you want to spin up individual workload elements in virtual machines.
- _msw_ 6y ago
  Disclosure: I work for AWS building cloud infrastructure. I wouldn't assume that VM migration does not exist in AWS. The overall design and implementation of Google's infrastructure somewhat mandated the development live migration support from day one. AWS was designed and built differently, and some types of events that force live migration in GCE do not exist in AWS. One specific example from Google's VM Live Migration At Scale paper [1] is "Regular maintenance on the power infrastructure in our data centers requires powering down subsets of machines for extended periods of time". The power infrastructure at AWS is designed to be redundant and concurrently maintainable, which removes a significant need for workload mobility within the datacenter. Personally, I think it was a very good idea to turn the thing that had to be built to launch into a marketed differentiated feature. But that doesn't mean that AWS doesn't have an ability to live migrate some workloads if it is able to do so without disrupting customers, or if it delivers a better experience than alternatives (e.g., instance degrade notices). [1] https://dl.acm.org/doi/10.1145/3186411.3186415 https://dl.acm.org/doi/10.1145/3186411.3186415
chousuke 6y ago
Likely they just had no need for it themselves. Might be that they didn't prioritize having a feature over the risk of someone taking over their hypervisors thanks to a buggy serial port emulator. Pretty much all hypervisors support serial consoles, but usually those interfaces are limited to trusted admins. For something like AWS, they'll also have to connect it from the hypervisor hosts into their public UI, and they can't trust the users.
- bigiain 6y ago
  They probably don't need it because they are much more likely to follow best practise "treat servers as cattle, not pets". If an instance wedges itself onto a state where I need console access, I'd just kill it and provision a replacement (ideally, my monitoring and automation will have done that already and not even have woken me up to tell me). I'm not sure I'd be at all comfortable having irreplaceable single points of failure in AWS. (Though I do recognise that people use it that way all the time...)
  - sargun 6y ago
    How do you debug broken instances?
    - x3n0ph3n3 6y ago
      Shut them down and mount the volume on a new host.
    - somethingAlex 6y ago
      Typically the goal is to architect the system such that you don't really care. If it's stateful, there's some other replica. Promote that and then spin up a new replica from a backup and roll it forward. If it's stateless then just kill it and spin up another.
      - sargun 6y ago
        But what if there is a bug that’s repeatable / pops up on the regular?
amelius 6y ago
Perhaps patents in HP's iLO system?
technion 6y ago
Lower end VPS providers have offered full console access for years. For example: https://www.linode.com/docs/guides/using-the-linode-shell-lish/ https://www.linode.com/docs/guides/using-the-linode-shell-li...

cperciva 6y ago

The biggest reason is probably security. This is not something you want to take chances with.

syncsynchalt 6y ago
That's my take too. They've gotten so far without this functionality that I have to wonder what finally tipped the balance into their offering it.
- cperciva 6y ago
  My guess is that an important vendor is shipping AMIs without sshd, and they need an "emergency back door".
  - bigiain 6y ago
    <cynical hat> s/vendor is shipping AMIs without sshd, and they/TLA/

pwarner 6y ago

I get this sense they are very conservative in their feature set there? Nested VMs for example are supposed in GCP, Azure and Oracle clouds I think, but not AWS. VM migration too I think exists in GCP at least, but not AWS. It's interesting.

oblio 6y ago
AWS is super slow to deprecate or remove features and services. A few people were noting that SimpleDB has been deprecated, it's not listed in the AWS web console, you can't find docs for it anymore, but if you have a running instance, your service API calls still work. And I think there have been many deprecation warnings since, plus migration messages. But they don't want to break existing clients. I'm guessing this is a similar case where they want to be really, really sure that it's worth offering the service.
- ignoramous 6y ago
  > ...you can't find docs for it anymore Here are the docs: https://aws.amazon.com/simpledb/ https://aws.amazon.com/simpledb/
  - oblio 6y ago
    Ah, my bad. But the service is no longer listed in the AWS Console menus.
    - Dunedan 6y ago
      Because it never was, as it predated the AWS Management Console.
      - oblio 6y ago
        S3 also predates the console, what kind of argument is that?
        detaro 6y ago
        "no longer" is weird phrasing if it never was there.
        oblio 6y ago
        You know those moments in sitcoms when some people discuss about something important and someone intervenes with some boring minutiae?
p0rkbelly 6y ago
Who needs nested virtualization when you have bare metal? VM Migration is only for maintenance on GCP -- and customers can't control it, just Google. AWS can hot patch live systems in place without any downtime, so, that's better than a migration (which has a brown out / maintenance period)
- boulos 6y ago
  Disclosure: I worked on nested virtualization for GCE. Even when you offer bare metal, it’s actually still nice to have nested virt! Otherwise, every node has to be a full sized host. So when you have a K8s cluster or similar with a pile of nodes and want to allow some teams to use it (e.g., Android emulator, firecracker, whatever), it’s really nice not to have to say “okay, this group requires full bare metal hosts that they manage themselves”. tl;dr: nested virt is still a nice to have so that all your infrastructure looks the same. Edit: Also, you can trigger migration yourself if you want (gcloud compute instances simulate-maintenance-event), but that's mostly to convince yourself that nothing bad will happen.
- JoshTriplett 6y ago
  > Who needs nested virtualization when you have bare metal? AWS's non-bare-metal systems can boot in ~10s with enough tuning. Their bare-metal systems take tens of minutes to boot. Nested virtualization would allow scaling up and starting new nodes much faster.
  - p0rkbelly 6y ago
    If you have a use case for spinning up nodes fast -- what is the use for nested virtualization? Sounds like that should be containers on any underlying single-virtualized layer?
    - JoshTriplett 6y ago
      Sometimes you want the safer isolation of virtualization, and you want to spin up individual workload elements in virtual machines.
_msw_ 6y ago
Disclosure: I work for AWS building cloud infrastructure. I wouldn't assume that VM migration does not exist in AWS. The overall design and implementation of Google's infrastructure somewhat mandated the development live migration support from day one. AWS was designed and built differently, and some types of events that force live migration in GCE do not exist in AWS. One specific example from Google's VM Live Migration At Scale paper [1] is "Regular maintenance on the power infrastructure in our data centers requires powering down subsets of machines for extended periods of time". The power infrastructure at AWS is designed to be redundant and concurrently maintainable, which removes a significant need for workload mobility within the datacenter. Personally, I think it was a very good idea to turn the thing that had to be built to launch into a marketed differentiated feature. But that doesn't mean that AWS doesn't have an ability to live migrate some workloads if it is able to do so without disrupting customers, or if it delivers a better experience than alternatives (e.g., instance degrade notices). [1] https://dl.acm.org/doi/10.1145/3186411.3186415 https://dl.acm.org/doi/10.1145/3186411.3186415

chousuke 6y ago

Likely they just had no need for it themselves. Might be that they didn't prioritize having a feature over the risk of someone taking over their hypervisors thanks to a buggy serial port emulator. Pretty much all hypervisors support serial consoles, but usually those interfaces are limited to trusted admins. For something like AWS, they'll also have to connect it from the hypervisor hosts into their public UI, and they can't trust the users.

bigiain 6y ago
They probably don't need it because they are much more likely to follow best practise "treat servers as cattle, not pets". If an instance wedges itself onto a state where I need console access, I'd just kill it and provision a replacement (ideally, my monitoring and automation will have done that already and not even have woken me up to tell me). I'm not sure I'd be at all comfortable having irreplaceable single points of failure in AWS. (Though I do recognise that people use it that way all the time...)
- sargun 6y ago
  How do you debug broken instances?
  - x3n0ph3n3 6y ago
    Shut them down and mount the volume on a new host.
  - somethingAlex 6y ago
    Typically the goal is to architect the system such that you don't really care. If it's stateful, there's some other replica. Promote that and then spin up a new replica from a backup and roll it forward. If it's stateless then just kill it and spin up another.
    - sargun 6y ago
      But what if there is a bug that’s repeatable / pops up on the regular?

amelius 6y ago

Perhaps patents in HP's iLO system?

technion 6y ago

Lower end VPS providers have offered full console access for years. For example: https://www.linode.com/docs/guides/using-the-linode-shell-lish/ https://www.linode.com/docs/guides/using-the-linode-shell-li...