5 ms·
Dunno about AWS, but GCP uses live migration, and will migrate your VM across physical machines as necessary. The disk volumes are all connected over the networ
by xyzzyz 4y ago
Dunno about AWS, but GCP uses live migration, and will migrate your VM across physical machines as necessary. The disk volumes are all connected over the network, nothing really depends on the actual physical machine your VM is ran on.
- lbhdc 4y agoHow does migrating a vm to another physical machine work?
- rejectfinite 4y agovsphere vmotion has been a thing for years lmao
- roomey 4y agoVMware has been doing this for years, it's called vmotion and there is a lot of documentation about it if you are interested (eg https://www.thegeekpub.com/8407/how-vmotion-works/ https://www.thegeekpub.com/8407/how-vmotion-works/ ) Essential, memory state is copied to the new host, the VM is stunned for a millisecond and the cpu states is copied and resumed on the new host (you may see a dropped ping). All the networking and storage is virtual anyway so that is "moved" (it's not really moved) in the background.
- mh- 4y agoUp to 500ms per your source, depending on how much churn there is in the memory from the source system. Very cool.
- davidro80 4y ago
- lbhdc 4y agoThat is really interesting I didn't realize it was so fast. Thanks for the post I will give it a read!
- jiggawatts 4y agoThe clever trick here is that they'll pre-copy most of the memory without bothering to do it consistently, but mark pages that the source had written to as "dirty". The network cutover is stop-the-world, but VMware doesn't copy the dirty pages during the stop. Instead, it simply treats them as "swapped to pagefile", where the pagefile is actually the source machine memory. When computation resumes at the target, the source is used to page memory back in on-demand. This allows very fast cutovers.
- politelemon 4y ago> VM is stunned for a millisecond This conjures up hilarious mental imagery, thanks
- kevincox 4y agoYou just bop it on the head, and move it to the new machine quickly. By the time the VM comes to it won't even realize that it is in a new home.
- valleyjo 4y agoStream the contents of ram from source to dest, pause the source, reprogram the network and copy and memory that changed since the initial stream, resume the dest, destroy the source, profit.
- pclmulqdq 4y agoThey pause your VM, copy everything about its state over to the new machine, and quickly start the other instance. It's pretty clever. I think there are tricks you can play with machines that have large memory footprints to copy most of it before the pause, and only copy what has changed since then during the pause. The disks are all on the network, so no need to move anything there.
- prmoustache 4y agoIn reality it sync the memory first to the other host and only pause the vm when the last state sync is small enough to be so quick the pause is barely measurable.
- water-your-self 4y agoIndian jones and the register states
- lbhdc 4y agoWhen its transferring the state to the target, how does it handle memory updates that are happening at that time? Is the programs execution paused at that point?
- outworlder 4y agoNo, they keep track of dirty pages.
- GauntletWizard 4y agoNo, but the memory accesses have hooks that say "This memory was written". Then, program execution is paused, and just the sections of memory that were written are copied again. This has memory performance characteristics - I ran a benchmark of memory read/write speed while this was happening once. It more than halved memory speed for the 30s or so it took from migration started to migration complete. The pause, too, was much longer.
- 4y ago
- the_duke 4y agoThis blog post is pretty old (2015) but gives a good introduction. https://cloudplatform.googleblog.com/2015/03/Google-Compute-Engine-uses-Live-Migration-technology-to-service-infrastructure-without-application-downtime.html?m=1 https://cloudplatform.googleblog.com/2015/03/Google-Compute-...
- lbhdc 4y agoThanks for sharing, I will give it a read!
- valleyjo 4y agoAzure, AWS and GCP all have live migration. VMWare has it too.
- free652 4y agoAre you sure, because AWS consistently requires me to migrate to a different host. They go as far as shutting down instances, but don't do any kind of live migrations.
- dilyevsky 4y agoEc2 does not have live migration. On azure it’s spotty so not every maintenance can offer it.
- deleted 4y ago[deleted]
- ta20200710 4y agoEC2 does support live migration, but it's not public and only for certain instance types/hypervisors. See: https://news.ycombinator.com/item?id=17815806 https://news.ycombinator.com/item?id=17815806
- _msw_ 4y agoHere's a comment that I made in a past thread. https://news.ycombinator.com/item?id=26650082 https://news.ycombinator.com/item?id=26650082
- _msw_ 4y agoDisclosure: I work for Amazon, and in the past I worked directly on EC2. From the FAQ: https://aws.amazon.com/ec2/faqs/ https://aws.amazon.com/ec2/faqs/ Q: How does EC2 perform maintenance? AWS regularly performs routine hardware, power, and network maintenance with minimal disruption across all EC2 instance types. To achieve this we employ a combination of tools and methods across the entire AWS Global infrastructure, such as redundant and concurrently maintainable systems, as well as live system updates and migration.
- darkwater 4y ago> AWS regularly performs routine hardware, power, and network maintenance with minimal disruption across all EC2 instance types. To achieve this we employ a combination of tools and methods across the entire AWS Global infrastructure, such as redundant and concurrently maintainable systems, as well as live system updates and migration. And yet, I keep getting almost every weeks emails like this: "EC2 has detected degradation of the underlying hardware hosting your Amazon EC2 instance (instance-ID: i-xxxxxxx) associated with your AWS account (AWS Account ID: NNNNN) in the eu-west-1 region. Due to this degradation your instance could already be unreachable. We will stop your instance after 2022-09-21 16:00:00 UTC" And we don't have tens of thousands of VMs in that region, just around 1k.
- _msw_ 4y agoLive migration can't be used to address every type of maintenance or underlying fault in a non-disruptive way.
- voiper1 4y agoLinode also uses live migrations now for most (all?) maintenance.