6 ms·
A few weeks ago I needed to change the volume type on an EC2 instance to gp3. Following the instructions, the change happened while the instance was running.
by politelemon 4y ago
A few weeks ago I needed to change the volume type on an EC2 instance to gp3. Following the instructions, the change happened while the instance was running. I didn't need to reboot or stop the instance, it just changed the type. While the instance was running.
I didn't understand how they were able to do this, I had thought volume types mapped to hardware clusters of some kind. And since I didn't understand, I wasn't able to distinguish it from magic.
- ArchOversight 4y agoChanging the volume type on AWS is somewhat magical. Seeing it happens on-line was amazing.
- xyzzyz 4y agoDunno about AWS, but GCP uses live migration, and will migrate your VM across physical machines as necessary. The disk volumes are all connected over the network, nothing really depends on the actual physical machine your VM is ran on.
- lbhdc 4y agoHow does migrating a vm to another physical machine work?
- rejectfinite 4y agovsphere vmotion has been a thing for years lmao
- roomey 4y agoVMware has been doing this for years, it's called vmotion and there is a lot of documentation about it if you are interested (eg https://www.thegeekpub.com/8407/how-vmotion-works/ https://www.thegeekpub.com/8407/how-vmotion-works/ ) Essential, memory state is copied to the new host, the VM is stunned for a millisecond and the cpu states is copied and resumed on the new host (you may see a dropped ping). All the networking and storage is virtual anyway so that is "moved" (it's not really moved) in the background.
- mh- 4y agoUp to 500ms per your source, depending on how much churn there is in the memory from the source system. Very cool.
- davidro80 4y ago
- lbhdc 4y agoThat is really interesting I didn't realize it was so fast. Thanks for the post I will give it a read!
- jiggawatts 4y agoThe clever trick here is that they'll pre-copy most of the memory without bothering to do it consistently, but mark pages that the source had written to as "dirty". The network cutover is stop-the-world, but VMware doesn't copy the dirty pages during the stop. Instead, it simply treats them as "swapped to pagefile", where the pagefile is actually the source machine memory. When computation resumes at the target, the source is used to page memory back in on-demand. This allows very fast cutovers.
- politelemon 4y ago> VM is stunned for a millisecond This conjures up hilarious mental imagery, thanks
- kevincox 4y agoYou just bop it on the head, and move it to the new machine quickly. By the time the VM comes to it won't even realize that it is in a new home.
- valleyjo 4y agoStream the contents of ram from source to dest, pause the source, reprogram the network and copy and memory that changed since the initial stream, resume the dest, destroy the source, profit.
- pclmulqdq 4y agoThey pause your VM, copy everything about its state over to the new machine, and quickly start the other instance. It's pretty clever. I think there are tricks you can play with machines that have large memory footprints to copy most of it before the pause, and only copy what has changed since then during the pause. The disks are all on the network, so no need to move anything there.
- prmoustache 4y agoIn reality it sync the memory first to the other host and only pause the vm when the last state sync is small enough to be so quick the pause is barely measurable.
- water-your-self 4y agoIndian jones and the register states
- lbhdc 4y agoWhen its transferring the state to the target, how does it handle memory updates that are happening at that time? Is the programs execution paused at that point?
- outworlder 4y agoNo, they keep track of dirty pages.
- GauntletWizard 4y agoNo, but the memory accesses have hooks that say "This memory was written". Then, program execution is paused, and just the sections of memory that were written are copied again. This has memory performance characteristics - I ran a benchmark of memory read/write speed while this was happening once. It more than halved memory speed for the 30s or so it took from migration started to migration complete. The pause, too, was much longer.
- 4y ago
- the_duke 4y agoThis blog post is pretty old (2015) but gives a good introduction. https://cloudplatform.googleblog.com/2015/03/Google-Compute-Engine-uses-Live-Migration-technology-to-service-infrastructure-without-application-downtime.html?m=1 https://cloudplatform.googleblog.com/2015/03/Google-Compute-...
- lbhdc 4y agoThanks for sharing, I will give it a read!
- valleyjo 4y agoAzure, AWS and GCP all have live migration. VMWare has it too.
- free652 4y agoAre you sure, because AWS consistently requires me to migrate to a different host. They go as far as shutting down instances, but don't do any kind of live migrations.
- dilyevsky 4y agoEc2 does not have live migration. On azure it’s spotty so not every maintenance can offer it.
- deleted 4y ago[deleted]
- ta20200710 4y agoEC2 does support live migration, but it's not public and only for certain instance types/hypervisors. See: https://news.ycombinator.com/item?id=17815806 https://news.ycombinator.com/item?id=17815806
- _msw_ 4y agoHere's a comment that I made in a past thread. https://news.ycombinator.com/item?id=26650082 https://news.ycombinator.com/item?id=26650082
- _msw_ 4y agoDisclosure: I work for Amazon, and in the past I worked directly on EC2. From the FAQ: https://aws.amazon.com/ec2/faqs/ https://aws.amazon.com/ec2/faqs/ Q: How does EC2 perform maintenance? AWS regularly performs routine hardware, power, and network maintenance with minimal disruption across all EC2 instance types. To achieve this we employ a combination of tools and methods across the entire AWS Global infrastructure, such as redundant and concurrently maintainable systems, as well as live system updates and migration.
- darkwater 4y ago> AWS regularly performs routine hardware, power, and network maintenance with minimal disruption across all EC2 instance types. To achieve this we employ a combination of tools and methods across the entire AWS Global infrastructure, such as redundant and concurrently maintainable systems, as well as live system updates and migration. And yet, I keep getting almost every weeks emails like this: "EC2 has detected degradation of the underlying hardware hosting your Amazon EC2 instance (instance-ID: i-xxxxxxx) associated with your AWS account (AWS Account ID: NNNNN) in the eu-west-1 region. Due to this degradation your instance could already be unreachable. We will stop your instance after 2022-09-21 16:00:00 UTC" And we don't have tens of thousands of VMs in that region, just around 1k.
- _msw_ 4y agoLive migration can't be used to address every type of maintenance or underlying fault in a non-disruptive way.
- voiper1 4y agoLinode also uses live migrations now for most (all?) maintenance.
- shrubble 4y agoAssuming this blurb is accurate: " General-purpose SSD volume (gp3) provides the consistent 125 MiB/s throughput and 3000 IOPS within the price of provisioned storage. Additional IOPS (up to 16,000) and throughput (1000 MiB/s) can be provisioned with an additional price. The General-purpose SSD volume (gp2) provides 3 IOPS per GiB storage provisioned with a minimum of 100 IOPS" ... then it seems like a device that limits bandwidth either on the storage cluster or between the node and storage cluster is present. 125MiB/s is right around the speed of a 1gbit link, I believe. That it was a networking setting changed in-switch doesn't seem to be surprising.
- nonameiguess 4y agoThis would have been my guess. All EBS volumes are stored on a physical disk that supports the highest bandwidth and IOPS you can live migrate to, and the actual rates you get are determined by something in the interconnect. Live migration is thus a matter of swapping out the interconnect between the VM and the disk or even just relaxing a logical rate-limiter, without having to migrate your data to a different disk.
- prmoustache 4y agoThe actual migration is not instantaneous despite the volume being immediately reported as gp3. You get a status change to "optimizing" if my memory is correct with a percentage. And the higher the volume the longer it takes so there is definitely a sync to faster storage.
- Salgat 4y agoIf I remember right they use the equivalent of a ledger of changes to manage volume state. So in this case, they copy over the contents (up to a certain point in time) to the new faster virtual volume, then append and direct all new changes to the new volume. This is also how they are able to snapshot a volume at a certain point in time without having any downtime or data inconsistencies.
- osti 4y agoLook up AWS Nitro on YouTube if you are interested in learning more about it.
- cavisne 4y agoEBS is already replicated so they probably just migrate behind the scenes, same as if the original physical disk was corrupted. It looks like only certain conditions allow this kindof migration. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/modify-volume-requirements.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/modify-v...