6 ms·

My guess is that they live migrate VMs only before a planned maintenance. Let's say they use a 10 Gbps network, and they use only 1 Gbps of bandwidth for migrat

by ngrilly 10y ago

My guess is that they live migrate VMs only before a planned maintenance. Let's say they use a 10 Gbps network, and they use only 1 Gbps of bandwidth for migrating data, then migrating a 200 GB disk would take something like 1 hour, which sounds OK. Can you share some details about your experience?

regularfry 10y ago
You're missing a subtlety: it's a live migration. You've also got to migrate the data the VM's writing while the migration is going on. Depending on how your network's set up, it might well be possible for the VM to saturate out the migration process. Plus this won't only be planned maintenance, it'll be to get off dodgy hardware, too, including spannered RAID controllers where you really don't want to risk hanging around. Add in that you're likely going to be moving sets of discs at once (so possibly a couple of TB at a time), rather than individually, and you're very quickly looking at spending a day (or a night) at a time watching in case there's a network blip which means you need to restart any of them. This does not lend itself to a peaceful, happy existence. It doesn't take doing this very often to make you realise that this is fundamentally backwards: you want the disc data already present on more than one storage server so that if one goes pop, you're not stuffed. Once you've done that, you can make the observation that hardware RAID is no longer necessary, and save yourself a layer of complexity.
- ngrilly 10y ago
  Thanks a lot for your detailed answer. I understand your arguments, but Google Compute Engine Local SSDs support live migration [1], which seems to prove it's possible, despite the difficulties. Any advice about this? [1] https://cloud.google.com/compute/docs/disks/#data_persistence_on_local_ssds https://cloud.google.com/compute/docs/disks/#data_persistenc...
  - regularfry 10y ago
    Yeah, it's possible. Just not pleasant. What they're doing (and how unpleasant it is to operate) will depend on exactly what they mean by: > Your instance might experience a short period of decreased performance and how tolerant they are of unplanned reboots. I suspect what it means is that they'll throttle your VM (or maybe pause your IO) so it can't interfere with finalising the migration. That being said, this is Google. They've probably thrown more man-hours at this than anyone else would think sane. I'll note that this is in the context of automatic migration away from "maintenance events" (whatever that covers) - it sounds like they think they've automated away a lot of the reasons we were having to keep an eye on things, but they're still vulnerable to hardware failure (obviously).

by ngrilly 10y ago

regularfry 10y ago

You're missing a subtlety: it's a live migration. You've also got to migrate the data the VM's writing while the migration is going on. Depending on how your network's set up, it might well be possible for the VM to saturate out the migration process. Plus this won't only be planned maintenance, it'll be to get off dodgy hardware, too, including spannered RAID controllers where you really don't want to risk hanging around. Add in that you're likely going to be moving sets of discs at once (so possibly a couple of TB at a time), rather than individually, and you're very quickly looking at spending a day (or a night) at a time watching in case there's a network blip which means you need to restart any of them. This does not lend itself to a peaceful, happy existence. It doesn't take doing this very often to make you realise that this is fundamentally backwards: you want the disc data already present on more than one storage server so that if one goes pop, you're not stuffed. Once you've done that, you can make the observation that hardware RAID is no longer necessary, and save yourself a layer of complexity.

ngrilly 10y ago
Thanks a lot for your detailed answer. I understand your arguments, but Google Compute Engine Local SSDs support live migration [1], which seems to prove it's possible, despite the difficulties. Any advice about this? [1] https://cloud.google.com/compute/docs/disks/#data_persistence_on_local_ssds https://cloud.google.com/compute/docs/disks/#data_persistenc...
- regularfry 10y ago
  Yeah, it's possible. Just not pleasant. What they're doing (and how unpleasant it is to operate) will depend on exactly what they mean by: > Your instance might experience a short period of decreased performance and how tolerant they are of unplanned reboots. I suspect what it means is that they'll throttle your VM (or maybe pause your IO) so it can't interfere with finalising the migration. That being said, this is Google. They've probably thrown more man-hours at this than anyone else would think sane. I'll note that this is in the context of automatic migration away from "maintenance events" (whatever that covers) - it sounds like they think they've automated away a lot of the reasons we were having to keep an eye on things, but they're still vulnerable to hardware failure (obviously).