3 ms·

Network block storage isn't inherently broken, the initial EBS implementation was frankly just unreliable. We've not had anything like those dark days with Per

by boulos 10y ago

Network block storage isn't inherently broken, the initial EBS implementation was frankly just unreliable. We've not had anything like those dark days with Persistent Disk. It's still true that having your storage across the network opens you to networking failures taking out your storage, but the gain in durability and maintenance pays for it (in our case, live migration would just be crazy with local spinning disks, we tried it didn't work). Disclaimer: I work on GCE, and we want your business ;)

ngrilly 10y ago
> in our case, live migration would just be crazy with local spinning disks, we tried it didn't work It looks like Exoscale does live migrations with locally attached SSDs.
- regularfry 10y ago
  Then they'll run into the same problem anyone does doing that: migrating reasonably-sized block devices across a reasonable network takes an unreasonable length of time. Been there, done that, don't want to sit staring at consoles waiting for live disc migrations ever again.
  - ngrilly 10y ago
    My guess is that they live migrate VMs only before a planned maintenance. Let's say they use a 10 Gbps network, and they use only 1 Gbps of bandwidth for migrating data, then migrating a 200 GB disk would take something like 1 hour, which sounds OK. Can you share some details about your experience?
    - regularfry 10y ago
      You're missing a subtlety: it's a live migration. You've also got to migrate the data the VM's writing while the migration is going on. Depending on how your network's set up, it might well be possible for the VM to saturate out the migration process. Plus this won't only be planned maintenance, it'll be to get off dodgy hardware, too, including spannered RAID controllers where you really don't want to risk hanging around. Add in that you're likely going to be moving sets of discs at once (so possibly a couple of TB at a time), rather than individually, and you're very quickly looking at spending a day (or a night) at a time watching in case there's a network blip which means you need to restart any of them. This does not lend itself to a peaceful, happy existence. It doesn't take doing this very often to make you realise that this is fundamentally backwards: you want the disc data already present on more than one storage server so that if one goes pop, you're not stuffed. Once you've done that, you can make the observation that hardware RAID is no longer necessary, and save yourself a layer of complexity.
      - ngrilly 10y ago
        Thanks a lot for your detailed answer. I understand your arguments, but Google Compute Engine Local SSDs support live migration [1], which seems to prove it's possible, despite the difficulties. Any advice about this? [1] https://cloud.google.com/compute/docs/disks/#data_persistence_on_local_ssds https://cloud.google.com/compute/docs/disks/#data_persistenc...
        regularfry 10y ago
        Yeah, it's possible. Just not pleasant. What they're doing (and how unpleasant it is to operate) will depend on exactly what they mean by: > Your instance might experience a short period of decreased performance and how tolerant they are of unplanned reboots. I suspect what it means is that they'll throttle your VM (or maybe pause your IO) so it can't interfere with finalising the migration. That being said, this is Google. They've probably thrown more man-hours at this than anyone else would think sane. I'll note that this is in the context of automatic migration away from "maintenance events" (whatever that covers) - it sounds like they think they've automated away a lot of the reasons we were having to keep an eye on things, but they're still vulnerable to hardware failure (obviously).
boulos 10y ago
Too late to edit, so I'll put that up here: we do migrate your local SSD but it's best effort and it takes a while (something like 30 minutes maybe?). But for comparison, our SSDs have crazy amounts of read/write bandwidth (1 gigabyte per second) compared to a single spinning disk. And while your VM is still going, it's not that disruptive to be re-syncing an SSD, randomly seeking around a hard drive while someone is trying to use it for say streaming reads will take their reasonable performance and tank it. tl;dr: Yes, we do this for our scratch disk SSDs, but you wouldn't want to rely on this for a guaranteed durable storage (as it'll take a really really long time to move 100 GiB of HDD).

by boulos 10y ago

ngrilly 10y ago

> in our case, live migration would just be crazy with local spinning disks, we tried it didn't work It looks like Exoscale does live migrations with locally attached SSDs.

regularfry 10y ago
Then they'll run into the same problem anyone does doing that: migrating reasonably-sized block devices across a reasonable network takes an unreasonable length of time. Been there, done that, don't want to sit staring at consoles waiting for live disc migrations ever again.
- ngrilly 10y ago
  My guess is that they live migrate VMs only before a planned maintenance. Let's say they use a 10 Gbps network, and they use only 1 Gbps of bandwidth for migrating data, then migrating a 200 GB disk would take something like 1 hour, which sounds OK. Can you share some details about your experience?
  - regularfry 10y ago
    You're missing a subtlety: it's a live migration. You've also got to migrate the data the VM's writing while the migration is going on. Depending on how your network's set up, it might well be possible for the VM to saturate out the migration process. Plus this won't only be planned maintenance, it'll be to get off dodgy hardware, too, including spannered RAID controllers where you really don't want to risk hanging around. Add in that you're likely going to be moving sets of discs at once (so possibly a couple of TB at a time), rather than individually, and you're very quickly looking at spending a day (or a night) at a time watching in case there's a network blip which means you need to restart any of them. This does not lend itself to a peaceful, happy existence. It doesn't take doing this very often to make you realise that this is fundamentally backwards: you want the disc data already present on more than one storage server so that if one goes pop, you're not stuffed. Once you've done that, you can make the observation that hardware RAID is no longer necessary, and save yourself a layer of complexity.
    - ngrilly 10y ago
      Thanks a lot for your detailed answer. I understand your arguments, but Google Compute Engine Local SSDs support live migration [1], which seems to prove it's possible, despite the difficulties. Any advice about this? [1] https://cloud.google.com/compute/docs/disks/#data_persistence_on_local_ssds https://cloud.google.com/compute/docs/disks/#data_persistenc...
      - regularfry 10y ago
        Yeah, it's possible. Just not pleasant. What they're doing (and how unpleasant it is to operate) will depend on exactly what they mean by: > Your instance might experience a short period of decreased performance and how tolerant they are of unplanned reboots. I suspect what it means is that they'll throttle your VM (or maybe pause your IO) so it can't interfere with finalising the migration. That being said, this is Google. They've probably thrown more man-hours at this than anyone else would think sane. I'll note that this is in the context of automatic migration away from "maintenance events" (whatever that covers) - it sounds like they think they've automated away a lot of the reasons we were having to keep an eye on things, but they're still vulnerable to hardware failure (obviously).

boulos 10y ago

Too late to edit, so I'll put that up here: we do migrate your local SSD but it's best effort and it takes a while (something like 30 minutes maybe?). But for comparison, our SSDs have crazy amounts of read/write bandwidth (1 gigabyte per second) compared to a single spinning disk. And while your VM is still going, it's not that disruptive to be re-syncing an SSD, randomly seeking around a hard drive while someone is trying to use it for say streaming reads will take their reasonable performance and tank it. tl;dr: Yes, we do this for our scratch disk SSDs, but you wouldn't want to rely on this for a guaranteed durable storage (as it'll take a really really long time to move 100 GiB of HDD).