9 ms·

EBS Storage Improvements

73 points by yaalok 6y ago

social_quotient 6y ago
For anyone curious about the durability of gen1 (io1) I had to look it up to compare to this new tier. “All other EBS volumes are designed to deliver 99.8% - 99.9% durability” Summary - the new offering (called io2) jumps to five 9’s. Great job AWS
- dx034 6y ago
  Not sure how they determine that though? At that level, drive failures or single host failures (for which they have plenty of data available) won't cause issues. The risk remaining are botched deployments or config changes that bring down whole data centers or services. They are very unlikely, but can you really quantify their likelihood and expected downtime? 5 9's to me sounds mostly like marketing. In the end you still have to have disaster recovery plans if the service fails and you also need to test your DR plans regularly. That extra 9 won't really save any money here.
  - social_quotient 6y ago
    I tend to look at it in terms of how safe you can be while being lazy. We should have replication, DR , backups at various intervals, snapshots (as someone else mentioned take forever with big disks). When we have less than what we “should” I think the 9s paint a picture of how risky we are playing. Keeping in mind, less than should is always lazy. We gotta call it what it is.
  - singron 6y ago
    This is the same company that declared s3 has 11 9s of annual durability. They might have enough objects that they can verify that number now. In the ebs outage of 2011, volumes were unable to remirror after losing 1 mirror. That increased the probability of losing both mirrors, possibly reducing the durability below the designed rate. I.e. the designed rate assumes everything is working as designed.
LogicX 6y ago
Max IOPS is still only 64,000 though?
- vcz_fr 6y ago
  This is correct, according to the documentation [^1]. I could not find any explanation of the differences between io1 and io2 that would justify the increase in IOPS per GiB and durability on the EC2 user guide [^2] or on Amazon Builders Library [^3], though. Note that 64k is the maximum IOPS per volume, not per instance the volume is attached to. Therefore you can use software RAID configurations to increase your performance or fault tolerance further [^4] Links: [^1]: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volume-types.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volu... [^2]: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-ug.pdf#AmazonEBS https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-ug.p... [^3]: https://aws.amazon.com/builders-library/ https://aws.amazon.com/builders-library/ [^4]: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/raid-config.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/raid-con...
usr1106 6y ago
And as always no word and no link to costs. EBS is not perfectly cheap, so in real life you need tedious studies to understand whether you want or can afford the nice options available. And understanding things in advance is close to impossible. You need to test and wait for enough billing data becoming available.
- m0xte 6y ago
  It’s expensive and a nightmare of cost complexity which leads to an entire meta-industry of cost management which I’m sure is a net loss. I’m really more concerned about the absolutely miserably shitty performance of snapshots myself which always goes unmentioned and unmaintained. So I’ve got a 4TiB volume online attached to a node which is blasting out fragmented writes galore (Jenkins - ugh). So we need to back up this volume for DR. Snapshots right? Well not when it takes 28 hours to snapshot that volume. So off to rdiff-backup it is. Back in the last infrastructure cult of virtualisation this was a 20 minute turnaround. Good job it’s 5 nines now. That means we only risk a 3 day recovery window 2000x less often. Actually that hurt writing that. I’m sure someone will now immediately sell me EKS to run Jenkins now which is ten fresh layers of eye gouging pain between idea to delivering software, something which I barely remember being able to do efficiently at this point.
  - codeduck 6y ago
    Heh. I feel your pain. EKS is a nice platform, but I have seen the light and try not to run anything that requires a persistent file system on it. I feel like I'm shouting in the wilderness though- just adout every day is a "can't we deploy mysql here?"
    - m0xte 6y ago
      Yes I’m a big proponent of stateless stuff. In fact 20 years ago I was crowing this to a deaf audience. Instead I find queues in databases that didn’t even need to exist. And our solutions, kubernetes for example, are complicated Rube Goldberg machines rather than things that make us sleep easy. However really the software industry is all about state and where to keep it because that’s where the trolls collect their gold coins. So what can we do?
siscia 6y ago
Somebody know what is the architecture behind those volumes? How do they work?
fideloper 6y ago
For RDS, the “trick” to getting max iops in gp2 cheaply is to get over 5334GB of storage, giving you the baseline of 16,000 iops (no burst at that level)[1]. This costs roughly $615 a month (double for multi az), way cheaper than io2 at that level. It looks like 16000 provisioned iops in RDS is more like $2266 for 5334gb (double for multi az) a month. Varies by storage amount of course - 500gb storage is $1600 for single-az at 16k iops. I’m not sure this change affects RDS storage pricing tho? The pricing calculator still calls Provisioned IOPs "IO1". [1] https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Storage.html https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_...
- jsmeaton 6y ago
  .. so it's cheaper to provision more disk and get free iops than to get a smaller disk and pay for provisioned iops? I recently bumped iops to 12k on one of my RDS instances, and I had to cringe when I did it.
- BillinghamJ 6y ago
  Bit of a one-way thing though, you can't reduce the amount of storage allocated on an RDS instance without downtime (at least with Postgres)
aclelland 6y ago
Will this new storage layer be available for AWS services like RDS and ElasticSearch? I checked the dashboards and the SSD options are still just "General Purpose SSD" or "Provisioned SSD". My understanding is that these services use EBS under the hood but I guess there is no way to check if the attached storage is using io1 or io2 for now?
- fideloper 6y ago
  Provisioned IOPs are io1 still according to the pricing calculator (to my knowledge, it is EBS under the hood).

73 points by yaalok 6y ago

For anyone curious about the durability of gen1 (io1) I had to look it up to compare to this new tier. “All other EBS volumes are designed to deliver 99.8% - 99.9% durability” Summary - the new offering (called io2) jumps to five 9’s. Great job AWS

dx034 6y ago
Not sure how they determine that though? At that level, drive failures or single host failures (for which they have plenty of data available) won't cause issues. The risk remaining are botched deployments or config changes that bring down whole data centers or services. They are very unlikely, but can you really quantify their likelihood and expected downtime? 5 9's to me sounds mostly like marketing. In the end you still have to have disaster recovery plans if the service fails and you also need to test your DR plans regularly. That extra 9 won't really save any money here.
- social_quotient 6y ago
  I tend to look at it in terms of how safe you can be while being lazy. We should have replication, DR , backups at various intervals, snapshots (as someone else mentioned take forever with big disks). When we have less than what we “should” I think the 9s paint a picture of how risky we are playing. Keeping in mind, less than should is always lazy. We gotta call it what it is.
- singron 6y ago
  This is the same company that declared s3 has 11 9s of annual durability. They might have enough objects that they can verify that number now. In the ebs outage of 2011, volumes were unable to remirror after losing 1 mirror. That increased the probability of losing both mirrors, possibly reducing the durability below the designed rate. I.e. the designed rate assumes everything is working as designed.

LogicX 6y ago

Max IOPS is still only 64,000 though?

vcz_fr 6y ago
This is correct, according to the documentation [^1]. I could not find any explanation of the differences between io1 and io2 that would justify the increase in IOPS per GiB and durability on the EC2 user guide [^2] or on Amazon Builders Library [^3], though. Note that 64k is the maximum IOPS per volume, not per instance the volume is attached to. Therefore you can use software RAID configurations to increase your performance or fault tolerance further [^4] Links: [^1]: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volume-types.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volu... [^2]: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-ug.pdf#AmazonEBS https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-ug.p... [^3]: https://aws.amazon.com/builders-library/ https://aws.amazon.com/builders-library/ [^4]: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/raid-config.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/raid-con...

usr1106 6y ago

And as always no word and no link to costs. EBS is not perfectly cheap, so in real life you need tedious studies to understand whether you want or can afford the nice options available. And understanding things in advance is close to impossible. You need to test and wait for enough billing data becoming available.

m0xte 6y ago
It’s expensive and a nightmare of cost complexity which leads to an entire meta-industry of cost management which I’m sure is a net loss. I’m really more concerned about the absolutely miserably shitty performance of snapshots myself which always goes unmentioned and unmaintained. So I’ve got a 4TiB volume online attached to a node which is blasting out fragmented writes galore (Jenkins - ugh). So we need to back up this volume for DR. Snapshots right? Well not when it takes 28 hours to snapshot that volume. So off to rdiff-backup it is. Back in the last infrastructure cult of virtualisation this was a 20 minute turnaround. Good job it’s 5 nines now. That means we only risk a 3 day recovery window 2000x less often. Actually that hurt writing that. I’m sure someone will now immediately sell me EKS to run Jenkins now which is ten fresh layers of eye gouging pain between idea to delivering software, something which I barely remember being able to do efficiently at this point.
- codeduck 6y ago
  Heh. I feel your pain. EKS is a nice platform, but I have seen the light and try not to run anything that requires a persistent file system on it. I feel like I'm shouting in the wilderness though- just adout every day is a "can't we deploy mysql here?"
  - m0xte 6y ago
    Yes I’m a big proponent of stateless stuff. In fact 20 years ago I was crowing this to a deaf audience. Instead I find queues in databases that didn’t even need to exist. And our solutions, kubernetes for example, are complicated Rube Goldberg machines rather than things that make us sleep easy. However really the software industry is all about state and where to keep it because that’s where the trolls collect their gold coins. So what can we do?

siscia 6y ago

Somebody know what is the architecture behind those volumes? How do they work?

fideloper 6y ago

For RDS, the “trick” to getting max iops in gp2 cheaply is to get over 5334GB of storage, giving you the baseline of 16,000 iops (no burst at that level)[1]. This costs roughly $615 a month (double for multi az), way cheaper than io2 at that level. It looks like 16000 provisioned iops in RDS is more like $2266 for 5334gb (double for multi az) a month. Varies by storage amount of course - 500gb storage is $1600 for single-az at 16k iops. I’m not sure this change affects RDS storage pricing tho? The pricing calculator still calls Provisioned IOPs "IO1". [1] https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Storage.html https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_...

jsmeaton 6y ago
.. so it's cheaper to provision more disk and get free iops than to get a smaller disk and pay for provisioned iops? I recently bumped iops to 12k on one of my RDS instances, and I had to cringe when I did it.
BillinghamJ 6y ago
Bit of a one-way thing though, you can't reduce the amount of storage allocated on an RDS instance without downtime (at least with Postgres)

aclelland 6y ago

Will this new storage layer be available for AWS services like RDS and ElasticSearch? I checked the dashboards and the SSD options are still just "General Purpose SSD" or "Provisioned SSD". My understanding is that these services use EBS under the hood but I guess there is no way to check if the attached storage is using io1 or io2 for now?

fideloper 6y ago
Provisioned IOPs are io1 still according to the pricing calculator (to my knowledge, it is EBS under the hood).