6 ms·

And as always no word and no link to costs. EBS is not perfectly cheap, so in real life you need tedious studies to understand whether you want or can afford th

And as always no word and no link to costs. EBS is not perfectly cheap, so in real life you need tedious studies to understand whether you want or can afford the nice options available. And understanding things in advance is close to impossible. You need to test and wait for enough billing data becoming available.

m0xte 6y ago
It’s expensive and a nightmare of cost complexity which leads to an entire meta-industry of cost management which I’m sure is a net loss. I’m really more concerned about the absolutely miserably shitty performance of snapshots myself which always goes unmentioned and unmaintained. So I’ve got a 4TiB volume online attached to a node which is blasting out fragmented writes galore (Jenkins - ugh). So we need to back up this volume for DR. Snapshots right? Well not when it takes 28 hours to snapshot that volume. So off to rdiff-backup it is. Back in the last infrastructure cult of virtualisation this was a 20 minute turnaround. Good job it’s 5 nines now. That means we only risk a 3 day recovery window 2000x less often. Actually that hurt writing that. I’m sure someone will now immediately sell me EKS to run Jenkins now which is ten fresh layers of eye gouging pain between idea to delivering software, something which I barely remember being able to do efficiently at this point.
- codeduck 6y ago
  Heh. I feel your pain. EKS is a nice platform, but I have seen the light and try not to run anything that requires a persistent file system on it. I feel like I'm shouting in the wilderness though- just adout every day is a "can't we deploy mysql here?"
  - m0xte 6y ago
    Yes I’m a big proponent of stateless stuff. In fact 20 years ago I was crowing this to a deaf audience. Instead I find queues in databases that didn’t even need to exist. And our solutions, kubernetes for example, are complicated Rube Goldberg machines rather than things that make us sleep easy. However really the software industry is all about state and where to keep it because that’s where the trolls collect their gold coins. So what can we do?
    - codeduck 6y ago
      I find gin helps to numb the pain...
    - simiones 6y ago
      So where would a queue go in a distributed system if not in some kind of database/file system? How do you handle resilience if the queue is not stored on disk somewhere? Paxos?
      - m0xte 6y ago
        Well the point is more that a lot of stuff doesn't even need to be in a queue. It's just cookie cutter engineering. "I did this before and it worked so I'll do it again" without any consideration on what it does and why it does it.
  - chrisbolt 6y ago
    Is this an EKS problem or a Kubernetes problem?
    - user5994461 6y ago
      Both. Containers are meant to be ephemeral, they don't play well with permanent storage by design. People have been trying to run databases anyway and hitting roadblock after roadblock. Here's an old rant mentioning broken container filesystems, a bit old from 2016 but storage didn't change much https://thehftguy.com/2016/11/01/docker-in-production-an-history-of-failure/ https://thehftguy.com/2016/11/01/docker-in-production-an-his...
      - cbsmith 6y ago
        Unless they are sharded and replicated distributed databases...
      - rad_gruchalski 6y ago
        > Both. Containers are meant to be ephemeral, they don't play well with permanent storage by design. People have been trying to run databases anyway and hitting roadblock after roadblock. Really? Can you share details? I'm asking because I know people who run databases in Docker for years and have no issues. Just store data on a volume. > Here's an old rant mentioning broken container filesystems, a bit old from 2016 but storage didn't change much Maybe not, haven't had a storage issue with Docker myself since maybe 2017. That article feels like a big rant but yes, it's ... old.
        Townley 6y ago
        Docker volumes are great. I've also run databases in docker without issue. But they're different from kubernetes "persistent volumes" and "persistent volume claims" The issues come when you move into high availability (specifically in a Kubernetes context). K8s may decide to move your database to a different node (server) on a whim, and suddenly your data is gone. You have to solve this by having mounted, block-level storage available to your cluster (eg AWS's EBS; the product being discussed) so you can use persistent volumes. That comes with added complexity, and usually the need to incorporate db operators into your stack. As a result, a lot of k8s admins (myself included) choose to pay crazy amounts for RDS and S3 to avoid the complexities of persistent data.
- joncrane 6y ago
  I was with you until >entire meta-industry of cost management which I’m sure is a net loss. I am paid hourly and I have literally, in the past week, billed ~14 hours to save ~$150,000 per month. We work for a client and we strive to make sure we're not spending more money in labor than we're saving in AWS costs. I don't know exactly the rate I'm billed at that the client sees, but I know for sure that on aggregate, aside from the actual productive work that I know adds value, I have justified every penny I've been paid in the last ~2 years through my cost savings work alone. Maybe some folks overcharge or undersave, but I take exception to your comment that you're "sure is a net loss." Edit: our monthly AWS spend is in the 7 figures. We go through cycles where our priorities change, but when we're in cost savings mode, we regularly discover cost savings measure that are a) implementable in double digit hours and b) save six figures per month.
  - m0xte 6y ago
    I’m not saying that it doesn’t result in a saving but that the fact that industry has to exist to keep costs constrained is a net loss. Imagine if pricing was transparent, calculable and easy to rationalise the cost of a change. But it isn’t unless you pull a third party in.
usr1106 6y ago
My older volumes are of type "standard", but that type is not on the price list. I cannot even send a link to the price list, because the region is not in the URL. https://aws.amazon.com/ebs/pricing/ https://aws.amazon.com/ebs/pricing/ Complete lack of transparency.
- BillinghamJ 6y ago
  It's not all that hard to find older generation pricing really... it's the same on every AWS product page. Many AWS products have thousands of older generation offerings which they don't want cluttering up the marketing pages. https://aws.amazon.com/ebs/previous-generation/ https://aws.amazon.com/ebs/previous-generation/ Also easy to see on the pricing calculator
- Androider 6y ago
  "standard" volumes are legacy. If you search for "previous generation" on a given service pricing page you'll usually find a link to the old stuff. In this case, https://aws.amazon.com/ebs/previous-generation/ https://aws.amazon.com/ebs/previous-generation/ Unless you're specifically looking for high-IOPS or massive HDD volume, you should just default to "gp2" SSD volumes these days.
vasco 6y ago
"Today we are launching the io2 volume type, with two important benefits, at the same price as the existing io1 volumes"
- usr1106 6y ago
  The same price as something that has been difficult to grasp before does not really help. My comment was not about the pricing of io2 in particular, but about the pricing of EBS in general.
  - phonon 6y ago
    Eh, it's not so complicated. io1/io2 is about $12/month per 100 GB, and $60/m per 1000 IOPS. So 200 GB and 10,000 iops would be $624 With the caveat that (io2) cannot support more than 500 IOPS per GB. gp2 is $10/m per 100 GB, and you get 3 IOPS/GB (with potential to burst to 3000 IOPS for volumes under 1 TB). GP2 IOPS max out at 16,000 per volume. io1/io2 IOPS max out at 64,000 Basically, use io2 if you need more than 16,000 IOPS. or you need a large amount of IOPS for only a small amount of storage. io2 also has a 0.001% annual failure rate, while the others have a .2% failure rate. E.g., to get 10,000 IOPS with gp2, you would get a 3.33 TB volume, which would cost $333/m. Local storage is far, far faster (up to millions of IOPS!)...but not as reliable, and does not have all the EBS features. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/storage-optimized-instances.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/storage-...
    - dx034 6y ago
      > io2 also has a 0.001% annual failure rate, while the others have a .2% failure rate. Is failing once every 5 years really different from failing once every 1000 years? You should have disaster recovery plans anyway and configuration errors are the most likely cause of failure, not the underlying storage system.
      - phonon 6y ago
        For companies that have a few hundred of these running at a time, it sure does! Also, it's a 2000x ratio, not 200..., and 1/500 years vs. 1/1,000,000 years.
jdub 6y ago
Literally in the first sentence of the new volume type intro: "… at the same price as the existing io1 volumes".
willvarfar 6y ago
"Today we are launching the io2 volume type, with two important benefits, at the same price as the existing io1 volumes"
skywhopper 6y ago
The cost isn't hard to find, Google "ebs pricing". There's plenty of things to complain about AWS billing, but EBS pricing is one of the simplest things to deal with. But if you aren't sure if you need these, you probably don't.
- usr1106 6y ago
  "ebs pricing" search does not find standard, at least not in top rank, because standard is not on the ebs pricing page. You need to know to add "previous generation" or "legacy" or something like that to your search terms. Easy when you remember it, but not great transparency from their side.
- usr1106 6y ago
  > But if you aren't sure if you need these, you probably don't. True. When inspired by this discussion checking out what I pay for a moment I had this oh shit experience for a moment, I pay twice as much as the cheapest option. When trying to convert the first volume I noticed that the minimum size for that is 500 GB, that's 100 times my volume size, 50 times the prize, so I really don't need it. However, as someone pointed out somewhere else in this discussion there might be a more suitable option on the low end called AWS lightsail.

m0xte 6y ago

It’s expensive and a nightmare of cost complexity which leads to an entire meta-industry of cost management which I’m sure is a net loss. I’m really more concerned about the absolutely miserably shitty performance of snapshots myself which always goes unmentioned and unmaintained. So I’ve got a 4TiB volume online attached to a node which is blasting out fragmented writes galore (Jenkins - ugh). So we need to back up this volume for DR. Snapshots right? Well not when it takes 28 hours to snapshot that volume. So off to rdiff-backup it is. Back in the last infrastructure cult of virtualisation this was a 20 minute turnaround. Good job it’s 5 nines now. That means we only risk a 3 day recovery window 2000x less often. Actually that hurt writing that. I’m sure someone will now immediately sell me EKS to run Jenkins now which is ten fresh layers of eye gouging pain between idea to delivering software, something which I barely remember being able to do efficiently at this point.

codeduck 6y ago
Heh. I feel your pain. EKS is a nice platform, but I have seen the light and try not to run anything that requires a persistent file system on it. I feel like I'm shouting in the wilderness though- just adout every day is a "can't we deploy mysql here?"
- m0xte 6y ago
  Yes I’m a big proponent of stateless stuff. In fact 20 years ago I was crowing this to a deaf audience. Instead I find queues in databases that didn’t even need to exist. And our solutions, kubernetes for example, are complicated Rube Goldberg machines rather than things that make us sleep easy. However really the software industry is all about state and where to keep it because that’s where the trolls collect their gold coins. So what can we do?
  - codeduck 6y ago
    I find gin helps to numb the pain...
  - simiones 6y ago
    So where would a queue go in a distributed system if not in some kind of database/file system? How do you handle resilience if the queue is not stored on disk somewhere? Paxos?
    - m0xte 6y ago
      Well the point is more that a lot of stuff doesn't even need to be in a queue. It's just cookie cutter engineering. "I did this before and it worked so I'll do it again" without any consideration on what it does and why it does it.
- chrisbolt 6y ago
  Is this an EKS problem or a Kubernetes problem?
  - user5994461 6y ago
    Both. Containers are meant to be ephemeral, they don't play well with permanent storage by design. People have been trying to run databases anyway and hitting roadblock after roadblock. Here's an old rant mentioning broken container filesystems, a bit old from 2016 but storage didn't change much https://thehftguy.com/2016/11/01/docker-in-production-an-history-of-failure/ https://thehftguy.com/2016/11/01/docker-in-production-an-his...
    - cbsmith 6y ago
      Unless they are sharded and replicated distributed databases...
    - rad_gruchalski 6y ago
      > Both. Containers are meant to be ephemeral, they don't play well with permanent storage by design. People have been trying to run databases anyway and hitting roadblock after roadblock. Really? Can you share details? I'm asking because I know people who run databases in Docker for years and have no issues. Just store data on a volume. > Here's an old rant mentioning broken container filesystems, a bit old from 2016 but storage didn't change much Maybe not, haven't had a storage issue with Docker myself since maybe 2017. That article feels like a big rant but yes, it's ... old.
      - Townley 6y ago
        Docker volumes are great. I've also run databases in docker without issue. But they're different from kubernetes "persistent volumes" and "persistent volume claims" The issues come when you move into high availability (specifically in a Kubernetes context). K8s may decide to move your database to a different node (server) on a whim, and suddenly your data is gone. You have to solve this by having mounted, block-level storage available to your cluster (eg AWS's EBS; the product being discussed) so you can use persistent volumes. That comes with added complexity, and usually the need to incorporate db operators into your stack. As a result, a lot of k8s admins (myself included) choose to pay crazy amounts for RDS and S3 to avoid the complexities of persistent data.
joncrane 6y ago
I was with you until >entire meta-industry of cost management which I’m sure is a net loss. I am paid hourly and I have literally, in the past week, billed ~14 hours to save ~$150,000 per month. We work for a client and we strive to make sure we're not spending more money in labor than we're saving in AWS costs. I don't know exactly the rate I'm billed at that the client sees, but I know for sure that on aggregate, aside from the actual productive work that I know adds value, I have justified every penny I've been paid in the last ~2 years through my cost savings work alone. Maybe some folks overcharge or undersave, but I take exception to your comment that you're "sure is a net loss." Edit: our monthly AWS spend is in the 7 figures. We go through cycles where our priorities change, but when we're in cost savings mode, we regularly discover cost savings measure that are a) implementable in double digit hours and b) save six figures per month.
- m0xte 6y ago
  I’m not saying that it doesn’t result in a saving but that the fact that industry has to exist to keep costs constrained is a net loss. Imagine if pricing was transparent, calculable and easy to rationalise the cost of a change. But it isn’t unless you pull a third party in.

usr1106 6y ago

My older volumes are of type "standard", but that type is not on the price list. I cannot even send a link to the price list, because the region is not in the URL. https://aws.amazon.com/ebs/pricing/ https://aws.amazon.com/ebs/pricing/ Complete lack of transparency.

BillinghamJ 6y ago
It's not all that hard to find older generation pricing really... it's the same on every AWS product page. Many AWS products have thousands of older generation offerings which they don't want cluttering up the marketing pages. https://aws.amazon.com/ebs/previous-generation/ https://aws.amazon.com/ebs/previous-generation/ Also easy to see on the pricing calculator
Androider 6y ago
"standard" volumes are legacy. If you search for "previous generation" on a given service pricing page you'll usually find a link to the old stuff. In this case, https://aws.amazon.com/ebs/previous-generation/ https://aws.amazon.com/ebs/previous-generation/ Unless you're specifically looking for high-IOPS or massive HDD volume, you should just default to "gp2" SSD volumes these days.

vasco 6y ago

"Today we are launching the io2 volume type, with two important benefits, at the same price as the existing io1 volumes"

usr1106 6y ago
The same price as something that has been difficult to grasp before does not really help. My comment was not about the pricing of io2 in particular, but about the pricing of EBS in general.
- phonon 6y ago
  Eh, it's not so complicated. io1/io2 is about $12/month per 100 GB, and $60/m per 1000 IOPS. So 200 GB and 10,000 iops would be $624 With the caveat that (io2) cannot support more than 500 IOPS per GB. gp2 is $10/m per 100 GB, and you get 3 IOPS/GB (with potential to burst to 3000 IOPS for volumes under 1 TB). GP2 IOPS max out at 16,000 per volume. io1/io2 IOPS max out at 64,000 Basically, use io2 if you need more than 16,000 IOPS. or you need a large amount of IOPS for only a small amount of storage. io2 also has a 0.001% annual failure rate, while the others have a .2% failure rate. E.g., to get 10,000 IOPS with gp2, you would get a 3.33 TB volume, which would cost $333/m. Local storage is far, far faster (up to millions of IOPS!)...but not as reliable, and does not have all the EBS features. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/storage-optimized-instances.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/storage-...
  - dx034 6y ago
    > io2 also has a 0.001% annual failure rate, while the others have a .2% failure rate. Is failing once every 5 years really different from failing once every 1000 years? You should have disaster recovery plans anyway and configuration errors are the most likely cause of failure, not the underlying storage system.
    - phonon 6y ago
      For companies that have a few hundred of these running at a time, it sure does! Also, it's a 2000x ratio, not 200..., and 1/500 years vs. 1/1,000,000 years.

jdub 6y ago

Literally in the first sentence of the new volume type intro: "… at the same price as the existing io1 volumes".

willvarfar 6y ago

"Today we are launching the io2 volume type, with two important benefits, at the same price as the existing io1 volumes"

skywhopper 6y ago

The cost isn't hard to find, Google "ebs pricing". There's plenty of things to complain about AWS billing, but EBS pricing is one of the simplest things to deal with. But if you aren't sure if you need these, you probably don't.

usr1106 6y ago
"ebs pricing" search does not find standard, at least not in top rank, because standard is not on the ebs pricing page. You need to know to add "previous generation" or "legacy" or something like that to your search terms. Easy when you remember it, but not great transparency from their side.
usr1106 6y ago
> But if you aren't sure if you need these, you probably don't. True. When inspired by this discussion checking out what I pay for a moment I had this oh shit experience for a moment, I pay twice as much as the cheapest option. When trying to convert the first volume I noticed that the minimum size for that is 500 GB, that's 100 times my volume size, 50 times the prize, so I really don't need it. However, as someone pointed out somewhere else in this discussion there might be a more suitable option on the low end called AWS lightsail.