4 ms·

It’s expensive and a nightmare of cost complexity which leads to an entire meta-industry of cost management which I’m sure is a net loss. I’m really more conce

by m0xte 6y ago

It’s expensive and a nightmare of cost complexity which leads to an entire meta-industry of cost management which I’m sure is a net loss. I’m really more concerned about the absolutely miserably shitty performance of snapshots myself which always goes unmentioned and unmaintained. So I’ve got a 4TiB volume online attached to a node which is blasting out fragmented writes galore (Jenkins - ugh). So we need to back up this volume for DR. Snapshots right? Well not when it takes 28 hours to snapshot that volume. So off to rdiff-backup it is. Back in the last infrastructure cult of virtualisation this was a 20 minute turnaround. Good job it’s 5 nines now. That means we only risk a 3 day recovery window 2000x less often. Actually that hurt writing that. I’m sure someone will now immediately sell me EKS to run Jenkins now which is ten fresh layers of eye gouging pain between idea to delivering software, something which I barely remember being able to do efficiently at this point.

codeduck 6y ago
Heh. I feel your pain. EKS is a nice platform, but I have seen the light and try not to run anything that requires a persistent file system on it. I feel like I'm shouting in the wilderness though- just adout every day is a "can't we deploy mysql here?"
- m0xte 6y ago
  Yes I’m a big proponent of stateless stuff. In fact 20 years ago I was crowing this to a deaf audience. Instead I find queues in databases that didn’t even need to exist. And our solutions, kubernetes for example, are complicated Rube Goldberg machines rather than things that make us sleep easy. However really the software industry is all about state and where to keep it because that’s where the trolls collect their gold coins. So what can we do?
  - codeduck 6y ago
    I find gin helps to numb the pain...
  - simiones 6y ago
    So where would a queue go in a distributed system if not in some kind of database/file system? How do you handle resilience if the queue is not stored on disk somewhere? Paxos?
    - m0xte 6y ago
      Well the point is more that a lot of stuff doesn't even need to be in a queue. It's just cookie cutter engineering. "I did this before and it worked so I'll do it again" without any consideration on what it does and why it does it.
- chrisbolt 6y ago
  Is this an EKS problem or a Kubernetes problem?
  - user5994461 6y ago
    Both. Containers are meant to be ephemeral, they don't play well with permanent storage by design. People have been trying to run databases anyway and hitting roadblock after roadblock. Here's an old rant mentioning broken container filesystems, a bit old from 2016 but storage didn't change much https://thehftguy.com/2016/11/01/docker-in-production-an-history-of-failure/ https://thehftguy.com/2016/11/01/docker-in-production-an-his...
    - cbsmith 6y ago
      Unless they are sharded and replicated distributed databases...
    - rad_gruchalski 6y ago
      > Both. Containers are meant to be ephemeral, they don't play well with permanent storage by design. People have been trying to run databases anyway and hitting roadblock after roadblock. Really? Can you share details? I'm asking because I know people who run databases in Docker for years and have no issues. Just store data on a volume. > Here's an old rant mentioning broken container filesystems, a bit old from 2016 but storage didn't change much Maybe not, haven't had a storage issue with Docker myself since maybe 2017. That article feels like a big rant but yes, it's ... old.
      - Townley 6y ago
        Docker volumes are great. I've also run databases in docker without issue. But they're different from kubernetes "persistent volumes" and "persistent volume claims" The issues come when you move into high availability (specifically in a Kubernetes context). K8s may decide to move your database to a different node (server) on a whim, and suddenly your data is gone. You have to solve this by having mounted, block-level storage available to your cluster (eg AWS's EBS; the product being discussed) so you can use persistent volumes. That comes with added complexity, and usually the need to incorporate db operators into your stack. As a result, a lot of k8s admins (myself included) choose to pay crazy amounts for RDS and S3 to avoid the complexities of persistent data.
joncrane 6y ago
I was with you until >entire meta-industry of cost management which I’m sure is a net loss. I am paid hourly and I have literally, in the past week, billed ~14 hours to save ~$150,000 per month. We work for a client and we strive to make sure we're not spending more money in labor than we're saving in AWS costs. I don't know exactly the rate I'm billed at that the client sees, but I know for sure that on aggregate, aside from the actual productive work that I know adds value, I have justified every penny I've been paid in the last ~2 years through my cost savings work alone. Maybe some folks overcharge or undersave, but I take exception to your comment that you're "sure is a net loss." Edit: our monthly AWS spend is in the 7 figures. We go through cycles where our priorities change, but when we're in cost savings mode, we regularly discover cost savings measure that are a) implementable in double digit hours and b) save six figures per month.
- m0xte 6y ago
  I’m not saying that it doesn’t result in a saving but that the fact that industry has to exist to keep costs constrained is a net loss. Imagine if pricing was transparent, calculable and easy to rationalise the cost of a change. But it isn’t unless you pull a third party in.

by m0xte 6y ago

codeduck 6y ago

Heh. I feel your pain. EKS is a nice platform, but I have seen the light and try not to run anything that requires a persistent file system on it. I feel like I'm shouting in the wilderness though- just adout every day is a "can't we deploy mysql here?"

m0xte 6y ago
Yes I’m a big proponent of stateless stuff. In fact 20 years ago I was crowing this to a deaf audience. Instead I find queues in databases that didn’t even need to exist. And our solutions, kubernetes for example, are complicated Rube Goldberg machines rather than things that make us sleep easy. However really the software industry is all about state and where to keep it because that’s where the trolls collect their gold coins. So what can we do?
- codeduck 6y ago
  I find gin helps to numb the pain...
- simiones 6y ago
  So where would a queue go in a distributed system if not in some kind of database/file system? How do you handle resilience if the queue is not stored on disk somewhere? Paxos?
  - m0xte 6y ago
    Well the point is more that a lot of stuff doesn't even need to be in a queue. It's just cookie cutter engineering. "I did this before and it worked so I'll do it again" without any consideration on what it does and why it does it.
chrisbolt 6y ago
Is this an EKS problem or a Kubernetes problem?
- user5994461 6y ago
  Both. Containers are meant to be ephemeral, they don't play well with permanent storage by design. People have been trying to run databases anyway and hitting roadblock after roadblock. Here's an old rant mentioning broken container filesystems, a bit old from 2016 but storage didn't change much https://thehftguy.com/2016/11/01/docker-in-production-an-history-of-failure/ https://thehftguy.com/2016/11/01/docker-in-production-an-his...
  - cbsmith 6y ago
    Unless they are sharded and replicated distributed databases...
  - rad_gruchalski 6y ago
    > Both. Containers are meant to be ephemeral, they don't play well with permanent storage by design. People have been trying to run databases anyway and hitting roadblock after roadblock. Really? Can you share details? I'm asking because I know people who run databases in Docker for years and have no issues. Just store data on a volume. > Here's an old rant mentioning broken container filesystems, a bit old from 2016 but storage didn't change much Maybe not, haven't had a storage issue with Docker myself since maybe 2017. That article feels like a big rant but yes, it's ... old.
    - Townley 6y ago
      Docker volumes are great. I've also run databases in docker without issue. But they're different from kubernetes "persistent volumes" and "persistent volume claims" The issues come when you move into high availability (specifically in a Kubernetes context). K8s may decide to move your database to a different node (server) on a whim, and suddenly your data is gone. You have to solve this by having mounted, block-level storage available to your cluster (eg AWS's EBS; the product being discussed) so you can use persistent volumes. That comes with added complexity, and usually the need to incorporate db operators into your stack. As a result, a lot of k8s admins (myself included) choose to pay crazy amounts for RDS and S3 to avoid the complexities of persistent data.

joncrane 6y ago

I was with you until >entire meta-industry of cost management which I’m sure is a net loss. I am paid hourly and I have literally, in the past week, billed ~14 hours to save ~$150,000 per month. We work for a client and we strive to make sure we're not spending more money in labor than we're saving in AWS costs. I don't know exactly the rate I'm billed at that the client sees, but I know for sure that on aggregate, aside from the actual productive work that I know adds value, I have justified every penny I've been paid in the last ~2 years through my cost savings work alone. Maybe some folks overcharge or undersave, but I take exception to your comment that you're "sure is a net loss." Edit: our monthly AWS spend is in the 7 figures. We go through cycles where our priorities change, but when we're in cost savings mode, we regularly discover cost savings measure that are a) implementable in double digit hours and b) save six figures per month.

m0xte 6y ago
I’m not saying that it doesn’t result in a saving but that the fact that industry has to exist to keep costs constrained is a net loss. Imagine if pricing was transparent, calculable and easy to rationalise the cost of a change. But it isn’t unless you pull a third party in.