6 ms·

False equivalence. The engineer will be doing more than just cloud work. This comparison is the #1 flawed sales tactic the cloud companies use to convince you

by nojito 6y ago

False equivalence. The engineer will be doing more than just cloud work. This comparison is the #1 flawed sales tactic the cloud companies use to convince you youre saving money

acdha 6y ago
It’s a “false equivalence” or “flawed sales tactic” to suggest planning using total costs? That’s what both engineers and business people are supposed to do - and reflexively attacking it really does not cast your motives in a good light.
- twic 6y ago
  It's a false equivalence to suggest that managed services have zero staff costs, and that using a DIY database has a staff cost measured in whole FTEs.
  - acdha 6y ago
    That could be the answer to the question which was actually asked but if you read the thread again, notice that you’re arguing against a claim nobody made.
    - twic 6y ago
      Scroll up, my dude: > False equivalence. The engineer will be doing more than just cloud work. > This comparison is the #1 flawed sales tactic the cloud companies use to convince you youre saving money "False equivalence. The engineer will be doing more than just cloud work" -> "It's a false equivalence to suggest that [...] using a DIY database has a staff cost measured in whole FTEs." Hey, maybe AWS should launch some kind of ML-powered reading-comprehension-as-a-service?
      - acdha 6y ago
        Yes, do scroll up — note that the portions you quoted were the strawmen which nojito tossed out, not the original question, and perhaps ponder whether accusing someone else of not reading for comprehension is adding anything to the conversation.
QuinnyPig 6y ago
I don't have a dog in this race; I'm not partnered with any cloud provider. I will say that based upon what the article discusses, they save a bunch of money on AWS Glue by... running their own ETL pipeline inside of ECS instead. What's the maintenance burden of that decision? It's certainly not zero.
- sidlls 6y ago
  That depends on the scale and other particulars that are only shared/known inside the company. In my experience, beyond a certain scale it simply doesn't make any sense to use managed services anymore. There is a high initial upfront cost in development hours and hardware that is amortized over a very long time after, and this upfront cost is partially paid for by the reduced cloud bill.
  - glogla 6y ago
    That also depends on the markup. EKS on EC2 costs EC2 costs + flat money for control plane, so that might make more sense than running your own Kubernetes on EC2. (although I have no experience to say how much time this actually saves you) Managed Kafka costing 2x the cost of EC2 infrastructure? Probably not worth it.
- kfk 6y ago
  Most ETL processes should be declarative and configured in text files like you configure a CloudFormation or Terraform. Once you have that, the execution piece is relatively straightforward. I think a lot of issues and costs with ETL come from poor architecture decisions. For instance, we all made the mistake of running Extract processes with too many transformations. An extract should be an extract, transformation should come later.
vageli 6y ago
> False equivalence. The engineer will be doing more than just cloud work. > This comparison is the #1 flawed sales tactic the cloud companies use to convince you youre saving money Time is of a limited quantity and time spent managing postgres backups (for example) is time not spent doing other (possibly more meaningful/impactful _to the business_) work.
- dylan604 6y ago
  what is involved in managing backups? isn't that just a cronjob?
  - pmelendez 6y ago
    There was a time when I used to think the same, then I found my backups were corrupted (or stopped because run out of space, etc) just in the moment when I needed them.
    - owenmarshall 6y ago
      I once worked at an IT shop that worked closely with the construction industry. A new sports stadium was being built and we were doing panoramic photos during each stage of construction, and rendering them in a web app where the facilities team could “peel back the layers” and see what was behind the wall or under the floor all the way down to the foundation. This was... 15 years ago? So it was pretty neat technology and a little less ubiquitous than today. Well, our storage server barfed and the data was gone. Went to restore from backups, all the hourly tar files were there... but were zero bytes. We looked at the backup script the engineer had put together and it was one of those classic “didn’t give the right parameter to have tar recurse” type bugs. Unfortunately we lost all the photos of the foundation and much of the photos of the electric being run. Oops.
      - jbn 6y ago
        One wonders why tar even has an option (and a default!) to not recurse. What would be the use case for that? The "normal" use case seems to have a recursive archival. Sounds like somebody chose the wrong default... would be an interesting software archaelogy project to figure out where this "feature" originated.
fiddlerwoaroof 6y ago
Yes. I’ve found that the amount you have to learn to use a managed service often equals or exceeds the amount you have to learn to run something on EC2 or on-prem. The automation/management costs of AWS or equivalent are a lot higher than people think and not significantly different from the costs to learn Linux and enough networking to do an “old-fashioned” deploy.
- sokoloff 6y ago
  Much of that rings true, but I find some of the cloud abstractions can help to make the steady-state ops time required lower (especially for a side project where you really don’t want to deal with life interruptions).
- conradfr 6y ago
  The cloud was supposed to help you get rid of all these pesky sysadmins. Imagine the savings! Now the cloud is so complicated that you have to hire "devops". It's the same people as before, with a higher salary.
brodouevencode 6y ago
While I don't entirely disagree I think it should be made clear that both can be true, even at the same time. To spin up a 9 node managed Elasticsearch cluster load-balanced across two regions takes a competent engineer with practice roughly a couple of hours, or twenty to thirty minutes if they were smart and terraformed it out previously. Now there's a whole host of potential problems that come along with using that managed Elasticsearch cluster too (no access to "cluster mode", no tunability, etc). But if those potential problems don't apply to you and a very vanilla ES cluster suits your use case then you're fine. Alternatively that practiced engineer could have spun up a self-managed ES cluster in a couple of DCs in about the same time, but now has the obligation to maintain those servers (patching, etc.). Maybe that marginal cost is damn near zero - chef has been deployed to all instances and enforces patching and there's already good security monitoring in place, etc. The cost of that engineer managing that box, as with a managed ES in AWS, is practically nothing. TL;DR: as in all cases, it depends.

by nojito 6y ago

False equivalence. The engineer will be doing more than just cloud work. This comparison is the #1 flawed sales tactic the cloud companies use to convince you youre saving money

acdha 6y ago

It’s a “false equivalence” or “flawed sales tactic” to suggest planning using total costs? That’s what both engineers and business people are supposed to do - and reflexively attacking it really does not cast your motives in a good light.

twic 6y ago
It's a false equivalence to suggest that managed services have zero staff costs, and that using a DIY database has a staff cost measured in whole FTEs.
- acdha 6y ago
  That could be the answer to the question which was actually asked but if you read the thread again, notice that you’re arguing against a claim nobody made.
  - twic 6y ago
    Scroll up, my dude: > False equivalence. The engineer will be doing more than just cloud work. > This comparison is the #1 flawed sales tactic the cloud companies use to convince you youre saving money "False equivalence. The engineer will be doing more than just cloud work" -> "It's a false equivalence to suggest that [...] using a DIY database has a staff cost measured in whole FTEs." Hey, maybe AWS should launch some kind of ML-powered reading-comprehension-as-a-service?
    - acdha 6y ago
      Yes, do scroll up — note that the portions you quoted were the strawmen which nojito tossed out, not the original question, and perhaps ponder whether accusing someone else of not reading for comprehension is adding anything to the conversation.

QuinnyPig 6y ago

I don't have a dog in this race; I'm not partnered with any cloud provider. I will say that based upon what the article discusses, they save a bunch of money on AWS Glue by... running their own ETL pipeline inside of ECS instead. What's the maintenance burden of that decision? It's certainly not zero.

sidlls 6y ago
That depends on the scale and other particulars that are only shared/known inside the company. In my experience, beyond a certain scale it simply doesn't make any sense to use managed services anymore. There is a high initial upfront cost in development hours and hardware that is amortized over a very long time after, and this upfront cost is partially paid for by the reduced cloud bill.
- glogla 6y ago
  That also depends on the markup. EKS on EC2 costs EC2 costs + flat money for control plane, so that might make more sense than running your own Kubernetes on EC2. (although I have no experience to say how much time this actually saves you) Managed Kafka costing 2x the cost of EC2 infrastructure? Probably not worth it.
kfk 6y ago
Most ETL processes should be declarative and configured in text files like you configure a CloudFormation or Terraform. Once you have that, the execution piece is relatively straightforward. I think a lot of issues and costs with ETL come from poor architecture decisions. For instance, we all made the mistake of running Extract processes with too many transformations. An extract should be an extract, transformation should come later.

vageli 6y ago

> False equivalence. The engineer will be doing more than just cloud work. > This comparison is the #1 flawed sales tactic the cloud companies use to convince you youre saving money Time is of a limited quantity and time spent managing postgres backups (for example) is time not spent doing other (possibly more meaningful/impactful _to the business_) work.

dylan604 6y ago
what is involved in managing backups? isn't that just a cronjob?
- pmelendez 6y ago
  There was a time when I used to think the same, then I found my backups were corrupted (or stopped because run out of space, etc) just in the moment when I needed them.
  - owenmarshall 6y ago
    I once worked at an IT shop that worked closely with the construction industry. A new sports stadium was being built and we were doing panoramic photos during each stage of construction, and rendering them in a web app where the facilities team could “peel back the layers” and see what was behind the wall or under the floor all the way down to the foundation. This was... 15 years ago? So it was pretty neat technology and a little less ubiquitous than today. Well, our storage server barfed and the data was gone. Went to restore from backups, all the hourly tar files were there... but were zero bytes. We looked at the backup script the engineer had put together and it was one of those classic “didn’t give the right parameter to have tar recurse” type bugs. Unfortunately we lost all the photos of the foundation and much of the photos of the electric being run. Oops.
    - jbn 6y ago
      One wonders why tar even has an option (and a default!) to not recurse. What would be the use case for that? The "normal" use case seems to have a recursive archival. Sounds like somebody chose the wrong default... would be an interesting software archaelogy project to figure out where this "feature" originated.

fiddlerwoaroof 6y ago

Yes. I’ve found that the amount you have to learn to use a managed service often equals or exceeds the amount you have to learn to run something on EC2 or on-prem. The automation/management costs of AWS or equivalent are a lot higher than people think and not significantly different from the costs to learn Linux and enough networking to do an “old-fashioned” deploy.

sokoloff 6y ago
Much of that rings true, but I find some of the cloud abstractions can help to make the steady-state ops time required lower (especially for a side project where you really don’t want to deal with life interruptions).
conradfr 6y ago
The cloud was supposed to help you get rid of all these pesky sysadmins. Imagine the savings! Now the cloud is so complicated that you have to hire "devops". It's the same people as before, with a higher salary.

brodouevencode 6y ago

While I don't entirely disagree I think it should be made clear that both can be true, even at the same time. To spin up a 9 node managed Elasticsearch cluster load-balanced across two regions takes a competent engineer with practice roughly a couple of hours, or twenty to thirty minutes if they were smart and terraformed it out previously. Now there's a whole host of potential problems that come along with using that managed Elasticsearch cluster too (no access to "cluster mode", no tunability, etc). But if those potential problems don't apply to you and a very vanilla ES cluster suits your use case then you're fine. Alternatively that practiced engineer could have spun up a self-managed ES cluster in a couple of DCs in about the same time, but now has the obligation to maintain those servers (patching, etc.). Maybe that marginal cost is damn near zero - chef has been deployed to all instances and enforces patching and there's already good security monitoring in place, etc. The cost of that engineer managing that box, as with a managed ES in AWS, is practically nothing. TL;DR: as in all cases, it depends.