16 ms·

Hmm. This looks to me like a lot of the savings were realized by moving away from managed services into a scenario where there’s more operator overhead. The AWS

Hmm. This looks to me like a lot of the savings were realized by moving away from managed services into a scenario where there’s more operator overhead. The AWS bill gets lower, but what about the cost of the engineering work?

alharith 6y ago
Does anyone else find the costs associated with running well-tested, well developed systems overblown? Like if you know how to adjust some basic parameters, you will solve for 99% use cases (adjust memory, adjust ram). Examples I can think of is Rabbit MQ and Cassandra. But in general, we have some really battle-tested software these days that has become simpler to configure and run over time. People seem scared to run their own these days.
- owenmarshall 6y ago
  I vouched for this comment because it’s a valid point and I’m not sure why it was killed. I happen to disagree strongly, though: lots of engineers in my experience undervalue the work of systems administrators and underestimate the effort needed to operationalize any technology. Running your own is absolutely fine if you are willing to keep your stack small and invest time learning the tools you pick. But there are still horror stories of people thinking snapshots are backups, turning the wrong knobs and turning off fsync on their databases, ...
- chrischen 6y ago
  Yea exactly and unless you are FB scale you can just run a single docker container and never really have to worry (granted you know how to use Docker). Most small startups are actually the ones who don’t really need SaaS services.
  - ckdarby 6y ago
    >Yea exactly and unless you are FB scale you can just run a single docker container and never really have to worry This has not been the case at multiple employers and or consulting clients. If you're providing software to an enterprise this almost will never fly. That single docker container will have an outage when basically anything happens. The container dies, systemd fails to restart, node dies, network switch dies, data center has basically any major issue, etc. I think your comment brings value just probably biased with your own experience of running a consumer to consumer startup.
    - chrischen 6y ago
      I think you're misinterpreting my comment. I meant specifically for most small time startups, not necessarily small time startups deploying enterprise apps. If you're deploying enterprise apps then by definition you're for all intents and purposes "fb scale." A lot of SaaS promise infinite scalability—a need which often never comes to most small time startups.
- csharptwdec19 6y ago
  Sometimes. But developers are part of this problem too. There's plenty of times where I see devs immediately reach for tools instead of learning just a little bit more about what they already have. My favorite example is when folks want to add a NoSQL db into the mix on top of a traditional db. Not because there's a real performance need, but because for their use case it is 'easier'. Never mind that their problem possibly could have been solved by just writing their own SQL instead of trusting a garbage ORM...
nojito 6y ago
False equivalence. The engineer will be doing more than just cloud work. This comparison is the #1 flawed sales tactic the cloud companies use to convince you youre saving money
- acdha 6y ago
  It’s a “false equivalence” or “flawed sales tactic” to suggest planning using total costs? That’s what both engineers and business people are supposed to do - and reflexively attacking it really does not cast your motives in a good light.
  - twic 6y ago
    It's a false equivalence to suggest that managed services have zero staff costs, and that using a DIY database has a staff cost measured in whole FTEs.
    - acdha 6y ago
      That could be the answer to the question which was actually asked but if you read the thread again, notice that you’re arguing against a claim nobody made.
      - twic 6y ago
        Scroll up, my dude: > False equivalence. The engineer will be doing more than just cloud work. > This comparison is the #1 flawed sales tactic the cloud companies use to convince you youre saving money "False equivalence. The engineer will be doing more than just cloud work" -> "It's a false equivalence to suggest that [...] using a DIY database has a staff cost measured in whole FTEs." Hey, maybe AWS should launch some kind of ML-powered reading-comprehension-as-a-service?
        acdha 6y ago
        Yes, do scroll up — note that the portions you quoted were the strawmen which nojito tossed out, not the original question, and perhaps ponder whether accusing someone else of not reading for comprehension is adding anything to the conversation.
runT1ME 6y ago
If you're already paying the cost(both engineering time and compute wise) for EMR, I can't imagine it takes more effort to create a new Flink job than a new Glue job? The advantage of Glue or the corresponding serverless GCP ETL option (dataflow) is that it's serverless elastic, but it sounds like their workload wasn't applicable.
- brodouevencode 6y ago
  Unfortunately that's not nearly how AWS works. AWS breaks down everything and charges you for it separately. Flink and Glue are entirely different animals.
  - runT1ME 6y ago
    Can you explain more? From what I understand, if you're already running a Flink cluster on AWS and you have capacity for another job, you aren't charged more, no? I haven't used Glue, but it seems like it's able to do stream processing on Kinesis and dumping to S3 or whatnot, so it seems like there's overlap with using EMR running Flink?
Cthulhu_ 6y ago
This is probably a tradeoff in a lot of AWS related stuff; you pay a premium for convenience. But depending on your workload it can pay itself off fairly quickly; AWS bills can go up quite fast, whereas personnel costs are fairly predictable.
brodouevencode 6y ago
Agreed - there is a tradeoff that must factor in many things: engineer competency or the ability to get competent engineers, state of the product itself (maybe Elasticsearch as a service was an interim step in a longer term vision), complexity of the managed service itself, integratability (is that a word?) into other AWS services, maturity of the managed service, and probably a few other things I'm missing. We've seen our teams go both from managed to non-managed and non-managed to managed with relative success - to give scale, across all of our accounts we spend way north of $3 million/month at AWS so this has happened within our realm a quite few times. The short, unsatisfying answer is that _it depends_. We have an internal policy from the suits that "if there's a managed version, use it" but most of our teams are thankfully smart enough to take that at face value and do their own analysis.
- gberger 6y ago
  > integratability interoperability?
  - brodouevencode 6y ago
    Thanks :)
cloudfalcon 6y ago
Hi Corey! I'm the author of the blog post - I definitely agree with you. 9 times out of 10 engineering teams underestimate cost of engineering work as well as opportunity cost lost due to managing non-core functionality internally or moving away from managed services. For us our pipeline was actually easier to work with Flink than Glue because of the restrictions that Amazon placed on it and so that factored into our decision.
dan_quixote 6y ago
My personal favorite is "Move off of AWS MSK". No big deal - we just fire up some kafka brokers and and zookeeper nodes in ECS! All we gotta do is run several more supporting services to keep the cluster healthy and deal with the nightmare of Apache security ourselves. As far as I'm concerned MSK is cheap - one broker is priced roughly same price as 2 equivalent EC2 instances. And you don't have to worry about zookeeper at all!

alharith 6y ago

Does anyone else find the costs associated with running well-tested, well developed systems overblown? Like if you know how to adjust some basic parameters, you will solve for 99% use cases (adjust memory, adjust ram). Examples I can think of is Rabbit MQ and Cassandra. But in general, we have some really battle-tested software these days that has become simpler to configure and run over time. People seem scared to run their own these days.

owenmarshall 6y ago
I vouched for this comment because it’s a valid point and I’m not sure why it was killed. I happen to disagree strongly, though: lots of engineers in my experience undervalue the work of systems administrators and underestimate the effort needed to operationalize any technology. Running your own is absolutely fine if you are willing to keep your stack small and invest time learning the tools you pick. But there are still horror stories of people thinking snapshots are backups, turning the wrong knobs and turning off fsync on their databases, ...
chrischen 6y ago
Yea exactly and unless you are FB scale you can just run a single docker container and never really have to worry (granted you know how to use Docker). Most small startups are actually the ones who don’t really need SaaS services.
- ckdarby 6y ago
  >Yea exactly and unless you are FB scale you can just run a single docker container and never really have to worry This has not been the case at multiple employers and or consulting clients. If you're providing software to an enterprise this almost will never fly. That single docker container will have an outage when basically anything happens. The container dies, systemd fails to restart, node dies, network switch dies, data center has basically any major issue, etc. I think your comment brings value just probably biased with your own experience of running a consumer to consumer startup.
  - chrischen 6y ago
    I think you're misinterpreting my comment. I meant specifically for most small time startups, not necessarily small time startups deploying enterprise apps. If you're deploying enterprise apps then by definition you're for all intents and purposes "fb scale." A lot of SaaS promise infinite scalability—a need which often never comes to most small time startups.
csharptwdec19 6y ago
Sometimes. But developers are part of this problem too. There's plenty of times where I see devs immediately reach for tools instead of learning just a little bit more about what they already have. My favorite example is when folks want to add a NoSQL db into the mix on top of a traditional db. Not because there's a real performance need, but because for their use case it is 'easier'. Never mind that their problem possibly could have been solved by just writing their own SQL instead of trusting a garbage ORM...

nojito 6y ago

False equivalence. The engineer will be doing more than just cloud work. This comparison is the #1 flawed sales tactic the cloud companies use to convince you youre saving money

acdha 6y ago
It’s a “false equivalence” or “flawed sales tactic” to suggest planning using total costs? That’s what both engineers and business people are supposed to do - and reflexively attacking it really does not cast your motives in a good light.
- twic 6y ago
  It's a false equivalence to suggest that managed services have zero staff costs, and that using a DIY database has a staff cost measured in whole FTEs.
  - acdha 6y ago
    That could be the answer to the question which was actually asked but if you read the thread again, notice that you’re arguing against a claim nobody made.
    - twic 6y ago
      Scroll up, my dude: > False equivalence. The engineer will be doing more than just cloud work. > This comparison is the #1 flawed sales tactic the cloud companies use to convince you youre saving money "False equivalence. The engineer will be doing more than just cloud work" -> "It's a false equivalence to suggest that [...] using a DIY database has a staff cost measured in whole FTEs." Hey, maybe AWS should launch some kind of ML-powered reading-comprehension-as-a-service?
      - acdha 6y ago
        Yes, do scroll up — note that the portions you quoted were the strawmen which nojito tossed out, not the original question, and perhaps ponder whether accusing someone else of not reading for comprehension is adding anything to the conversation.

runT1ME 6y ago

If you're already paying the cost(both engineering time and compute wise) for EMR, I can't imagine it takes more effort to create a new Flink job than a new Glue job? The advantage of Glue or the corresponding serverless GCP ETL option (dataflow) is that it's serverless elastic, but it sounds like their workload wasn't applicable.

brodouevencode 6y ago
Unfortunately that's not nearly how AWS works. AWS breaks down everything and charges you for it separately. Flink and Glue are entirely different animals.
- runT1ME 6y ago
  Can you explain more? From what I understand, if you're already running a Flink cluster on AWS and you have capacity for another job, you aren't charged more, no? I haven't used Glue, but it seems like it's able to do stream processing on Kinesis and dumping to S3 or whatnot, so it seems like there's overlap with using EMR running Flink?

Cthulhu_ 6y ago

This is probably a tradeoff in a lot of AWS related stuff; you pay a premium for convenience. But depending on your workload it can pay itself off fairly quickly; AWS bills can go up quite fast, whereas personnel costs are fairly predictable.

brodouevencode 6y ago

Agreed - there is a tradeoff that must factor in many things: engineer competency or the ability to get competent engineers, state of the product itself (maybe Elasticsearch as a service was an interim step in a longer term vision), complexity of the managed service itself, integratability (is that a word?) into other AWS services, maturity of the managed service, and probably a few other things I'm missing. We've seen our teams go both from managed to non-managed and non-managed to managed with relative success - to give scale, across all of our accounts we spend way north of $3 million/month at AWS so this has happened within our realm a quite few times. The short, unsatisfying answer is that _it depends_. We have an internal policy from the suits that "if there's a managed version, use it" but most of our teams are thankfully smart enough to take that at face value and do their own analysis.

gberger 6y ago
> integratability interoperability?
- brodouevencode 6y ago
  Thanks :)

cloudfalcon 6y ago

Hi Corey! I'm the author of the blog post - I definitely agree with you. 9 times out of 10 engineering teams underestimate cost of engineering work as well as opportunity cost lost due to managing non-core functionality internally or moving away from managed services. For us our pipeline was actually easier to work with Flink than Glue because of the restrictions that Amazon placed on it and so that factored into our decision.

dan_quixote 6y ago

My personal favorite is "Move off of AWS MSK". No big deal - we just fire up some kafka brokers and and zookeeper nodes in ECS! All we gotta do is run several more supporting services to keep the cluster healthy and deal with the nightmare of Apache security ourselves. As far as I'm concerned MSK is cheap - one broker is priced roughly same price as 2 equivalent EC2 instances. And you don't have to worry about zookeeper at all!