5 ms·

Unfortunately, it's a false dichotomy you present, it's not a binary choice of fully managed or entirely roll your own. E.g., if you're running K8s (one thing

Unfortunately, it's a false dichotomy you present, it's not a binary choice of fully managed or entirely roll your own. E.g., if you're running K8s (one thing I typically recommend you buy a managed one of), you can install your own Kafka in it, using an operator that does about 85% of what MSK does. Sure, you'll need to dedicate person hours to support the operator, but is supporting that any more expensive than supporting AWS products? That you're already paying through the nose for?

xyzzy123 2y ago
It's also, what kind of startup are you? What kind of workload do you have? If you are bootstrapping a crud app business then 1 beefy hetzner box (or something slightly more reliable) with postgresql is probably fine until you reach scale where you sell the business. You care about burn rate above all. If you are VC backed go all in on gcp or aws because thats what you're expected to do and and what the expensive people you hire are going to know.
- heisenbit 2y ago
  I really wonder whether a VC would rather invest into a startup with an architect focusing on KISS or one where the architect goes all in on cloud.
- infecto 2y ago
  I agree but would slightly modify it in that if you have taken VC money, growth probably matters above all else. Don't waste time on activities not related to the product being sold.
redserk 2y ago
You can open a ticket and make the weirdest of issues with MSK Amazon’s problem to deal with. Same with RDS, etc. It’s pretty great not to waste time when the lottery for the bizarrest of 0.000001% issues arise. The operator only solves the happy path. An AWS support ticket usually can solve the unhappy path.
- EdwardDiego 2y ago
  Sure, but you can also go to a Slack channel and get help from the people who wrote the FOSS code you're using. For free. Yep, if your Kafka is mission critical and crashes hard, that is bad. But things like Kafka are _never_ a black box you just spin up and never worry about, if anyone thinks so, CAP theorem will give them an awful surprise one day. You're always going to need someone in your team who understands the tech and how to make best use of it. MSK won't tell you how many partitions your topic needs, or whether your retention strategy should be delete, or compact, or both. You still need that knowledge of the "managed" service to make effective use of it. And that knowledge sits rather close to knowledge of how the system works, so given you'll need that knowledge anyway, may as well cultivate it instead. Oh, and the operators also solve a lot of the unhappy paths too, FYI. I tend to describe the operator approach as "half-managed" because things like multiple-AZ stretch clusters need some configuration. But then, maybe you didn't want a 3-AZ cluster? Maybe a 2.5? MSK says no.
  - CPLX 2y ago
    > For free Not sure we agree on the meaning of this phrase in this context.
    - EdwardDiego 2y ago
      If you ever hit an issue with Kafka or Strimzi, go to their Slack, some of the most intelligent people I've ever had the privilege to work alongside will be there, helping you. For 0 money. That kinda free.
      - oh_fiddlesticks 2y ago
        I would prefer to say "free of charge", because that support is not actually free, it has a cost, you're just not required to pay for it. But you as well as I know, that what the other participant in this conversation means, is that if a for-profit entity relies on support that is "free of charge" in this way, such that it can continue to profit on the back of their product support, then the for-profit entity really ought to seriously consider a voluntary donation of some kind to support the continued maintenance and support of the product.
infecto 2y ago
That's ultimately the question. It comes down to cost and time. If you have enough scale that hiring a full-time person is more cost effective than paying for managed, great. On the flip side, you don't necessarily want to take engineering hours away from building the product you sell.

xyzzy123 2y ago

It's also, what kind of startup are you? What kind of workload do you have? If you are bootstrapping a crud app business then 1 beefy hetzner box (or something slightly more reliable) with postgresql is probably fine until you reach scale where you sell the business. You care about burn rate above all. If you are VC backed go all in on gcp or aws because thats what you're expected to do and and what the expensive people you hire are going to know.

heisenbit 2y ago
I really wonder whether a VC would rather invest into a startup with an architect focusing on KISS or one where the architect goes all in on cloud.
infecto 2y ago
I agree but would slightly modify it in that if you have taken VC money, growth probably matters above all else. Don't waste time on activities not related to the product being sold.

redserk 2y ago

You can open a ticket and make the weirdest of issues with MSK Amazon’s problem to deal with. Same with RDS, etc. It’s pretty great not to waste time when the lottery for the bizarrest of 0.000001% issues arise. The operator only solves the happy path. An AWS support ticket usually can solve the unhappy path.

EdwardDiego 2y ago
Sure, but you can also go to a Slack channel and get help from the people who wrote the FOSS code you're using. For free. Yep, if your Kafka is mission critical and crashes hard, that is bad. But things like Kafka are _never_ a black box you just spin up and never worry about, if anyone thinks so, CAP theorem will give them an awful surprise one day. You're always going to need someone in your team who understands the tech and how to make best use of it. MSK won't tell you how many partitions your topic needs, or whether your retention strategy should be delete, or compact, or both. You still need that knowledge of the "managed" service to make effective use of it. And that knowledge sits rather close to knowledge of how the system works, so given you'll need that knowledge anyway, may as well cultivate it instead. Oh, and the operators also solve a lot of the unhappy paths too, FYI. I tend to describe the operator approach as "half-managed" because things like multiple-AZ stretch clusters need some configuration. But then, maybe you didn't want a 3-AZ cluster? Maybe a 2.5? MSK says no.
- CPLX 2y ago
  > For free Not sure we agree on the meaning of this phrase in this context.
  - EdwardDiego 2y ago
    If you ever hit an issue with Kafka or Strimzi, go to their Slack, some of the most intelligent people I've ever had the privilege to work alongside will be there, helping you. For 0 money. That kinda free.
    - oh_fiddlesticks 2y ago
      I would prefer to say "free of charge", because that support is not actually free, it has a cost, you're just not required to pay for it. But you as well as I know, that what the other participant in this conversation means, is that if a for-profit entity relies on support that is "free of charge" in this way, such that it can continue to profit on the back of their product support, then the for-profit entity really ought to seriously consider a voluntary donation of some kind to support the continued maintenance and support of the product.

infecto 2y ago

That's ultimately the question. It comes down to cost and time. If you have enough scale that hiring a full-time person is more cost effective than paying for managed, great. On the flip side, you don't necessarily want to take engineering hours away from building the product you sell.