4 ms·

Amazon Managed Streaming for Kafka in Public Preview

62 points by abd12 8y ago

selfind 8y ago
This is awesome for what I heard about kafka operations.
hendzen 8y ago
Amazon taking more open source and selling it with no value created for the developers. Maybe Kafka should switch to the SSPL [0]. [0] - https://www.mongodb.com/licensing/server-side-public-license https://www.mongodb.com/licensing/server-side-public-license
- t0mbstone 8y ago
  I would argue that making Kafka more approachable and easy to deploy and use out of box is creating value for the developers, because it helps increase potential user adoption (and therefore increases the potential open source contributor base)
  - zedpm 8y ago
    I agree. We started using Redis with AWS' managed offering (Elasticache), then migrated to running it on our own once we were comfortable with it. I can imagine the same path for Kafka use.
- ijung 8y ago
  operations could be headaches tho. I always think AWS as seperate operation contracto teams for running large scale distributed softwares
- kevin_b_er 8y ago
  Then you couldn't run it on Linux, as you'd be unable to relicense the storage system that is the kernel file system or storage drivers from GPL to SSPL.
- manigandham 8y ago
  Open-source software is free, that's the point. If you wanted to make money with it then don't make it open-source. This entitlement is just silly. Many vendors make money by selling services on top, and there are at least a dozen that offer managed Kafka right now.
alienreborn 8y ago
> Broker instances start at $0.21/hr and broker storage is $0.10 per GB-month Kafka requires Zookeeper, do you pay for zookeeper instances too? Anyway, this pricing seems prohibitively costly unless you really need hands-off approach to your infra. Kafka is pretty low-maintenance and stable in my experience.
- davidstoker 8y ago
  From the pricing page: https://aws.amazon.com/msk/pricing/ https://aws.amazon.com/msk/pricing/ "You do not pay for Apache Zookeeper nodes that Amazon MSK provisions for you, or data transfer that occurs between brokers and nodes within clusters."
- chosenken 8y ago
  Just to note, the $.21/hr broker is on an m5.large (2 cpu, 8 GB mem), which goes for $.096/hr. We run three nodes right now on m5.xlarge instances. At $.42/hr for the managed kafka, compared to $.192/hr self hosted kafka, I think we'll keep it self hosted for now...
  - NikolaeVarius 8y ago
    What about the cost of the ZK nodes and associated EBS?
- NikolaeVarius 8y ago
  I find that this is true, but when it breaks, man it breaks REALLY hard.
  - sofaofthedamned 8y ago
    It does. And until last year even the major support company's containers (for Kafka etc) were not very good, we didn't upgrade a k8s cluster purely because of the state of their stack. This stuff scared me and this is a shocking affair these days. In some ways it's a step back in reliability, and i'd have taken this AWS option in a heartbeat, price-gouging aside.
reilly3000 8y ago
Awesome stuff. Kafka has been out of reach for our small org because we simply don't have the resources to manage a prod Kafka+zookeeper cluster. Kinesis has never anywhere near feature parity with Kafka. Ktables on AWS here we come!
kureikain 8y ago
This is awesome. Kafka were super fragile to me. Every version will come with all kind of issues and I have to restart here and there :(. Right now I'm trying to debug this issue: https://community.hortonworks.com/questions/208764/kafka-error-attempting-to-send-response-via-channe.html https://community.hortonworks.com/questions/208764/kafka-err... I'm not a fan of Kafka though but you sometimes have to managed stuf fyou don't like anyway. So this is great news. AWS truly listen to their customers I think.
bradhe 8y ago
Huh interesting. I’ve actually had really good luck with Kinesis, this must be their Postgres vs MySQL thing.
wuunderbar 8y ago
What would be the advantage of using this over Kinesis?

62 points by abd12 8y ago

This is awesome for what I heard about kafka operations.

Amazon taking more open source and selling it with no value created for the developers. Maybe Kafka should switch to the SSPL [0]. [0] - https://www.mongodb.com/licensing/server-side-public-license https://www.mongodb.com/licensing/server-side-public-license

t0mbstone 8y ago
I would argue that making Kafka more approachable and easy to deploy and use out of box is creating value for the developers, because it helps increase potential user adoption (and therefore increases the potential open source contributor base)
- zedpm 8y ago
  I agree. We started using Redis with AWS' managed offering (Elasticache), then migrated to running it on our own once we were comfortable with it. I can imagine the same path for Kafka use.
ijung 8y ago
operations could be headaches tho. I always think AWS as seperate operation contracto teams for running large scale distributed softwares
kevin_b_er 8y ago
Then you couldn't run it on Linux, as you'd be unable to relicense the storage system that is the kernel file system or storage drivers from GPL to SSPL.
manigandham 8y ago
Open-source software is free, that's the point. If you wanted to make money with it then don't make it open-source. This entitlement is just silly. Many vendors make money by selling services on top, and there are at least a dozen that offer managed Kafka right now.

alienreborn 8y ago

> Broker instances start at $0.21/hr and broker storage is $0.10 per GB-month Kafka requires Zookeeper, do you pay for zookeeper instances too? Anyway, this pricing seems prohibitively costly unless you really need hands-off approach to your infra. Kafka is pretty low-maintenance and stable in my experience.

davidstoker 8y ago
From the pricing page: https://aws.amazon.com/msk/pricing/ https://aws.amazon.com/msk/pricing/ "You do not pay for Apache Zookeeper nodes that Amazon MSK provisions for you, or data transfer that occurs between brokers and nodes within clusters."
chosenken 8y ago
Just to note, the $.21/hr broker is on an m5.large (2 cpu, 8 GB mem), which goes for $.096/hr. We run three nodes right now on m5.xlarge instances. At $.42/hr for the managed kafka, compared to $.192/hr self hosted kafka, I think we'll keep it self hosted for now...
- NikolaeVarius 8y ago
  What about the cost of the ZK nodes and associated EBS?
NikolaeVarius 8y ago
I find that this is true, but when it breaks, man it breaks REALLY hard.
- sofaofthedamned 8y ago
  It does. And until last year even the major support company's containers (for Kafka etc) were not very good, we didn't upgrade a k8s cluster purely because of the state of their stack. This stuff scared me and this is a shocking affair these days. In some ways it's a step back in reliability, and i'd have taken this AWS option in a heartbeat, price-gouging aside.

reilly3000 8y ago

Awesome stuff. Kafka has been out of reach for our small org because we simply don't have the resources to manage a prod Kafka+zookeeper cluster. Kinesis has never anywhere near feature parity with Kafka. Ktables on AWS here we come!

kureikain 8y ago

This is awesome. Kafka were super fragile to me. Every version will come with all kind of issues and I have to restart here and there :(. Right now I'm trying to debug this issue: https://community.hortonworks.com/questions/208764/kafka-error-attempting-to-send-response-via-channe.html https://community.hortonworks.com/questions/208764/kafka-err... I'm not a fan of Kafka though but you sometimes have to managed stuf fyou don't like anyway. So this is great news. AWS truly listen to their customers I think.

bradhe 8y ago

Huh interesting. I’ve actually had really good luck with Kinesis, this must be their Postgres vs MySQL thing.

wuunderbar 8y ago

What would be the advantage of using this over Kinesis?