10 ms·

Heroku Kafka

276 points by sixwing 10y ago

cachemiss 10y ago
Kudos to Heroku. As someone who has had to make Kafka into a managed service, I know what a pain it is (I'm not a Kafka fan for a lot of reasons) to administer in a cloud environment.
- sethammons 10y ago
  Would love to hear what you don't care for in Kafka and what alterative solution(s) you prefer.
  - cachemiss 10y ago
    To clarify, my feelings towards Kafka are from the POV of someone who has had to build a managed service on top of it, which is not the common use case (for which many people seem to be happy with). Other people may have more positive experiences. In my experience, Kafka is a solid system when you work in its wheelhouse, which is a relatively static set of servers / topics, that you add to slowly and deliberately. If you can't use something like Kinesis, then its a good choice. In Kafka, programmatic administration is generally an afterthought. They have APIs for doing things, but they generally involve directly modifying znodes. Simple things don't work or have bugs, deleting topics didn't work at all until 0.8.2, and even now has bugs. We've seen cases where if you delete a topic while an ISR is shrinking or expanding, your cluster can get into an unrecoverable state where you have to reboot everything, and even then it doesn't always get fixed. Most of the time you are expected to use scripts to modify everything (there's a wide variety of systems out there that try to build mgmt on top of kafka). Its dependency on Zookeeper is a pain, and limits scalability of topic / partition counts. Rebalancing topics will reset retention periods because they use the last modified ts of the segment files to check for oldness, meaning if you rebalance often, you need extra disk space laying around. ZK has some bugs with its DNS handling, which affects Kafka if you try and use DNS. It has throttling, but its by client id, what you'd like in some cases, is to say that a node has X throughput, and have the broker be able to somewhat guarantee that throughput, and create backpressure when clients are overwhelming it. Otherwise your latency can go through the roof. You also want replication to play nice with client requests, and it doesn't (if you add a new broker and move a bunch of partitions to it, you'll light up all your other brokers while it replicates, and cause timeouts). Its replication story can cause issues when network partitions come into play. It's highly configurable like many Apache projects, which is a blessing and a curse, as your team has to know all the knobs, both consumer / producer / broker side. The alternative if you are at a company with the resources to do so (mine is), is to build something that fits your use case better than Kafka, or to use a hosted service like this, or Kinesis.
    - Dr_tldr 10y ago
      Your comment is highly technical, critical, but still very fair. This is why I love HN.
    - sethammons 10y ago
      Thanks for the thoughtful and detailed response! Very helpful.
    - emfree 10y ago
      Thanks for the insightful comment! > The alternative if you are at a company with the resources to do so (mine is), is to build something that fits your use case better than Kafka I'd love to hear more about this :) What did you end up doing differently from Kafka? How's it working out for you?
  - deleted 10y ago
    [deleted]
plunchete 10y ago
Is the pricing public?
- neovintage 10y ago
  Not yet. We're working it during our early access program. Well be looking for lots of feedback from customers.
  - plunchete 10y ago
    Thanks! Looking forward to be able to try it :)
jonahx 10y ago
> What is Kafka? > Apache Kafka is a distributed commit log for fast, fault-tolerant communication between producers and consumers using message based topics. Kafka provides the messaging backbone for building a new generation of distributed applications capable of handling billions of events and millions of transactions Can anyone translate this into meaningful English for me?
- amock 10y ago
  It's a distributed message queue.
  - gshx 10y ago
    It can be used as a queue but the bigger benefit is for streaming use cases. One of the key differences, among others, is that streaming assumes somewhat faster consumers as opposed to queueing. There's also the pub-sub use-case which is generally considered separate from that of a queue (considered a point to point transport).
    - amock 10y ago
      That is more descriptive, but it still sounds like queue functionality. Streaming processing is just a queue that gets emptied quickly and pub-sub is just a set of queues.
      - dkersten 10y ago
        Kafka doesn't generally get emptied quickly, but rather retains messages for a configured time/size. Because of this, consumers can choose to replay previously consumed messages, if they wish to do so.
        gshx 10y ago
        You're right. I was mostly commenting on the common idiomatic ways ppl differentiate streams vs queues. Indeed, it can be used in both scenarios.
koolba 10y ago
I've wondered why there isn't a "big player" in the cloud space for this. Felt like a hole. My operating theory is that the people who would really make use of something like this have grown beyond managed offerings and would take it in house. For smaller operations Redis is more than enough for pub/sub. Ditto for SQS for externally triggered eventing.
- amock 10y ago
  Depending on what you mean by "this" there are offerings by the big players. Google has Cloud Pub/Sub and AWS has Kinesis in addition to SQS, so two of the big players do have offerings. I'm not familiar enough with Azure to know what it has.
  - koolba 10y ago
    By "this" I meant a managed Kafka cloud offering. I generally a fan of these types of services as there isn't as tight a binding as proprietary ones. Migrating from Heroku Postgres to RDS or self hosted is well defined. Ditto for Redis migrations. SQS, Kinesis, and other proprietary ones not so much. You can insulate your code base but if you're really going to leverage the ecosystem of those services then you're going to be stuck there. That's why I find something like this interesting. The "out" is there so it makes it easier to accept getting in.
    - manigandham 10y ago
      There really isn't much lock-in when it comes to event logging systems. Just change the interface your code uses to whatever service you need. There might be a little refactoring to handle topics in the different ways but it's all ultimately the same thing. Since logging by nature offers asynchronous processing, you can migrate your publishers first and then the consumers without any downtime.
  - manigandham 10y ago
    Azure has Event Hubs that are very similar to Kinesis/Kafka. https://azure.microsoft.com/en-us/services/event-hubs/ https://azure.microsoft.com/en-us/services/event-hubs/ They also have simpler Queues and Service Bus for RPC/lightweight message handling.
tibbon 10y ago
Kafka vs Redis. I've only used Redis... what should I know?
- manigandham 10y ago
  Redis is an in-memory (with persistence) key-value database that also implements some basic structures like lists, sets and hashes natively. Kafka is a distributed logging system that can ingest large amounts of data straight to disk, then allows for multiple consumers to read this data through a simple abstraction of topics and partitions. Consumers maintain their own position of where they last read up to (or re-read things if they want) and everything is sequential I/O which creates very high throughput.
nodesocket 10y ago
Can somebody provide a real-life use case for Kafka? I've seen comparisons between Redis, but what specifically does Kafka solve that Redis cannot?
- manigandham 10y ago
  Kafka and Redis are very different things - see this: https://news.ycombinator.com/item?id=11577312 https://news.ycombinator.com/item?id=11577312 Redis is a database, Kakfa is a data logging system built for scale and throughput. Event processing (of any kind like stocks, ad impressions, ecommerce purchases) are a great fit. Also good as a message queue unless you need ultra low-latency RPC.
  - nodesocket 10y ago
    Gotcha, so then advantage of Kafka over Logstash + ElasticSearch?
    - manigandham 10y ago
      ElasticSearch is a database optimized for searching, not related at all. You can somewhat compare Kafka to Logstash but Kafka has no processing, it's purely a distributed log writing/reading/storage system that also scales far more than logstash can. You write data to it and then read from it with a basic messaging abstraction of topics and partitions.
      - ec109685 10y ago
        ElasticSearch can store sequenced number data, which is really all that Kafka is doing, so I don't think it is fair to say it isn't related at all.
        allengeorge 10y ago
        So can a RDBMS... But that doesn't mean that Kafka and databases are related. As multiple comments have stated above, Kafka is really a distributed message subsystem. Its core interface is a set of topics that one can publish to, and that consumers can read from (in other words, a pub-sub system). Kafka doesn't inspect the message payload at all. Elasticsearch is a unstructured (to some extent) document store that's optimized around document search. So at the very least, the payload is important when using Elasticsearch.
manigandham 10y ago
This will be interesting to try out. I've used all the major cloud event/logging systems (Kinesis, Azure EventHubs, etc) and so far Google PubSub is the best in features and performance. Only downside with Google Pubsub can be latency (which I'm working on fixing by building a gRPC driver) but Kafka has proven to be too complicated to maintain in-house. If heroku can provide the speed without the ops overhead, it'll be some good competition to Google's option. Also want to note that Jay Kreps who helped build Kafka at LinkedIn is now behind http://www.confluent.io/ http://www.confluent.io/ which is like a better/enterprise version of Kafka.
- alexatkeplar 10y ago
  Not sure why you are comparing Google Cloud Pub/Sub to Kinesis - the former is a MQ system, not a distributed commit log. When creating a Kinesis consumer, I can specify whether I want to start reading a stream from a) TRIM_HORIZON (which is the earliest events in the stream which haven't yet been expired aka "trimmed"), b) LATEST which is the Cloud Pub/Sub capability, c) AT_SEQUENCE_NUMBER {x} which means from the event in the stream with the given offset ID, d) AFTER_SEQUENCE_NUMBER {x} which is the event immediately after c), e) AT_TIMESTAMP to read records from an arbitrary point in time. A Kinesis stream (like a Kafka topic) is a very special form of database - it exists independently of any consumers. By contrast, with Google Cloud Pub/Sub [1]: > When you create a subscription, the system establishes a sync point. That is, your subscriber is guaranteed to receive any message published after this point. [1] https://cloud.google.com/pubsub/subscriber https://cloud.google.com/pubsub/subscriber So the stream is not a first class entity in Cloud Pub/Sub - it's just a consumer-tied message queue.
  - nivertech 10y ago
    Is there something like Kinesis' AT_TIMESTAMP in Kafka? I think the only way in to replay events in Google Cloud Pub/Sub is to create multiple subscriptions in advance, right after topic creation. But then I think you need to pay for the storage and event traversal requests.
- rtehfm 10y ago
  What are your thoughts on Kafka vs Flume?
ChartsNGraffs 10y ago
For anyone wanting to play with Kafka, Spotify's Kafka container was an invaluable resource for getting me up and running with Kafka. All the Zookeeper dependencies are taken care of allowing you to just start playing with Kafka right away. https://github.com/spotify/docker-kafka https://github.com/spotify/docker-kafka https://hub.docker.com/r/spotify/kafka/ https://hub.docker.com/r/spotify/kafka/
- Jarmo 10y ago
  I never tried spotify's container. Tried wurstmeister's, and was able to run it on a single server for testing purposes, but kept running into issues while clustering on different servers. Decided to use Ambari and have it do all the work for me instead.
mbseid 10y ago
As a former user of Kafka, this is awesome and it would have been a huge help for our company if this was available then. I'm glad to hear that a company is offering Kafka as opposed to other propriety versions(AWS Kinesis etc). One thing is odd though, there is no mention of disk space at all and only a configuration of retention time. One of Kafka's best features is the use of disk to store large amounts of messages, you are not RAM bound. Heroku seems to only allows you to set retention times? This could be awesome if they are giving you "unlimited" disk space, but could also be a beta oversight. Interested to see how this progresses.
- uhoh-itsmaciek 10y ago
  Hi, I'm Maciek and I work on the Heroku Kafka team. You don't have to think about disk space--it's on us to make sure there's enough to satisfy the retention settings you configure. We're excited to provide another great open-source project as a managed service!
  - mbseid 10y ago
    Thanks for the update. That is awesome. Excited to see what people do with it.
- ktamura 10y ago
  Don't forget that Heroku is the original multi-tenant shop. I wouldnt be surprised if a single Kafka instance stores multiple customers's messages and elastically scale as more customers/data is added.
  - sixwing 10y ago
    I'm Rand Fitzpatrick, and this is one of the products I work on at Heroku. None of our current Kafka offerings are multi-tenant.
franciscop 10y ago
I love Heroku and everything they are doing, it's doubtless a push forward for the web as a whole. However, the pricing for hobby sites (including SSL) is crazy from a personal point of view so I'm slowly moving my projects out of it [1][2]. I wish they had some kind of "Hobby Bundle". [1] http://umbrellajs.com/ http://umbrellajs.com/ [2] http://picnicss.com/ http://picnicss.com/
- sudhirj 10y ago
  Their pricing for hobby sites is 7$ + 10$ DB, which is very comparable with a self setup IaaS like DO and AWS. Personally I think the developer experience is much better on Heroku and quite worth it. SSL is a pain point, though I do empathize with them - I think they're doing something expensive for that. What I do is to use AWS Cloudfront and ACM for a free cert and site speedup - if they are personal projects the CF bill ought to be in the low few dollars anyway.
  - why-el 10y ago
    + 7$ for a worker I think.
  - flurdy 10y ago
    The $7 is comparable to one app per DO or AWS micro/nano server. So Heroku wins on convenience. If you have say 10 apps then Heroku costs 10*$7, but you might still only have used 1-3 servers depending on memory use of apps etc so then Heroku looses on cost. Naturally I got a total mix of quite a few on Heroku's classic or new free plan, some on their hobby plan, some on AWS, some on docker cloud, most proxied behind a SSL certificate running on AWS..... (https://flurdy.com/docs/letsencrypt/nginx.html https://flurdy.com/docs/letsencrypt/nginx.html)
  - balamaci 10y ago
    DO has 5$ + VAT price for 512MB instance. I have no problem accommodating mysql + web on that.
  - franciscop 10y ago
    Yeah, but I started with PHP I could just choose among many hosting companies for 5-10$/month and you get your shared space with unlimited domains, which was perfectly suited for my needs at that point. Of course as I learned more, Node and the such I needed better technology and that's why I moved to Heroku. So I'd love to see a "shared hosting for heroku" or similar. I think it will happen given some time, when the Node.js environment stabilizes more and more big players come.
andreasklinger 10y ago
For those wondering (all imo and only best guess) The biggest advantage of kafka is that all of the heroku marketplace all of a sudden becomes "plug and play" Essentially it's the "backend data" equivalent of what segment does for "frontend data". Example: What's the benefit of having a graphDB service in the marketplace if most people dont want to / cant invest engineering in keeping the data in (realtime) sync. With kafka they can establish standards that all partners can adapt to, they will simply offer piping of all heroku postgres/redis changes.
hmottestad 10y ago
Does anyone know if Kafka has improved on their data loss issues since tested by Aphyr? https://aphyr.com/posts/293-jepsen-kafka https://aphyr.com/posts/293-jepsen-kafka A quote from the article: "At the end of the run, Kafka typically acknowledges 98–100% of writes. However, half of those writes (all those made during the partition) are lost."
- lars_francke 10y ago
  Yes, the suggestion discussed by Aphyr has been implemented. You can now set up a lower bound on the ISR size (min.insync.replicas). Together with required.acks=-1 you can wait for a message to be committed to at least min.insync.replicas nodes. https://issues.apache.org/jira/browse/KAFKA-1555 https://issues.apache.org/jira/browse/KAFKA-1555
elcct 10y ago
My impression of Kafka was that this thing is bloated. How it compares to something like NSQ?
- kasey_junk 10y ago
  Its a completely different use case. Many times people call Kafka a "message queue" but its not. It's a distributed log service. Its possible to build a message queue on top of a distributed log service but there are reasons not to. Its better to think of Kafka as a database for events, not as a transport mechanism for those events. As for being bloated, Kafka lives in a very empty space, that is it supports fully ordered events to all consumers (and it has good HA options). The only other tool that I've come across that gives you the same data guarantees is Kinesis and it requires AWS. I've found that yes Kafka is complex, but its complex because its solving a complex problem, not because its bloated. That said, if you want a non-ordered message queue, use NSQ instead of Kafka.
  - elcct 10y ago
    Thanks for explanation. I didn't know those things.
tenismyanswer 10y ago
All kafkaesque to me ;->
mtw 10y ago
What kind of companies or startups usually use this service?
- rhodin 10y ago
  Companies dealing with large amounts of data. A list with some companies using Apache Kafka can be found here: https://cwiki.apache.org/confluence/display/KAFKA/Powered+By https://cwiki.apache.org/confluence/display/KAFKA/Powered+By
  - mtw 10y ago
    thanks. I guess my sites are not big enough (yet)
- poooogles 10y ago
  It's pretty big in ad tech, or anywhere that really does lots and lots of centralised logging (Datadog/Loggly both use Kafka). Lots of places also use it just as a message queue, some places for example write time series metrics to Kafka for monitoring.
jbob2000 10y ago
The comments in this thread are funny; Hey, what is Kafka? "It's a distributed logging system, not a message queue" Ok, what's the use case? describes a case when its used as a message queue

276 points by sixwing 10y ago

Kudos to Heroku. As someone who has had to make Kafka into a managed service, I know what a pain it is (I'm not a Kafka fan for a lot of reasons) to administer in a cloud environment.

sethammons 10y ago
Would love to hear what you don't care for in Kafka and what alterative solution(s) you prefer.
- cachemiss 10y ago
  To clarify, my feelings towards Kafka are from the POV of someone who has had to build a managed service on top of it, which is not the common use case (for which many people seem to be happy with). Other people may have more positive experiences. In my experience, Kafka is a solid system when you work in its wheelhouse, which is a relatively static set of servers / topics, that you add to slowly and deliberately. If you can't use something like Kinesis, then its a good choice. In Kafka, programmatic administration is generally an afterthought. They have APIs for doing things, but they generally involve directly modifying znodes. Simple things don't work or have bugs, deleting topics didn't work at all until 0.8.2, and even now has bugs. We've seen cases where if you delete a topic while an ISR is shrinking or expanding, your cluster can get into an unrecoverable state where you have to reboot everything, and even then it doesn't always get fixed. Most of the time you are expected to use scripts to modify everything (there's a wide variety of systems out there that try to build mgmt on top of kafka). Its dependency on Zookeeper is a pain, and limits scalability of topic / partition counts. Rebalancing topics will reset retention periods because they use the last modified ts of the segment files to check for oldness, meaning if you rebalance often, you need extra disk space laying around. ZK has some bugs with its DNS handling, which affects Kafka if you try and use DNS. It has throttling, but its by client id, what you'd like in some cases, is to say that a node has X throughput, and have the broker be able to somewhat guarantee that throughput, and create backpressure when clients are overwhelming it. Otherwise your latency can go through the roof. You also want replication to play nice with client requests, and it doesn't (if you add a new broker and move a bunch of partitions to it, you'll light up all your other brokers while it replicates, and cause timeouts). Its replication story can cause issues when network partitions come into play. It's highly configurable like many Apache projects, which is a blessing and a curse, as your team has to know all the knobs, both consumer / producer / broker side. The alternative if you are at a company with the resources to do so (mine is), is to build something that fits your use case better than Kafka, or to use a hosted service like this, or Kinesis.
  - Dr_tldr 10y ago
    Your comment is highly technical, critical, but still very fair. This is why I love HN.
  - sethammons 10y ago
    Thanks for the thoughtful and detailed response! Very helpful.
  - emfree 10y ago
    Thanks for the insightful comment! > The alternative if you are at a company with the resources to do so (mine is), is to build something that fits your use case better than Kafka I'd love to hear more about this :) What did you end up doing differently from Kafka? How's it working out for you?
- deleted 10y ago
  [deleted]

plunchete 10y ago

Is the pricing public?

neovintage 10y ago
Not yet. We're working it during our early access program. Well be looking for lots of feedback from customers.
- plunchete 10y ago
  Thanks! Looking forward to be able to try it :)

jonahx 10y ago

> What is Kafka? > Apache Kafka is a distributed commit log for fast, fault-tolerant communication between producers and consumers using message based topics. Kafka provides the messaging backbone for building a new generation of distributed applications capable of handling billions of events and millions of transactions Can anyone translate this into meaningful English for me?

amock 10y ago
It's a distributed message queue.
- gshx 10y ago
  It can be used as a queue but the bigger benefit is for streaming use cases. One of the key differences, among others, is that streaming assumes somewhat faster consumers as opposed to queueing. There's also the pub-sub use-case which is generally considered separate from that of a queue (considered a point to point transport).
  - amock 10y ago
    That is more descriptive, but it still sounds like queue functionality. Streaming processing is just a queue that gets emptied quickly and pub-sub is just a set of queues.
    - dkersten 10y ago
      Kafka doesn't generally get emptied quickly, but rather retains messages for a configured time/size. Because of this, consumers can choose to replay previously consumed messages, if they wish to do so.
      - gshx 10y ago
        You're right. I was mostly commenting on the common idiomatic ways ppl differentiate streams vs queues. Indeed, it can be used in both scenarios.

koolba 10y ago

I've wondered why there isn't a "big player" in the cloud space for this. Felt like a hole. My operating theory is that the people who would really make use of something like this have grown beyond managed offerings and would take it in house. For smaller operations Redis is more than enough for pub/sub. Ditto for SQS for externally triggered eventing.

amock 10y ago
Depending on what you mean by "this" there are offerings by the big players. Google has Cloud Pub/Sub and AWS has Kinesis in addition to SQS, so two of the big players do have offerings. I'm not familiar enough with Azure to know what it has.
- koolba 10y ago
  By "this" I meant a managed Kafka cloud offering. I generally a fan of these types of services as there isn't as tight a binding as proprietary ones. Migrating from Heroku Postgres to RDS or self hosted is well defined. Ditto for Redis migrations. SQS, Kinesis, and other proprietary ones not so much. You can insulate your code base but if you're really going to leverage the ecosystem of those services then you're going to be stuck there. That's why I find something like this interesting. The "out" is there so it makes it easier to accept getting in.
  - manigandham 10y ago
    There really isn't much lock-in when it comes to event logging systems. Just change the interface your code uses to whatever service you need. There might be a little refactoring to handle topics in the different ways but it's all ultimately the same thing. Since logging by nature offers asynchronous processing, you can migrate your publishers first and then the consumers without any downtime.
- manigandham 10y ago
  Azure has Event Hubs that are very similar to Kinesis/Kafka. https://azure.microsoft.com/en-us/services/event-hubs/ https://azure.microsoft.com/en-us/services/event-hubs/ They also have simpler Queues and Service Bus for RPC/lightweight message handling.

tibbon 10y ago

Kafka vs Redis. I've only used Redis... what should I know?

manigandham 10y ago
Redis is an in-memory (with persistence) key-value database that also implements some basic structures like lists, sets and hashes natively. Kafka is a distributed logging system that can ingest large amounts of data straight to disk, then allows for multiple consumers to read this data through a simple abstraction of topics and partitions. Consumers maintain their own position of where they last read up to (or re-read things if they want) and everything is sequential I/O which creates very high throughput.

nodesocket 10y ago

Can somebody provide a real-life use case for Kafka? I've seen comparisons between Redis, but what specifically does Kafka solve that Redis cannot?

manigandham 10y ago
Kafka and Redis are very different things - see this: https://news.ycombinator.com/item?id=11577312 https://news.ycombinator.com/item?id=11577312 Redis is a database, Kakfa is a data logging system built for scale and throughput. Event processing (of any kind like stocks, ad impressions, ecommerce purchases) are a great fit. Also good as a message queue unless you need ultra low-latency RPC.
- nodesocket 10y ago
  Gotcha, so then advantage of Kafka over Logstash + ElasticSearch?
  - manigandham 10y ago
    ElasticSearch is a database optimized for searching, not related at all. You can somewhat compare Kafka to Logstash but Kafka has no processing, it's purely a distributed log writing/reading/storage system that also scales far more than logstash can. You write data to it and then read from it with a basic messaging abstraction of topics and partitions.
    - ec109685 10y ago
      ElasticSearch can store sequenced number data, which is really all that Kafka is doing, so I don't think it is fair to say it isn't related at all.
      - allengeorge 10y ago
        So can a RDBMS... But that doesn't mean that Kafka and databases are related. As multiple comments have stated above, Kafka is really a distributed message subsystem. Its core interface is a set of topics that one can publish to, and that consumers can read from (in other words, a pub-sub system). Kafka doesn't inspect the message payload at all. Elasticsearch is a unstructured (to some extent) document store that's optimized around document search. So at the very least, the payload is important when using Elasticsearch.

manigandham 10y ago

This will be interesting to try out. I've used all the major cloud event/logging systems (Kinesis, Azure EventHubs, etc) and so far Google PubSub is the best in features and performance. Only downside with Google Pubsub can be latency (which I'm working on fixing by building a gRPC driver) but Kafka has proven to be too complicated to maintain in-house. If heroku can provide the speed without the ops overhead, it'll be some good competition to Google's option. Also want to note that Jay Kreps who helped build Kafka at LinkedIn is now behind http://www.confluent.io/ http://www.confluent.io/ which is like a better/enterprise version of Kafka.

alexatkeplar 10y ago
Not sure why you are comparing Google Cloud Pub/Sub to Kinesis - the former is a MQ system, not a distributed commit log. When creating a Kinesis consumer, I can specify whether I want to start reading a stream from a) TRIM_HORIZON (which is the earliest events in the stream which haven't yet been expired aka "trimmed"), b) LATEST which is the Cloud Pub/Sub capability, c) AT_SEQUENCE_NUMBER {x} which means from the event in the stream with the given offset ID, d) AFTER_SEQUENCE_NUMBER {x} which is the event immediately after c), e) AT_TIMESTAMP to read records from an arbitrary point in time. A Kinesis stream (like a Kafka topic) is a very special form of database - it exists independently of any consumers. By contrast, with Google Cloud Pub/Sub [1]: > When you create a subscription, the system establishes a sync point. That is, your subscriber is guaranteed to receive any message published after this point. [1] https://cloud.google.com/pubsub/subscriber https://cloud.google.com/pubsub/subscriber So the stream is not a first class entity in Cloud Pub/Sub - it's just a consumer-tied message queue.
- nivertech 10y ago
  Is there something like Kinesis' AT_TIMESTAMP in Kafka? I think the only way in to replay events in Google Cloud Pub/Sub is to create multiple subscriptions in advance, right after topic creation. But then I think you need to pay for the storage and event traversal requests.
rtehfm 10y ago
What are your thoughts on Kafka vs Flume?

ChartsNGraffs 10y ago

For anyone wanting to play with Kafka, Spotify's Kafka container was an invaluable resource for getting me up and running with Kafka. All the Zookeeper dependencies are taken care of allowing you to just start playing with Kafka right away. https://github.com/spotify/docker-kafka https://github.com/spotify/docker-kafka https://hub.docker.com/r/spotify/kafka/ https://hub.docker.com/r/spotify/kafka/

Jarmo 10y ago
I never tried spotify's container. Tried wurstmeister's, and was able to run it on a single server for testing purposes, but kept running into issues while clustering on different servers. Decided to use Ambari and have it do all the work for me instead.

mbseid 10y ago

As a former user of Kafka, this is awesome and it would have been a huge help for our company if this was available then. I'm glad to hear that a company is offering Kafka as opposed to other propriety versions(AWS Kinesis etc). One thing is odd though, there is no mention of disk space at all and only a configuration of retention time. One of Kafka's best features is the use of disk to store large amounts of messages, you are not RAM bound. Heroku seems to only allows you to set retention times? This could be awesome if they are giving you "unlimited" disk space, but could also be a beta oversight. Interested to see how this progresses.

uhoh-itsmaciek 10y ago
Hi, I'm Maciek and I work on the Heroku Kafka team. You don't have to think about disk space--it's on us to make sure there's enough to satisfy the retention settings you configure. We're excited to provide another great open-source project as a managed service!
- mbseid 10y ago
  Thanks for the update. That is awesome. Excited to see what people do with it.
ktamura 10y ago
Don't forget that Heroku is the original multi-tenant shop. I wouldnt be surprised if a single Kafka instance stores multiple customers's messages and elastically scale as more customers/data is added.
- sixwing 10y ago
  I'm Rand Fitzpatrick, and this is one of the products I work on at Heroku. None of our current Kafka offerings are multi-tenant.

franciscop 10y ago

I love Heroku and everything they are doing, it's doubtless a push forward for the web as a whole. However, the pricing for hobby sites (including SSL) is crazy from a personal point of view so I'm slowly moving my projects out of it [1][2]. I wish they had some kind of "Hobby Bundle". [1] http://umbrellajs.com/ http://umbrellajs.com/ [2] http://picnicss.com/ http://picnicss.com/

sudhirj 10y ago
Their pricing for hobby sites is 7$ + 10$ DB, which is very comparable with a self setup IaaS like DO and AWS. Personally I think the developer experience is much better on Heroku and quite worth it. SSL is a pain point, though I do empathize with them - I think they're doing something expensive for that. What I do is to use AWS Cloudfront and ACM for a free cert and site speedup - if they are personal projects the CF bill ought to be in the low few dollars anyway.
- why-el 10y ago
  + 7$ for a worker I think.
- flurdy 10y ago
  The $7 is comparable to one app per DO or AWS micro/nano server. So Heroku wins on convenience. If you have say 10 apps then Heroku costs 10*$7, but you might still only have used 1-3 servers depending on memory use of apps etc so then Heroku looses on cost. Naturally I got a total mix of quite a few on Heroku's classic or new free plan, some on their hobby plan, some on AWS, some on docker cloud, most proxied behind a SSL certificate running on AWS..... (https://flurdy.com/docs/letsencrypt/nginx.html https://flurdy.com/docs/letsencrypt/nginx.html)
- balamaci 10y ago
  DO has 5$ + VAT price for 512MB instance. I have no problem accommodating mysql + web on that.
- franciscop 10y ago
  Yeah, but I started with PHP I could just choose among many hosting companies for 5-10$/month and you get your shared space with unlimited domains, which was perfectly suited for my needs at that point. Of course as I learned more, Node and the such I needed better technology and that's why I moved to Heroku. So I'd love to see a "shared hosting for heroku" or similar. I think it will happen given some time, when the Node.js environment stabilizes more and more big players come.

andreasklinger 10y ago

For those wondering (all imo and only best guess) The biggest advantage of kafka is that all of the heroku marketplace all of a sudden becomes "plug and play" Essentially it's the "backend data" equivalent of what segment does for "frontend data". Example: What's the benefit of having a graphDB service in the marketplace if most people dont want to / cant invest engineering in keeping the data in (realtime) sync. With kafka they can establish standards that all partners can adapt to, they will simply offer piping of all heroku postgres/redis changes.

hmottestad 10y ago

Does anyone know if Kafka has improved on their data loss issues since tested by Aphyr? https://aphyr.com/posts/293-jepsen-kafka https://aphyr.com/posts/293-jepsen-kafka A quote from the article: "At the end of the run, Kafka typically acknowledges 98–100% of writes. However, half of those writes (all those made during the partition) are lost."

lars_francke 10y ago
Yes, the suggestion discussed by Aphyr has been implemented. You can now set up a lower bound on the ISR size (min.insync.replicas). Together with required.acks=-1 you can wait for a message to be committed to at least min.insync.replicas nodes. https://issues.apache.org/jira/browse/KAFKA-1555 https://issues.apache.org/jira/browse/KAFKA-1555

elcct 10y ago

My impression of Kafka was that this thing is bloated. How it compares to something like NSQ?

kasey_junk 10y ago
Its a completely different use case. Many times people call Kafka a "message queue" but its not. It's a distributed log service. Its possible to build a message queue on top of a distributed log service but there are reasons not to. Its better to think of Kafka as a database for events, not as a transport mechanism for those events. As for being bloated, Kafka lives in a very empty space, that is it supports fully ordered events to all consumers (and it has good HA options). The only other tool that I've come across that gives you the same data guarantees is Kinesis and it requires AWS. I've found that yes Kafka is complex, but its complex because its solving a complex problem, not because its bloated. That said, if you want a non-ordered message queue, use NSQ instead of Kafka.
- elcct 10y ago
  Thanks for explanation. I didn't know those things.

tenismyanswer 10y ago

All kafkaesque to me ;->

mtw 10y ago

What kind of companies or startups usually use this service?

rhodin 10y ago
Companies dealing with large amounts of data. A list with some companies using Apache Kafka can be found here: https://cwiki.apache.org/confluence/display/KAFKA/Powered+By https://cwiki.apache.org/confluence/display/KAFKA/Powered+By
- mtw 10y ago
  thanks. I guess my sites are not big enough (yet)
poooogles 10y ago
It's pretty big in ad tech, or anywhere that really does lots and lots of centralised logging (Datadog/Loggly both use Kafka). Lots of places also use it just as a message queue, some places for example write time series metrics to Kafka for monitoring.

jbob2000 10y ago

The comments in this thread are funny; Hey, what is Kafka? "It's a distributed logging system, not a message queue" Ok, what's the use case? describes a case when its used as a message queue