22 ms·

What every software engineer should know about Apache Kafka

597 points by aloknnikhil 6y ago

cosmotic 6y ago
Cool click-bait title
- fmjrey 6y ago
  Not necessarily, it's most likely a reference to the popular article from 2013: https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha...
  - ken 6y ago
    Or directly from 1991's "What Every Computer Scientist Should Know About Floating-Point Arithmetic".
    - pierrec 6y ago
      Where the "every" was actually justified, contrarily to its descendants.
gigatexal 6y ago
this is so timely, thank you!
unohoo 6y ago
Use pulsar - so much better than kafka
- lytedev 6y ago
  Why is that?
  - dominotw 6y ago
    millions of topics, no zookeeper ect. Kafka is addressing these shortcomings on the roadmap.
    - nitwit005 6y ago
      The Pulsar documentation says it requires Zookeeper: https://pulsar.apache.org/docs/en/administration-zk-bk/ https://pulsar.apache.org/docs/en/administration-zk-bk/
      - dominotw 6y ago
        oh sorry i meant storing topic info in zookeper that limits kafka to a certain number of topics.
        EdwardDiego 6y ago
        Nope, it also stores topic metadata in ZK - it's not exactly going to store that in the (near) stateless brokers, or in Bookkeeper - and BK also relies on ZK, but it's common to reuse the ZK quorum between the brokers and the bookies. It also needs an additional ZK for cluster replication.
    - oweiler 6y ago
      For a lot a projects this is hardly a problem. On the other hand Kafka is more mature and has a huge ecosystem (Kafka Connect, Kafka Streams, KSQL, ...).
      - math 6y ago
        Kafka also has less moving parts even today before zookeeper removal is complete (2 vs pulsar 3).
        biggestlou 6y ago
        But one of those moving parts of Pulsar, BookKeeper, means that you're no longer storing data on message brokers. Worth the extra puzzle piece for a lot of use cases.
- mrunkel 6y ago
  I am all for self-reliance, but if you really want to influence someone else, you might want to include a link to the project, especially when the only word you share has a much more prevalent meaning.
  - jooz 6y ago
    There was a Pulsar post on HN a few months ago, with interesting comments, some of the related to kafka. I keep Pulsar on the radar since that post: https://news.ycombinator.com/item?id=21936252 https://news.ycombinator.com/item?id=21936252
kevindeasis 6y ago
anyone wanna share their thoughts about deploying their own messaging system vs using a messaging system provided by their cloud provider?
- dtech 6y ago
  Distributed messaging is really hard to get right. It'll seem to work fine right up until you get weird bugs and unreliability during at the worst moments. I wouldn't recommend relying primarily on something vendor-specific like Amazon SQS, but there are very good out-of-the-box tools like RabbitMQ or Kafka available. Writing your own messaging system is like writing your own database, it's the wrong choice 99.9% of the time.
  - skyde 6y ago
    While I agree I see a big problem that people often pick Kafka when they really needed Rabbitmq because of the hype around Kafka and not understanding the difference between a transactional message broker ( Rabbitmq and ibm mq ...) as used in banking and distributed log store (pulsar/ Kafka)
    - LgWoodenBadger 6y ago
      Not sure that I’d want to rely too much on Rabbit’s transactional behavior based on their limitations. https://www.rabbitmq.com/semantics.html https://www.rabbitmq.com/semantics.html
- TheBlight 6y ago
  Avoiding vendor lock-in is the only quasi-sane reason I can think of.
- realtalk_sp 6y ago
  The GCP Pub/Sub API has largely replicated all the features you'd want out of Kafka (including Consumer Groups). The primary consideration at this point is cost. There's an inflection point in size (at some very large message volume) where it makes sense to start running your own Kafka cluster and hire a dedicated person or two to manage it. Most companies will never get anywhere close. Any project just starting out should use Pub/Sub. One thing I really like is that GCP provides emulators of Pub/Sub et al for local testing. That used to be a bit of an obstacle not too long ago. In terms of lock-in, I don't see how that applies to an AMQ. The data moving through it should only be transiently persisted, up to a week or two at most in the usual case. If you want to avoid cloud lock-in, have DB backups, use Postgres/MySQL/etc, containerize your service(s), replicate data in object storage, etc. Common sense stuff, if that's something that's of concern. Personally, I've seen "vendor lock-in" weaponized as an excuse for a lot of costly NIH bullshit. It's painful to reflect back on a project that could have involved literally a tenth of the time and pain it ended up taking because of that one choice alone.
seemslegit 6y ago
Hmm, I'm pretty sure that a software engineer developing safety-critical firmware for embedded medical systems does not need to know anything about Apache Kafka. Or a game developer. Or a web frontend developer. Given the title it's surprising how many software engineers can in fact go through life and career without ever knowing anything about Apache Kafka.
- vsareto 6y ago
  By now, "What Every Software Engineer Should Know" headlines aren't intended to be serious.
  - seemslegit 6y ago
    I wish, instead they are just not intended to be taken literally.
- DenisM 6y ago
  One can only make assumptions about the needs of every engineer out of profound ignorance, which accordingly sets expectations for the article contents. Conveniently so, I might add.
georgewfraser 6y ago
This notion of “stream-table duality” might be the most misleading, damaging idea floating around in software engineering today. Yes, you can turn a stream of events into a table of the present state. However, during that process you will eventually confront every single hard problem that relational database management systems have faced for decades. You will more or less have to write a full-fledged DBMS in your application code. And you will probably not do a great job, and will end up with dirty reads, phantoms, and all the other symptoms of a buggy database. Kafka is a message broker. It’s not a database and it’s not close to being a database. This idea of stream-table duality is not nearly as profound or important as it seems at first.
- zok3102 6y ago
  Reminds me of that time when database vendors were overreaching to be message queues/brokers - OracleAQ, MSMQ, etc.
- math 6y ago
  At the moment you more or less need to write a DBMS in your app code, but I don't think that's the end state. I think what we're seeing the beginnings of something big - it just might not seem like it yet because it's the v1 / no where near complete version. I think having all your data in a single system (Kafka, KsqlDb, ..) that allows you to work with it in cross paradigm ways will turn out to be very compelling.
  - eternalban 6y ago
    > At the moment you more or less need to write a DBMS in your app code Since we're discussing misunderstandings in the community, it should be pointed out that a Database Management System (DBMS) is not merely a database, to say nothing of a data store. Oracle, Postgres, et al are genuine "DBMS". ATM you are very likely putting together a data store in your app code.
    - tomnipotent 6y ago
      You're conflating DBMS with RDBMS. I interpreted the OP as DBMS=database, which absolutely includes application code that stores and retrieves data in proprietary formats. Even a linked list mmap'd to disk can be a database, just maybe not a very good one.
deleted 6y ago
[deleted]
gnfargbl 6y ago
Something that I wish I had known about Apache Kafka a year or so ago is that it essentially has no support for long-running tasks, i.e. tasks where longest-possible-worker-execution-time >> longest-tolerable-group-rebalance-time. After much angst in trying to work around this issue, I finally gave up and switched to Pulsar. Pulsar isn’t without it’s own issues (mostly around bugs and general maturity) but it handles this particular scenario admirably.
- ketralnis 6y ago
  It's true, message buses and work queues have different characteristics. It sounds like you want a work queue, not a message bus. I have very successful experience with using rabbitmq for work queueing, but as you mention there are others too.
  - biggestlou 6y ago
    Pulsar works quite well as a message queue: https://pulsar.apache.org/docs/en/cookbooks-message-queue/ https://pulsar.apache.org/docs/en/cookbooks-message-queue/
    - skyde 6y ago
      Pulsar can also support infinite data retention using data tiering and by spreading data on all nodes. It’s much better than Kafka when you don’t know when the consumer will come back looking for what changed
      - biggestlou 6y ago
        Spreading data on all BookKeeper nodes, yes :) Pulsar brokers are themselves stateless.
  - gnfargbl 6y ago
    You’re right. The issue is that in this particular application I need _both_ a work queue and a message bus. I’ve also successfully used Rabbit as a work queue, but it’s not high-throughput enough to meet my messaging needs. Pulsar seems to cope well in both roles. All that said, if I had my time again I’d probably just use one of the cloud providers’ solutions and spend my efforts elsewhere...
Traster 6y ago
There seems to be this common problem among relatively new technologies, that they're not actually aware of what the average person knows about them. So let me be the moron in the room. I work at a company that uses Kafka. What I know so far is that Kafka is broken. It seems to me that this article is more about what every software engineer who plans to re-skill as a kafka engineer should know.
- oweiler 6y ago
  In which way is Kafka broken?
  - skyde 6y ago
    The main aspect for me after 5 years of running several Kafka cluster to run production critical system at Microsoft is that it’s very fragile because of the bad design. distributed system are hard to design correctly. When using HDFS you can create a single 1TB file if the cluster have enough capacity. And with (hdfs hbase cassandra ..) if 5 server permanently fail in the span of 24 hours you don’t need manual intervention and your data is not lost. In Kafka your partition data need to fit entirely on a single server and you might still run into issue if that server also hosting other partition. And with Kafka if one by one the 3 server manually assigned as replica to one partition fail before an engineer is wake up in middle of the night and have time to fix it manually you just lost all your data.
    - LgWoodenBadger 6y ago
      Do you have examples of other systems that won’t lose all your data if all its replicas fail? This seems unavoidable in any scenario.
      - skyde 6y ago
        Assuming replication factor of 3 i’m not talking about 3 machine going down at exact same time, but in sequence. like 5 minutes apart. All other distributed system handle this with no issue. Ex: Cassandra, Hbase/hdfs, CockroachDB
sixhobbits 6y ago
So much criticism here! I've read a lot about Kafka over the last few years and I wish I had read this article earlier -- even basic questions like "Can Kafka store data persistently?" are not adequately answered in many intros to it. That said, I do find the tutorial flip-flops a bit in target audience. It's mainly "this is what Kafka is", but sometimes has weird asides like "This is how to optimise Kafka" (redundancy, number of partitions, etc) which are pretty distracting from the more fundamental points.
opportune 6y ago
>With Kafka, such a stream may record the history of your business for hundreds of years Do not do this. Kafka is not a database! Kafka should never be the source of truth for your business. The source of truth should be in whatever consumes data from Kafka downstream when messages are committed as read. Why? Because in your middle layer you can do all the data normalization, sanity checking, processing, and interaction with a REAL database system downstream that can give you things like transactions, ACID, etc. Of course Confluent wants you to try to use Kafka as a DB, so your usage of it is very high and you pay for the top support package and they have you by the cajones, but that doesn't mean you should do that. You will miss out on all the benefits of using a real database, with what benefit? Having a simple client API?
- sixdimensional 6y ago
  So, I've been having a back and forth with a colleague on this and I'm genuinely interested in why you so strongly suggest this. For the record, I have good real world experience with all kinds of databases (relational, NoSQL, and even legacy multivalue and hierarchical ones), and I don't see why what you say has to be "always true". One way of looking at Kafka is that is an unbundled transaction log, nothing more or less, so it could be used to permanently store and replay transactional activity, if one wishes. Noting that, even most databases don't store an immutable, permanent transaction log (as they often grow to be huge and are truncated every so often, and tables are used as the current state). This article by Confluent seems to cover the topic (yes, recognizing it is written by the very vendor you suggest is trying to lock us in): https://www.confluent.io/blog/okay-store-data-apache-kafka/ https://www.confluent.io/blog/okay-store-data-apache-kafka/. Ok, so how about the idea of a persistent, immutable, never-ending transaction log (uhoh, sounds like blockchain now!)? Setting aside Kafka for now, what do you think about the basic design pattern? To me it sounds a bit like it could represent a temporal database in raw transactional log form. Why not? EDIT: After rereading your comment I see your main concern is using Kafka as a database management system (DBMS). I would agree, that's not what Kafka is for. But, I don't think Confluent intends that use case, do they? I look at it more as an unbundled single component that is very useful by itself, and is part of a more complex data platform/architecture (ex. Lambda or Kappa architecture).
skyde 6y ago
What every engineer should know about Kafka is that it should not be used for anything critical like you would use Cassandra or Hbase. But if you are ok with partitions not being available for many hours or losing all written data because the cluster did not automatically move parution to 3 new replica after 2 of the replica failed ... then it’s a good scalable(speed) product. There is also no serious multi tenant support. So if you need multitenancy you gotta use kubernete and do one cluster per tenant and automate that yourself.
Rebelgecko 6y ago
I'm surprised by the amount of criticism in this thread. I've used Kafka in the past and it definitely got the job done (as a message bus, not using stream processing or the other more whiz-bang features). What do people use instead?
- skyde 6y ago
  If you need. Ordered log abstraction (Apache pulsar, Facebook log device ...) if you need a real transactional and highly available message broker any (my/jms) server like Rabbitmq IBM MQ .... Kafka in my experience is always the worse solution unless you need to aggregate http server log to do offline analytics using something like spark
- halbritt 6y ago
  For streaming data kafka is great. It gets misused in many ways. It can be made to guarantee order and has semantics for exactly once processing which may lead one to believe it can be used for transactions, but that forces all kinds of things to occur under the hood that makes performance at scale nearly impossible. I know folks like Netflix are using it for streaming UI logs into Flink for real time sentiment analysis. There are some very serious use cases for Kafka at scale. If one just needs a message broker, well, there are other, better ones.
  - skyde 6y ago
    exactly once messaging is a lie. it just mean you have duplicates message but let Kafka try to detect duplicate for you, which is almost never what you want. it’s much simpler to make your message processing indempotent.
    - opportune 6y ago
      Great comment. I have seen multiple people in my short career get burned by trying to figure out how to get exactly once semantics in a distributed system. The solution almost always, as you said, is idempotency (perhaps with the message producer appending a UUID field).
ckdarby 6y ago
What every software engineer should know about Kafka, it's dead. If you're not already technically chained into it and Confluence hasn't already upsold your poor organization avoid it. If you want the early flexibility and the rapid PoC just look at AWS Kinesis/Firehouse. If you're looking at large scale (+1 gbit ingest, 100k/s, kind of stuff) then Apache Pulsar is where to go.
- studmuffin650 6y ago
  I would argue that Kinesis is not the way to go for quick POC unless you're tied to the JVM. Pulsar is still niche in most enterprise. Kafka is not dead, there are many enterprises (including 2 successful ones I've worked at in the past 5 years) that have built POCs and successful products on Kafka. Its supports all language performantly and has tons of community support. I would argue there is nothing better to build a POC on.
  - andonisus 6y ago
    Why would you be tied to the JVM by using Kinesis? You can write a client library for any language. We did it for Go.
    - agacera 6y ago
      Because the official client from AWS is written in Java. It is possible to write clients in any language, however, it is not that simple especially when you need to handle logic of scalling out or in your kinesis stream (that will split or join shards) and when you have multiple consumers in the same consumer group (you will need a distributed locking mechanism and a logic to steal locks if one consumer dies). So it is not trivial.
      - etxm 6y ago
        Sure it is. AWS ships a bunch of clients for Kinesis as well as a kinesis agent for shipping log files. https://docs.aws.amazon.com/sdk-for-go/api/service/kinesis/ https://docs.aws.amazon.com/sdk-for-go/api/service/kinesis/
effnorwood 6y ago
It sucks
skyde 6y ago
Anyone know how large scale chat system ( facebook messenger, ...) are implemented? My guess is message are in a data store like hbase and a very simple notification system let user that are online know to fetch for new entry
- gaogao 6y ago
  It was roughly that at one point, http://highscalability.com/blog/2010/11/16/facebooks-new-real-time-messaging-system-hbase-to-store-135.html http://highscalability.com/blog/2010/11/16/facebooks-new-rea..., but now does something else, https://engineering.fb.com/data-infrastructure/messenger/ https://engineering.fb.com/data-infrastructure/messenger/
  - skyde 6y ago
    thanks a lot. This seem to be a very simple but smart design.
analognoise 6y ago
"Every engineer". The whole human activity of reducing science to practical art will do just fine without knowing Apache Kafka, thanks.
PaulWaldman 6y ago
Is there a reason Kafka wouldn't choose to leverage an existing mature RDBMS for their table storage instead of rolling their own?
- geodel 6y ago
  Well they use rocksDB internally. Though that is not RDBMS.
deleted 6y ago
[deleted]
badrabbit 6y ago
Some times you should just use Graylog (kafka+elastic) ,especially if you are already comfortable with Elastic. You get to scale,retain and monitor your data in addition to stream processing. If I have fairly small Go webapp that needs stream processing, I would just use Graylog instead of trying to use Kafka directly.
bonquesha99 6y ago
As a Kafka alternative, has anyone attempted to use PostgreSQL logical replication with table partitioning for async service communication? Proof of concept (with diagrams in the comments): https://gist.github.com/shuber/8e53d42d0de40e90edaf4fb182b59dfc https://gist.github.com/shuber/8e53d42d0de40e90edaf4fb182b59... Services would commit messages to their own databases along with the rest of their data (with the same transactional guarantees) and then messages are "realtime" replicated (with all of its features and guarantees) to the receiving service's database where their workers (e.g. https://github.com/que-rb/que https://github.com/que-rb/que, skip locked polling, etc) are waiting to respond by inserting messages into their database to be replicated back. Throw in a trigger to automatically acknowledge/cleanup/notify messages and I think we've got something that resembles a queue? Maybe make that same trigger match incoming messages against a "routes" table (based on message type, certain JSON schemas in the payload, etc) and write matches to the que-rb jobs table instead for some kind of distributed/replicated work queue hybrid? I'm looking to poke holes in this concept before sinking anymore time exploring the idea. Any feedback/warnings/concerns would be much appreciated, thanks for your time! Other discussions: * https://old.reddit.com/r/PostgreSQL/comments/gkdp6p/logical_replication_for_async_service/ https://old.reddit.com/r/PostgreSQL/comments/gkdp6p/logical_... * https://dba.stackexchange.com/questions/267266/postgresql-logical-replication-for-async-service-communication https://dba.stackexchange.com/questions/267266/postgresql-lo... * https://www.postgresql.org/message-id/CAM8f5Mi1Ftj%2B48PZxN1AbM-P%3D4YMLENY5zRaPwTbmbkFwCsTkA%40mail.gmail.com https://www.postgresql.org/message-id/CAM8f5Mi1Ftj%2B48PZxN1...
- skyde 6y ago
  interesting idea: I think one issue is that the write throughout of a single master Instance is very limited. But if working set fit in memory on the instances processing the writes, you could use PostgreSQL logical replication to update materialized view on other server easily. but then it start looking like Amazon aurora or Datomic database.
- PopeDotNinja 6y ago
  Just as Kafka isn't a database, PostgreSQL isn't a queue/broker. You can use it that way, but you'll spend a lot of time tweaking it, and I suspect you'll find it's too slow for non-trivial workloads.
tspann 6y ago
Kafka for events Flink for continous sql against topic events Kudu for event storage. Databases for data storage and queries
Twisell 6y ago
Just a remark to writers, when you redact an "introduction to smth" please refrain from writing down the name of your product 50 or more time in the first paragraph. It's totally frustrating and made me run away to just look up wikipedia instead to get a grasp of the general idea. Example: You've probably heard of smth thing and wonder how smth differ from smth else. In this article about smth we will dive into smth things to discover how smth is well better suited to do things thanks to smth things and other things that are really specific to smth! The power of smth enable things to things in a way things do things that make smth something you need to learn about. So during this journey about smth will be cut in four part the first being an introduction. (...) --sudo click the link for first part This is the first part of a fourth series about smth to learn more about things and things in smth. Smth is very specific about things, and that's specifically why smth is well suited for things. Smth is a new way to do something that will make you think more about things and things. Let's now dive into smth... Smth use things as a things to do the smth things with tings on smth......... --sudo repeat marketing ad nauseam No matter of wether the actual product is pure gold or pure garbage you probably just lost 50% of the readers at some point.
- unphased 6y ago
  i totally agree with what you're saying but you should also know that redact doesnt fit in your sentence
  - Twisell 6y ago
    Too late to edit sorry. English is not my native language and I clearly see my translation mistake now. Thank for pointing out ;)
a_c 6y ago
1st rule is you don't need kafka. Second rule is don't forget about 1st rule.
xrisk 6y ago
can someone give me a tldr for what problem kafka solves / what situation calls for the usage of kafka?
simonjgreen 6y ago
"Your scientists were so preoccupied with whether or not they could, they didn’t stop to think if they should."
est 6y ago
One thing Kafka bite me is for each partition there can be only one consumer. If your consumer had performance issue (e.g using CPython) then you are out of luck.
- bootlooped 6y ago
  The thing this article pointed out to me, which I didn't know before, it that is why you should just set the partition count very high to begin with. Then you just horizontally scale consumers.
  - est 6y ago
    Yeas that's the lesson I learned the hard way. The publisher was from another department and had only 1 partition with large volume of data.
nemetroid 6y ago
I read the introduction to the series, and then the introduction to the first part, and I’m still not sure exactly what Kafka is, or why I (as part of ”every software engineer”) need to know anything about it. The title suggests that the article(s) will convey some concepts that are useful in a broad sense, but from a skim, this looks like a lot of details about some database-ish product, which probably are good to know if you use that product, but not so much otherwise.

597 points by aloknnikhil 6y ago

cosmotic 6y ago

Cool click-bait title

fmjrey 6y ago
Not necessarily, it's most likely a reference to the popular article from 2013: https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha...
- ken 6y ago
  Or directly from 1991's "What Every Computer Scientist Should Know About Floating-Point Arithmetic".
  - pierrec 6y ago
    Where the "every" was actually justified, contrarily to its descendants.

gigatexal 6y ago

this is so timely, thank you!

unohoo 6y ago

Use pulsar - so much better than kafka

lytedev 6y ago
Why is that?
- dominotw 6y ago
  millions of topics, no zookeeper ect. Kafka is addressing these shortcomings on the roadmap.
  - nitwit005 6y ago
    The Pulsar documentation says it requires Zookeeper: https://pulsar.apache.org/docs/en/administration-zk-bk/ https://pulsar.apache.org/docs/en/administration-zk-bk/
    - dominotw 6y ago
      oh sorry i meant storing topic info in zookeper that limits kafka to a certain number of topics.
      - EdwardDiego 6y ago
        Nope, it also stores topic metadata in ZK - it's not exactly going to store that in the (near) stateless brokers, or in Bookkeeper - and BK also relies on ZK, but it's common to reuse the ZK quorum between the brokers and the bookies. It also needs an additional ZK for cluster replication.
  - oweiler 6y ago
    For a lot a projects this is hardly a problem. On the other hand Kafka is more mature and has a huge ecosystem (Kafka Connect, Kafka Streams, KSQL, ...).
    - math 6y ago
      Kafka also has less moving parts even today before zookeeper removal is complete (2 vs pulsar 3).
      - biggestlou 6y ago
        But one of those moving parts of Pulsar, BookKeeper, means that you're no longer storing data on message brokers. Worth the extra puzzle piece for a lot of use cases.
mrunkel 6y ago
I am all for self-reliance, but if you really want to influence someone else, you might want to include a link to the project, especially when the only word you share has a much more prevalent meaning.
- jooz 6y ago
  There was a Pulsar post on HN a few months ago, with interesting comments, some of the related to kafka. I keep Pulsar on the radar since that post: https://news.ycombinator.com/item?id=21936252 https://news.ycombinator.com/item?id=21936252

kevindeasis 6y ago

anyone wanna share their thoughts about deploying their own messaging system vs using a messaging system provided by their cloud provider?

dtech 6y ago
Distributed messaging is really hard to get right. It'll seem to work fine right up until you get weird bugs and unreliability during at the worst moments. I wouldn't recommend relying primarily on something vendor-specific like Amazon SQS, but there are very good out-of-the-box tools like RabbitMQ or Kafka available. Writing your own messaging system is like writing your own database, it's the wrong choice 99.9% of the time.
- skyde 6y ago
  While I agree I see a big problem that people often pick Kafka when they really needed Rabbitmq because of the hype around Kafka and not understanding the difference between a transactional message broker ( Rabbitmq and ibm mq ...) as used in banking and distributed log store (pulsar/ Kafka)
  - LgWoodenBadger 6y ago
    Not sure that I’d want to rely too much on Rabbit’s transactional behavior based on their limitations. https://www.rabbitmq.com/semantics.html https://www.rabbitmq.com/semantics.html
TheBlight 6y ago
Avoiding vendor lock-in is the only quasi-sane reason I can think of.
realtalk_sp 6y ago
The GCP Pub/Sub API has largely replicated all the features you'd want out of Kafka (including Consumer Groups). The primary consideration at this point is cost. There's an inflection point in size (at some very large message volume) where it makes sense to start running your own Kafka cluster and hire a dedicated person or two to manage it. Most companies will never get anywhere close. Any project just starting out should use Pub/Sub. One thing I really like is that GCP provides emulators of Pub/Sub et al for local testing. That used to be a bit of an obstacle not too long ago. In terms of lock-in, I don't see how that applies to an AMQ. The data moving through it should only be transiently persisted, up to a week or two at most in the usual case. If you want to avoid cloud lock-in, have DB backups, use Postgres/MySQL/etc, containerize your service(s), replicate data in object storage, etc. Common sense stuff, if that's something that's of concern. Personally, I've seen "vendor lock-in" weaponized as an excuse for a lot of costly NIH bullshit. It's painful to reflect back on a project that could have involved literally a tenth of the time and pain it ended up taking because of that one choice alone.

seemslegit 6y ago

Hmm, I'm pretty sure that a software engineer developing safety-critical firmware for embedded medical systems does not need to know anything about Apache Kafka. Or a game developer. Or a web frontend developer. Given the title it's surprising how many software engineers can in fact go through life and career without ever knowing anything about Apache Kafka.

vsareto 6y ago
By now, "What Every Software Engineer Should Know" headlines aren't intended to be serious.
- seemslegit 6y ago
  I wish, instead they are just not intended to be taken literally.
DenisM 6y ago
One can only make assumptions about the needs of every engineer out of profound ignorance, which accordingly sets expectations for the article contents. Conveniently so, I might add.

georgewfraser 6y ago

This notion of “stream-table duality” might be the most misleading, damaging idea floating around in software engineering today. Yes, you can turn a stream of events into a table of the present state. However, during that process you will eventually confront every single hard problem that relational database management systems have faced for decades. You will more or less have to write a full-fledged DBMS in your application code. And you will probably not do a great job, and will end up with dirty reads, phantoms, and all the other symptoms of a buggy database. Kafka is a message broker. It’s not a database and it’s not close to being a database. This idea of stream-table duality is not nearly as profound or important as it seems at first.

zok3102 6y ago
Reminds me of that time when database vendors were overreaching to be message queues/brokers - OracleAQ, MSMQ, etc.
math 6y ago
At the moment you more or less need to write a DBMS in your app code, but I don't think that's the end state. I think what we're seeing the beginnings of something big - it just might not seem like it yet because it's the v1 / no where near complete version. I think having all your data in a single system (Kafka, KsqlDb, ..) that allows you to work with it in cross paradigm ways will turn out to be very compelling.
- eternalban 6y ago
  > At the moment you more or less need to write a DBMS in your app code Since we're discussing misunderstandings in the community, it should be pointed out that a Database Management System (DBMS) is not merely a database, to say nothing of a data store. Oracle, Postgres, et al are genuine "DBMS". ATM you are very likely putting together a data store in your app code.
  - tomnipotent 6y ago
    You're conflating DBMS with RDBMS. I interpreted the OP as DBMS=database, which absolutely includes application code that stores and retrieves data in proprietary formats. Even a linked list mmap'd to disk can be a database, just maybe not a very good one.

deleted 6y ago

[deleted]

gnfargbl 6y ago

Something that I wish I had known about Apache Kafka a year or so ago is that it essentially has no support for long-running tasks, i.e. tasks where longest-possible-worker-execution-time >> longest-tolerable-group-rebalance-time. After much angst in trying to work around this issue, I finally gave up and switched to Pulsar. Pulsar isn’t without it’s own issues (mostly around bugs and general maturity) but it handles this particular scenario admirably.

ketralnis 6y ago
It's true, message buses and work queues have different characteristics. It sounds like you want a work queue, not a message bus. I have very successful experience with using rabbitmq for work queueing, but as you mention there are others too.
- biggestlou 6y ago
  Pulsar works quite well as a message queue: https://pulsar.apache.org/docs/en/cookbooks-message-queue/ https://pulsar.apache.org/docs/en/cookbooks-message-queue/
  - skyde 6y ago
    Pulsar can also support infinite data retention using data tiering and by spreading data on all nodes. It’s much better than Kafka when you don’t know when the consumer will come back looking for what changed
    - biggestlou 6y ago
      Spreading data on all BookKeeper nodes, yes :) Pulsar brokers are themselves stateless.
- gnfargbl 6y ago
  You’re right. The issue is that in this particular application I need _both_ a work queue and a message bus. I’ve also successfully used Rabbit as a work queue, but it’s not high-throughput enough to meet my messaging needs. Pulsar seems to cope well in both roles. All that said, if I had my time again I’d probably just use one of the cloud providers’ solutions and spend my efforts elsewhere...

Traster 6y ago

There seems to be this common problem among relatively new technologies, that they're not actually aware of what the average person knows about them. So let me be the moron in the room. I work at a company that uses Kafka. What I know so far is that Kafka is broken. It seems to me that this article is more about what every software engineer who plans to re-skill as a kafka engineer should know.

oweiler 6y ago
In which way is Kafka broken?
- skyde 6y ago
  The main aspect for me after 5 years of running several Kafka cluster to run production critical system at Microsoft is that it’s very fragile because of the bad design. distributed system are hard to design correctly. When using HDFS you can create a single 1TB file if the cluster have enough capacity. And with (hdfs hbase cassandra ..) if 5 server permanently fail in the span of 24 hours you don’t need manual intervention and your data is not lost. In Kafka your partition data need to fit entirely on a single server and you might still run into issue if that server also hosting other partition. And with Kafka if one by one the 3 server manually assigned as replica to one partition fail before an engineer is wake up in middle of the night and have time to fix it manually you just lost all your data.
  - LgWoodenBadger 6y ago
    Do you have examples of other systems that won’t lose all your data if all its replicas fail? This seems unavoidable in any scenario.
    - skyde 6y ago
      Assuming replication factor of 3 i’m not talking about 3 machine going down at exact same time, but in sequence. like 5 minutes apart. All other distributed system handle this with no issue. Ex: Cassandra, Hbase/hdfs, CockroachDB

sixhobbits 6y ago

So much criticism here! I've read a lot about Kafka over the last few years and I wish I had read this article earlier -- even basic questions like "Can Kafka store data persistently?" are not adequately answered in many intros to it. That said, I do find the tutorial flip-flops a bit in target audience. It's mainly "this is what Kafka is", but sometimes has weird asides like "This is how to optimise Kafka" (redundancy, number of partitions, etc) which are pretty distracting from the more fundamental points.

opportune 6y ago

>With Kafka, such a stream may record the history of your business for hundreds of years Do not do this. Kafka is not a database! Kafka should never be the source of truth for your business. The source of truth should be in whatever consumes data from Kafka downstream when messages are committed as read. Why? Because in your middle layer you can do all the data normalization, sanity checking, processing, and interaction with a REAL database system downstream that can give you things like transactions, ACID, etc. Of course Confluent wants you to try to use Kafka as a DB, so your usage of it is very high and you pay for the top support package and they have you by the cajones, but that doesn't mean you should do that. You will miss out on all the benefits of using a real database, with what benefit? Having a simple client API?

sixdimensional 6y ago
So, I've been having a back and forth with a colleague on this and I'm genuinely interested in why you so strongly suggest this. For the record, I have good real world experience with all kinds of databases (relational, NoSQL, and even legacy multivalue and hierarchical ones), and I don't see why what you say has to be "always true". One way of looking at Kafka is that is an unbundled transaction log, nothing more or less, so it could be used to permanently store and replay transactional activity, if one wishes. Noting that, even most databases don't store an immutable, permanent transaction log (as they often grow to be huge and are truncated every so often, and tables are used as the current state). This article by Confluent seems to cover the topic (yes, recognizing it is written by the very vendor you suggest is trying to lock us in): https://www.confluent.io/blog/okay-store-data-apache-kafka/ https://www.confluent.io/blog/okay-store-data-apache-kafka/. Ok, so how about the idea of a persistent, immutable, never-ending transaction log (uhoh, sounds like blockchain now!)? Setting aside Kafka for now, what do you think about the basic design pattern? To me it sounds a bit like it could represent a temporal database in raw transactional log form. Why not? EDIT: After rereading your comment I see your main concern is using Kafka as a database management system (DBMS). I would agree, that's not what Kafka is for. But, I don't think Confluent intends that use case, do they? I look at it more as an unbundled single component that is very useful by itself, and is part of a more complex data platform/architecture (ex. Lambda or Kappa architecture).

skyde 6y ago

What every engineer should know about Kafka is that it should not be used for anything critical like you would use Cassandra or Hbase. But if you are ok with partitions not being available for many hours or losing all written data because the cluster did not automatically move parution to 3 new replica after 2 of the replica failed ... then it’s a good scalable(speed) product. There is also no serious multi tenant support. So if you need multitenancy you gotta use kubernete and do one cluster per tenant and automate that yourself.

Rebelgecko 6y ago

I'm surprised by the amount of criticism in this thread. I've used Kafka in the past and it definitely got the job done (as a message bus, not using stream processing or the other more whiz-bang features). What do people use instead?

skyde 6y ago
If you need. Ordered log abstraction (Apache pulsar, Facebook log device ...) if you need a real transactional and highly available message broker any (my/jms) server like Rabbitmq IBM MQ .... Kafka in my experience is always the worse solution unless you need to aggregate http server log to do offline analytics using something like spark
halbritt 6y ago
For streaming data kafka is great. It gets misused in many ways. It can be made to guarantee order and has semantics for exactly once processing which may lead one to believe it can be used for transactions, but that forces all kinds of things to occur under the hood that makes performance at scale nearly impossible. I know folks like Netflix are using it for streaming UI logs into Flink for real time sentiment analysis. There are some very serious use cases for Kafka at scale. If one just needs a message broker, well, there are other, better ones.
- skyde 6y ago
  exactly once messaging is a lie. it just mean you have duplicates message but let Kafka try to detect duplicate for you, which is almost never what you want. it’s much simpler to make your message processing indempotent.
  - opportune 6y ago
    Great comment. I have seen multiple people in my short career get burned by trying to figure out how to get exactly once semantics in a distributed system. The solution almost always, as you said, is idempotency (perhaps with the message producer appending a UUID field).

ckdarby 6y ago

What every software engineer should know about Kafka, it's dead. If you're not already technically chained into it and Confluence hasn't already upsold your poor organization avoid it. If you want the early flexibility and the rapid PoC just look at AWS Kinesis/Firehouse. If you're looking at large scale (+1 gbit ingest, 100k/s, kind of stuff) then Apache Pulsar is where to go.

studmuffin650 6y ago
I would argue that Kinesis is not the way to go for quick POC unless you're tied to the JVM. Pulsar is still niche in most enterprise. Kafka is not dead, there are many enterprises (including 2 successful ones I've worked at in the past 5 years) that have built POCs and successful products on Kafka. Its supports all language performantly and has tons of community support. I would argue there is nothing better to build a POC on.
- andonisus 6y ago
  Why would you be tied to the JVM by using Kinesis? You can write a client library for any language. We did it for Go.
  - agacera 6y ago
    Because the official client from AWS is written in Java. It is possible to write clients in any language, however, it is not that simple especially when you need to handle logic of scalling out or in your kinesis stream (that will split or join shards) and when you have multiple consumers in the same consumer group (you will need a distributed locking mechanism and a logic to steal locks if one consumer dies). So it is not trivial.
    - etxm 6y ago
      Sure it is. AWS ships a bunch of clients for Kinesis as well as a kinesis agent for shipping log files. https://docs.aws.amazon.com/sdk-for-go/api/service/kinesis/ https://docs.aws.amazon.com/sdk-for-go/api/service/kinesis/

effnorwood 6y ago

It sucks

skyde 6y ago

Anyone know how large scale chat system ( facebook messenger, ...) are implemented? My guess is message are in a data store like hbase and a very simple notification system let user that are online know to fetch for new entry

gaogao 6y ago
It was roughly that at one point, http://highscalability.com/blog/2010/11/16/facebooks-new-real-time-messaging-system-hbase-to-store-135.html http://highscalability.com/blog/2010/11/16/facebooks-new-rea..., but now does something else, https://engineering.fb.com/data-infrastructure/messenger/ https://engineering.fb.com/data-infrastructure/messenger/
- skyde 6y ago
  thanks a lot. This seem to be a very simple but smart design.

analognoise 6y ago

"Every engineer". The whole human activity of reducing science to practical art will do just fine without knowing Apache Kafka, thanks.

PaulWaldman 6y ago

Is there a reason Kafka wouldn't choose to leverage an existing mature RDBMS for their table storage instead of rolling their own?

geodel 6y ago
Well they use rocksDB internally. Though that is not RDBMS.

deleted 6y ago

[deleted]

badrabbit 6y ago

Some times you should just use Graylog (kafka+elastic) ,especially if you are already comfortable with Elastic. You get to scale,retain and monitor your data in addition to stream processing. If I have fairly small Go webapp that needs stream processing, I would just use Graylog instead of trying to use Kafka directly.

bonquesha99 6y ago

As a Kafka alternative, has anyone attempted to use PostgreSQL logical replication with table partitioning for async service communication? Proof of concept (with diagrams in the comments): https://gist.github.com/shuber/8e53d42d0de40e90edaf4fb182b59dfc https://gist.github.com/shuber/8e53d42d0de40e90edaf4fb182b59... Services would commit messages to their own databases along with the rest of their data (with the same transactional guarantees) and then messages are "realtime" replicated (with all of its features and guarantees) to the receiving service's database where their workers (e.g. https://github.com/que-rb/que https://github.com/que-rb/que, skip locked polling, etc) are waiting to respond by inserting messages into their database to be replicated back. Throw in a trigger to automatically acknowledge/cleanup/notify messages and I think we've got something that resembles a queue? Maybe make that same trigger match incoming messages against a "routes" table (based on message type, certain JSON schemas in the payload, etc) and write matches to the que-rb jobs table instead for some kind of distributed/replicated work queue hybrid? I'm looking to poke holes in this concept before sinking anymore time exploring the idea. Any feedback/warnings/concerns would be much appreciated, thanks for your time! Other discussions: * https://old.reddit.com/r/PostgreSQL/comments/gkdp6p/logical_replication_for_async_service/ https://old.reddit.com/r/PostgreSQL/comments/gkdp6p/logical_... * https://dba.stackexchange.com/questions/267266/postgresql-logical-replication-for-async-service-communication https://dba.stackexchange.com/questions/267266/postgresql-lo... * https://www.postgresql.org/message-id/CAM8f5Mi1Ftj%2B48PZxN1AbM-P%3D4YMLENY5zRaPwTbmbkFwCsTkA%40mail.gmail.com https://www.postgresql.org/message-id/CAM8f5Mi1Ftj%2B48PZxN1...

skyde 6y ago
interesting idea: I think one issue is that the write throughout of a single master Instance is very limited. But if working set fit in memory on the instances processing the writes, you could use PostgreSQL logical replication to update materialized view on other server easily. but then it start looking like Amazon aurora or Datomic database.
PopeDotNinja 6y ago
Just as Kafka isn't a database, PostgreSQL isn't a queue/broker. You can use it that way, but you'll spend a lot of time tweaking it, and I suspect you'll find it's too slow for non-trivial workloads.

tspann 6y ago

Kafka for events Flink for continous sql against topic events Kudu for event storage. Databases for data storage and queries

Twisell 6y ago

Just a remark to writers, when you redact an "introduction to smth" please refrain from writing down the name of your product 50 or more time in the first paragraph. It's totally frustrating and made me run away to just look up wikipedia instead to get a grasp of the general idea. Example: You've probably heard of smth thing and wonder how smth differ from smth else. In this article about smth we will dive into smth things to discover how smth is well better suited to do things thanks to smth things and other things that are really specific to smth! The power of smth enable things to things in a way things do things that make smth something you need to learn about. So during this journey about smth will be cut in four part the first being an introduction. (...) --sudo click the link for first part This is the first part of a fourth series about smth to learn more about things and things in smth. Smth is very specific about things, and that's specifically why smth is well suited for things. Smth is a new way to do something that will make you think more about things and things. Let's now dive into smth... Smth use things as a things to do the smth things with tings on smth......... --sudo repeat marketing ad nauseam No matter of wether the actual product is pure gold or pure garbage you probably just lost 50% of the readers at some point.

unphased 6y ago
i totally agree with what you're saying but you should also know that redact doesnt fit in your sentence
- Twisell 6y ago
  Too late to edit sorry. English is not my native language and I clearly see my translation mistake now. Thank for pointing out ;)

a_c 6y ago

1st rule is you don't need kafka. Second rule is don't forget about 1st rule.

xrisk 6y ago

can someone give me a tldr for what problem kafka solves / what situation calls for the usage of kafka?

simonjgreen 6y ago

"Your scientists were so preoccupied with whether or not they could, they didn’t stop to think if they should."

est 6y ago

One thing Kafka bite me is for each partition there can be only one consumer. If your consumer had performance issue (e.g using CPython) then you are out of luck.

bootlooped 6y ago
The thing this article pointed out to me, which I didn't know before, it that is why you should just set the partition count very high to begin with. Then you just horizontally scale consumers.
- est 6y ago
  Yeas that's the lesson I learned the hard way. The publisher was from another department and had only 1 partition with large volume of data.

nemetroid 6y ago

I read the introduction to the series, and then the introduction to the first part, and I’m still not sure exactly what Kafka is, or why I (as part of ”every software engineer”) need to know anything about it. The title suggests that the article(s) will convey some concepts that are useful in a broad sense, but from a skim, this looks like a lot of details about some database-ish product, which probably are good to know if you use that product, but not so much otherwise.