6 ms·

I got tricked :) This is not about Apache Kafka but about Bohemian novelist Franz Kafka. Nothing wrong, just saying my mind.

libraryofbabel 6y ago
Not the first HN thread to have this happen: https://news.ycombinator.com/item?id=23599427 https://news.ycombinator.com/item?id=23599427 Come for the distributed, replicated, partitioned log. Stay for the existential angst, sinister bureaucracy, and paranoia in the face of the absurd.
- whoisburbansky 6y ago
  Realistically, aren't they the same thing in the end? Can you have the former without truly feeling the latter?
  - libraryofbabel 6y ago
    The defining characteristics of distributed systems are random failure and unreliable communication. So yes. Sounds pretty Kafkaesque to me. (Or Byzantine.) Distributed consensus in particular strikes me as a problem quite worthy of The Castle or The Trial.
- inopinatus 6y ago
  back in the day we combined the two and called it Usenet.
  - 082349872349872 6y ago
    Does https://olduse.net https://olduse.net still work for anyone else?
- ritter2a 6y ago
  I consider it a fun little game to guess for each mention of "Kafka" in a HN title, whether it means the author or the software. Definitely not trivial, since this time, I would have guessed "Kafka in Pieces" to be an introduction to the software, component by component.
  - danpeddle 6y ago
    If there is something out there which is this introduction, would love to read it!
geodel 6y ago
Sure. I have heard this now so many time here. To me however Kafka the software does resemble a Kafkaesque way: the literary style found in Kafka novels. Apache kafka is nightmarishly complex software which barely works. And I never fully grasped what I would use it for.
- prerok 6y ago
  We came for the gems but all we got is shards :)
- nullsense 6y ago
  It's named Kafka because LinkedIn's data access story was Kafkaesque and they wanted a way to wrangle it, so they wrote a tool to do that and named it Kafka. It's good if you have a use for it.
- jpgvm 6y ago
  I think Kafka inherits too much of this "heinously complex" reputation from Zookeeper (which itself is also not heinously complex, just some people have an allergy to either JVM, ZAB or both). Internally it's one of the simpler pieces of software I rely on. It's base API is small and effective and works "as you would expect". Replication uses the same fetch API consumers use, because why wouldn't it? Controller election uses tried and true ZK patterns, it's log storage is pretty simple, even the compaction logic is understandable. To be fair I have spent a lot of time with it, patched it and it's pretty much the core of my toolbox but I don't think this is a controversial opinion among data infrastructure engineers. Compared to other software we work with Kafka is some of the simpler, dumber stuff which is refreshing. On the other hand I really want to learn Pulsar but it definitely is more complex, my hope is that complexity pays off with big architectural advantages but we will see. :) On the topic of what to use it for chances are if you don't know what it's for you probably don't have the problems it's meant to solve. I don't mean that in a derogatory way just that it's designed for large distributed architectures where many interested parties want to consume the same data. Or where a small number of very high throughput applications need a buffered transport that can take the load and spill to disk etc. I.e it is inherently niche, most companies don't have these problems.
wanghq 6y ago
"The state in which Kafka left The Castle is representative of the condition of his entire oeuvre."
azureel 6y ago
Me too. Maybe this can be solved by updating the title as "Franz Kafka in Pieces". I know, respecting the original title is a nice thing. On the other hand, HN is a tech oriented web site and +90% of the times if there is a Kafka in the title it is related to the stream processing framework, not the writer. So, just writing the writer's full name (in those rare occurances where the writer is mentioned here in this news site) can solve that tiny problem. Those are my two cents...
- geodel 6y ago
  That is one way. I think may be Kafka the software need to be called Apache Kafka here. Not sure if kafka the software will remain popular in next few years as cool developers seem already moving to Apache Pulsar, NATS and so on. > +90% of the times if there is a Kafka in the title it is related to the stream processing framework, not the writer This makes me think why these crappily implemented software choose such pretentious literary names. 'Linkedin Sprof' would be a suitable name for this below average software.

libraryofbabel 6y ago

Not the first HN thread to have this happen: https://news.ycombinator.com/item?id=23599427 https://news.ycombinator.com/item?id=23599427 Come for the distributed, replicated, partitioned log. Stay for the existential angst, sinister bureaucracy, and paranoia in the face of the absurd.

whoisburbansky 6y ago
Realistically, aren't they the same thing in the end? Can you have the former without truly feeling the latter?
- libraryofbabel 6y ago
  The defining characteristics of distributed systems are random failure and unreliable communication. So yes. Sounds pretty Kafkaesque to me. (Or Byzantine.) Distributed consensus in particular strikes me as a problem quite worthy of The Castle or The Trial.
inopinatus 6y ago
back in the day we combined the two and called it Usenet.
- 082349872349872 6y ago
  Does https://olduse.net https://olduse.net still work for anyone else?
ritter2a 6y ago
I consider it a fun little game to guess for each mention of "Kafka" in a HN title, whether it means the author or the software. Definitely not trivial, since this time, I would have guessed "Kafka in Pieces" to be an introduction to the software, component by component.
- danpeddle 6y ago
  If there is something out there which is this introduction, would love to read it!

geodel 6y ago

Sure. I have heard this now so many time here. To me however Kafka the software does resemble a Kafkaesque way: the literary style found in Kafka novels. Apache kafka is nightmarishly complex software which barely works. And I never fully grasped what I would use it for.

prerok 6y ago
We came for the gems but all we got is shards :)
nullsense 6y ago
It's named Kafka because LinkedIn's data access story was Kafkaesque and they wanted a way to wrangle it, so they wrote a tool to do that and named it Kafka. It's good if you have a use for it.
jpgvm 6y ago
I think Kafka inherits too much of this "heinously complex" reputation from Zookeeper (which itself is also not heinously complex, just some people have an allergy to either JVM, ZAB or both). Internally it's one of the simpler pieces of software I rely on. It's base API is small and effective and works "as you would expect". Replication uses the same fetch API consumers use, because why wouldn't it? Controller election uses tried and true ZK patterns, it's log storage is pretty simple, even the compaction logic is understandable. To be fair I have spent a lot of time with it, patched it and it's pretty much the core of my toolbox but I don't think this is a controversial opinion among data infrastructure engineers. Compared to other software we work with Kafka is some of the simpler, dumber stuff which is refreshing. On the other hand I really want to learn Pulsar but it definitely is more complex, my hope is that complexity pays off with big architectural advantages but we will see. :) On the topic of what to use it for chances are if you don't know what it's for you probably don't have the problems it's meant to solve. I don't mean that in a derogatory way just that it's designed for large distributed architectures where many interested parties want to consume the same data. Or where a small number of very high throughput applications need a buffered transport that can take the load and spill to disk etc. I.e it is inherently niche, most companies don't have these problems.

wanghq 6y ago

"The state in which Kafka left The Castle is representative of the condition of his entire oeuvre."

azureel 6y ago

Me too. Maybe this can be solved by updating the title as "Franz Kafka in Pieces". I know, respecting the original title is a nice thing. On the other hand, HN is a tech oriented web site and +90% of the times if there is a Kafka in the title it is related to the stream processing framework, not the writer. So, just writing the writer's full name (in those rare occurances where the writer is mentioned here in this news site) can solve that tiny problem. Those are my two cents...

geodel 6y ago
That is one way. I think may be Kafka the software need to be called Apache Kafka here. Not sure if kafka the software will remain popular in next few years as cool developers seem already moving to Apache Pulsar, NATS and so on. > +90% of the times if there is a Kafka in the title it is related to the stream processing framework, not the writer This makes me think why these crappily implemented software choose such pretentious literary names. 'Linkedin Sprof' would be a suitable name for this below average software.