6 ms·

Comment I made last summer: It’s rare a piece of tech has a more fitting name! “Is your orgs politics so complicated that direct team-to-team communication has

by zwkrt 5y ago

Comment I made last summer: It’s rare a piece of tech has a more fitting name! “Is your orgs politics so complicated that direct team-to-team communication has broken down? Is your business process subject to unannounced violent change? Bogged down by consistent DB schemas and API versioning? Tired of retries on failed messages? Introducing Kafka by Apache: an easy to use, highly configurable, persistently stored ephemeral centralized federated event based messaging data analytics Single Source of Truth as a Service that can scale with any enterprise-sized dysfunction!”

abledon 5y ago
i wish there was some CLI like homebrew / manpages... but had paragraphs like this comment for each techonology.... so when your team starts talking about it you can look it up. > hninsight -q kafka
- jacquesm 5y ago
  I often do this when I find some interesting new tech: https://www.google.com/search?q=site:news.ycombinator.com%20productname https://www.google.com/search?q=site:news.ycombinator.com%20... it almost always turns up good stuff.
almeria 5y ago
Anything more about Kafka you can tell us? Seriously, the one time I was in a situation where much of the team seemed hellbent on this "just put all in Kafka" idea (without really understanding why, exactly) the arguments they came up with were not too dissimilar from what you've shared with us above. It all seemed to come down to "OMG databases are hard, schemas are hard, our customers don't understand the data they're shoving at us. But Kafka will take care of all of that for us. Because, you know, shiny." That said I'd still like to have a more ... balanced understanding of why Kafka may not necessarily be The Answer, and/or have more hidden complexity or other negative tradeoffs than we may have bargained for.
- yongjik 5y ago
  I only had a brief exposure to it, but my impression is that it's sort of a message queue optimized for very large data (TB or more). So, for example, there's no way to easily answer questions like "How many requests did server X generate between 1pm and 2pm and how many of them were served by server Y?" because when your data doesn't fit in a single machine, supporting such queries requires a lot of bookkeeping. If you never need them, you don't want to pay for them. Of course, when have a few megabytes of data and you route it through Kafka, then all you get is an opaque message queue where you can't see which message went from where to where. Good luck debugging any issues. But, hey, you got to use Kafka.
  - EdwardDiego 5y ago
    > for example, there's no way to easily answer questions like "How many requests did server X generate between 1pm and 2pm and how many of them were served by server Y?" There's many ways to answer that using data streamed over Kafka - ingest it into your preferred query engine, go query it. Kafka is a distributed log, that's it.
- zwkrt 5y ago
  My take was humorous but it didn’t hide anything. Kafka was built so that LinkedIn could shove all its real-time click data through a single funnel—terabytes upon terabytes. It has since been evangelized and created a cottage industry of Confluent salespeople who will give your manager a course in how to lobby their engineers into using Kafka. Have scaling problems? Kafka. Have business events that need to be ordered? Kafka! Have “changing schemas”? KAFKA!! I’m always suspicious when a company gives a product away for free tbut then charges $$$ for “support”. I worked for a high profile recently-failed project from a company that rhymes with Brillo, and our data was just beginning to be too big for google sheets (!). However, we were also having organizational problems because the higher ups were seeing the failing project losing money so they of course decided to hire 100 extra engineers. Our communications (both human and programmatic) were failing and the confluent salespeople began circling like buzzards. Of course by the time it was suggested we we use it the project was already 6 months past the point of no return. My advice is that if your data fits in a database, use a database. Anyone who says that isn’t scalable should have to tell you the actual reason it doesn’t scale and the number of requests/users/GBs/uptime/ etc that is the bottleneck.
damontal 5y ago
In The Castle, any phone call from the town to the castle causes all telephones in the castle to ring. Anyone can pick up. I thought this was the origin of the name but I’m probably wrong.
- wodenokoto 5y ago
  I’ve never read the castle so I’m sure I’m missing some context, but isn’t that how landline phones always work?
  - damontal 5y ago
    The castle is full of administrative offices and various bureaucratic departments that run the town. The protagonist gets the number of a bureaucrat and wants to contact him at his office in the castle. The fact that the number he’s given causes all phones in all offices in the castle to ring, and the people who answer don’t know or care about what he’s after, just adds a layer of confusion and difficulty to his goal of getting to the castle.

by zwkrt 5y ago

abledon 5y ago

i wish there was some CLI like homebrew / manpages... but had paragraphs like this comment for each techonology.... so when your team starts talking about it you can look it up. > hninsight -q kafka

jacquesm 5y ago
I often do this when I find some interesting new tech: https://www.google.com/search?q=site:news.ycombinator.com%20productname https://www.google.com/search?q=site:news.ycombinator.com%20... it almost always turns up good stuff.

almeria 5y ago

Anything more about Kafka you can tell us? Seriously, the one time I was in a situation where much of the team seemed hellbent on this "just put all in Kafka" idea (without really understanding why, exactly) the arguments they came up with were not too dissimilar from what you've shared with us above. It all seemed to come down to "OMG databases are hard, schemas are hard, our customers don't understand the data they're shoving at us. But Kafka will take care of all of that for us. Because, you know, shiny." That said I'd still like to have a more ... balanced understanding of why Kafka may not necessarily be The Answer, and/or have more hidden complexity or other negative tradeoffs than we may have bargained for.

yongjik 5y ago
I only had a brief exposure to it, but my impression is that it's sort of a message queue optimized for very large data (TB or more). So, for example, there's no way to easily answer questions like "How many requests did server X generate between 1pm and 2pm and how many of them were served by server Y?" because when your data doesn't fit in a single machine, supporting such queries requires a lot of bookkeeping. If you never need them, you don't want to pay for them. Of course, when have a few megabytes of data and you route it through Kafka, then all you get is an opaque message queue where you can't see which message went from where to where. Good luck debugging any issues. But, hey, you got to use Kafka.
- EdwardDiego 5y ago
  > for example, there's no way to easily answer questions like "How many requests did server X generate between 1pm and 2pm and how many of them were served by server Y?" There's many ways to answer that using data streamed over Kafka - ingest it into your preferred query engine, go query it. Kafka is a distributed log, that's it.
zwkrt 5y ago
My take was humorous but it didn’t hide anything. Kafka was built so that LinkedIn could shove all its real-time click data through a single funnel—terabytes upon terabytes. It has since been evangelized and created a cottage industry of Confluent salespeople who will give your manager a course in how to lobby their engineers into using Kafka. Have scaling problems? Kafka. Have business events that need to be ordered? Kafka! Have “changing schemas”? KAFKA!! I’m always suspicious when a company gives a product away for free tbut then charges $$$ for “support”. I worked for a high profile recently-failed project from a company that rhymes with Brillo, and our data was just beginning to be too big for google sheets (!). However, we were also having organizational problems because the higher ups were seeing the failing project losing money so they of course decided to hire 100 extra engineers. Our communications (both human and programmatic) were failing and the confluent salespeople began circling like buzzards. Of course by the time it was suggested we we use it the project was already 6 months past the point of no return. My advice is that if your data fits in a database, use a database. Anyone who says that isn’t scalable should have to tell you the actual reason it doesn’t scale and the number of requests/users/GBs/uptime/ etc that is the bottleneck.

damontal 5y ago

In The Castle, any phone call from the town to the castle causes all telephones in the castle to ring. Anyone can pick up. I thought this was the origin of the name but I’m probably wrong.

wodenokoto 5y ago
I’ve never read the castle so I’m sure I’m missing some context, but isn’t that how landline phones always work?
- damontal 5y ago
  The castle is full of administrative offices and various bureaucratic departments that run the town. The protagonist gets the number of a bureaucrat and wants to contact him at his office in the castle. The fact that the number he’s given causes all phones in all offices in the castle to ring, and the people who answer don’t know or care about what he’s after, just adds a layer of confusion and difficulty to his goal of getting to the castle.