3 ms·

Interesting. To me, you seem to be conflating some pieces of data infra plumbing with very sensible and concrete goals (Kafka, Hadoop) with others that are adja

by the_af 5y ago

Interesting. To me, you seem to be conflating some pieces of data infra plumbing with very sensible and concrete goals (Kafka, Hadoop) with others that are adjacent tools which may or may not make sense. Regardless of any tech details, the idea behind Kafka makes a lot of sense: https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha... Or did you mean, "I have no idea what business needs all this data for", a much broader question?

pram 5y ago
It can become the opposite of sensible, very easily. I've had devs demand to use Kafka for storing 100mb+ blobs (which of course required a custom broker config + cluster)
- geodel 5y ago
  Indeed. The idea that somehow Kafka is beyond reproach seems amusing to me.
  - the_af 5y ago
    Thankfully nobody here said that Kafka is beyond reproach. I just said the idea behind Kafka, the distributed log, makes a lot of sense.
- the_af 5y ago
  Anything can become the opposite of sensible, so that's not saying much. I'm saying there are logical, sensible ideas behind things such as Kafka -- "you could have invented the distributed log" -- and that it's not snake oil. Of course it can be misused, not everything calls for it, and it comes with tradeoffs and pitfalls, but which tool doesn't?
  - pram 5y ago
    There is a logical idea behind it, of course! I didn't say it was snake oil anywhere. Kafka was even pretty novel 7 years ago when I started deploying it (although 0.9 was an enormous dumpster fire) Today you also have Pulsar, Redpanda, Kinesis, PubSub, and NATS off the top of my head. You can do the same streaming patterns with all of them, with the exception of log compaction probably. It's an architecture so amazing that the world needs 6 redundant implementations. I think my OP implied something regarding that.
    - the_af 5y ago
      I agree with you that the slightly different implementations of the same idea are confusing and slightly embarrassing. No argument from me there!
    - agallego 5y ago
      for the record, you can do compaction with redpanda - all the same compaction strategies you know w/ kafka work w/ redpanda. https://github.com/vectorizedio/redpanda/tree/dev/src/v/storage - see all 'compaction_*' for deets
geodel 5y ago
Well ideas behind all the other stuff will also make lot of sense if one align with perspective of snake oil salesman. And I do think Kafka/hadoop etc fit in snake oil category. Linkedin saying Kafka is good for them is not saying much.
- EdwardDiego 5y ago
  Apache Kafka is FOSS, plenty of companies use it who chose to without the input of a salesperson. Must be solving some problem for them, eh?
- the_af 5y ago
  Did you read the article? Because it makes a ton of sense regardless of whether it's LinkedIn saying whatever. It reads more like a paper than an advertisement -- which is more than what can be said about many articles peddling "solutions".

by the_af 5y ago

pram 5y ago

It can become the opposite of sensible, very easily. I've had devs demand to use Kafka for storing 100mb+ blobs (which of course required a custom broker config + cluster)

geodel 5y ago
Indeed. The idea that somehow Kafka is beyond reproach seems amusing to me.
- the_af 5y ago
  Thankfully nobody here said that Kafka is beyond reproach. I just said the idea behind Kafka, the distributed log, makes a lot of sense.
the_af 5y ago
Anything can become the opposite of sensible, so that's not saying much. I'm saying there are logical, sensible ideas behind things such as Kafka -- "you could have invented the distributed log" -- and that it's not snake oil. Of course it can be misused, not everything calls for it, and it comes with tradeoffs and pitfalls, but which tool doesn't?
- pram 5y ago
  There is a logical idea behind it, of course! I didn't say it was snake oil anywhere. Kafka was even pretty novel 7 years ago when I started deploying it (although 0.9 was an enormous dumpster fire) Today you also have Pulsar, Redpanda, Kinesis, PubSub, and NATS off the top of my head. You can do the same streaming patterns with all of them, with the exception of log compaction probably. It's an architecture so amazing that the world needs 6 redundant implementations. I think my OP implied something regarding that.
  - the_af 5y ago
    I agree with you that the slightly different implementations of the same idea are confusing and slightly embarrassing. No argument from me there!
  - agallego 5y ago
    for the record, you can do compaction with redpanda - all the same compaction strategies you know w/ kafka work w/ redpanda. https://github.com/vectorizedio/redpanda/tree/dev/src/v/storage - see all 'compaction_*' for deets

geodel 5y ago

Well ideas behind all the other stuff will also make lot of sense if one align with perspective of snake oil salesman. And I do think Kafka/hadoop etc fit in snake oil category. Linkedin saying Kafka is good for them is not saying much.

EdwardDiego 5y ago
Apache Kafka is FOSS, plenty of companies use it who chose to without the input of a salesperson. Must be solving some problem for them, eh?
the_af 5y ago
Did you read the article? Because it makes a ton of sense regardless of whether it's LinkedIn saying whatever. It reads more like a paper than an advertisement -- which is more than what can be said about many articles peddling "solutions".