5 ms·

I’ve spent the last 10 years of my life building “Big Data” platforms for companies (Kafka, Druid, Hadoop, Databricks, Presto, Teradata, Informatica, MicroStrat

by pram 5y ago

I’ve spent the last 10 years of my life building “Big Data” platforms for companies (Kafka, Druid, Hadoop, Databricks, Presto, Teradata, Informatica, MicroStrategy, etc etc) and I still have no idea what people need most of this crap for. There’s so much duplicative functionality, and a basically never ending list of OLAP adjacent products emerging every year. I mean it’s good for me personally, it’s pretty much a gravy train, but I always question if a lot of it is a solution searching for a problem lol

the_af 5y ago
Interesting. To me, you seem to be conflating some pieces of data infra plumbing with very sensible and concrete goals (Kafka, Hadoop) with others that are adjacent tools which may or may not make sense. Regardless of any tech details, the idea behind Kafka makes a lot of sense: https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha... Or did you mean, "I have no idea what business needs all this data for", a much broader question?
- pram 5y ago
  It can become the opposite of sensible, very easily. I've had devs demand to use Kafka for storing 100mb+ blobs (which of course required a custom broker config + cluster)
  - geodel 5y ago
    Indeed. The idea that somehow Kafka is beyond reproach seems amusing to me.
    - the_af 5y ago
      Thankfully nobody here said that Kafka is beyond reproach. I just said the idea behind Kafka, the distributed log, makes a lot of sense.
  - the_af 5y ago
    Anything can become the opposite of sensible, so that's not saying much. I'm saying there are logical, sensible ideas behind things such as Kafka -- "you could have invented the distributed log" -- and that it's not snake oil. Of course it can be misused, not everything calls for it, and it comes with tradeoffs and pitfalls, but which tool doesn't?
    - pram 5y ago
      There is a logical idea behind it, of course! I didn't say it was snake oil anywhere. Kafka was even pretty novel 7 years ago when I started deploying it (although 0.9 was an enormous dumpster fire) Today you also have Pulsar, Redpanda, Kinesis, PubSub, and NATS off the top of my head. You can do the same streaming patterns with all of them, with the exception of log compaction probably. It's an architecture so amazing that the world needs 6 redundant implementations. I think my OP implied something regarding that.
      - the_af 5y ago
        I agree with you that the slightly different implementations of the same idea are confusing and slightly embarrassing. No argument from me there!
      - agallego 5y ago
        for the record, you can do compaction with redpanda - all the same compaction strategies you know w/ kafka work w/ redpanda. https://github.com/vectorizedio/redpanda/tree/dev/src/v/storage - see all 'compaction_*' for deets
- geodel 5y ago
  Well ideas behind all the other stuff will also make lot of sense if one align with perspective of snake oil salesman. And I do think Kafka/hadoop etc fit in snake oil category. Linkedin saying Kafka is good for them is not saying much.
  - EdwardDiego 5y ago
    Apache Kafka is FOSS, plenty of companies use it who chose to without the input of a salesperson. Must be solving some problem for them, eh?
  - the_af 5y ago
    Did you read the article? Because it makes a ton of sense regardless of whether it's LinkedIn saying whatever. It reads more like a paper than an advertisement -- which is more than what can be said about many articles peddling "solutions".
clusterhacks 5y ago
Similar experience for me. Years ago I made a decision to jump into data engineering and reporting instead of pushing more into enterprise app development (meaning, Java/C# stuff). I share the question about a solution searching for a problem. I will say that I continue to see efforts being made that seem focused on adding a bullet point to some executive leader's yearly accomplishments rather than specifically providing value back to an org . . .
eyeball 5y ago
I wonder about this a lot. The IT department where I work is rolling out a new “data platform”. I’m in the finance unit and a prime customer but fairly up to date on the tech. This thing has soooo many logos on it. I suspect there is someone just saying yes to everything their consultant pitches. Like why have databricks, azure synapse, AND snowflake on the same picture. I’m certain I could do everything they say they’re going to do with this shit with an integration tool, snowflake, a machine to run python on, and our BI tool in the same cloud.
- akdor1154 5y ago
  > Like why have databricks, azure synapse, AND snowflake on the same picture. This hits painfully close to home. Our org just "finished" such a build, mainly done by a large contractor, and the result is a messy nightmare. > integration tool, snowflake, a machine to run python on, and our BI tool Yep, my next campaign will be to gradually pare it down to this. Snowflake is wonderfully capable.

by pram 5y ago

the_af 5y ago

Interesting. To me, you seem to be conflating some pieces of data infra plumbing with very sensible and concrete goals (Kafka, Hadoop) with others that are adjacent tools which may or may not make sense. Regardless of any tech details, the idea behind Kafka makes a lot of sense: https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha... Or did you mean, "I have no idea what business needs all this data for", a much broader question?

pram 5y ago
It can become the opposite of sensible, very easily. I've had devs demand to use Kafka for storing 100mb+ blobs (which of course required a custom broker config + cluster)
- geodel 5y ago
  Indeed. The idea that somehow Kafka is beyond reproach seems amusing to me.
  - the_af 5y ago
    Thankfully nobody here said that Kafka is beyond reproach. I just said the idea behind Kafka, the distributed log, makes a lot of sense.
- the_af 5y ago
  Anything can become the opposite of sensible, so that's not saying much. I'm saying there are logical, sensible ideas behind things such as Kafka -- "you could have invented the distributed log" -- and that it's not snake oil. Of course it can be misused, not everything calls for it, and it comes with tradeoffs and pitfalls, but which tool doesn't?
  - pram 5y ago
    There is a logical idea behind it, of course! I didn't say it was snake oil anywhere. Kafka was even pretty novel 7 years ago when I started deploying it (although 0.9 was an enormous dumpster fire) Today you also have Pulsar, Redpanda, Kinesis, PubSub, and NATS off the top of my head. You can do the same streaming patterns with all of them, with the exception of log compaction probably. It's an architecture so amazing that the world needs 6 redundant implementations. I think my OP implied something regarding that.
    - the_af 5y ago
      I agree with you that the slightly different implementations of the same idea are confusing and slightly embarrassing. No argument from me there!
    - agallego 5y ago
      for the record, you can do compaction with redpanda - all the same compaction strategies you know w/ kafka work w/ redpanda. https://github.com/vectorizedio/redpanda/tree/dev/src/v/storage - see all 'compaction_*' for deets
geodel 5y ago
Well ideas behind all the other stuff will also make lot of sense if one align with perspective of snake oil salesman. And I do think Kafka/hadoop etc fit in snake oil category. Linkedin saying Kafka is good for them is not saying much.
- EdwardDiego 5y ago
  Apache Kafka is FOSS, plenty of companies use it who chose to without the input of a salesperson. Must be solving some problem for them, eh?
- the_af 5y ago
  Did you read the article? Because it makes a ton of sense regardless of whether it's LinkedIn saying whatever. It reads more like a paper than an advertisement -- which is more than what can be said about many articles peddling "solutions".

clusterhacks 5y ago

Similar experience for me. Years ago I made a decision to jump into data engineering and reporting instead of pushing more into enterprise app development (meaning, Java/C# stuff). I share the question about a solution searching for a problem. I will say that I continue to see efforts being made that seem focused on adding a bullet point to some executive leader's yearly accomplishments rather than specifically providing value back to an org . . .

eyeball 5y ago

I wonder about this a lot. The IT department where I work is rolling out a new “data platform”. I’m in the finance unit and a prime customer but fairly up to date on the tech. This thing has soooo many logos on it. I suspect there is someone just saying yes to everything their consultant pitches. Like why have databricks, azure synapse, AND snowflake on the same picture. I’m certain I could do everything they say they’re going to do with this shit with an integration tool, snowflake, a machine to run python on, and our BI tool in the same cloud.

akdor1154 5y ago
> Like why have databricks, azure synapse, AND snowflake on the same picture. This hits painfully close to home. Our org just "finished" such a build, mainly done by a large contractor, and the result is a messy nightmare. > integration tool, snowflake, a machine to run python on, and our BI tool Yep, my next campaign will be to gradually pare it down to this. Snowflake is wonderfully capable.