6 ms·

It is a feelpiece, but I don't think that's necessarily a bad thing, at least it invites discussion. Also, framing it as "us vs them" misses the point. For ins

It is a feelpiece, but I don't think that's necessarily a bad thing, at least it invites discussion. Also, framing it as "us vs them" misses the point. For instance, in my world (Data & Analytics), there's no universe where Spark makes sense for <1TB data or using Kafka for what could be solved using with more simple and predictable batch processing, yet we still see them used in wild implementations that clearly don't need that scale. Those implementations can end up needing a lot of headcount and budget for support without adding any value whatsoever and sometimes even ending up in a more brittle platform. Complexity often comes from misguided choices, not just ego. I've seen enormous data teams at startups, by that I mean bigger than WhatsApp engineering team in its prime pre-Meta (~30 people). That's a red flag, especially when they're processing far less data.

moritzwarhier 1y ago
Inviting discussion through underspecified warnings or appeal to feelings of superiority, safety/caution or "best practices" is a very pervasive issue in software. I admit I did not read the article. But I have done so before I guess, I remember the headline. The headline is enough to provoke thought. See https://world.hey.com/dhh/merchants-of-complexity-4851301b https://world.hey.com/dhh/merchants-of-complexity-4851301b And after clicking, I see, this article is very recent. Strange...
- mrbungie 1y ago
  I'm not sure what to tell you, dismissing the article as posturing or superiority, without reading it, is at the very least ironic. That being said, I'd just add (and perhaps the article fails to acknowledge this, maybe contributing to the perceived tone of superiority) that no one is completely safe from falling into these traps. We're more human than engineer, after all. It's easy to slip into complexity, for all sorts of reasons: ego, incentives, time pressure, misguided intellectual curiosity, or simply chasing what's familiar, even when familiar isn't the right fit for the context. In my context, one funny pattern always shows up. I think of it as a data-related 'rite of passage': junior data engineers and data scientists often dismiss SQL as an old language, assuming it can be easily replaced by something shinier. But over time, they hit a wall: they can't match the speed, performance, or clarity of more experienced peers. Eventually, they come back to SQL and recognize its strengths (especially when paired with an imperative language for tasks like ML, plotting, or report generation). I was that junior once too, stubbornly trying to do everything in Python, only to learn the hard way that joins in pandas are not only slower but also far more painful to write than in good old SQL. It turns out that writing code which outperforms a mature SQL engine is extremely difficult, and realizing that takes a bit of humility and experience (which younger engineers usually haven't developed yet, and honestly, it's not expected from them either).
  - moritzwarhier 1y ago
    I have read (tbh, more skimmed, but this is because it's late here) the article by now. It's a general discussion and very Yin/Yang like (or dialectical, I'm not 100% sure if these are appropriate terms). Second the comment by mrkeen. With concrete examples, this could be more interesting. And I poorly introduced the thought I wanted to communicate. The "appeal to feelings of superiority" that I mentioned were mangled, I started to also think about said "merchants of complexity" and software design "evangelism" in general. And I didn't read the post initially because the writing, structure etc combined with the vague familiarity of the title threw me off. Especially starting with a bulleted list, ending with a summary, and still: me not seeing any interesting example at first glance. Though only the top comment in this chain reminded me of the importance of examples / real-world anecdotes.
    - mrbungie 1y ago
      Yep, I share the sentiment about missing examples. The whole overengineering problem does have a yin/yang quality to it, especially around themes like designing "for the future or for scale" (generalization) vs YAGNI/"premature optimization is the root of all evil", and engineering is, after all, largely about calibration. But it's not just abstract philosophy. There are plenty of concrete signals and they tend to appear in cycles, which suggests the industry collectively relearns how to navigate complexity over time: the rise of DuckDB and Iceberg over Spark or Hive, the cycle of people embracing MongoDB then returning to PostgreSQL, and even companies moving workloads off the cloud and back to colocated metal after complexity hit them hard, both in direct costs and in hidden ones like engineering headcount. These aren't just trends, they’re signs that overengineering is a concrete problem with real-world, objective consequences, and not merely a matter of opinion.
      - moritzwarhier 1y ago
        True, but what comes out of it? That's why I shared the link to DHH's post: his post appeared as part of others by him and the decisions about how and if the Rails community should concern themselves with JavaScript. Hey also pitched to HN initially with their technical approach (simplicity!). So for that reason the post felt derivative to me. It's still cool to reflect on all these longstanding general SE tradeoffs and truisms.
mrkeen 1y ago
> Also, framing it as "us vs them" misses the point. For instance, We would never use Spark or Kafka for small data, not like them. Their choices are misguided. Their data teams are too big, ours are the right size. Fwiw, I use Kafka regardless of scale. It's the only mainstream system out there that doesn't encourage throwing away your receipts. It's the programming equivalent of "accountants don't use erasers".
- mrbungie 1y ago
  > Their data teams are too big, ours are the right size. Come on dude, tech we can discuss a lot and there is no totally correct objective response. Irresponsible financials are an entirely different beast. A proportionally big data team in a non-data focused / non-tech company is a low-key red flag, and that's what I'm referring to (i.e. >10% of headcount). CFOs don't (and should not talk) in relatives, like "us vs them". > Fwiw, I use Kafka regardless of scale. It's the only mainstream system out there that doesn't encourage throwing away your receipts. It's the programming equivalent of "accountants don't use erasers". About Kafka, as far as I've experienced it and also read about, it is mostly safe if you are paying for managed solutions like Confluent or heavily investing on infra for covering for ordering and other issues. But Kafka is as good a technology as it is recognizedly messy, up to a point where I've seen people loving it and then hating it months-years down the line. I would stick with batch processing if I don't care for receipts. You need to take care for them? Cool, then take charge but if not then don't tackle complexity your business don't/won't need. That's the point, even big banks have survived without taking care about the receipts.

moritzwarhier 1y ago

Inviting discussion through underspecified warnings or appeal to feelings of superiority, safety/caution or "best practices" is a very pervasive issue in software. I admit I did not read the article. But I have done so before I guess, I remember the headline. The headline is enough to provoke thought. See https://world.hey.com/dhh/merchants-of-complexity-4851301b https://world.hey.com/dhh/merchants-of-complexity-4851301b And after clicking, I see, this article is very recent. Strange...

mrbungie 1y ago
I'm not sure what to tell you, dismissing the article as posturing or superiority, without reading it, is at the very least ironic. That being said, I'd just add (and perhaps the article fails to acknowledge this, maybe contributing to the perceived tone of superiority) that no one is completely safe from falling into these traps. We're more human than engineer, after all. It's easy to slip into complexity, for all sorts of reasons: ego, incentives, time pressure, misguided intellectual curiosity, or simply chasing what's familiar, even when familiar isn't the right fit for the context. In my context, one funny pattern always shows up. I think of it as a data-related 'rite of passage': junior data engineers and data scientists often dismiss SQL as an old language, assuming it can be easily replaced by something shinier. But over time, they hit a wall: they can't match the speed, performance, or clarity of more experienced peers. Eventually, they come back to SQL and recognize its strengths (especially when paired with an imperative language for tasks like ML, plotting, or report generation). I was that junior once too, stubbornly trying to do everything in Python, only to learn the hard way that joins in pandas are not only slower but also far more painful to write than in good old SQL. It turns out that writing code which outperforms a mature SQL engine is extremely difficult, and realizing that takes a bit of humility and experience (which younger engineers usually haven't developed yet, and honestly, it's not expected from them either).
- moritzwarhier 1y ago
  I have read (tbh, more skimmed, but this is because it's late here) the article by now. It's a general discussion and very Yin/Yang like (or dialectical, I'm not 100% sure if these are appropriate terms). Second the comment by mrkeen. With concrete examples, this could be more interesting. And I poorly introduced the thought I wanted to communicate. The "appeal to feelings of superiority" that I mentioned were mangled, I started to also think about said "merchants of complexity" and software design "evangelism" in general. And I didn't read the post initially because the writing, structure etc combined with the vague familiarity of the title threw me off. Especially starting with a bulleted list, ending with a summary, and still: me not seeing any interesting example at first glance. Though only the top comment in this chain reminded me of the importance of examples / real-world anecdotes.
  - mrbungie 1y ago
    Yep, I share the sentiment about missing examples. The whole overengineering problem does have a yin/yang quality to it, especially around themes like designing "for the future or for scale" (generalization) vs YAGNI/"premature optimization is the root of all evil", and engineering is, after all, largely about calibration. But it's not just abstract philosophy. There are plenty of concrete signals and they tend to appear in cycles, which suggests the industry collectively relearns how to navigate complexity over time: the rise of DuckDB and Iceberg over Spark or Hive, the cycle of people embracing MongoDB then returning to PostgreSQL, and even companies moving workloads off the cloud and back to colocated metal after complexity hit them hard, both in direct costs and in hidden ones like engineering headcount. These aren't just trends, they’re signs that overengineering is a concrete problem with real-world, objective consequences, and not merely a matter of opinion.
    - moritzwarhier 1y ago
      True, but what comes out of it? That's why I shared the link to DHH's post: his post appeared as part of others by him and the decisions about how and if the Rails community should concern themselves with JavaScript. Hey also pitched to HN initially with their technical approach (simplicity!). So for that reason the post felt derivative to me. It's still cool to reflect on all these longstanding general SE tradeoffs and truisms.

mrkeen 1y ago

> Also, framing it as "us vs them" misses the point. For instance, We would never use Spark or Kafka for small data, not like them. Their choices are misguided. Their data teams are too big, ours are the right size. Fwiw, I use Kafka regardless of scale. It's the only mainstream system out there that doesn't encourage throwing away your receipts. It's the programming equivalent of "accountants don't use erasers".

mrbungie 1y ago
> Their data teams are too big, ours are the right size. Come on dude, tech we can discuss a lot and there is no totally correct objective response. Irresponsible financials are an entirely different beast. A proportionally big data team in a non-data focused / non-tech company is a low-key red flag, and that's what I'm referring to (i.e. >10% of headcount). CFOs don't (and should not talk) in relatives, like "us vs them". > Fwiw, I use Kafka regardless of scale. It's the only mainstream system out there that doesn't encourage throwing away your receipts. It's the programming equivalent of "accountants don't use erasers". About Kafka, as far as I've experienced it and also read about, it is mostly safe if you are paying for managed solutions like Confluent or heavily investing on infra for covering for ordering and other issues. But Kafka is as good a technology as it is recognizedly messy, up to a point where I've seen people loving it and then hating it months-years down the line. I would stick with batch processing if I don't care for receipts. You need to take care for them? Cool, then take charge but if not then don't tackle complexity your business don't/won't need. That's the point, even big banks have survived without taking care about the receipts.