4 ms·

My opinion: the high-prevalence of implementations using Spark, Pandas, etc. are mostly driven by (1) people's tendency to work with tools that use APIs they a

My opinion: the high-prevalence of implementations using Spark, Pandas, etc. are mostly driven by (1) people's tendency to work with tools that use APIs they are already familiar with, (2) resume driven development, and/or to a much lesser degree (3) sustainability with regard to future maintainers, versus what may be technically sensible with regard to performance. A decade ago we saw similar articles referencing misapplications of Hadoop/Mapreduce, and today it is Spark as its successor. Pandas' use of the dataframe concepts and APIs were informed by R and a desire to provide something familiar and accessible to R users (i.e. ease of user adoption). Likewise, when the Spark development community somewhere around the version 0.11 days began implementing the dataframe abstraction over its original native RDD abstractions, it understood the need to provide a robust Python API similar to the Pandas APIs for accessibility (i.e. ease of user adoption). At some point those familiar APIs also became a burden, or were not-great to begin with, in several ways and we see new tools emerge like DuckDB and Polars. However, we now have a non-unique issue where people are learning and applying specific tools versus general problem-solving skills and tradecraft in the related domain (i.e. the common pattern of people with hammers seeing everything as nails). Note all of the "learn these -n- tools/packages to become a great ____ engineer and make xyz dollars" type tutorials and starter-packs on the internet today.

threeseed 2y ago
It's always easy to ignorantly criticise technology choices. But from my experience in almost all cases it is misguided requirements e.g. we want to support 100x data requirements in 5 years that drive in hindsight bad choices. Not resume driven development. And at least in enterprise space having a vendor who can support the technology is just as important as the merits of the technology itself. And vendors tend to spring up from popular, trendy technologies.
- bdcravens 2y ago
  It's not binary. Valid uses of a technology don't mean there aren't others using that same technology in a resume-driven manner.
  - com 2y ago
    Yeah, I’ve seen enough of both to know that this is genuine. The problem with resume-driven technology choices are a kind of tech debt that typically costs a lot of cash to operate and perhaps worse, delivers terrible opportunity costs, both which really do sink businesses. Premature scaling doesn’t. The challenge is not leaving it too late. Even doing that though only has led to a very few high-profile business failures.
esafak 2y ago
You can use Fugue as a translation layer. https://github.com/fugue-project/fugue https://github.com/fugue-project/fugue

threeseed 2y ago

It's always easy to ignorantly criticise technology choices. But from my experience in almost all cases it is misguided requirements e.g. we want to support 100x data requirements in 5 years that drive in hindsight bad choices. Not resume driven development. And at least in enterprise space having a vendor who can support the technology is just as important as the merits of the technology itself. And vendors tend to spring up from popular, trendy technologies.

bdcravens 2y ago
It's not binary. Valid uses of a technology don't mean there aren't others using that same technology in a resume-driven manner.
- com 2y ago
  Yeah, I’ve seen enough of both to know that this is genuine. The problem with resume-driven technology choices are a kind of tech debt that typically costs a lot of cash to operate and perhaps worse, delivers terrible opportunity costs, both which really do sink businesses. Premature scaling doesn’t. The challenge is not leaving it too late. Even doing that though only has led to a very few high-profile business failures.

esafak 2y ago

You can use Fugue as a translation layer. https://github.com/fugue-project/fugue https://github.com/fugue-project/fugue