4 ms·

Something I've been asking for a long time as well. Java/JVM are great, but it would be great to see _some_ diversity in the Big Data ecosystem when it comes to

Something I've been asking for a long time as well. Java/JVM are great, but it would be great to see _some_ diversity in the Big Data ecosystem when it comes to implementations. :)

jpgvm 4y ago
Why? JVM is essentially perfectly suited for data workloads. Diversity of languages serves no technical purpose.
- FridgeSeal 4y ago
  You have to deal with the JVM? You have to deal with the 13m JVM config options? You have to deal with the confusion and complexity that is “java dependencies” Everything is OOP-abstraction-heavy API’s? I’ve spent too much of my time and effort recently debugging Scala/Spark/JVM resource issues/dependency issues and the more I have to deal with it, the less I want anything to do with JVM-based solutions. The closest I want to get to another awful JVM application is a docker container. I will rejoice the day that spark-alternatives progress enough for our team to replace our workloads and I can throw our spark stuff into the literal bin.
  - jpgvm 4y ago
    Yeah but you are basically asking for a world which has fragmented implementations such that some stuff will interoperate and be usable within Spark et al and "the new stuff" which could either all be on the same platform or much more likely spread across a number of platforms. Once you have gotten the hang of running Spark (and working knowledge of JVM itself) then you have paid the cost and it's done. Adding a whole bunch of new stuff just means more costs to pay, less integration, more fragmentation of knowledge. Doesn't seem like a net win on any front to me.
    - FridgeSeal 4y ago
      If we were to stabilise on a framework, I don’t think Java/spark is the right one. It’s notoriously fickle fragile to run, it’s obscenely resource heavy, often not actually that fast, and it’s construction is locked into a set of implementations that have aged poorly-you basically need a JVM wizard to tune everything correctly for you, and you need to run so much supporting infrastructure that I think it greatly erodes the benefits of spark. > "the new stuff" which could either all be on the same platform or much more likely spread across a number of platforms. Yes. I’d like to see more variety in approaches, more variety in specialisations, and for interop between “platforms” to occur at a slightly “higher” level (I.e. maybe common sql dialect, common set of messaging/interop protocols). I think rather than pouring all of our effort into a singular platform, we should put our effort towards improving our tooling and languages so that it is more straightforward and viable to build these platforms. Everyone language has got their own web-server framework(s), I see Spark et al as the web-server framework of the data space.
      - EdwardDiego 4y ago
        I feel like you're conflating Spark and the JVM. I get your point on tuning Spark, been there, suffered the frustration where a job fails 4 hours in, but yeah, that's Spark for you.

Something I've been asking for a long time as well. Java/JVM are great, but it would be great to see _some_ diversity in the Big Data ecosystem when it comes to implementations. :)

jpgvm 4y ago

Why? JVM is essentially perfectly suited for data workloads. Diversity of languages serves no technical purpose.

FridgeSeal 4y ago
You have to deal with the JVM? You have to deal with the 13m JVM config options? You have to deal with the confusion and complexity that is “java dependencies” Everything is OOP-abstraction-heavy API’s? I’ve spent too much of my time and effort recently debugging Scala/Spark/JVM resource issues/dependency issues and the more I have to deal with it, the less I want anything to do with JVM-based solutions. The closest I want to get to another awful JVM application is a docker container. I will rejoice the day that spark-alternatives progress enough for our team to replace our workloads and I can throw our spark stuff into the literal bin.
- jpgvm 4y ago
  Yeah but you are basically asking for a world which has fragmented implementations such that some stuff will interoperate and be usable within Spark et al and "the new stuff" which could either all be on the same platform or much more likely spread across a number of platforms. Once you have gotten the hang of running Spark (and working knowledge of JVM itself) then you have paid the cost and it's done. Adding a whole bunch of new stuff just means more costs to pay, less integration, more fragmentation of knowledge. Doesn't seem like a net win on any front to me.
  - FridgeSeal 4y ago
    If we were to stabilise on a framework, I don’t think Java/spark is the right one. It’s notoriously fickle fragile to run, it’s obscenely resource heavy, often not actually that fast, and it’s construction is locked into a set of implementations that have aged poorly-you basically need a JVM wizard to tune everything correctly for you, and you need to run so much supporting infrastructure that I think it greatly erodes the benefits of spark. > "the new stuff" which could either all be on the same platform or much more likely spread across a number of platforms. Yes. I’d like to see more variety in approaches, more variety in specialisations, and for interop between “platforms” to occur at a slightly “higher” level (I.e. maybe common sql dialect, common set of messaging/interop protocols). I think rather than pouring all of our effort into a singular platform, we should put our effort towards improving our tooling and languages so that it is more straightforward and viable to build these platforms. Everyone language has got their own web-server framework(s), I see Spark et al as the web-server framework of the data space.
    - EdwardDiego 4y ago
      I feel like you're conflating Spark and the JVM. I get your point on tuning Spark, been there, suffered the frustration where a job fails 4 hours in, but yeah, that's Spark for you.