6 ms·

tangent - what is it about Apache or big data that the associated softwares are mostly written in Java?

by mi_lk 4y ago

zekrioca 4y ago
Good question, but maybe due to Java's stability and portability.
oaiey 4y ago
Java, like .NET, are just solid application platforms which are statically typed and their performance is good. Java has a history in big systems for soon 30 years. Rust, Python and Go are just not there yet. Rust is too low level, Python is not statically typed and will always suffer performance wise and Go ... I is a youngster :). And .NET is always not everyone's free choice. And Apache, well they just liked Java for their applications. They started with some C/C++ code but then quickly aggregated a lot of Java tech.
- GordonS 4y ago
  > And .NET is always not everyone's free choice. Hey, sometimes it really is!!
  - oaiey 4y ago
    I am a big fanboy actually. The wording is a bit screwed. The "free" was as in (perceived) free/open/... software.
ojhughes 4y ago
MapReduce and HDFS were written in Java and they paved the way for a lot of the other big data tools
- erik_seaberg 4y ago
  I get the impression that Google used C++ (their dialect) for the first MapReduce system, while open source Hadoop and HDFS came later in Java.
  - tomwheeler 4y ago
    That is correct.
mrtweetyhack 4y ago
manish_gill 4y ago
Something I've been asking for a long time as well. Java/JVM are great, but it would be great to see _some_ diversity in the Big Data ecosystem when it comes to implementations. :)
- jpgvm 4y ago
  Why? JVM is essentially perfectly suited for data workloads. Diversity of languages serves no technical purpose.
  - FridgeSeal 4y ago
    You have to deal with the JVM? You have to deal with the 13m JVM config options? You have to deal with the confusion and complexity that is “java dependencies” Everything is OOP-abstraction-heavy API’s? I’ve spent too much of my time and effort recently debugging Scala/Spark/JVM resource issues/dependency issues and the more I have to deal with it, the less I want anything to do with JVM-based solutions. The closest I want to get to another awful JVM application is a docker container. I will rejoice the day that spark-alternatives progress enough for our team to replace our workloads and I can throw our spark stuff into the literal bin.
    - jpgvm 4y ago
      Yeah but you are basically asking for a world which has fragmented implementations such that some stuff will interoperate and be usable within Spark et al and "the new stuff" which could either all be on the same platform or much more likely spread across a number of platforms. Once you have gotten the hang of running Spark (and working knowledge of JVM itself) then you have paid the cost and it's done. Adding a whole bunch of new stuff just means more costs to pay, less integration, more fragmentation of knowledge. Doesn't seem like a net win on any front to me.
      - FridgeSeal 4y ago
        If we were to stabilise on a framework, I don’t think Java/spark is the right one. It’s notoriously fickle fragile to run, it’s obscenely resource heavy, often not actually that fast, and it’s construction is locked into a set of implementations that have aged poorly-you basically need a JVM wizard to tune everything correctly for you, and you need to run so much supporting infrastructure that I think it greatly erodes the benefits of spark. > "the new stuff" which could either all be on the same platform or much more likely spread across a number of platforms. Yes. I’d like to see more variety in approaches, more variety in specialisations, and for interop between “platforms” to occur at a slightly “higher” level (I.e. maybe common sql dialect, common set of messaging/interop protocols). I think rather than pouring all of our effort into a singular platform, we should put our effort towards improving our tooling and languages so that it is more straightforward and viable to build these platforms. Everyone language has got their own web-server framework(s), I see Spark et al as the web-server framework of the data space.
qbasic_forever 4y ago
From like 2000 to 2010 or even 2015 either java or .NET was the default choice for big enterprise companies. Nobody ever got fired for picking Microsoft or Java (I would add), as they say. A lot of these Apache projects have been donated from work at big enterprises so it comes out of that background from enterprise I imagine.
- Tostino 4y ago
  Even now, what options are a huge step up from Java/.NET for your standard tech business backend/webbapp? They integrate well with so many other languages, and both have great cross-platform stories...
  - oaiey 4y ago
    Outside of the SV bubble actually none. When you look from a conservative angle (static typing, developer availability, productivity, tool support, library support, performance, etc) it goes quickly down to Java and .NET. The dynamic interpretation of Python and JavaScript are their core deal breakers.
zmmmmm 4y ago
Performance, portability, stability, scalability, concurrency, ecosystem (libraries, etc) .... despite all the new languages around, there actually still aren't many alternatives that give you the same combination of all these to the same level as Java does.
- vips7L 4y ago
  Don’t forget observability and tooling!
62951413 4y ago
https://arrow.apache.org/datafusion/ https://arrow.apache.org/datafusion/ (aka Ballista) is the first serious contender that started on the JVM and quickly moved to Rust.
erik_seaberg 4y ago
Java classloading is a natural fit for calling into a typesafe plugin fetched from a URL. Spark can even serialize lambdas and distribute their execution. It seems to be more of a hassle to do this kind of thing over IPC with native binaries, and if ARM starts displacing a lot of x86-64 in datacenters it gets more complicated.

tangent - what is it about Apache or big data that the associated softwares are mostly written in Java?

by mi_lk 4y ago

tangent - what is it about Apache or big data that the associated softwares are mostly written in Java?

zekrioca 4y ago

Good question, but maybe due to Java's stability and portability.

oaiey 4y ago

Java, like .NET, are just solid application platforms which are statically typed and their performance is good. Java has a history in big systems for soon 30 years. Rust, Python and Go are just not there yet. Rust is too low level, Python is not statically typed and will always suffer performance wise and Go ... I is a youngster :). And .NET is always not everyone's free choice. And Apache, well they just liked Java for their applications. They started with some C/C++ code but then quickly aggregated a lot of Java tech.

GordonS 4y ago
> And .NET is always not everyone's free choice. Hey, sometimes it really is!!
- oaiey 4y ago
  I am a big fanboy actually. The wording is a bit screwed. The "free" was as in (perceived) free/open/... software.

ojhughes 4y ago

MapReduce and HDFS were written in Java and they paved the way for a lot of the other big data tools

erik_seaberg 4y ago
I get the impression that Google used C++ (their dialect) for the first MapReduce system, while open source Hadoop and HDFS came later in Java.
- tomwheeler 4y ago
  That is correct.

mrtweetyhack 4y ago

manish_gill 4y ago

Something I've been asking for a long time as well. Java/JVM are great, but it would be great to see _some_ diversity in the Big Data ecosystem when it comes to implementations. :)

jpgvm 4y ago
Why? JVM is essentially perfectly suited for data workloads. Diversity of languages serves no technical purpose.
- FridgeSeal 4y ago
  You have to deal with the JVM? You have to deal with the 13m JVM config options? You have to deal with the confusion and complexity that is “java dependencies” Everything is OOP-abstraction-heavy API’s? I’ve spent too much of my time and effort recently debugging Scala/Spark/JVM resource issues/dependency issues and the more I have to deal with it, the less I want anything to do with JVM-based solutions. The closest I want to get to another awful JVM application is a docker container. I will rejoice the day that spark-alternatives progress enough for our team to replace our workloads and I can throw our spark stuff into the literal bin.
  - jpgvm 4y ago
    Yeah but you are basically asking for a world which has fragmented implementations such that some stuff will interoperate and be usable within Spark et al and "the new stuff" which could either all be on the same platform or much more likely spread across a number of platforms. Once you have gotten the hang of running Spark (and working knowledge of JVM itself) then you have paid the cost and it's done. Adding a whole bunch of new stuff just means more costs to pay, less integration, more fragmentation of knowledge. Doesn't seem like a net win on any front to me.
    - FridgeSeal 4y ago
      If we were to stabilise on a framework, I don’t think Java/spark is the right one. It’s notoriously fickle fragile to run, it’s obscenely resource heavy, often not actually that fast, and it’s construction is locked into a set of implementations that have aged poorly-you basically need a JVM wizard to tune everything correctly for you, and you need to run so much supporting infrastructure that I think it greatly erodes the benefits of spark. > "the new stuff" which could either all be on the same platform or much more likely spread across a number of platforms. Yes. I’d like to see more variety in approaches, more variety in specialisations, and for interop between “platforms” to occur at a slightly “higher” level (I.e. maybe common sql dialect, common set of messaging/interop protocols). I think rather than pouring all of our effort into a singular platform, we should put our effort towards improving our tooling and languages so that it is more straightforward and viable to build these platforms. Everyone language has got their own web-server framework(s), I see Spark et al as the web-server framework of the data space.

qbasic_forever 4y ago

From like 2000 to 2010 or even 2015 either java or .NET was the default choice for big enterprise companies. Nobody ever got fired for picking Microsoft or Java (I would add), as they say. A lot of these Apache projects have been donated from work at big enterprises so it comes out of that background from enterprise I imagine.

Tostino 4y ago
Even now, what options are a huge step up from Java/.NET for your standard tech business backend/webbapp? They integrate well with so many other languages, and both have great cross-platform stories...
- oaiey 4y ago
  Outside of the SV bubble actually none. When you look from a conservative angle (static typing, developer availability, productivity, tool support, library support, performance, etc) it goes quickly down to Java and .NET. The dynamic interpretation of Python and JavaScript are their core deal breakers.

zmmmmm 4y ago

Performance, portability, stability, scalability, concurrency, ecosystem (libraries, etc) .... despite all the new languages around, there actually still aren't many alternatives that give you the same combination of all these to the same level as Java does.

vips7L 4y ago
Don’t forget observability and tooling!

62951413 4y ago

https://arrow.apache.org/datafusion/ https://arrow.apache.org/datafusion/ (aka Ballista) is the first serious contender that started on the JVM and quickly moved to Rust.

erik_seaberg 4y ago

Java classloading is a natural fit for calling into a typesafe plugin fetched from a URL. Spark can even serialize lambdas and distribute their execution. It seems to be more of a hassle to do this kind of thing over IPC with native binaries, and if ARM starts displacing a lot of x86-64 in datacenters it gets more complicated.