3 ms·

Just checked and Spark has over a million lines of Scala code, as well as 100k+ loc of Java and Python. Which would you prefer, rewriting a million lines of Sca

• nobody0 5y ago
  Let alone lines of code would be even more when it's written in C++.
  • tormeh 5y ago
    A lot more. Scala, the Akka ecosystem, and the Hadoop ecosystem is uniquely suited to Spark's problem domain. And Scala is, for both better and worse, an extremely compact language too. I would expect about 5x more lines of code just from that.
    • mumblemumble 5y ago
      One interesting thing here is that there are people who know some real numbers, and not just hypotheticals, but they probably can't talk about it publicly. Google has their own distributed compute engine, written in C++. Everyone I know who's worked with it says it's much nicer than Spark, and much more performant. I wouldn't necessarily just assume that it's 5x more code. I've spent a fair bit of time digging around in the Spark code, and it's not clear to me that it's as efficiently designed or cleanly implemented as one might like. And, while I can't speak to the influence of Google's style guide, I don't personally feel like modern C++ is 5x as verbose as modern Scala. A bit more, maybe? But it's not like Scala (2, at least) doesn't have its own bizarre sources of unnecessary verbosity. It mostly looks great by comparison to Java.
• bpodgursky 5y ago
  Spark is also heavily dependent on a bajillion other OSS libraries, many of which don't directly translate to C++ libs -- Apache, Guava, etc. It's probably at least 10x this line count if you include all the utilities they pull in.