5 ms·

Scala was a terrible technology that was adopted by my organization five years ago and were still paying the price. How to deprecate this at scale? The extent

Scala was a terrible technology that was adopted by my organization five years ago and were still paying the price. How to deprecate this at scale? The extent to which i like scala is as a dsl for describing etl jobs with spark and i prefer the RDD api. Engineers are consistently crippled by it, we dont even use shapeless or scalaz. Spec2 is a terrible dsl that offers a shitty vocabular to write unit tests and makes the common path incredible difficult compared to junit.

eweise 5y ago
Well, you know, you can just use Junit with Scala. That's what I do.
mhzsh 5y ago
Spark is really its own world that’s fairly separate from the Scala community at large. What makes it a ”terrible technology”?
- waynesonfire 5y ago
  Scala moved the needle five years ago and made java better among other languages. Using scala helped me grow as an engineer. in my view, it has no place in an enterprise. I require a mature environment that favors consistency. It boils down to reducing complexity. I can barely handle the complexity growth due to the demands of the business, whether it be software engineering or people and these reasons do not justify scalas pluses. Ymmv.
  - kaba0 5y ago
    I think scala can absolutely account for that complexity, like it is not a one-way street. It can scale both up and down on expressivity. But I agree that often times it is not used that way.
- tharne 5y ago
  I wouldn't say spark is terrible, but it is a sort of a "reverse bicycle". If you put a spark codebase down for a week or two and pick it back up, it feels like you're learning everything all over again.
fnord77 5y ago
newer java has many similar features that scala had, so it seems moot to use it now. unless you like ridiculously long compilation times and having libs tied to what version of the compiler you're using (serious WTF)
- ebruchez 5y ago
  > it seems moot to use it now It's true that some features of Scala have made it to Java, but I am not sure the gap between the two languages has shrunk much. For example while Java has added first-class functions and closures, it is still clumsy to attempt any serious functional programming approach with Java; Scala's type system is still way ahead; Scala compile to the JVM, JavaScript, and now native via LLVM. > having libs tied to what version of the compiler you're using It's never been that. However libraries used to be tied to the major version of Scala (for example 2.11, 2.12, 2.13) as major versions were not binary-compatible with each other. Scala 3 addresses this thanks to the TASTY format which creates a strong foundation for binary compatibility. [1] You can use Scala 3 libraries with Scala 2.13 and, more importantly, Scala 2.13 libraries with Scala 3. In short, Scala is not standing still. [1] https://docs.scala-lang.org/scala3/guides/tasty-overview.html https://docs.scala-lang.org/scala3/guides/tasty-overview.htm...
lmm 5y ago
> Spec2 is a terrible dsl that offers a shitty vocabular to write unit tests and makes the common path incredible difficult compared to junit. Yes it is. Deprecate using it and migrate away from it. I hate that these shitty overengineered Cucumber wannabees are what "Scala" means to most people. It's a great language IMO, but you need to make sure fancy frameworks pay their weight. IMO Shapeless and Scalaz-or-equivalent do - replacing reflection or AOP with plain old compiled code, even complex code, is a huge win. But SBT and the mess of Scala testing libraries are just huge complexity for marginal benefit.
amir734jj 5y ago
In my previous company we used Scala + Spark and I hated it initially. I had no idea how syntax worked, no clue about functional programming or the ecosystem. I got interested in the Scala as a language and decided to pursue PhD in computer science focusing on compilers. Long story short, I am finishing my PhD on Monday :) Don't be negative about technology. Be positive.
- dikei 5y ago
  > In my previous company we used Scala + Spark and I hated it initially. I had no idea how syntax worked, no clue about functional programming or the ecosystem. It's funny that if you tell Cat-purist that Spark is functional-style, they would laugh at your face :)
  - caoilte 5y ago
    That's an unhelpful thing to say. Most people agree that the Spark scala DSL is functional style. It's not as advanced in the style as other scala ecosystems but it has been a very good on ramp for people discovering functional style and is widely celebrated for that
- burntoutfire 5y ago
  Scala + Spark is absolutely the sane face of Scala. Try using stuff like cats (basically Haskel ported to Scala) to experience the deep end of the pool.
  - esarbe 5y ago
    To discard Cats as 'basically Haskell ported to Scala' is doing it a bit of a disservice. Yes, it explores pure functional programming, but that design space is huge.
zalequin 5y ago
It saddens me to see that the HN crowd had been diluted to such an extent in terms of engineering quality that straight up luddism has become the norm, often accompanied with silly ad hominems.
- _vertigo 5y ago
  I don't agree with him, but that's a biiit smug..
random314 5y ago
As a counterpoint, I used scala at my previous job and it was the best coding experience and environment I ever experienced. Got rid of 1000s of lines of Java and SQL for 100s of lines of scala, scalding and Spark. The codebase size on the first migration was 10% of the original Java+SQL. overall, a team of 6 worked for 4 years to produce 10K lines of high quality Scala code for a very complex app and platform after the first transition from 10K lines of java+sql to 1K lines of scala. Never experienced something like this before or after. At the next place I wrote 400 lines of C++ mapreduce over 3 days, which could have been 40 lines of sql or 10 lines of scalding/spark. Jarring transition.
- chrisjc 5y ago
  Regarding the reduction in lines of SQL. Doesn't this imply a tradeoff with where compute is utilized? By reducing SQL, wouldn't you be "pushing up" the compute from the database to the JVM while also increasing the amount of data that has to be moved around? Edit: Wait, did you imply that the SQL you replaced with Scala ultimately gets transpiled into SQL (spark or other dataframe engine) and "pushed down" to the database? Now you're just maintaining Scala and not a rats nest of SQL strings concatenated together? If so, that's pretty awesome.
  - random314 5y ago
    Most of this was Hadoop. Which executes mapreduce jobs written in sql on the jvm. The scalding jobs executed faster, except for one job for which we had to stick with Apache pig. Spark jobs were native RDDs before the SQL Spark era.
  - random314 5y ago
    No. There was no transpiling to SQL. This was several years ago, before Spark SQL. The SQL was HiveQL. Spark didn't really support SQL at the time and was native JVM Scala RDDs. HiveQL with UDFs compiled to JVM bytecode typically ended up performing worse than scalding that corresponds to straight raw jvm map reduce jobs that are the real native interface of Hadoop. The real performance drag from HQL came from developers writing inefficient map reduce jobs with extra joins to avoid writing Java UDFs because they were a huge PITA. In scalding, developers wrote map reduce jobs with as few group bys as possible. Our hadoop infra team actually complimented us on how great the execution profile of our teams jobs were.

eweise 5y ago

Well, you know, you can just use Junit with Scala. That's what I do.

mhzsh 5y ago

Spark is really its own world that’s fairly separate from the Scala community at large. What makes it a ”terrible technology”?

waynesonfire 5y ago
Scala moved the needle five years ago and made java better among other languages. Using scala helped me grow as an engineer. in my view, it has no place in an enterprise. I require a mature environment that favors consistency. It boils down to reducing complexity. I can barely handle the complexity growth due to the demands of the business, whether it be software engineering or people and these reasons do not justify scalas pluses. Ymmv.
- kaba0 5y ago
  I think scala can absolutely account for that complexity, like it is not a one-way street. It can scale both up and down on expressivity. But I agree that often times it is not used that way.
tharne 5y ago
I wouldn't say spark is terrible, but it is a sort of a "reverse bicycle". If you put a spark codebase down for a week or two and pick it back up, it feels like you're learning everything all over again.

fnord77 5y ago

newer java has many similar features that scala had, so it seems moot to use it now. unless you like ridiculously long compilation times and having libs tied to what version of the compiler you're using (serious WTF)

ebruchez 5y ago
> it seems moot to use it now It's true that some features of Scala have made it to Java, but I am not sure the gap between the two languages has shrunk much. For example while Java has added first-class functions and closures, it is still clumsy to attempt any serious functional programming approach with Java; Scala's type system is still way ahead; Scala compile to the JVM, JavaScript, and now native via LLVM. > having libs tied to what version of the compiler you're using It's never been that. However libraries used to be tied to the major version of Scala (for example 2.11, 2.12, 2.13) as major versions were not binary-compatible with each other. Scala 3 addresses this thanks to the TASTY format which creates a strong foundation for binary compatibility. [1] You can use Scala 3 libraries with Scala 2.13 and, more importantly, Scala 2.13 libraries with Scala 3. In short, Scala is not standing still. [1] https://docs.scala-lang.org/scala3/guides/tasty-overview.html https://docs.scala-lang.org/scala3/guides/tasty-overview.htm...

lmm 5y ago

> Spec2 is a terrible dsl that offers a shitty vocabular to write unit tests and makes the common path incredible difficult compared to junit. Yes it is. Deprecate using it and migrate away from it. I hate that these shitty overengineered Cucumber wannabees are what "Scala" means to most people. It's a great language IMO, but you need to make sure fancy frameworks pay their weight. IMO Shapeless and Scalaz-or-equivalent do - replacing reflection or AOP with plain old compiled code, even complex code, is a huge win. But SBT and the mess of Scala testing libraries are just huge complexity for marginal benefit.

amir734jj 5y ago

In my previous company we used Scala + Spark and I hated it initially. I had no idea how syntax worked, no clue about functional programming or the ecosystem. I got interested in the Scala as a language and decided to pursue PhD in computer science focusing on compilers. Long story short, I am finishing my PhD on Monday :) Don't be negative about technology. Be positive.

dikei 5y ago
> In my previous company we used Scala + Spark and I hated it initially. I had no idea how syntax worked, no clue about functional programming or the ecosystem. It's funny that if you tell Cat-purist that Spark is functional-style, they would laugh at your face :)
- caoilte 5y ago
  That's an unhelpful thing to say. Most people agree that the Spark scala DSL is functional style. It's not as advanced in the style as other scala ecosystems but it has been a very good on ramp for people discovering functional style and is widely celebrated for that
burntoutfire 5y ago
Scala + Spark is absolutely the sane face of Scala. Try using stuff like cats (basically Haskel ported to Scala) to experience the deep end of the pool.
- esarbe 5y ago
  To discard Cats as 'basically Haskell ported to Scala' is doing it a bit of a disservice. Yes, it explores pure functional programming, but that design space is huge.

zalequin 5y ago

It saddens me to see that the HN crowd had been diluted to such an extent in terms of engineering quality that straight up luddism has become the norm, often accompanied with silly ad hominems.

_vertigo 5y ago
I don't agree with him, but that's a biiit smug..

random314 5y ago

As a counterpoint, I used scala at my previous job and it was the best coding experience and environment I ever experienced. Got rid of 1000s of lines of Java and SQL for 100s of lines of scala, scalding and Spark. The codebase size on the first migration was 10% of the original Java+SQL. overall, a team of 6 worked for 4 years to produce 10K lines of high quality Scala code for a very complex app and platform after the first transition from 10K lines of java+sql to 1K lines of scala. Never experienced something like this before or after. At the next place I wrote 400 lines of C++ mapreduce over 3 days, which could have been 40 lines of sql or 10 lines of scalding/spark. Jarring transition.

chrisjc 5y ago
Regarding the reduction in lines of SQL. Doesn't this imply a tradeoff with where compute is utilized? By reducing SQL, wouldn't you be "pushing up" the compute from the database to the JVM while also increasing the amount of data that has to be moved around? Edit: Wait, did you imply that the SQL you replaced with Scala ultimately gets transpiled into SQL (spark or other dataframe engine) and "pushed down" to the database? Now you're just maintaining Scala and not a rats nest of SQL strings concatenated together? If so, that's pretty awesome.
- random314 5y ago
  Most of this was Hadoop. Which executes mapreduce jobs written in sql on the jvm. The scalding jobs executed faster, except for one job for which we had to stick with Apache pig. Spark jobs were native RDDs before the SQL Spark era.
- random314 5y ago
  No. There was no transpiling to SQL. This was several years ago, before Spark SQL. The SQL was HiveQL. Spark didn't really support SQL at the time and was native JVM Scala RDDs. HiveQL with UDFs compiled to JVM bytecode typically ended up performing worse than scalding that corresponds to straight raw jvm map reduce jobs that are the real native interface of Hadoop. The real performance drag from HQL came from developers writing inefficient map reduce jobs with extra joins to avoid writing Java UDFs because they were a huge PITA. In scalding, developers wrote map reduce jobs with as few group bys as possible. Our hadoop infra team actually complimented us on how great the execution profile of our teams jobs were.