7 ms·

Scaling Scala

93 points by amzans 9y ago

FLGMwt 9y ago
I'm not quite sure why the Scalaz IRC behavior issue got such a focus here, but here's the chat logs of one of the incidents https://gist.github.com/betehess/77de5b4b35ae11801936 https://gist.github.com/betehess/77de5b4b35ae11801936
- killin_dan 9y ago
  Idk, tony is my programming role model. Fantastic guy, extremely knowledgeable, explains things in very approachable way. Love him to death. He's helped me fix MANY stupid type puzzles, and scalaz scumbags ran him out of the entire language. I mainly do f# and erlang now :( Scala has plenty of problems aside from the users. I can't begin to explain how terrible sbt is to people who haven't suffered its grasp. Sbt imo is the single most critical obstacle preventing widespread adoption. It's awful. There's too much politics in the scalaverse. The people are too whiny, the tooling whines even more than the people, and its a damn shame because it's really great code at the end of the day. Its the most fascinating objfunctional I know of, though F# isn't bad at all. Great tooling too, but poor native support.
  - walkingolof 9y ago
    What do you find problematic with SBT ? I'm from the C++ camp and I just love SBT, simple and powerful, so I never really understood what the issue was with SBT (i've seen other people complain about it too)
    - killin_dan 9y ago
      Too many weird macros, too many non-english operators, too much weird bullshit with Ivy, that if you install scala from source sbt will fuck it up if you try and do coverage testing on different versions, build times are ridiculous (this has improved quite a bit since I first used sbt, but it's still embarrassingly slow compared to other build systems (FAKE comes to mind immediately, or rebar3 is real nice too, hell I still use rake for when I have an obnoxious code structure with ten million subdirs, and it's great if you use jruby) What's good about it? What about sbt makes it better than any other build tool OTHER than the fact that it lets you use scala? I don't mind that SBT uses a bunch of advanced concepts and type safety, that's fucking AWESOME, but it's so difficult to wrangle it to just build something correctly, it seems like there's such a huge time investment to learning it to a threshold where its features are useful enough to make that investment seem worth it. Several years later, I still have not reached that point. It seemed like such a pain at first to learn other languages (Erlang syntax still confuses me sometimes) but then I started completing components in these languages and there were moments when I started working on prod builds, and using rake, FAKE, rebar3, etc, and it really put into perspective how much time I had been spending internally screaming at sbt to just DO IT already. There will always be a special place in my heart for Scala as a language, and I think it's one of those languages we'll look back on in 3017 and think "Damn, we learned a lot then" but I'm getting over it in my day-to-day work. I'm also being slowly enslaved by .NET corp overlords now, but I've been complaining about SBT for a LONG time before this, and a lot of other people have too, sometimes for the same reasons that I hate it, sometimes for their own special reasons. It causes way too much friction for Lightbend to continue to look the other way. It's cool when you read the list of features, but write a complicated build and look me in the eyes and say you enjoyed it, you just can't.
      - eweise 9y ago
        why not just use Maven?
        killin_dan 9y ago
        I don't want to trade a shit for a turd. And I also don't want to learn another jvm build system. I've long awaited a day where scala native can start replacing things, but they've already put their sbt in their coffin.
  - tekacs 9y ago
    https://github.com/cvogt/cbt https://github.com/cvogt/cbt [0] is under very active, heavy development and seems like a potential beacon of hope to SBT-haters (including myself). The CBT author and community seem determined to genuinely make it a completely viable SBT replacement (it already works pretty well). Given that during the earlier part of my career starting a new SBT project would involve 1 - 2 days of wrangling every time (there's always something simple that needs changing, that's inevitably harder than it need be in SBT), I can't wait to see CBT mature. [0]: In their own words: 'fun, fast, intuitive, compositional, statically checked builds written in Scala'
    - cdegroot 9y ago
      It'd be the first build system for any JVM language that would be nice. I don't know what it is, but they all seem to come with various amounts of suckage (started with Java when GNU Make was the most advanced option, via ant, maven, gradle, buildr, sbt, ...). It must be the JVM. I worked with Scala for close to a decade now, and my conclusion is that its advantages (type stuff) are mostly theoretical, and its drawbacks (sbt, type stuff, the JVM, slow compiler, the mess of features) are mostly practical. I'm happy to have mostly moved on.
      - killin_dan 9y ago
        What do you use now? Have you tried F#?
      - pimeys 9y ago
        While not on the same level as Cargo, I'd say l Leiningen is a very robust and nice build system for the JVM. Miles ahead of SBT.
        killin_dan 9y ago
        Yeah but who would want to use clojure of they had a choice to use scala? Lein is what I always wished sbcl had.
    - killin_dan 9y ago
- tekacs 9y ago
  The Scalaz vs Cats issue (and seeing what happens to them over time as a 'community judgement' on whether the CoC decision was correct, as in the article) is muddied further by the fact that Cats is different on a technical level too (it's not just a fork of Scalaz). Cats is more approachable than Scalaz. Whilst you could ostensibly see it as an extension of their CoC principles, I think Cats mostly just benefits from being much, much newer and so has more freedom and power of hindsight.
spenrose 9y ago
Two points: article is 5 months old, and it's fantastic: informed journalism rather than "one coder's opinion."
ludicast 9y ago
What strikes me about the Scala community is how amazingly anti-fragile it is. Odersky often uses criticism to refine future versions of the language. He is currently my favorite benevolent dictator.
- happyhippo333 9y ago
  With handling criticism it all comes down to the mental attitude: Is your project just a tool to get a job done or do you draw some sense of self-worth and life purpose from it? Former is a healthy attitude, latter would make your life miserable.
bollockitis 9y ago
Aren't alternative JVM languages subject to the same potential legal issues that sparked the Oracle lawsuit? Don't languages like Scala, Groovy, and Kotlin still make use of many Java APIs?
- ex3ndr 9y ago
  kotlin does not reimplement java API. Instead it have it's own "wrappers" for collections and primitive types. This wrappers are just platform specific types that is replaced directly to native ArrayList and others by compiler. This is more like just using Java API instead of reimplementing it. If Oracle will sue someone for using their API to build Java apps - it will be the end for everything.
  - sjrd 9y ago
    True, and Scala is exactly the same. It uses Java APIs.
    - vorg 9y ago
      Apache Groovy is also the same.
      - happyhippo333 9y ago
        Why is that in every discussion about Scala some Groovy fanboy pops out? sometimes with a ridiculous claim like "groovy is more popular", or "groovy has better java interoperability"
        dang 9y ago
        This account's comments have already been violating the HN guidelines. We ban accounts that do that. Please read the following and post civilly and substantively, or not at all. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newswelcome.html https://news.ycombinator.com/newswelcome.html
jakozaur 9y ago
The main long-term of Scala is nobody is making money on it. Lightbend tried (formerly called Typesafe), but haven't been that succesful and pivoted to other tools (Akka). Java also stagnated, because money obssesed Oracle is not earning much money on it.
- BatFastard 9y ago
  Good point. Seems like a language either needs to be non profit and open sourced, or profitable and closed source. Non profit and closed source seems like a formula for doom and destruction.
  - hocuspocus 9y ago
    > Non profit and closed source seems like a formula for doom and destruction. Which is not the situation here.
    - BatFastard 9y ago
      It seems very hard to make money off of a language unless you can get support contracts from corporations. Which in the language rich world is very hard.
      - hocuspocus 9y ago
        No company is trying to sell Scala on its own. Several startups emerged around major open source Scala projects and are doing fine. Lightbend employs core contributors to the compiler and stdlib. There are also very successful consulting shops around Scala technologies that are contributing where they can. The language and its ecosystem haven't seen anything but growth, I'm not worried.
- blackoil 9y ago
  Is Java really stagnated? I was under impression Java 8/9/10 have stable schedules and are delivering some long sought features like lambda, value types and REPL. It is much better than when Sun had lost all money and future was bleak.
AheadOfTime295 9y ago
The community both shrinks and grows. An example of the former: https://www.reddit.com/r/programming/comments/6bh8xv/leaving_scala_after_six_years_of_development/ https://www.reddit.com/r/programming/comments/6bh8xv/leaving...
blackoil 9y ago
One of the main problems with Scala is that it has a clusterfuck of niche use features, which even experienced developers have problem using. Scala core library is littered with :: +: and other nonsensical operators. Things like implicit may have few good use cases, but are mostly abused and make code hard to read. Blindly following immutability in even local function variables and stack makes code slow and unreadable.
- RBerenguel 9y ago
  Eventually you get to understand these operators (and, after all they essentially mean append/concatenate/join in one sense or another... so it's basically add stuff to other stuff), and as for implicits, they make code much easier to follow if they are used sparingly and for good reasons. What makes Scala code (with implicits) hard to follow can be type classes, and understanding how all the pieces of the language fit together in these cases. As for immutability making things slow... Well, personally I start writing code for humans (and immutability makes the code much easier to read for humans). _If_ speed is an issue then I consider dropping immutability. So far have had no issues with Scala. Also depending on how much speed I need I wouldn't even think about using Scala.
  - killin_dan 9y ago
    That's what they say about perl. Keep trying and itll make sense! Granted, scala can be written to read much easier than perl, but that argument in particular don't carry much weight imo.
    - RBerenguel 9y ago
      The thing is, in Scala you can do really crazy and complicated stuff (see Shapeless, Cats, etc) that in other languages (with exceptions, of course) would be extremely hard or downright impossible. In Scala, it is just hard, but doable, but implies code being hard to read because what you are doing _is_ hard. This does not detract from doing as much as you can to make it readable, but sometimes it is impossible, or at least impossible to someone not used to it. I'm not good enough with Scala to understand some of it (yet, I hope) but this is something that is bugging me lately. Some things (in general, I'm not talking just about programming here) are hard. No matter how much sugar-coating, some stuff is hard to understand, complex and not everyone can get to the bottom of it. It just happens in some programming languages, happens in most of mathematics, physics and computer science.
MichaelBurge 9y ago
Before you start trying to scale with Scala/Spark, consider if my preferred progression makes more sense: * < 50 million rows: Can your data fit into a standard RDBMS? Writing all your business logic in SQL might make more sense. * < 1 billion rows: Can your data fit into a few flat files on disk, with some Python/Perl scripts to manipulate them? * < 1 trillion rows: Can you use an off-the-shelf solution like Google BigQuery or Amazon Redshift? You can also use this rather than flat files depending on the skillset of your team. * < 1 trillion rows: Can you engineer smart data structures & algorithms, write some custom C++, tune the compiler flags, inspect the resulting assembly, cache intermediate results on disk, etc. for a small number of core queries? For anything larger, good luck. You seem to genuinely need a dedicated data engineering team, and similar tools may very well help you scale up. Even so, try to keep things simple. A single binary with well-defined inputs & outputs(even poorly written in C++) is intrinsically simpler than a distributed network(even if the Scala code itself is simpler and well-written), just due to deployment considerations. If you need things to be accessible in real-time from a website, that may also require a sophisticated solution. Consider if you can queue the query and use one of the above approaches, however. It's even harder to deploy a real-time distributed query system with worst-case performance guaranteed to be queryable from the front-end of your website.
- vikiomega9 9y ago
  I'm averse to writing SQL simply because it's a bit harder to test business logic with and things get hard to read very quickly. Also consider the example where my logs are in json for whatever reason, now using SQL just makes things harder especially when I need to run some sort of transformation.
  - tormeh 9y ago
    Also SQL is in clear text, which invites small manual changes outside of version control.
- lmm 9y ago
  I find Scala/Spark easier to write and work with than any of the options you list (yes, SQL included). It gives you a real first-class programming language and lets you reuse the same code in your analysis/aggregation as you're using for your main product.
geodel 9y ago
> (Spark, Mesos, Akka, Cassandra, Kafka), most of which has been written in Scala. Regarding Kafka, I noticed a large amount of Scala API is deprecated and replaced with Java API in the latest version. There is about twice the Java code in Kafka as compared to Scala. Mesos is C++. Cassandra is Java. So it seems to me Spark and Akka is mainly Scala. And in this field I am expecting Rust to make big inroads with high performance, no JVM like overhead for memory safety, and lately emphasis on async IO with tokio.
- throwaway91111 9y ago
  I wouldn't expect rust to take much scala market share. A big part of he scala advantage is being able to take advantage of the massive number of jvm libraries available.
- kod 9y ago
  The kafka client api moved to java because of fewer dependencies to distribute. The brokers are still a scala project.

93 points by amzans 9y ago

I'm not quite sure why the Scalaz IRC behavior issue got such a focus here, but here's the chat logs of one of the incidents https://gist.github.com/betehess/77de5b4b35ae11801936 https://gist.github.com/betehess/77de5b4b35ae11801936

killin_dan 9y ago
Idk, tony is my programming role model. Fantastic guy, extremely knowledgeable, explains things in very approachable way. Love him to death. He's helped me fix MANY stupid type puzzles, and scalaz scumbags ran him out of the entire language. I mainly do f# and erlang now :( Scala has plenty of problems aside from the users. I can't begin to explain how terrible sbt is to people who haven't suffered its grasp. Sbt imo is the single most critical obstacle preventing widespread adoption. It's awful. There's too much politics in the scalaverse. The people are too whiny, the tooling whines even more than the people, and its a damn shame because it's really great code at the end of the day. Its the most fascinating objfunctional I know of, though F# isn't bad at all. Great tooling too, but poor native support.
- walkingolof 9y ago
  What do you find problematic with SBT ? I'm from the C++ camp and I just love SBT, simple and powerful, so I never really understood what the issue was with SBT (i've seen other people complain about it too)
  - killin_dan 9y ago
    Too many weird macros, too many non-english operators, too much weird bullshit with Ivy, that if you install scala from source sbt will fuck it up if you try and do coverage testing on different versions, build times are ridiculous (this has improved quite a bit since I first used sbt, but it's still embarrassingly slow compared to other build systems (FAKE comes to mind immediately, or rebar3 is real nice too, hell I still use rake for when I have an obnoxious code structure with ten million subdirs, and it's great if you use jruby) What's good about it? What about sbt makes it better than any other build tool OTHER than the fact that it lets you use scala? I don't mind that SBT uses a bunch of advanced concepts and type safety, that's fucking AWESOME, but it's so difficult to wrangle it to just build something correctly, it seems like there's such a huge time investment to learning it to a threshold where its features are useful enough to make that investment seem worth it. Several years later, I still have not reached that point. It seemed like such a pain at first to learn other languages (Erlang syntax still confuses me sometimes) but then I started completing components in these languages and there were moments when I started working on prod builds, and using rake, FAKE, rebar3, etc, and it really put into perspective how much time I had been spending internally screaming at sbt to just DO IT already. There will always be a special place in my heart for Scala as a language, and I think it's one of those languages we'll look back on in 3017 and think "Damn, we learned a lot then" but I'm getting over it in my day-to-day work. I'm also being slowly enslaved by .NET corp overlords now, but I've been complaining about SBT for a LONG time before this, and a lot of other people have too, sometimes for the same reasons that I hate it, sometimes for their own special reasons. It causes way too much friction for Lightbend to continue to look the other way. It's cool when you read the list of features, but write a complicated build and look me in the eyes and say you enjoyed it, you just can't.
    - eweise 9y ago
      why not just use Maven?
      - killin_dan 9y ago
        I don't want to trade a shit for a turd. And I also don't want to learn another jvm build system. I've long awaited a day where scala native can start replacing things, but they've already put their sbt in their coffin.
- tekacs 9y ago
  https://github.com/cvogt/cbt https://github.com/cvogt/cbt [0] is under very active, heavy development and seems like a potential beacon of hope to SBT-haters (including myself). The CBT author and community seem determined to genuinely make it a completely viable SBT replacement (it already works pretty well). Given that during the earlier part of my career starting a new SBT project would involve 1 - 2 days of wrangling every time (there's always something simple that needs changing, that's inevitably harder than it need be in SBT), I can't wait to see CBT mature. [0]: In their own words: 'fun, fast, intuitive, compositional, statically checked builds written in Scala'
  - cdegroot 9y ago
    It'd be the first build system for any JVM language that would be nice. I don't know what it is, but they all seem to come with various amounts of suckage (started with Java when GNU Make was the most advanced option, via ant, maven, gradle, buildr, sbt, ...). It must be the JVM. I worked with Scala for close to a decade now, and my conclusion is that its advantages (type stuff) are mostly theoretical, and its drawbacks (sbt, type stuff, the JVM, slow compiler, the mess of features) are mostly practical. I'm happy to have mostly moved on.
    - killin_dan 9y ago
      What do you use now? Have you tried F#?
    - pimeys 9y ago
      While not on the same level as Cargo, I'd say l Leiningen is a very robust and nice build system for the JVM. Miles ahead of SBT.
      - killin_dan 9y ago
        Yeah but who would want to use clojure of they had a choice to use scala? Lein is what I always wished sbcl had.
  - killin_dan 9y ago
tekacs 9y ago
The Scalaz vs Cats issue (and seeing what happens to them over time as a 'community judgement' on whether the CoC decision was correct, as in the article) is muddied further by the fact that Cats is different on a technical level too (it's not just a fork of Scalaz). Cats is more approachable than Scalaz. Whilst you could ostensibly see it as an extension of their CoC principles, I think Cats mostly just benefits from being much, much newer and so has more freedom and power of hindsight.

spenrose 9y ago

Two points: article is 5 months old, and it's fantastic: informed journalism rather than "one coder's opinion."

ludicast 9y ago

What strikes me about the Scala community is how amazingly anti-fragile it is. Odersky often uses criticism to refine future versions of the language. He is currently my favorite benevolent dictator.

happyhippo333 9y ago
With handling criticism it all comes down to the mental attitude: Is your project just a tool to get a job done or do you draw some sense of self-worth and life purpose from it? Former is a healthy attitude, latter would make your life miserable.

bollockitis 9y ago

Aren't alternative JVM languages subject to the same potential legal issues that sparked the Oracle lawsuit? Don't languages like Scala, Groovy, and Kotlin still make use of many Java APIs?

ex3ndr 9y ago
kotlin does not reimplement java API. Instead it have it's own "wrappers" for collections and primitive types. This wrappers are just platform specific types that is replaced directly to native ArrayList and others by compiler. This is more like just using Java API instead of reimplementing it. If Oracle will sue someone for using their API to build Java apps - it will be the end for everything.
- sjrd 9y ago
  True, and Scala is exactly the same. It uses Java APIs.
  - vorg 9y ago
    Apache Groovy is also the same.
    - happyhippo333 9y ago
      Why is that in every discussion about Scala some Groovy fanboy pops out? sometimes with a ridiculous claim like "groovy is more popular", or "groovy has better java interoperability"
      - dang 9y ago
        This account's comments have already been violating the HN guidelines. We ban accounts that do that. Please read the following and post civilly and substantively, or not at all. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newswelcome.html https://news.ycombinator.com/newswelcome.html

jakozaur 9y ago

The main long-term of Scala is nobody is making money on it. Lightbend tried (formerly called Typesafe), but haven't been that succesful and pivoted to other tools (Akka). Java also stagnated, because money obssesed Oracle is not earning much money on it.

BatFastard 9y ago
Good point. Seems like a language either needs to be non profit and open sourced, or profitable and closed source. Non profit and closed source seems like a formula for doom and destruction.
- hocuspocus 9y ago
  > Non profit and closed source seems like a formula for doom and destruction. Which is not the situation here.
  - BatFastard 9y ago
    It seems very hard to make money off of a language unless you can get support contracts from corporations. Which in the language rich world is very hard.
    - hocuspocus 9y ago
      No company is trying to sell Scala on its own. Several startups emerged around major open source Scala projects and are doing fine. Lightbend employs core contributors to the compiler and stdlib. There are also very successful consulting shops around Scala technologies that are contributing where they can. The language and its ecosystem haven't seen anything but growth, I'm not worried.
blackoil 9y ago
Is Java really stagnated? I was under impression Java 8/9/10 have stable schedules and are delivering some long sought features like lambda, value types and REPL. It is much better than when Sun had lost all money and future was bleak.

AheadOfTime295 9y ago

The community both shrinks and grows. An example of the former: https://www.reddit.com/r/programming/comments/6bh8xv/leaving_scala_after_six_years_of_development/ https://www.reddit.com/r/programming/comments/6bh8xv/leaving...

blackoil 9y ago

One of the main problems with Scala is that it has a clusterfuck of niche use features, which even experienced developers have problem using. Scala core library is littered with :: +: and other nonsensical operators. Things like implicit may have few good use cases, but are mostly abused and make code hard to read. Blindly following immutability in even local function variables and stack makes code slow and unreadable.

RBerenguel 9y ago
Eventually you get to understand these operators (and, after all they essentially mean append/concatenate/join in one sense or another... so it's basically add stuff to other stuff), and as for implicits, they make code much easier to follow if they are used sparingly and for good reasons. What makes Scala code (with implicits) hard to follow can be type classes, and understanding how all the pieces of the language fit together in these cases. As for immutability making things slow... Well, personally I start writing code for humans (and immutability makes the code much easier to read for humans). _If_ speed is an issue then I consider dropping immutability. So far have had no issues with Scala. Also depending on how much speed I need I wouldn't even think about using Scala.
- killin_dan 9y ago
  That's what they say about perl. Keep trying and itll make sense! Granted, scala can be written to read much easier than perl, but that argument in particular don't carry much weight imo.
  - RBerenguel 9y ago
    The thing is, in Scala you can do really crazy and complicated stuff (see Shapeless, Cats, etc) that in other languages (with exceptions, of course) would be extremely hard or downright impossible. In Scala, it is just hard, but doable, but implies code being hard to read because what you are doing _is_ hard. This does not detract from doing as much as you can to make it readable, but sometimes it is impossible, or at least impossible to someone not used to it. I'm not good enough with Scala to understand some of it (yet, I hope) but this is something that is bugging me lately. Some things (in general, I'm not talking just about programming here) are hard. No matter how much sugar-coating, some stuff is hard to understand, complex and not everyone can get to the bottom of it. It just happens in some programming languages, happens in most of mathematics, physics and computer science.

MichaelBurge 9y ago

Before you start trying to scale with Scala/Spark, consider if my preferred progression makes more sense: * < 50 million rows: Can your data fit into a standard RDBMS? Writing all your business logic in SQL might make more sense. * < 1 billion rows: Can your data fit into a few flat files on disk, with some Python/Perl scripts to manipulate them? * < 1 trillion rows: Can you use an off-the-shelf solution like Google BigQuery or Amazon Redshift? You can also use this rather than flat files depending on the skillset of your team. * < 1 trillion rows: Can you engineer smart data structures & algorithms, write some custom C++, tune the compiler flags, inspect the resulting assembly, cache intermediate results on disk, etc. for a small number of core queries? For anything larger, good luck. You seem to genuinely need a dedicated data engineering team, and similar tools may very well help you scale up. Even so, try to keep things simple. A single binary with well-defined inputs & outputs(even poorly written in C++) is intrinsically simpler than a distributed network(even if the Scala code itself is simpler and well-written), just due to deployment considerations. If you need things to be accessible in real-time from a website, that may also require a sophisticated solution. Consider if you can queue the query and use one of the above approaches, however. It's even harder to deploy a real-time distributed query system with worst-case performance guaranteed to be queryable from the front-end of your website.

vikiomega9 9y ago
I'm averse to writing SQL simply because it's a bit harder to test business logic with and things get hard to read very quickly. Also consider the example where my logs are in json for whatever reason, now using SQL just makes things harder especially when I need to run some sort of transformation.
- tormeh 9y ago
  Also SQL is in clear text, which invites small manual changes outside of version control.
lmm 9y ago
I find Scala/Spark easier to write and work with than any of the options you list (yes, SQL included). It gives you a real first-class programming language and lets you reuse the same code in your analysis/aggregation as you're using for your main product.

geodel 9y ago

> (Spark, Mesos, Akka, Cassandra, Kafka), most of which has been written in Scala. Regarding Kafka, I noticed a large amount of Scala API is deprecated and replaced with Java API in the latest version. There is about twice the Java code in Kafka as compared to Scala. Mesos is C++. Cassandra is Java. So it seems to me Spark and Akka is mainly Scala. And in this field I am expecting Rust to make big inroads with high performance, no JVM like overhead for memory safety, and lately emphasis on async IO with tokio.

throwaway91111 9y ago
I wouldn't expect rust to take much scala market share. A big part of he scala advantage is being able to take advantage of the massive number of jvm libraries available.
kod 9y ago
The kafka client api moved to java because of fewer dependencies to distribute. The brokers are still a scala project.