23 ms·

From First Principles: Why Scala?

286 points by lihaoyi 6y ago

dbsmith83 6y ago
Scala is nice for doing things with Spark compared to say, Java, but I really think that Kotlin can do just as well. The only advantage Scala has with Spark is the syntactic sugar. Once there is an API for Kotlin (beyond the preview version), I will be dropping Scala like a bad habit. Kotlin is also concise, has better IntelliJ support, less "implicit magic", and better Java interop. https://github.com/JetBrains/kotlin-spark-api https://github.com/JetBrains/kotlin-spark-api
- bjarneh 6y ago
  > has better IntelliJ support That's gonna be hard to beat for most languages I guess, the best IntelliJ support will most likely always go to Kotlin :-)
  - brabel 6y ago
    That's not actually true, IntelliJ support for Java has always been much better, and still is today! Just try auto-completion in Java VS Kotlin in your IDE... it's much faster in Java files, something I've been complaining about as I've been using more Kotlin than Java, recently. Check also how many refactorings are available in your Java files compared to the Kotlin ones. Jetbrains knows its main source of income is still, by far, Java, so they don't drop that ball.
- zmmmmm 6y ago
  If only Kotlin would introduce a non-ridiculous way to declare map and list literals it would be a great data science language. As it is, it falls over at the first hurdle where you want to specify some data inline. One reason I've really stuck with Groovy for JVM-based data sciency stuff.
  - dbsmith83 6y ago
    val list = listOf('a', 'b', 'c') val map = hashMapOf( "John" to "Doe", "Jane" to "Smith" ) Seems pretty non-ridiculous to me. Maybe you were using an old version? I guess you can switch now :)
- lmm 6y ago
  Kotlin has lots of syntax sugar but the semantics are crazy. It's essentially an imperative language where the only way to reason about effects (suspend functions) is by reasoning about control flow. That already makes things like error handling incomprehensible, and it'll be a really bad fit for Spark's way of doing things (where values have to be serializable and shipped around the cluster) - imagine trying to use a suspend function from a Spark anything, it's completely impossible to implement in a sane way.
  - dbsmith83 6y ago
    I'm curious what your use case is for using a suspend function in spark. I have written several spark jobs with Kotlin and have never had a reason to create one. The only difference from the Scala/Java jobs is that I had to be much more explicit with certain things (encoders in higher order funcs, etc.).
    - lmm 6y ago
      All of the reasons you'd want to use a suspend function at all - e.g. mutually recursive functions, async I/O (e.g. database queries), error handling - suspend functions seem to be kotlin's answer to anything even slightly non-vanilla. Writing in Scala I can, with care, use async and iteratees in spark and be able to reason about what's going to get executed where and when, because these things are represented as values.
adenozine 6y ago
If I've said it once I've said it a thousand times: the day Scala kills sbt officially will be the day thousands of devs consider a return to it. Until then, I couldn't dare.
- denimnerd42 6y ago
  we use it with Maven. no issues
- holtalanm 6y ago
  curious. what exactly is wrong with sbt? I used it once of twice when I was learning scala and I liked it a LOT more than gradle/maven. To the point where I actually opted to use it with my Kotlin projects later.
  - arcsin 6y ago
    From the same author as OP: https://www.lihaoyi.com/post/SowhatswrongwithSBT.html https://www.lihaoyi.com/post/SowhatswrongwithSBT.html
  - Fiahil 6y ago
    Being better than maven is not a high bar, you should try cargo/rust, to have a meaningful comparison of a good build system. The only problem is, once you've tried out Rust, it will be difficult for you to come back to Scala.
  - the_af 6y ago
    To me, sbt is confusing as hell. I no longer program in Scala, but I used to for my previous job and I regularly came across build files I couldn't understand -- and when I asked the person who wrote them, they couldn't explain their build properly either. And this happened in enough cases that I wouldn't consider it an exception or a problem with a particular developer. SBT used to have two entirely separate syntaxes, and it was a mess translating from one to the other. I don't know if this is still the case (I know one of the two syntaxes got deprecated, can't remember which one now).
  - 6y ago
agbell 6y ago
Great Analysis! I agree that languages are becoming more like Scala. I'm not sure about the JIT part though. Aren't many of the recent language success stories now about AOT compiled languages like Go, Rust and Swift?
- mark_l_watson 6y ago
  I am working on an AI book using Swift. Yesterday in a meeting, my manager was asking why Swift. I compared it to Scala, but built on LLVM, and a rich CoreML and other library infrastructure. REPL + closures, very fast build times. I haven’t had much opportunity to use Scala, but I did take Martin’s Functional Programming with Scala course years ago, and that was great.
  - agbell 6y ago
    Scala is a great language. I haven't done that coursera course, but many people swear by it. To me, Swift and Kotlin look like they have taken inspiration from Scala. I think Kotlin explicitly did. That idea transfer from one area to another is how we get better langauges.
    - zepto 6y ago
      Having programmed professionally in both Scala, and Swift, and enjoyed both, I can see basically nothing that Swift takes from Scala. The common ancestor is OCaml, and the ML languages in general. To be fair, that may change as Swift adopts Actor based concurrency.
  - asimpletune 6y ago
    I wish swift were like Scala. It attempts to look and feel like it, but there’s one huge aspect of Scala that is missing in swift that basically makes swift a world apart, which is expressions vs statements. In Scala everything is an expression, meaning everything evaluates to some result even if that result is void. In swift everything is a statement. There’s a big difference because statements don’t return things, only functions do. It sounds minor, especially since swift has pattern matching and first class supper for functions etc, but the truth is these are all improvements over a more oop (in the Java sense) way of programming. They don’t however make the languages very similar because at a more basic so level they’re very different. I would even go as far as to say that rust is more like Scala than swift is, although it’s been a while since I coded in rust.
occupy_paul_st 6y ago
I definitely agree that many newer languages are strongly inspired by Scala, but I don't see a strong argument for why to choose Scala over one of a newer hybrid language. My own intuition is that these newer languages have had the benefit of being able to learn from Scala's mistakes. A lot of them have intentionally sacrificed some of the Scala's flexibility to be easier to learn and use. I found Scala to be an extremely enlightening language: it taught me a lot about functional programming. But, the complex type system and the sacrifices needed to ensure Java compatibility entail many quirks that I find frustrating. I think it's actually a similar dynamic to what you mention: "retro-fitting these features onto an existing language never fits quite as nicely as a language that was designed with them from scratch." BTW, I am a huge fan of your writing: the Principle of Least Power is a sacred programming text to me .
- playing_colours 6y ago
  What newer hybrid languages besides Kotlin can cover the benefits you can get with Scala: a solid ecosystem (both for Scala and Java) for developing web applications, microservices, data processing, and good performance. Julia, Rust, and Elixir are all great, but popularity, ecosystem are not there yet.
  - dunefox 6y ago
    For Julia there are PyCall and RCall, so the ecosystem is less of an issue than with other languages. Also, its ecosystem has been growing quickly over the last 2 years or so. https://github.com/JuliaPy/PyCall.jl https://github.com/JuliaPy/PyCall.jl https://github.com/JuliaInterop/RCall.jl https://github.com/JuliaInterop/RCall.jl
  - jolux 6y ago
    F#
    - lihaoyi 6y ago
      I actually came to Scala via F#. You can even find exact dates of those transitions in my stackoverflow history. Scala and F# are sinilar enough to basically be the same language, though I found F# had a number of warts and idiosyncracies that made me move on to Scala. Scala didn't have all of these (though of course it had warts of its own!) and is what stuck with me for the long term
esarbe 6y ago
Scala is just an amazing language. I'm very happy that the worst of the 'Scala as a worse Haskell' days is over and that the Scala community is finding it's own programming style. I remember years ago when I had a heated discussion about local mutable state. An that time it felt like heresy to even suggest it. I just recently had another discussion with the same programmer and he was like: "Yeah, that works". Good times.
- asimpletune 6y ago
  Local mutable state as in a var in a function or how did you mean?
  - chaorace 6y ago
    Basically that, yes. The idea is, as long as the mutable state starts and ends within the confines of an individual function, you can get away with treating that function as if it were "truly" FP. The main argument against this is that it's a concession that makes the codebase less coherent. The more hardline argument being that the encapsulated mutable state is still more prone to bugs and should be almost always be avoided, except at great cost.
    - asimpletune 6y ago
      Yeah, this is totally fine. It’s rare that you need to use vars in Scala, but still if it’s scoped to the function then you know you can mentally discount it once the function terminates.
    - kaba0 6y ago
      There are certain algorithms that either can’t or only insanely error prone way can be expressed “purely”. There is nothing wrong with locally mutable state. It is basically the same as a State monad that is evaluated/executed — which can arguably be worse
- randmeerkat 6y ago
  I still find that Scala devs think they’re writing Haskell. From what I’ve seen in the community, no one cares about shipping code, they care about whether or not it’s “pure”.
aynyc 6y ago
Is Scala used outside of Spark and Akka? Spark is my company chosen analytic engine, Scala was/is the main language for majority of the production workload. In the last two years or so, more devs are writing PySpark and SparkSQL jobs than Scala.
- dominotw 6y ago
  def mirrors my experience too. Vast majority of spark jobs were easily ported to sql/dbt and the remaining ones are in pyspark. I used to use a lot of scala spark in backend data processing in 2016 but now its almost down to zero. scala is real big impediment to making data processing accessible to general public in your company. order of preference now at my company is, 1. sql 2. pyspark 3. java spark 4. scala spark eg: shopify found that 70% of their pyspark could be converted to just sql https://shopify.engineering/build-production-grade-workflow-sql-modelling https://shopify.engineering/build-production-grade-workflow-...
  - aynyc 6y ago
    > scala is real big impediment to making data processing accessible to general public in your company Ding Ding Ding! Presto/Athena now is becoming huge in BI ecosystem. We don't really use Spark for ad-hoc BI anymore, we use it for data science and large repetitive workload.
  - MrPowers 6y ago
    I've rolled out Scala based Spark interfaces to non-programmers in Databricks notebooks, so it's definitely possible, but only if you stick with the basic language features. Here's a more detailed PySpark vs Scala comparison in case folks are interested: https://mungingdata.com/apache-spark/python-pyspark-scala-which-better/ https://mungingdata.com/apache-spark/python-pyspark-scala-wh... I think Scala Spark (using 10% of the language features) is a better technical decision (because it provides huge benefits like fat JARs, shading, better text editor support, etc), but the worse overall choice for most organizations because people are generally terrified of Scala. They'd rather do nothing than write Scala code. I can empathize with their position.
bjarneh 6y ago
Scala is one of those great languages with that one Achilles heel. The speed of that compiler: $ time scalac Hello.scala real 0m2,907s user 0m7,901s sys 0m0,308s $ cat Hello.scala object Hello{ def main(args: Array[String]){ println("hello world") } } Implicit type conversion can be debated I guess, but otherwise a very beautiful language.
- lihaoyi 6y ago
  Yeah cold compiles are slow, but most modern tools keep a hot compiler to re-use and once warm a hot compile of hello world is more like ~200ms. There's a bit of complexity managing these daemons, but that's something that the build tool normaly handles for you transparently
  - bjarneh 6y ago
    Yes it gets better, and the 'fsc' (Fast Scala Compiler) does improve on the situation somewhat; but I've always found it annoyingly slow. It should be mentioned that I haven't looked very hard at that language in many years for that very reason; so things could have improved somewhat. And from what I understand, the next version of Scala (Dotty) is going address this issue (build speed).
    - lihaoyi 6y ago
      The compiler has sped up 3x in the past few years. That's a very big speedup, and the current experience is likely nothing like what you remember
      - bjarneh 6y ago
        Sounds good, perhaps a language that needs a revisit.
- _a1_ 6y ago
  Using scalac is not a standard use case. Standard use case is to use sbt (well, or mill, since we're in lihaoyi's thread :D). Sbt starts up slow, but with bloop, or sbtn (native client of sbt), the compilation is really fast (much faster than C++ for example): $ time sbtc compile [info] entering *experimental* thin client - BEEP WHIRR [info] terminate the server with `shutdown` > compile [info] compiling 1 Scala source to /home/.../target/scala-2.13/classes ... [info] compile completed [success] Total time: 0 s, completed Feb 11, 2021 4:21:51 PM sbtc compile 0,08s user 0,02s system 21% cpu 0,492 total
mwcampbell 6y ago
I dabbled with Scala several years ago, but I've been using Kotlin for a JVM-based project and am happy with it. My main reason for choosing Kotlin is smoother interop with the JVM world. For example: - Scala adds an Option type, whereas Kotlin adds nullability checking for existing object types. - Scala has its own convention for getters and setters, whereas Kotlin automatically turns JVM getters and setters into properties. - Scala defines its own set of collection classes, whereas Kotlin has zero-overhead wrappers over the JVM collection types. I haven't tried Kotlin/JS lately, but I expect that Kotlin's strong focus on interop with existing stuff would serve it well there too, especially when it comes to minimizing bundle size. Scala may be more powerful in some ways, but Kotlin gives me the conciseness I want, with great interop with the huge Java ecosystem.
- the_af 6y ago
  > Scala defines its own set of collection classes [...] One (arguably) negative consequence of this is that Scala's collections don't play nice with Hibernate and similar ORMs. Due to the way they work, both Scala and Hibernate wanted to "take over" your collections, and obviously they cannot do so at the same time. Of course, it's arguable whether this is a flaw with Scala or with Hibernate, but for the programmers who just wants to get things done that's irrelevant. It used to be a big deal a few years ago, to the point teams in my company decided to either drop Hibernate or drop Scala. PS: if I remember correctly, early Scala adopters had reached the consensus that ORMs didn't play nice with Scala's FP style anyway, and so using Hibernate was a mistake. Possibly this attitude changed later.
  - thelittlenag 6y ago
    The two major SQL libraries in the Scala ecosystem these days are Doobie (https://tpolecat.github.io/doobie/ https://tpolecat.github.io/doobie/) and Slick (https://scala-slick.org/ https://scala-slick.org/). With Doobie you manually write your queries, and then map the results into the objects in your domain model. Nothing is generated for you. OTOH, nothing is hidden and you are free to write queries as optimized and specialized as you need. The real selling point of Doobie is a typesafe API for manipulating and combining queries, and fragments of queries, into larger wholes. This works very well when your application interfaces with a database it doesn't own. With Slick you get access to a DSL that lets you layout how your tables look. From there Slick offers an api that let's you treat SQL tables as-if they are basically mutable collections, with Slick handling all the SQL generation itself. You also get DDL, so that you can automate db creation and upgrades. This work very well when your application owns and controls the database it is connecting to. Both of these have diverged from that traditional ORM model. Slick bills itself as FRM, or Functional Relational Mapping. And Doobie is embedded queries on steroids.
MrPowers 6y ago
I am a Scala programmer & think it's a great language. Here are some arguments for why not Scala: * Li's libs (os-lib, upickle, utest) have clean public interfaces, but most Scala ecosystem libs are hard to use, see the JSON alternatives for examples: https://www.lihaoyi.com/post/uJsonfastflexibleandintuitiveJSONforScala.html#scala-alternatives https://www.lihaoyi.com/post/uJsonfastflexibleandintuitiveJS... * The Mill build tool looks a lot better than SBT, but seems like everyone is still using SBT * Scala minor version are binary incompatible, so maintaining Scala projects is a big pain. Upgrading Spark from Scala 2.11 to Scala 2.12 was a massive undertaking for example. * Scala has tons of language features and lets people do crazy things in the code. Hard to win technical arguments with Scala geniuses that like using complicated language features. * Scalatest is stil used by most projects and is annoying to use, as described here: https://github.com/lihaoyi/utest#why-utest https://github.com/lihaoyi/utest#why-utest. The overuse of DSLs in Scala is really annoying. Too many DSLs is another example of something I consider to be an antipattern, but there is no Scala community consensus on the responsible use of DSLs. I'm optimistic about Scala. There are some folks that love the language and are continuously improving the ecosystem. Scala 3 will have to sell a better story about ditching legacy tooling and giving users a better default stack if it wants to compete with modern Go/Rust/Python.
- dominotw 6y ago
  > Hard to win technical arguments with Scala geniuses that like using complicated language features. I was on a team building good old crud apps using monads, monoids, categories, combinators, effects cats, seamless and bunch of other nonsense that i've now purged from my brain. You could've easily mistaken our team for a programming language research group at a university. This is literally the number one reason i would never choose scala. There is no upper bound to amount of stupidity one can indulge in. Our code was so convoluted that even intellij had trouble understanding what the hell was going on and would spit out compilation errors when there were none. I now work with clojure team and i feel a sense of relief and weight taken off of my head from all the useless stuff i had to learn and use.
  - Cthulhu_ 6y ago
atbpaca 6y ago
I think that the Scala experience will get even better with Scala 3, tons of well thought refactorings for metaprogramming and typeclasses, among others. Also, looking forward to (optional) indentation based syntax.
- neysofu 6y ago
  It's funny, because my experience is almost the opposite. My Scala skills are a bit rusty at this point and I promised myself to get back in the game once Scala 3 was released. Most of it I like. The new syntax, however, is really putting me off. My #1 problem with Scala has always been its huge upfront complexity and the quality of libraries, which are often over-engineered and not nearly enough UX-centric (cough implicits everywhere). Scala 3 was supposed to be a breath of fresh air, but I don't want to invest more time in a language that instead of getting simpler further complicates something as basic as the syntax. This change tells me that I strongly disagree with the core leadership when it comes to the future of Scala and what kind of language it should be.
  - user1241320 6y ago
    Opposite for me. After years of Scala (which I like) I recently tried to port a few things to Scala 3 and it really feels like some of the wtf-implicit part is gone. We'll still need to make a complete idea about this version 3 though.
liquidify 6y ago
A lot of my buddies who use Scala seem to like it. I'm sure it has some to do with the fact that they can use Intellij, which I consider to be one of the best IDE's available.
ibains 6y ago
Prophecy.io - low code product for data engineering on Spark - is Scala based - we love it! We do compiler work - and packrat parser combinators are great. Basic things are supported well. We use Play. We have GraphQL in web services - use Sangria. Slick FRM for database access is very clean. We use Akka for live web socket connections to Spark for interactive execution. We extend Spark code from outside when needed - adding functions to classes in imported libraries. It’s a learning curve for new engineers, but we hire the best and they like Scala after using it for a bit. SBT is confusing as hell, build times were painful but getting better. Debugging is sometimes hard (implicits) One thing I’ve learnt us that if you’re handed a gun, you don’t have to start shooting at your feet just because you can. We used C++ at NVIDIA sensibly a decade back - using OOP but still avoiding horrible parts of C++. We use Scala sensibly now. Can’t imagine another language that would work for us. I’ve used C, C++, F# before (building compilers and databases)- don’t know Kotlin. Only other language we use is Golang for our kubernetes operator (excluding UI)
dlandis 6y ago
> While most programming languages allow you to take an expression (1213, "moo") and infer the type (Int, String), Scala allows you to go the other direction: take the type (Int, String) and infer a value (1213, "moo")! This is tremendously useful in a whole range of different scenarios That's an interesting way to put it (and pretty neat), but I guess I'm not totally convinced it is that useful in practice. I saw the linked example about the parser using implicit objects but I think that could be rewritten in a number of different ways without relying on that feature.
- valenterry 6y ago
  It is for example very usefull to generate random test data (property based testing) in a very elegant way, without manually wiring up a lot of stuff.
- yw3410 6y ago
  DI in Java world is another example where people require this type of feature (unsafely) via reflection.
deleted 6y ago
[deleted]
polote 6y ago
Damn, you submitted the link 5 times in 5 days, at what number did you plan to stop? https://news.ycombinator.com/submitted?id=lihaoyi https://news.ycombinator.com/submitted?id=lihaoyi
- sdfhbdf 6y ago
  I guess 8. They did that with a YouTube link sometime before [1]. Stopped after 8 failed submissions. [1]: https://news.ycombinator.com/item?id=23810499 https://news.ycombinator.com/item?id=23810499
- deleted 6y ago
  [deleted]
- ta988 6y ago
  Looks like this time it did stick to the wall. Thankfully not everybody is playing that game.
lfmunoz4 6y ago
What's the benefit of Scala over Kotlin? To me it seems Kotlin has all the advantages of Scala plus none of the downsides. Also with Android going to Kotlin the community is bigger and Kotlin Co-routines is probably the best multi-threading library ever created.
vemv 6y ago
> In contrast, cross-compilers for dynamic languages like Clojure tend to have a long-list of caveats and incompatibilites I don't think the author fully comprehended the linked resource (namely https://clojurescript.org/about/differences https://clojurescript.org/about/differences). Perhaps he just noted the its size. A good chunk of the document lists things in common, not differences. The actual differences have nothing to do with dynamic typing and are deliberate design decisions: - don't pretend the JVM and JS have identical runtimes when it comes to concurrency or numerics. Clojure is a language, not a platform abstraction. - Emit efficient, optimally minifiable javascript code, at the cost of offering something less traditionally lisp-y when it comes to eval, macros, and other forms of code loading. Those are still possible, but some discipline is imposed.
- lihaoyi 6y ago
  > - don't pretend the JVM and JS have identical runtimes when it comes to concurrency or numerics. Clojure is a language, not a platform abstraction. > - Emit efficient, optimally minifiable javascript code, at the cost of offering something less traditionally lisp-y when it comes to eval, macros, and other forms of code loading. Those are still possible, but some discipline is imposed. This is exactly the kind of thing I'm talking about: Scala.js doesn't need to make these tradeoffs at all! Concurrency works identically (just no parallelism), numerics work identically, macros work identically. The list of pure-Scala things that behave differently in Scala.js is shockingly short (https://www.scala-js.org/doc/semantics.html https://www.scala-js.org/doc/semantics.html) Scala.js sacrifices none of these and still emits efficient (sometimes faster than idiomatic JS!), optimally minifiable javascript code. The fact that Clojure needs to make these tradeoffs and Scala doesn't is largely due to Scala's static nature allowing more aggressive analysis and optimisation at compile time. For example, the reason Scala.js can provide exact semantics for numerics while maintaining high performance is precisely because it knows what exactly types things are, and can use that information to optimize the generated JS in a semantics-preserving way (http://lampwww.epfl.ch/~doeraene/presentations/jslongs-vmm2017/ http://lampwww.epfl.ch/~doeraene/presentations/jslongs-vmm20...)
  - yw3410 6y ago
rubyn00bie 6y ago
I loved Scala until a minor version updated somewhere around 2.9-2.11 destroyed all my hope. Eight years later I still refused to use it to build anything. Yet, with that said, IMHO, it can be fucking great _second_ language. If you haven't used a statically typed language before, i.e. you're from Ruby, JS, Python, et. al then it's one bangin' ass brain rodeo. I'd highly recommend spending a couple months trying to learn it. Once you're done, you will almost certainly spend near zero effort learning any other typesystem save the MLs (which would, in some ways, probably be a better rodeo). One of my favorite "ah-ha! that is totally how it fucking works" moments with Scala was going to see how `Function` worked only to find this bad boi: https://www.scala-lang.org/api/current/scala/Function22.html https://www.scala-lang.org/api/current/scala/Function22.html A lot of time people think there's more magic going on than there is and just seeing that's how a function works with 22 arguments was pretty illuminating for me coming from Ruby.
- fmakunbound 6y ago
  > only to find this bad boi: https://www.scala-lang.org/api/current/scala/Function22.html https://www.scala-lang.org/api/current/scala/Function22.html Dang. I just checked in the Common Lisp I use and it's a bit higher. What's the reason for having to have at least 22 concrete function signatures like that? CL-USER> call-arguments-limit 4611686018427387903 (62 bits, #x3FFFFFFFFFFFFFFF)
  - rbonvall 6y ago
    Originally there was no way of abstracting over an arbitrary number of type parameters in the language, so a different trait was required for each arity. The choice of 22 was arbitrary, I think. This is no longer true in Scala 3, so there will be no such limit: http://dotty.epfl.ch/docs/reference/dropped-features/limit22.html http://dotty.epfl.ch/docs/reference/dropped-features/limit22...
  - lmm 6y ago
    The JVM had no support for functions as values (it was designed around the idea of only having objects with methods). So if you want to very closely mirror Java at runtime (e.g. you want Java reflection libraries to work with your language) but you want to use functions as values, you have to bridge that gap with classes whose instances will represent functions.
deleted 6y ago
[deleted]
xvilka 6y ago
Problem with Scala that most projects use sbt but it's not even packaged in many Linux distributions. With the Haskell doing strong and Rust, both compile into the native out of the box there are no reasons to use Scala unless you need particular Java libraries.
specialist 6y ago
I'm most interested in Scala's structural typing. Are there any studies comparing real world impact of structural vs nominal typing? Defects? Code size? Team size? Architectural choices, big and small, like use of Visitor design patterns? I've looked, but no joy. -- FWIW, metaprogramming should be reserved for personal projects and small high trust teams. Definitely not for bog standard data processing and CRUD apps. I'm kinda curious how structural typing story plays out in this dichotomy.
- jatcwang 6y ago
  Scala _can_ do structural typing, but you almost never see it in the wild so I don't think you'll be able to find any data on it. The main reason why it isn't used in Scala is because it's implemented using run-time reflection which can be slow. From the time I've spent working with Typescript (Probably the most mainstream with with structural types), I don't think structural-by-default is the right approach if we are to design a language for safety & productivity. The error messages are a lot worse than nominal typing, and useful patterns like new type are either not safe or has a runtime cost. [1] I suspect nominal first with good structural typing support from the compiler is where the sweet spot is, but to integrate it nicely in any language used today is probably an open research question. [1] https://kubyshkin.name/posts/newtype-in-typescript/ https://kubyshkin.name/posts/newtype-in-typescript/
davidfstr 6y ago
The author argues that dynamic languages like (JavaScript, Python) have Poor IDE and Tooling Support and are Unmanageable in large programs. I'll counter that if you add a type checker to one of these languages, such as TypeScript (for JavaScript) or mypy (for Python), then all of those problems go away. I'm personally particularly excited about the possibilities unlocked in the Python community by type checking, which I wrote about recently here: https://dafoster.net/articles/2021/01/26/python%27s-type-checking-renaissance/ https://dafoster.net/articles/2021/01/26/python%27s-type-che... Additionally, RE "Poor performance" the author is presumably talking about CPU-bound performance. If you happen to be writing network-based or I/O bound programs, which you always are in JS/TypeScript and sometimes are in Python (for web applications), then the effective performance is more affected by the network/database/disk than the programming language.
- valenterry 6y ago
  I think the author as a point though. Typescript and type annotations for python will never be able to be as good. Not only because they are not "first class", but also because they always have to interopt with potentially non-typed code. Then, creating a good typesystem is really really hard. Even professional language designers struggle with that. Take Martin Odersky, the creater of Scala. He didn't just invent Scala, no, he learnt under his professor, then created pizza-lang and did other experiments and worked with other languages (e.g. he was the one who added generics to Java) before he used all his expertise to create Scala. And even then mistakes were made. You can see it from the language. I work with both Scala and python (with type annotations) and you can see that one is created by a specialized professional, where the other one is created by smart people but not with the same sophistication. > Additionally, RE "Poor performance" the author is presumably talking about CPU-bound performance. He probably does, but these languages are most often also better for IO-bound programs. The reason is that writing asynchronous and concurrently executed code is very difficult and pretty much adds another layer of problems on top of everything someone does. Languages with static types (not only Scala) can leverage a big advantage here, because the compiler can help you to indicate what is synchronous and what isn't.
chrisjarvis 6y ago
Great article, you have convinced me to give Scala another try :D The first time around made me feel very, very stupid.
prionassembly 6y ago
I want to switch away from Python, at least for personal projects. I want to broaden my range, experiment with weird paradigms (Raku) or get code that’s a little faster (eg Rust). The problem is that I use Python as a “glue language” and heavily rely on excellent third party libraries. I don’t want to learn how to write a minimum spanning theorem algorithm; I’m used to have it next to several other convenient graph utilities in networkx. There’s so much stuff for Python — where else will you find something like giotto-tda? I’m not gloating about Python, I feel stuck.
- kaba0 6y ago
  The JVM ecosystem is similarly rich (and basically the only one comparable; there are many js packages but they are more opinionated towards web development), with plenty of languages to choose from (and with Graal’s python implementation you can even write python code that interoperates with other JVM code) As for concrete languages, I like Java, because it has just enough abstraction to make it useful and is not overly complex. Scala is a real gem in that it was one of the first OOP+FP hybrid, a marriage that seems really fruitful since then basically every OOP lang introduced more and more FP concepts and vice versa. There is also Clojure if you are into LISPs. If overly functional programming is not your thing, Kotlin is a modern language with some additional features to Java though I am not particularly fond of it. And there are even more niche languages running on top of it. But the real gem is the JVM itself, and it is really actively developed and many new, exciting features are in the works.
- rodelrod 6y ago
  So you've found a tool that works great for your purposes. Why is that a bad thing?
- lizmat 6y ago
  Raku also has the Inline::Python module, that allows you to use just about any Python (3) module "natively" in your Raku program. The interface is not as polished yet as it is with the Inline::Perl5 module, but there is a GSoC proposal to improve on that! https://github.com/perl-foundation-outreach/gsoc-2021-ideas/blob/main/raku/Inline::Python-Update.md https://github.com/perl-foundation-outreach/gsoc-2021-ideas/...
MrMan 6y ago
I love kotlin because I don’t have to ever worry about all this stuff
tanin 6y ago
I'm a big fan of Scala, though I think the sweet spot for Scala is for a small team with pre-existing Scala's expertise. For a one-person project, I definitely use Scala. I used to use Ruby a lot. You can move fast with safety with the caveat that the code quality is harder to manage/control. Contrasting this with golang where they really take the guard rails to the other extreme (e.g. can't compile if a variable is unused, not even compiling just to test it out locally). Another weak-point of Scala is that the learning curve is steep, especially when it involves SBT, the defacto build system for Scala.
- lihaoyi 6y ago
  We're scaling out Scala to a larger team here at work, and part of the effort is certainly about putting in guardrails. In fact, a lot of it is putting in thr guardrails people like from Go: enforced autoformatting, banning unused variables, fatal warnings, etc. are all just a flip of a switch these days (though we only enforce these things on-merge, people are free to test locally regardless, which I understand is a big pain point in Go!)
  - tanin 6y ago
    Yup, golang's guardrails are great, but they should be enforced on CI... not while coding locally.
AzzieElbab 6y ago
Scala is awesome. It is meant to be a scalable language where you go from level 1 to say 5 as you learn. Problems arise when you put a level 5 guy into a level 1 team and letting him loose. No common style guide, no team training, just write those type signatures approaching tweets max length and let the rest wonder wth is going on
- lihaoyi 6y ago
  I'd argue that the real problem is finding people who think they're level 5 but aren't. Writing dense impenetrable code certainly doesn't sound like a high-level engineer to me! Scala certainly has no shortage of these people, but at my work we throw senior engineers with zero Scala experience or training at the language and it generally turns out OK. Turns out that a 10x faster Python or a 10x more concise Java are selling points enough, even without any fancy code!
  - AzzieElbab 6y ago
    I meant people who are comfortable with scala’s advanced features, not necessarily better engineers.
einpoklum 6y ago
The article mostly lost me with two points: * It claims to be a compiled language "like C++ and Java" - not making a distinction between targeting multiple hardware platforms and targeting a cozy VM. * It ties itself strongly to the Java ecosystem. So, I guess I might consider it in the 'nicer Java' category - and it does seem nicer than Java, subjectively - but not outside of that ecosystem.
- ebruchez 6y ago
  It might be worth mentioning that Scala now compiles to 3 different targets: - JVM - JavaScript (Scala.js) - native (Scala Native) The tie to the Java ecosystem has significantly loosened over the last few years.
  - einpoklum 6y ago
    That is interesting... but it's also the opposite message from the article. It even emphasizes again how Scala "leans heavily" on the JVM: > The last thing that Scala does well is to lean heavily on the host language for both its language semantics as well as its implementation. Scala is typically run on the JVM, which together with the Java ecosystem provides a host of useful things that Scala doesn't need to worry about: etc. etc. So, what do you do with Scala when you don't use the JVM at all, nor any Java libraries?
    - lihaoyi 6y ago
      In Scala.js, you lean heavily on the Javascript runtime, and on the large ecosystem of Javascript libraries. That integration into the existing ecosystem is in fact one of the big selling points of Scala.js, just like how it is a big selling point of Scala-JVM Scala has multiple host languages, and it leans heavily on each of then whenever used on that platform
    - ebruchez 6y ago
      I should add to lihaoyi's reply that in addition to the target environment's libraries, many Scala libraries cross-compile for Scala JVM, Scala.js, and (to a lesser extent due to its relative novelty) Scala Native. For example I am now cross-compiling JVM/JS a fairly large project which uses, directly or indirectly, Circe, Cats, ScalaTest, Enumeratum, Shapeless, and Parboiled2, and all of that works flawlessly.
deleted 6y ago
[deleted]
elmalto 6y ago
I have used python and scala for work and I find myself wishing to be able to use scala a lot when I use python (especially when using pandas or trying to multi process/thread). Not so much the other way around even though I also love python and think it’s amazing. Main reason: The type system is great and there are so many things that I can express/enforce using it that are missing for me in python even using typing libraries. Yes, there is a large crew of FP fans around scala that can be obnoxious at times, but they have also brought a lot of good stuff around. The way I see it: You need groups that “overshoot” and then settle on the middle ground. Compromise... I use FP when it’s not too complicated, but stay away from the larger monad systems that are hard to explain to newcomers (effects/IO/Kleisli etc) because I honestly feel like it’s too much. Let me just run my debug logger when I want to... (yes I know that’s not what it’s about) I probably fall in the category of using scala as the (way) better Java with sprinkles of FP and non-mutable data objects everywhere. I really like the authors style (his work is awesome in general and I point to your posts a lot for our juniors, thank you for your contributions if you read this) It has its flaws, just like any other language, but it’s also incredibly expressive and fast (enough) for the backend and data engineering that we do
- dionian 6y ago
  My favorite thing about the language is its flexibility, if i want to have pure imperative java style code i can do it. I dont think everyone should blame it just because there are so many haskell fans. I think ZIO is a good sign
  - ibraheemdev 6y ago
    Funny, ZIO was also on the front page today: https://news.ycombinator.com/item?id=26101463 https://news.ycombinator.com/item?id=26101463
- jmfldn 6y ago
  Your point about the FP crew being seeming obnoxious sometimes is true. The style itself can be great, I use IO daily and the monadic style is the glue that holds our apps together. The difference with my team, or approaches that work generally, is that we don't go on and on about these abstractions. I barely mention the 'M' word and just talk about chaining things together. Once that is appreciated then IO isn't that hard to get a basic idea of either. We talk a bit about lazy evaluation but I try not to freak everyone out by mentioning purity, higher kinder types and functors every second breath. It's enough to gain a working intuition for these things without bringing theory to the foreground. That can come later if / when ppl are interested. I guess we also choose our style and abstractions carefully and only use things when we can justify it as a team. If someone raised a PR full of novel concepts that we hadn't grokked as a team, it might well get rejected. For example we started using monad transformers a while ago only after the concept and benefits in our code was explained and bought into by the team. This doesn't just apply to Scala but is especially important when using it. Reach consensus with your team about the style and evolve it together. I get the feeling that other language cultures, maybe Rust being one, are good at just using FP, without talking endlessly about it.
non-e-moose 6y ago
Because EMACS is not an efficient programming languge
u678u 6y ago
Old articles of "Why Scala" talked about how great lambdas are, pattern matching, lightweight classes without getters/setters. Now Java does those things the reason to use Scala are few.
- valenterry 6y ago
  That's because these concepts where new and difficult for many people at the time. Now that they already know them from other languages (which is great!) we can directly start with the next batch of helpful techniques in Scala. Java is still light years behind in terms of productivity through language features. (however, therefore it has higher backwards compatibility, which also helps productivity)
quilombodigital 6y ago
When you go to an auto repair shop, the mechanic uses four different screwdrivers or just one super ultra incredible 8-in-1 screwdriver for everything? Why not? is the mechanic stupid? the 8-in-1 screwdriver is marvelous, has the DOUBLE of features! At the end of the day, he has only the tools he need for his job, and thats it.
d_burfoot 6y ago
I dispute the claim that compiled languages have "heavyweight build setups" or are "inconvenient for small programs". I am constantly writing small Java programs to do minor tasks. I compile and run those programs with Python scripts, not heavyweight IDEs or build tools.
- RealityVoid 6y ago
  In my mind, I, perhaps wrongly, equate compiled with - "runs machine code". Java is considered compiled but runs on the JVM. Python also has compilation as one of the steps and runs in its virtual machine yet it is considered interpreted. Sooo... Where do you draw the line? Regarding your main point, I agree, compiled languages do not necessarily have complicated build setups. But maybe it is tainted by my experience in this field.
karna1112 6y ago
It is rare to find a language as simple and yet powerful. Much awaited Scala 3 brings more features. It is not better Java. It is an independent language which can interop with Java . It has richer set of specifications compared to any other JVM language. People think most of Scala community is about functional programming . It is not . It is a hybrid language. It is best suited for every facet of data engineering where we can make use of state of the art data abstractions and concurrent programming. There is ongoing work on improving tooling and ecosystem. I would ask of you people not to get scared of language based on others opinions. Don't go for advanced concepts when all you need as basics.

286 points by lihaoyi 6y ago

Scala is nice for doing things with Spark compared to say, Java, but I really think that Kotlin can do just as well. The only advantage Scala has with Spark is the syntactic sugar. Once there is an API for Kotlin (beyond the preview version), I will be dropping Scala like a bad habit. Kotlin is also concise, has better IntelliJ support, less "implicit magic", and better Java interop. https://github.com/JetBrains/kotlin-spark-api https://github.com/JetBrains/kotlin-spark-api

bjarneh 6y ago
> has better IntelliJ support That's gonna be hard to beat for most languages I guess, the best IntelliJ support will most likely always go to Kotlin :-)
- brabel 6y ago
  That's not actually true, IntelliJ support for Java has always been much better, and still is today! Just try auto-completion in Java VS Kotlin in your IDE... it's much faster in Java files, something I've been complaining about as I've been using more Kotlin than Java, recently. Check also how many refactorings are available in your Java files compared to the Kotlin ones. Jetbrains knows its main source of income is still, by far, Java, so they don't drop that ball.
zmmmmm 6y ago
If only Kotlin would introduce a non-ridiculous way to declare map and list literals it would be a great data science language. As it is, it falls over at the first hurdle where you want to specify some data inline. One reason I've really stuck with Groovy for JVM-based data sciency stuff.
- dbsmith83 6y ago
  val list = listOf('a', 'b', 'c') val map = hashMapOf( "John" to "Doe", "Jane" to "Smith" ) Seems pretty non-ridiculous to me. Maybe you were using an old version? I guess you can switch now :)
lmm 6y ago
Kotlin has lots of syntax sugar but the semantics are crazy. It's essentially an imperative language where the only way to reason about effects (suspend functions) is by reasoning about control flow. That already makes things like error handling incomprehensible, and it'll be a really bad fit for Spark's way of doing things (where values have to be serializable and shipped around the cluster) - imagine trying to use a suspend function from a Spark anything, it's completely impossible to implement in a sane way.
- dbsmith83 6y ago
  I'm curious what your use case is for using a suspend function in spark. I have written several spark jobs with Kotlin and have never had a reason to create one. The only difference from the Scala/Java jobs is that I had to be much more explicit with certain things (encoders in higher order funcs, etc.).
  - lmm 6y ago
    All of the reasons you'd want to use a suspend function at all - e.g. mutually recursive functions, async I/O (e.g. database queries), error handling - suspend functions seem to be kotlin's answer to anything even slightly non-vanilla. Writing in Scala I can, with care, use async and iteratees in spark and be able to reason about what's going to get executed where and when, because these things are represented as values.

adenozine 6y ago

If I've said it once I've said it a thousand times: the day Scala kills sbt officially will be the day thousands of devs consider a return to it. Until then, I couldn't dare.

denimnerd42 6y ago
we use it with Maven. no issues
holtalanm 6y ago
curious. what exactly is wrong with sbt? I used it once of twice when I was learning scala and I liked it a LOT more than gradle/maven. To the point where I actually opted to use it with my Kotlin projects later.
- arcsin 6y ago
  From the same author as OP: https://www.lihaoyi.com/post/SowhatswrongwithSBT.html https://www.lihaoyi.com/post/SowhatswrongwithSBT.html
- Fiahil 6y ago
  Being better than maven is not a high bar, you should try cargo/rust, to have a meaningful comparison of a good build system. The only problem is, once you've tried out Rust, it will be difficult for you to come back to Scala.
- the_af 6y ago
  To me, sbt is confusing as hell. I no longer program in Scala, but I used to for my previous job and I regularly came across build files I couldn't understand -- and when I asked the person who wrote them, they couldn't explain their build properly either. And this happened in enough cases that I wouldn't consider it an exception or a problem with a particular developer. SBT used to have two entirely separate syntaxes, and it was a mess translating from one to the other. I don't know if this is still the case (I know one of the two syntaxes got deprecated, can't remember which one now).
- 6y ago

agbell 6y ago

Great Analysis! I agree that languages are becoming more like Scala. I'm not sure about the JIT part though. Aren't many of the recent language success stories now about AOT compiled languages like Go, Rust and Swift?

mark_l_watson 6y ago
I am working on an AI book using Swift. Yesterday in a meeting, my manager was asking why Swift. I compared it to Scala, but built on LLVM, and a rich CoreML and other library infrastructure. REPL + closures, very fast build times. I haven’t had much opportunity to use Scala, but I did take Martin’s Functional Programming with Scala course years ago, and that was great.
- agbell 6y ago
  Scala is a great language. I haven't done that coursera course, but many people swear by it. To me, Swift and Kotlin look like they have taken inspiration from Scala. I think Kotlin explicitly did. That idea transfer from one area to another is how we get better langauges.
  - zepto 6y ago
    Having programmed professionally in both Scala, and Swift, and enjoyed both, I can see basically nothing that Swift takes from Scala. The common ancestor is OCaml, and the ML languages in general. To be fair, that may change as Swift adopts Actor based concurrency.
- asimpletune 6y ago
  I wish swift were like Scala. It attempts to look and feel like it, but there’s one huge aspect of Scala that is missing in swift that basically makes swift a world apart, which is expressions vs statements. In Scala everything is an expression, meaning everything evaluates to some result even if that result is void. In swift everything is a statement. There’s a big difference because statements don’t return things, only functions do. It sounds minor, especially since swift has pattern matching and first class supper for functions etc, but the truth is these are all improvements over a more oop (in the Java sense) way of programming. They don’t however make the languages very similar because at a more basic so level they’re very different. I would even go as far as to say that rust is more like Scala than swift is, although it’s been a while since I coded in rust.

occupy_paul_st 6y ago

I definitely agree that many newer languages are strongly inspired by Scala, but I don't see a strong argument for why to choose Scala over one of a newer hybrid language. My own intuition is that these newer languages have had the benefit of being able to learn from Scala's mistakes. A lot of them have intentionally sacrificed some of the Scala's flexibility to be easier to learn and use. I found Scala to be an extremely enlightening language: it taught me a lot about functional programming. But, the complex type system and the sacrifices needed to ensure Java compatibility entail many quirks that I find frustrating. I think it's actually a similar dynamic to what you mention: "retro-fitting these features onto an existing language never fits quite as nicely as a language that was designed with them from scratch." BTW, I am a huge fan of your writing: the Principle of Least Power is a sacred programming text to me .

playing_colours 6y ago
What newer hybrid languages besides Kotlin can cover the benefits you can get with Scala: a solid ecosystem (both for Scala and Java) for developing web applications, microservices, data processing, and good performance. Julia, Rust, and Elixir are all great, but popularity, ecosystem are not there yet.
- dunefox 6y ago
  For Julia there are PyCall and RCall, so the ecosystem is less of an issue than with other languages. Also, its ecosystem has been growing quickly over the last 2 years or so. https://github.com/JuliaPy/PyCall.jl https://github.com/JuliaPy/PyCall.jl https://github.com/JuliaInterop/RCall.jl https://github.com/JuliaInterop/RCall.jl
- jolux 6y ago
  F#
  - lihaoyi 6y ago
    I actually came to Scala via F#. You can even find exact dates of those transitions in my stackoverflow history. Scala and F# are sinilar enough to basically be the same language, though I found F# had a number of warts and idiosyncracies that made me move on to Scala. Scala didn't have all of these (though of course it had warts of its own!) and is what stuck with me for the long term

esarbe 6y ago

Scala is just an amazing language. I'm very happy that the worst of the 'Scala as a worse Haskell' days is over and that the Scala community is finding it's own programming style. I remember years ago when I had a heated discussion about local mutable state. An that time it felt like heresy to even suggest it. I just recently had another discussion with the same programmer and he was like: "Yeah, that works". Good times.

asimpletune 6y ago
Local mutable state as in a var in a function or how did you mean?
- chaorace 6y ago
  Basically that, yes. The idea is, as long as the mutable state starts and ends within the confines of an individual function, you can get away with treating that function as if it were "truly" FP. The main argument against this is that it's a concession that makes the codebase less coherent. The more hardline argument being that the encapsulated mutable state is still more prone to bugs and should be almost always be avoided, except at great cost.
  - asimpletune 6y ago
    Yeah, this is totally fine. It’s rare that you need to use vars in Scala, but still if it’s scoped to the function then you know you can mentally discount it once the function terminates.
  - kaba0 6y ago
    There are certain algorithms that either can’t or only insanely error prone way can be expressed “purely”. There is nothing wrong with locally mutable state. It is basically the same as a State monad that is evaluated/executed — which can arguably be worse
randmeerkat 6y ago
I still find that Scala devs think they’re writing Haskell. From what I’ve seen in the community, no one cares about shipping code, they care about whether or not it’s “pure”.

aynyc 6y ago

Is Scala used outside of Spark and Akka? Spark is my company chosen analytic engine, Scala was/is the main language for majority of the production workload. In the last two years or so, more devs are writing PySpark and SparkSQL jobs than Scala.

dominotw 6y ago
def mirrors my experience too. Vast majority of spark jobs were easily ported to sql/dbt and the remaining ones are in pyspark. I used to use a lot of scala spark in backend data processing in 2016 but now its almost down to zero. scala is real big impediment to making data processing accessible to general public in your company. order of preference now at my company is, 1. sql 2. pyspark 3. java spark 4. scala spark eg: shopify found that 70% of their pyspark could be converted to just sql https://shopify.engineering/build-production-grade-workflow-sql-modelling https://shopify.engineering/build-production-grade-workflow-...
- aynyc 6y ago
  > scala is real big impediment to making data processing accessible to general public in your company Ding Ding Ding! Presto/Athena now is becoming huge in BI ecosystem. We don't really use Spark for ad-hoc BI anymore, we use it for data science and large repetitive workload.
- MrPowers 6y ago
  I've rolled out Scala based Spark interfaces to non-programmers in Databricks notebooks, so it's definitely possible, but only if you stick with the basic language features. Here's a more detailed PySpark vs Scala comparison in case folks are interested: https://mungingdata.com/apache-spark/python-pyspark-scala-which-better/ https://mungingdata.com/apache-spark/python-pyspark-scala-wh... I think Scala Spark (using 10% of the language features) is a better technical decision (because it provides huge benefits like fat JARs, shading, better text editor support, etc), but the worse overall choice for most organizations because people are generally terrified of Scala. They'd rather do nothing than write Scala code. I can empathize with their position.

bjarneh 6y ago

Scala is one of those great languages with that one Achilles heel. The speed of that compiler: $ time scalac Hello.scala real 0m2,907s user 0m7,901s sys 0m0,308s $ cat Hello.scala object Hello{ def main(args: Array[String]){ println("hello world") } } Implicit type conversion can be debated I guess, but otherwise a very beautiful language.

lihaoyi 6y ago
Yeah cold compiles are slow, but most modern tools keep a hot compiler to re-use and once warm a hot compile of hello world is more like ~200ms. There's a bit of complexity managing these daemons, but that's something that the build tool normaly handles for you transparently
- bjarneh 6y ago
  Yes it gets better, and the 'fsc' (Fast Scala Compiler) does improve on the situation somewhat; but I've always found it annoyingly slow. It should be mentioned that I haven't looked very hard at that language in many years for that very reason; so things could have improved somewhat. And from what I understand, the next version of Scala (Dotty) is going address this issue (build speed).
  - lihaoyi 6y ago
    The compiler has sped up 3x in the past few years. That's a very big speedup, and the current experience is likely nothing like what you remember
    - bjarneh 6y ago
      Sounds good, perhaps a language that needs a revisit.
_a1_ 6y ago
Using scalac is not a standard use case. Standard use case is to use sbt (well, or mill, since we're in lihaoyi's thread :D). Sbt starts up slow, but with bloop, or sbtn (native client of sbt), the compilation is really fast (much faster than C++ for example): $ time sbtc compile [info] entering *experimental* thin client - BEEP WHIRR [info] terminate the server with `shutdown` > compile [info] compiling 1 Scala source to /home/.../target/scala-2.13/classes ... [info] compile completed [success] Total time: 0 s, completed Feb 11, 2021 4:21:51 PM sbtc compile 0,08s user 0,02s system 21% cpu 0,492 total

mwcampbell 6y ago

I dabbled with Scala several years ago, but I've been using Kotlin for a JVM-based project and am happy with it. My main reason for choosing Kotlin is smoother interop with the JVM world. For example: - Scala adds an Option type, whereas Kotlin adds nullability checking for existing object types. - Scala has its own convention for getters and setters, whereas Kotlin automatically turns JVM getters and setters into properties. - Scala defines its own set of collection classes, whereas Kotlin has zero-overhead wrappers over the JVM collection types. I haven't tried Kotlin/JS lately, but I expect that Kotlin's strong focus on interop with existing stuff would serve it well there too, especially when it comes to minimizing bundle size. Scala may be more powerful in some ways, but Kotlin gives me the conciseness I want, with great interop with the huge Java ecosystem.

the_af 6y ago
> Scala defines its own set of collection classes [...] One (arguably) negative consequence of this is that Scala's collections don't play nice with Hibernate and similar ORMs. Due to the way they work, both Scala and Hibernate wanted to "take over" your collections, and obviously they cannot do so at the same time. Of course, it's arguable whether this is a flaw with Scala or with Hibernate, but for the programmers who just wants to get things done that's irrelevant. It used to be a big deal a few years ago, to the point teams in my company decided to either drop Hibernate or drop Scala. PS: if I remember correctly, early Scala adopters had reached the consensus that ORMs didn't play nice with Scala's FP style anyway, and so using Hibernate was a mistake. Possibly this attitude changed later.
- thelittlenag 6y ago
  The two major SQL libraries in the Scala ecosystem these days are Doobie (https://tpolecat.github.io/doobie/ https://tpolecat.github.io/doobie/) and Slick (https://scala-slick.org/ https://scala-slick.org/). With Doobie you manually write your queries, and then map the results into the objects in your domain model. Nothing is generated for you. OTOH, nothing is hidden and you are free to write queries as optimized and specialized as you need. The real selling point of Doobie is a typesafe API for manipulating and combining queries, and fragments of queries, into larger wholes. This works very well when your application interfaces with a database it doesn't own. With Slick you get access to a DSL that lets you layout how your tables look. From there Slick offers an api that let's you treat SQL tables as-if they are basically mutable collections, with Slick handling all the SQL generation itself. You also get DDL, so that you can automate db creation and upgrades. This work very well when your application owns and controls the database it is connecting to. Both of these have diverged from that traditional ORM model. Slick bills itself as FRM, or Functional Relational Mapping. And Doobie is embedded queries on steroids.

MrPowers 6y ago

I am a Scala programmer & think it's a great language. Here are some arguments for why not Scala: * Li's libs (os-lib, upickle, utest) have clean public interfaces, but most Scala ecosystem libs are hard to use, see the JSON alternatives for examples: https://www.lihaoyi.com/post/uJsonfastflexibleandintuitiveJSONforScala.html#scala-alternatives https://www.lihaoyi.com/post/uJsonfastflexibleandintuitiveJS... * The Mill build tool looks a lot better than SBT, but seems like everyone is still using SBT * Scala minor version are binary incompatible, so maintaining Scala projects is a big pain. Upgrading Spark from Scala 2.11 to Scala 2.12 was a massive undertaking for example. * Scala has tons of language features and lets people do crazy things in the code. Hard to win technical arguments with Scala geniuses that like using complicated language features. * Scalatest is stil used by most projects and is annoying to use, as described here: https://github.com/lihaoyi/utest#why-utest https://github.com/lihaoyi/utest#why-utest. The overuse of DSLs in Scala is really annoying. Too many DSLs is another example of something I consider to be an antipattern, but there is no Scala community consensus on the responsible use of DSLs. I'm optimistic about Scala. There are some folks that love the language and are continuously improving the ecosystem. Scala 3 will have to sell a better story about ditching legacy tooling and giving users a better default stack if it wants to compete with modern Go/Rust/Python.

dominotw 6y ago
> Hard to win technical arguments with Scala geniuses that like using complicated language features. I was on a team building good old crud apps using monads, monoids, categories, combinators, effects cats, seamless and bunch of other nonsense that i've now purged from my brain. You could've easily mistaken our team for a programming language research group at a university. This is literally the number one reason i would never choose scala. There is no upper bound to amount of stupidity one can indulge in. Our code was so convoluted that even intellij had trouble understanding what the hell was going on and would spit out compilation errors when there were none. I now work with clojure team and i feel a sense of relief and weight taken off of my head from all the useless stuff i had to learn and use.
- Cthulhu_ 6y ago

atbpaca 6y ago

I think that the Scala experience will get even better with Scala 3, tons of well thought refactorings for metaprogramming and typeclasses, among others. Also, looking forward to (optional) indentation based syntax.

neysofu 6y ago
It's funny, because my experience is almost the opposite. My Scala skills are a bit rusty at this point and I promised myself to get back in the game once Scala 3 was released. Most of it I like. The new syntax, however, is really putting me off. My #1 problem with Scala has always been its huge upfront complexity and the quality of libraries, which are often over-engineered and not nearly enough UX-centric (cough implicits everywhere). Scala 3 was supposed to be a breath of fresh air, but I don't want to invest more time in a language that instead of getting simpler further complicates something as basic as the syntax. This change tells me that I strongly disagree with the core leadership when it comes to the future of Scala and what kind of language it should be.
- user1241320 6y ago
  Opposite for me. After years of Scala (which I like) I recently tried to port a few things to Scala 3 and it really feels like some of the wtf-implicit part is gone. We'll still need to make a complete idea about this version 3 though.

liquidify 6y ago

A lot of my buddies who use Scala seem to like it. I'm sure it has some to do with the fact that they can use Intellij, which I consider to be one of the best IDE's available.

ibains 6y ago

Prophecy.io - low code product for data engineering on Spark - is Scala based - we love it! We do compiler work - and packrat parser combinators are great. Basic things are supported well. We use Play. We have GraphQL in web services - use Sangria. Slick FRM for database access is very clean. We use Akka for live web socket connections to Spark for interactive execution. We extend Spark code from outside when needed - adding functions to classes in imported libraries. It’s a learning curve for new engineers, but we hire the best and they like Scala after using it for a bit. SBT is confusing as hell, build times were painful but getting better. Debugging is sometimes hard (implicits) One thing I’ve learnt us that if you’re handed a gun, you don’t have to start shooting at your feet just because you can. We used C++ at NVIDIA sensibly a decade back - using OOP but still avoiding horrible parts of C++. We use Scala sensibly now. Can’t imagine another language that would work for us. I’ve used C, C++, F# before (building compilers and databases)- don’t know Kotlin. Only other language we use is Golang for our kubernetes operator (excluding UI)

dlandis 6y ago

> While most programming languages allow you to take an expression (1213, "moo") and infer the type (Int, String), Scala allows you to go the other direction: take the type (Int, String) and infer a value (1213, "moo")! This is tremendously useful in a whole range of different scenarios That's an interesting way to put it (and pretty neat), but I guess I'm not totally convinced it is that useful in practice. I saw the linked example about the parser using implicit objects but I think that could be rewritten in a number of different ways without relying on that feature.

valenterry 6y ago
It is for example very usefull to generate random test data (property based testing) in a very elegant way, without manually wiring up a lot of stuff.
yw3410 6y ago
DI in Java world is another example where people require this type of feature (unsafely) via reflection.

deleted 6y ago

[deleted]

polote 6y ago

Damn, you submitted the link 5 times in 5 days, at what number did you plan to stop? https://news.ycombinator.com/submitted?id=lihaoyi https://news.ycombinator.com/submitted?id=lihaoyi

sdfhbdf 6y ago
I guess 8. They did that with a YouTube link sometime before [1]. Stopped after 8 failed submissions. [1]: https://news.ycombinator.com/item?id=23810499 https://news.ycombinator.com/item?id=23810499
deleted 6y ago
[deleted]
ta988 6y ago
Looks like this time it did stick to the wall. Thankfully not everybody is playing that game.

lfmunoz4 6y ago

What's the benefit of Scala over Kotlin? To me it seems Kotlin has all the advantages of Scala plus none of the downsides. Also with Android going to Kotlin the community is bigger and Kotlin Co-routines is probably the best multi-threading library ever created.

vemv 6y ago

> In contrast, cross-compilers for dynamic languages like Clojure tend to have a long-list of caveats and incompatibilites I don't think the author fully comprehended the linked resource (namely https://clojurescript.org/about/differences https://clojurescript.org/about/differences). Perhaps he just noted the its size. A good chunk of the document lists things in common, not differences. The actual differences have nothing to do with dynamic typing and are deliberate design decisions: - don't pretend the JVM and JS have identical runtimes when it comes to concurrency or numerics. Clojure is a language, not a platform abstraction. - Emit efficient, optimally minifiable javascript code, at the cost of offering something less traditionally lisp-y when it comes to eval, macros, and other forms of code loading. Those are still possible, but some discipline is imposed.

lihaoyi 6y ago
> - don't pretend the JVM and JS have identical runtimes when it comes to concurrency or numerics. Clojure is a language, not a platform abstraction. > - Emit efficient, optimally minifiable javascript code, at the cost of offering something less traditionally lisp-y when it comes to eval, macros, and other forms of code loading. Those are still possible, but some discipline is imposed. This is exactly the kind of thing I'm talking about: Scala.js doesn't need to make these tradeoffs at all! Concurrency works identically (just no parallelism), numerics work identically, macros work identically. The list of pure-Scala things that behave differently in Scala.js is shockingly short (https://www.scala-js.org/doc/semantics.html https://www.scala-js.org/doc/semantics.html) Scala.js sacrifices none of these and still emits efficient (sometimes faster than idiomatic JS!), optimally minifiable javascript code. The fact that Clojure needs to make these tradeoffs and Scala doesn't is largely due to Scala's static nature allowing more aggressive analysis and optimisation at compile time. For example, the reason Scala.js can provide exact semantics for numerics while maintaining high performance is precisely because it knows what exactly types things are, and can use that information to optimize the generated JS in a semantics-preserving way (http://lampwww.epfl.ch/~doeraene/presentations/jslongs-vmm2017/ http://lampwww.epfl.ch/~doeraene/presentations/jslongs-vmm20...)
- yw3410 6y ago

rubyn00bie 6y ago

I loved Scala until a minor version updated somewhere around 2.9-2.11 destroyed all my hope. Eight years later I still refused to use it to build anything. Yet, with that said, IMHO, it can be fucking great _second_ language. If you haven't used a statically typed language before, i.e. you're from Ruby, JS, Python, et. al then it's one bangin' ass brain rodeo. I'd highly recommend spending a couple months trying to learn it. Once you're done, you will almost certainly spend near zero effort learning any other typesystem save the MLs (which would, in some ways, probably be a better rodeo). One of my favorite "ah-ha! that is totally how it fucking works" moments with Scala was going to see how `Function` worked only to find this bad boi: https://www.scala-lang.org/api/current/scala/Function22.html https://www.scala-lang.org/api/current/scala/Function22.html A lot of time people think there's more magic going on than there is and just seeing that's how a function works with 22 arguments was pretty illuminating for me coming from Ruby.

fmakunbound 6y ago
> only to find this bad boi: https://www.scala-lang.org/api/current/scala/Function22.html https://www.scala-lang.org/api/current/scala/Function22.html Dang. I just checked in the Common Lisp I use and it's a bit higher. What's the reason for having to have at least 22 concrete function signatures like that? CL-USER> call-arguments-limit 4611686018427387903 (62 bits, #x3FFFFFFFFFFFFFFF)
- rbonvall 6y ago
  Originally there was no way of abstracting over an arbitrary number of type parameters in the language, so a different trait was required for each arity. The choice of 22 was arbitrary, I think. This is no longer true in Scala 3, so there will be no such limit: http://dotty.epfl.ch/docs/reference/dropped-features/limit22.html http://dotty.epfl.ch/docs/reference/dropped-features/limit22...
- lmm 6y ago
  The JVM had no support for functions as values (it was designed around the idea of only having objects with methods). So if you want to very closely mirror Java at runtime (e.g. you want Java reflection libraries to work with your language) but you want to use functions as values, you have to bridge that gap with classes whose instances will represent functions.

deleted 6y ago

[deleted]

xvilka 6y ago

Problem with Scala that most projects use sbt but it's not even packaged in many Linux distributions. With the Haskell doing strong and Rust, both compile into the native out of the box there are no reasons to use Scala unless you need particular Java libraries.

specialist 6y ago

I'm most interested in Scala's structural typing. Are there any studies comparing real world impact of structural vs nominal typing? Defects? Code size? Team size? Architectural choices, big and small, like use of Visitor design patterns? I've looked, but no joy. -- FWIW, metaprogramming should be reserved for personal projects and small high trust teams. Definitely not for bog standard data processing and CRUD apps. I'm kinda curious how structural typing story plays out in this dichotomy.

jatcwang 6y ago
Scala _can_ do structural typing, but you almost never see it in the wild so I don't think you'll be able to find any data on it. The main reason why it isn't used in Scala is because it's implemented using run-time reflection which can be slow. From the time I've spent working with Typescript (Probably the most mainstream with with structural types), I don't think structural-by-default is the right approach if we are to design a language for safety & productivity. The error messages are a lot worse than nominal typing, and useful patterns like new type are either not safe or has a runtime cost. [1] I suspect nominal first with good structural typing support from the compiler is where the sweet spot is, but to integrate it nicely in any language used today is probably an open research question. [1] https://kubyshkin.name/posts/newtype-in-typescript/ https://kubyshkin.name/posts/newtype-in-typescript/

davidfstr 6y ago

The author argues that dynamic languages like (JavaScript, Python) have Poor IDE and Tooling Support and are Unmanageable in large programs. I'll counter that if you add a type checker to one of these languages, such as TypeScript (for JavaScript) or mypy (for Python), then all of those problems go away. I'm personally particularly excited about the possibilities unlocked in the Python community by type checking, which I wrote about recently here: https://dafoster.net/articles/2021/01/26/python%27s-type-checking-renaissance/ https://dafoster.net/articles/2021/01/26/python%27s-type-che... Additionally, RE "Poor performance" the author is presumably talking about CPU-bound performance. If you happen to be writing network-based or I/O bound programs, which you always are in JS/TypeScript and sometimes are in Python (for web applications), then the effective performance is more affected by the network/database/disk than the programming language.

valenterry 6y ago
I think the author as a point though. Typescript and type annotations for python will never be able to be as good. Not only because they are not "first class", but also because they always have to interopt with potentially non-typed code. Then, creating a good typesystem is really really hard. Even professional language designers struggle with that. Take Martin Odersky, the creater of Scala. He didn't just invent Scala, no, he learnt under his professor, then created pizza-lang and did other experiments and worked with other languages (e.g. he was the one who added generics to Java) before he used all his expertise to create Scala. And even then mistakes were made. You can see it from the language. I work with both Scala and python (with type annotations) and you can see that one is created by a specialized professional, where the other one is created by smart people but not with the same sophistication. > Additionally, RE "Poor performance" the author is presumably talking about CPU-bound performance. He probably does, but these languages are most often also better for IO-bound programs. The reason is that writing asynchronous and concurrently executed code is very difficult and pretty much adds another layer of problems on top of everything someone does. Languages with static types (not only Scala) can leverage a big advantage here, because the compiler can help you to indicate what is synchronous and what isn't.

chrisjarvis 6y ago

Great article, you have convinced me to give Scala another try :D The first time around made me feel very, very stupid.

prionassembly 6y ago

I want to switch away from Python, at least for personal projects. I want to broaden my range, experiment with weird paradigms (Raku) or get code that’s a little faster (eg Rust). The problem is that I use Python as a “glue language” and heavily rely on excellent third party libraries. I don’t want to learn how to write a minimum spanning theorem algorithm; I’m used to have it next to several other convenient graph utilities in networkx. There’s so much stuff for Python — where else will you find something like giotto-tda? I’m not gloating about Python, I feel stuck.

kaba0 6y ago
The JVM ecosystem is similarly rich (and basically the only one comparable; there are many js packages but they are more opinionated towards web development), with plenty of languages to choose from (and with Graal’s python implementation you can even write python code that interoperates with other JVM code) As for concrete languages, I like Java, because it has just enough abstraction to make it useful and is not overly complex. Scala is a real gem in that it was one of the first OOP+FP hybrid, a marriage that seems really fruitful since then basically every OOP lang introduced more and more FP concepts and vice versa. There is also Clojure if you are into LISPs. If overly functional programming is not your thing, Kotlin is a modern language with some additional features to Java though I am not particularly fond of it. And there are even more niche languages running on top of it. But the real gem is the JVM itself, and it is really actively developed and many new, exciting features are in the works.
rodelrod 6y ago
So you've found a tool that works great for your purposes. Why is that a bad thing?
lizmat 6y ago
Raku also has the Inline::Python module, that allows you to use just about any Python (3) module "natively" in your Raku program. The interface is not as polished yet as it is with the Inline::Perl5 module, but there is a GSoC proposal to improve on that! https://github.com/perl-foundation-outreach/gsoc-2021-ideas/blob/main/raku/Inline::Python-Update.md https://github.com/perl-foundation-outreach/gsoc-2021-ideas/...

MrMan 6y ago

I love kotlin because I don’t have to ever worry about all this stuff

tanin 6y ago

I'm a big fan of Scala, though I think the sweet spot for Scala is for a small team with pre-existing Scala's expertise. For a one-person project, I definitely use Scala. I used to use Ruby a lot. You can move fast with safety with the caveat that the code quality is harder to manage/control. Contrasting this with golang where they really take the guard rails to the other extreme (e.g. can't compile if a variable is unused, not even compiling just to test it out locally). Another weak-point of Scala is that the learning curve is steep, especially when it involves SBT, the defacto build system for Scala.

lihaoyi 6y ago
We're scaling out Scala to a larger team here at work, and part of the effort is certainly about putting in guardrails. In fact, a lot of it is putting in thr guardrails people like from Go: enforced autoformatting, banning unused variables, fatal warnings, etc. are all just a flip of a switch these days (though we only enforce these things on-merge, people are free to test locally regardless, which I understand is a big pain point in Go!)
- tanin 6y ago
  Yup, golang's guardrails are great, but they should be enforced on CI... not while coding locally.

AzzieElbab 6y ago

Scala is awesome. It is meant to be a scalable language where you go from level 1 to say 5 as you learn. Problems arise when you put a level 5 guy into a level 1 team and letting him loose. No common style guide, no team training, just write those type signatures approaching tweets max length and let the rest wonder wth is going on

lihaoyi 6y ago
I'd argue that the real problem is finding people who think they're level 5 but aren't. Writing dense impenetrable code certainly doesn't sound like a high-level engineer to me! Scala certainly has no shortage of these people, but at my work we throw senior engineers with zero Scala experience or training at the language and it generally turns out OK. Turns out that a 10x faster Python or a 10x more concise Java are selling points enough, even without any fancy code!
- AzzieElbab 6y ago
  I meant people who are comfortable with scala’s advanced features, not necessarily better engineers.

einpoklum 6y ago

The article mostly lost me with two points: * It claims to be a compiled language "like C++ and Java" - not making a distinction between targeting multiple hardware platforms and targeting a cozy VM. * It ties itself strongly to the Java ecosystem. So, I guess I might consider it in the 'nicer Java' category - and it does seem nicer than Java, subjectively - but not outside of that ecosystem.

ebruchez 6y ago
It might be worth mentioning that Scala now compiles to 3 different targets: - JVM - JavaScript (Scala.js) - native (Scala Native) The tie to the Java ecosystem has significantly loosened over the last few years.
- einpoklum 6y ago
  That is interesting... but it's also the opposite message from the article. It even emphasizes again how Scala "leans heavily" on the JVM: > The last thing that Scala does well is to lean heavily on the host language for both its language semantics as well as its implementation. Scala is typically run on the JVM, which together with the Java ecosystem provides a host of useful things that Scala doesn't need to worry about: etc. etc. So, what do you do with Scala when you don't use the JVM at all, nor any Java libraries?
  - lihaoyi 6y ago
    In Scala.js, you lean heavily on the Javascript runtime, and on the large ecosystem of Javascript libraries. That integration into the existing ecosystem is in fact one of the big selling points of Scala.js, just like how it is a big selling point of Scala-JVM Scala has multiple host languages, and it leans heavily on each of then whenever used on that platform
  - ebruchez 6y ago
    I should add to lihaoyi's reply that in addition to the target environment's libraries, many Scala libraries cross-compile for Scala JVM, Scala.js, and (to a lesser extent due to its relative novelty) Scala Native. For example I am now cross-compiling JVM/JS a fairly large project which uses, directly or indirectly, Circe, Cats, ScalaTest, Enumeratum, Shapeless, and Parboiled2, and all of that works flawlessly.

deleted 6y ago

[deleted]

elmalto 6y ago

I have used python and scala for work and I find myself wishing to be able to use scala a lot when I use python (especially when using pandas or trying to multi process/thread). Not so much the other way around even though I also love python and think it’s amazing. Main reason: The type system is great and there are so many things that I can express/enforce using it that are missing for me in python even using typing libraries. Yes, there is a large crew of FP fans around scala that can be obnoxious at times, but they have also brought a lot of good stuff around. The way I see it: You need groups that “overshoot” and then settle on the middle ground. Compromise... I use FP when it’s not too complicated, but stay away from the larger monad systems that are hard to explain to newcomers (effects/IO/Kleisli etc) because I honestly feel like it’s too much. Let me just run my debug logger when I want to... (yes I know that’s not what it’s about) I probably fall in the category of using scala as the (way) better Java with sprinkles of FP and non-mutable data objects everywhere. I really like the authors style (his work is awesome in general and I point to your posts a lot for our juniors, thank you for your contributions if you read this) It has its flaws, just like any other language, but it’s also incredibly expressive and fast (enough) for the backend and data engineering that we do

dionian 6y ago
My favorite thing about the language is its flexibility, if i want to have pure imperative java style code i can do it. I dont think everyone should blame it just because there are so many haskell fans. I think ZIO is a good sign
- ibraheemdev 6y ago
  Funny, ZIO was also on the front page today: https://news.ycombinator.com/item?id=26101463 https://news.ycombinator.com/item?id=26101463
jmfldn 6y ago
Your point about the FP crew being seeming obnoxious sometimes is true. The style itself can be great, I use IO daily and the monadic style is the glue that holds our apps together. The difference with my team, or approaches that work generally, is that we don't go on and on about these abstractions. I barely mention the 'M' word and just talk about chaining things together. Once that is appreciated then IO isn't that hard to get a basic idea of either. We talk a bit about lazy evaluation but I try not to freak everyone out by mentioning purity, higher kinder types and functors every second breath. It's enough to gain a working intuition for these things without bringing theory to the foreground. That can come later if / when ppl are interested. I guess we also choose our style and abstractions carefully and only use things when we can justify it as a team. If someone raised a PR full of novel concepts that we hadn't grokked as a team, it might well get rejected. For example we started using monad transformers a while ago only after the concept and benefits in our code was explained and bought into by the team. This doesn't just apply to Scala but is especially important when using it. Reach consensus with your team about the style and evolve it together. I get the feeling that other language cultures, maybe Rust being one, are good at just using FP, without talking endlessly about it.

non-e-moose 6y ago

Because EMACS is not an efficient programming languge

u678u 6y ago

Old articles of "Why Scala" talked about how great lambdas are, pattern matching, lightweight classes without getters/setters. Now Java does those things the reason to use Scala are few.

valenterry 6y ago
That's because these concepts where new and difficult for many people at the time. Now that they already know them from other languages (which is great!) we can directly start with the next batch of helpful techniques in Scala. Java is still light years behind in terms of productivity through language features. (however, therefore it has higher backwards compatibility, which also helps productivity)

quilombodigital 6y ago

When you go to an auto repair shop, the mechanic uses four different screwdrivers or just one super ultra incredible 8-in-1 screwdriver for everything? Why not? is the mechanic stupid? the 8-in-1 screwdriver is marvelous, has the DOUBLE of features! At the end of the day, he has only the tools he need for his job, and thats it.

d_burfoot 6y ago

I dispute the claim that compiled languages have "heavyweight build setups" or are "inconvenient for small programs". I am constantly writing small Java programs to do minor tasks. I compile and run those programs with Python scripts, not heavyweight IDEs or build tools.

RealityVoid 6y ago
In my mind, I, perhaps wrongly, equate compiled with - "runs machine code". Java is considered compiled but runs on the JVM. Python also has compilation as one of the steps and runs in its virtual machine yet it is considered interpreted. Sooo... Where do you draw the line? Regarding your main point, I agree, compiled languages do not necessarily have complicated build setups. But maybe it is tainted by my experience in this field.

karna1112 6y ago

It is rare to find a language as simple and yet powerful. Much awaited Scala 3 brings more features. It is not better Java. It is an independent language which can interop with Java . It has richer set of specifications compared to any other JVM language. People think most of Scala community is about functional programming . It is not . It is a hybrid language. It is best suited for every facet of data engineering where we can make use of state of the art data abstractions and concurrent programming. There is ongoing work on improving tooling and ecosystem. I would ask of you people not to get scared of language based on others opinions. Don't go for advanced concepts when all you need as basics.