10 ms·

Just curious, what companies (or kinds of companies) are betting on Scala nowadays? It seems that the people wanting "better Java" all decided they like Kotlin,

by nnq 9y ago

Just curious, what companies (or kinds of companies) are betting on Scala nowadays? It seems that the people wanting "better Java" all decided they like Kotlin, and the functional programming people now gravitate more towards either F# (for .NET ecosystem) or OCaml/Reason (for the more unixy world). And the academic/research/learning crowd seems to like Haskell more. Who's still in Scala boat? Are Google or Fb or other big cos known to give back to open-source growing any Scala codebases now?

kprybol 9y ago
Scala is used pretty heavily in the big data world, particularly if you are working with Spark.
- joshlemer 9y ago
  Not to mention, Flink, Akka, Apache Beam (via the Scio library or the Java api).
  - kprybol 9y ago
    Kafka too
    - dominotw 9y ago
      Kafka switched to java 100%, afaik.
- bitL 9y ago
  Python is still king there, even DataBricks market it that way. Scala is for advanced stuff that matters.
  - RBerenguel 9y ago
    Python is king in analysis, but for big data engineering, most of the building blocks (as mentioned elsewhere in this thread, Akka, Kafka, Flink, Spark, etc) are written in Scala
    - sandGorgon 9y ago
      It doesn't matter. Because pyspark is still the go-to language. In fact with spark 2.3 python UDF, the performance gap has also reduced. https://mindfulmachines.io/blog/2018/4/3/spark-rdds-and-dataframes https://mindfulmachines.io/blog/2018/4/3/spark-rdds-and-data...
      - RBerenguel 9y ago
        Being able to pass data through Arrow is a big improvement, but there's also a lot of serialisation going on you pay in Python. Also, if you want to do anything in the fancy areas (like, write your own optimisation rule for the SparkSQL optimiser) it's Scala. Even something simple as writing a custom aggregator is impossible in Python (at least it was in 2.2, haven't checked in 2.3 or "current" 2.4)
      - kprybol 9y ago
        It depends on the use case. Our work primarily revolves around extending Spark with custom pipelines, models, ensembles, etc. to be deployed into our production systems (petabyte scale). Scala was really the only way to go for us.
        sandGorgon 9y ago
        I can understand performance difference, but I have not generally seen a difference in building custom pipelines and ensembles .. although I grant I'm not at your scale yet. What kind of specific pipelines did you have trouble in pyspark ?
        RBerenguel 9y ago
        Although we decided to start using Scala specifically because PySpark was not as performant (2.0 is not so far ago), a reasonable use case I keep always in mind is aggregation (and in general any API which is still not solid/experimental/under work). Python bindings are always the last to be available (because all groundwork is being done in Scala). We have a relatively large scale process that takes advantage of custom-built aggregation methods on top of groupedDatasets, where we can pack a good deal of logic in the merge and reduce steps of aggregation. We could replicate this in Python using reducers, but aggregating makes more sense semantically, which makes the code is easier to understand. Also, the testing facilities for Spark code under Scala are a bit more advanced than under Python (they are not super-great, but are better), even without considering that being strongly typed makes a whole kind of errors impossible, right out of the compiler. I very, very rarely think of using PySpark (and I have way more experience with Python than with Scala) when working with Spark. In a kitchen setting, it would be like having to prepare a cake and having to choose between a fork and a whisker. I can get it done with the fork, but I'll do a better and faster job with the whisker.
  - wenc 9y ago
    Scala is still primarily used for data engineering workloads due to the fact it is a JVM language. (There's Java too, but no one wants to write Java code) PySpark is often used for data science experimentation, but is not as frequently found in production pipelines due to the serialization/deserialization overhead between Python and the JVM. In recent years this problem is less pronounced due to the introduction of Spark dataframes which obviates the performance differences between PySpark and Scala Spark, but for UDFs, Scala Spark is still faster. A newer development that may change all this is the introduction (in Spark 2.3) of Apache Arrow, a in-memory column store engine which lets Python UDFs work with the in-memory object without serializing/deserializing. This is very exciting as this lets Python get closer to the performance of JVM languages. I've played around with it on Spark 2.3 -- the Arrow interface works but still not quite production-ready but I expect it will only get better. Many folks are making strategic bets on Arrow technology due to the AI/GPU craze (and an in-memory standard enables multiple parties to build GPU-based analytics [1]), so there is tremendous momentum there. At some point I expect the relative importance of Scala on Spark will decrease with respect to Python. (even though Spark APIs are Scala native) [1] https://www.nextplatform.com/2017/05/09/goai-keeping-databases-analytics-machine-learning-gpu/ https://www.nextplatform.com/2017/05/09/goai-keeping-databas...
phdp 9y ago
I know quite a few finance companies use it - Morgan Stanley and Goldman Sachs come to mind.
pjmlp 9y ago
F# only has access to a subset of .NET deployment scenarios, where Scala can be used pretty much everywhere there is a JVM. So most companies actually are more keen to bet on Scala than F#. OCaml/Reason world lacks the wealth of Java libraries, which are an import away on Scala. Regarding Kotlin, so far its killer use case is targeting Android, where one is stuck with an ageing Java subset. Outside of Android, it remains to be seen how it will keep up with the Java improvements coming up every 6 months now.
- stephen123 9y ago
  Not only on JVM, scala can be used in the browser with scala.js or without a JVM with scala native.
  - GiorgioG 9y ago
    Yep - you're right... F# definitely can't be used in the browser: http://fable.io/ http://fable.io/
- GiorgioG 9y ago
  > F# only has access to a subset of .NET deployment scenarios And what subset would that be? The only place you can't use F# IIRC is UWP application which is likely not the deciding factor in choosing F# or Scala.
  - pjmlp 9y ago
    Anything GUI related, and their respective tooling support on Visual Studio, which it never got. It is still playing catch up with VS 2017 tooling for VB.NET and C#.
    - GiorgioG 9y ago
      If you're waiting for feature parity with C# in VS, you'll never get it. There are millions of C# developers and F# devs are counted in the tens of thousands (sadly) so there's a reason for that. I don't believe most organizations considering adopting F# or Scala are considering them for GUI development so I'm not sure why you'd rule out F# because of that
boubiyeah 9y ago
Scala has an insanely productive and efficient streaming ecosystem with the likes of Kafka and Akka streams. You can use those with other languages but it's not nearly as nice.
- sidlls 9y ago
  Why isn't it nearly as nice? Is there something particular about Scala such that Kafka or Akka are best implemented in it? Could you give some concrete examples (e.g. compare it with Java, C, C++, Rust, Haskell)?
  - chamakits 9y ago
    I think the parent comment was talking more about using those libraries with Scala, not necessarily implementing them. Can't speak too much about the others, but I recently looked into using Akka with Java. Akka itself had a good deal of documentation for Java users. However, other related products (look tools for monitoring Akka, etc) were clearly treating Java as a second class citizen. When there was documentation, it would be outdated, and unmaintained. I ran into this over and over again, which weighted heavily on my decision to not use Akka. I'm not quite sure if there is anything about the language that would have made it particularly challenging (doubt it), but it definitely seems like the community built around it is more Scala centric. And that in itself makes it challenging to use with anything else.
    - sidlls 9y ago
      I took the comment to mean that Scala is the best choice for having implemented, e.g. Kafka, and that other languages somehow aren't suited for it, and asked the question with that in mind.
  - lmm 9y ago
    (Compared to Java, C or C++ there's all the usual ML-family goodness - first-class functions, algebraic data types with pattern matching, type inference). Scala has a for/yield construct similar to Haskell's "do notation", which is the perfect way to work with async code - it strikes the right balance of avoiding an unreadable callback pyramid of doom, but still your async yield points visible (the difference between <- and = is about as concise as it gets, but still very visible). And having HKT and allowing you to express concepts like monads means that a whole library ecosystem can build up of standard operatons operations (e.g. traverse, cataM) that work on async futures and also on other contextual/effecty types (e.g. audit logging, database transactions). That's the big advantage it has over Rust and most other competitors, and it particularly shows in things like Kafka/Akka that need to work with async a lot, but also in large programming projects generally. (What it shows up as in practice is that you can do everything in "plain old code" - all the things that need reflection/interceptors/macros/metaclasses/agents/... in other languages are just libraries in Scala - and that makes development so much easier and bugs so much rarer) Haskell has all that - the trouble with Haskell is that "there's no way to get to there from here". I was able to go from writing Java on Friday to writing Scala on Monday - not great Scala, but working Scala, and I was just as productive as I was in Java. I couldn't've done that with Haskell.
sgeb 9y ago
Here's one datapoint: AutoScout24 (the vehicle vertical of Scout24, a Europe-based digital marketplace) has fully committed to Scala a couple of years ago as part of a re-platforming (microservices on AWS). It's still the recommended default language. Not a "big" company by global standards, but not small either, with around 120 engineers.
joslin01 9y ago
Twitter created a website "Scala School" awhile ago, so I'm guessing they didn't jump ship yet (but could be wrong). https://twitter.github.io/scala_school/ https://twitter.github.io/scala_school/
- joshlemer 9y ago
  Twitter is still very much heavily committed to Scala.
Bizarro 9y ago
Yeah, I don't think Scala 3 changes anything with regards to perception and/or adoption of Scala. That ship has pretty much sailed. Martin Odersky seems like a really good language designer. I took a look at the Scala 3 languages features, and a more radical departure from historic Scala probably would have helped Scala.
- lmm 9y ago
  Three or four years ago the biggest criticism of Scala was that releases were breaking backwards compatibility too much. On this very thread you'll find people worrying that Scala 3 will create a Python 2/3-style endless awkward transition. At this stage it's a mature language with a big established ecosystem, it can't afford to break too radically. (Did you have specific breaking changes in mind? The most important changes I'd want to make to the language - having a syntax for guaranteed-safe equality comparison and guaranteed-safe pattern matching - could be done in a backwards compatible way, and the only other thing I can think that I'd like would be Idris-style totality checking which could also be backwards compatible.)
  - Bizarro 9y ago
    I wouldn't have called it Scala 3. And if I had the big brain of Martin Odersky, 5 years ago I would've had started thinking about an intermediate step in programming productivity, correctness - instead of doing Doty. I don't follow Scala that much anymore, but it seems that it's mostly a reworking of the internal consistency of Scala from an implementation perspective. At least that was the gist of it I got from watching Odersky give a talk about it a couple years ago.
    - lmm 9y ago
      > 5 years ago I would've had started thinking about an intermediate step in programming productivity, correctness - instead of doing Doty. Wait what? In the last post you were complaining it was too close to existing scala, now you're saying it's too big a step? Again, what is the change you're actually advocating?
afroboy 9y ago
Twitter now is running its Scala code on GraalVM.
- polskibus 9y ago
  Woah, can you provide a reference? I'm curious about the rationale and performance implications.
  - ErunamoJAZZ 9y ago
    https://twitter.com/graalvm/status/986587968667291648 https://twitter.com/graalvm/status/986587968667291648
    - polskibus 9y ago
      Did you find any benchmarks with hard numbers regarding performance improvement?
misja111 9y ago
> Just curious, what companies (or kinds of companies) are betting on Scala nowadays? LinkedIn, Twitter, The Guardian, Morgan Stanley, Barclays, Zalando. Generally speaking, Scala is used a lot by companies involved with Big Data (because they use Spark).
- sulam 9y ago
  I’m not sure if your info is dated or mine is, but all the systems development at LNKD that was being done in Scala is now done in Java. There is some legacy Scala still but as of a couple years ago they decided no new development. Perhaps they are still using Scala in a data science context, since it is hard to avoid there, these days.
  - sidlls 9y ago
    It's trivial to avoid it in data science, with even modest to large (although not "Big") data in the few TB range. My data science team's entire stack is basically Python, with a smidge of Java and Rust for infrastructure (soa, pipelines) development.
    - sulam 9y ago
      I almost added the clarifying statement "on the JVM" but got lazy. Yes, Python is the obvious choice for data science until you hit a certain scale. It's not even clear to me that most companies doing data science in Scala have that scale -- they're just using tools and libraries companies at that scale have open sourced. You could call it cargo culting, but I think it's more nuanced than that. I think engineers can be separated into two camps roughly: those who are passionate about the language(s) they use, and those who are simply trying to get the result they need and don't care what language they use to get it. A lot of data science engineers naturally fall into the second bucket, so using Scala because a library they want to use is written in it comes naturally, even if they could get the job done in Python (possibly with a bit more wheel reinvention).
      - sidlls 9y ago
noelwelsh 9y ago
Interesting that you see it that way. I see Haskell and Purescript usage growing, and I don't know many people interested in O'Caml / Reason. Not trying to say you're wrong---just commenting on the effect of our respective filter bubbles. As for Scala usage. I'm a Scala consultant, so I'm biased, but I'm seeing a lot of adoption. Most finance companies and most media companies are using Scala.
joneholland 9y ago
Expedia is a big time scala shop, but new stuff is trending towards Kotlin.
dudul 9y ago
F# and OCaml? What source do you use to say that? Scala is way more popular in the industry that these 2 combined.
CookWithMe 9y ago
I'm working since 3 years at a growing startup that has been hiring Scala Engineers at least every half year since I started. In the last ~18 months, the amount of CVs we're getting has been constantly increasing. Part of that is probably that our startups hiring matures, but it's also the kind of CV that is changing. I'd say that 3 years ago, there was an 80% chance that the applicant was highly self-motivated to learn Scala in their freetime, and tried/did introduce it at his/her current workplace. Today, there is an 80% chance that the applicant either "had to" learn it in their current workplace, or learned Scala when switching jobs. (Don't get me wrong, they're still motivated, and they took the chance when it was there!) So there is a switch from the Early Adopters to the Early Majority (where the Early Majority now has worked 1 or 2 years with Scala at their current job, and is confident enough to look for a new one). One driving force was definitely Spark, but there are a lot of Enterprise apps, unrelated to ML (usually with higher traffic requirements). The sort that would have most likely been done with Java or C# 5 years ago. It seems a lot of Enterprises introduce Scala when they try to break up their (Java) monolith into microservices. So it seems that Scala has been carving out it's place in backend/microservices with scalability requirements, and is eating part of Javas cake there.
saosebastiao 9y ago
I still think Scala is a better better-jav than Kotlin, but Kotlin captures that market better because of the much lower learning overhead. I'm much more of an ML-style programmer, but the thing keeping me away from OCaml is the lack of a multicore runtime. Not only is Scala's concurrency mature and performant, implicits (like having an ExecutionContext) make it extremely easy to use. Even if OCaml released it's multicore runtime today, it would take years to get to the maturity of the Scala ecosystem. Plus, occasionally I run into areas where OOP really blows away functional programming. It's nice to not have to learn a new language for those occasional times.
- yawaramin 9y ago
  OCaml fully supports OOP (the 'O' in OCaml).
JackFr 9y ago
> It seems that the people wanting "better Java" all decided they like Kotlin I've been using Java and Scala exclusively for the past 10 years at two large NY banks and two fintech start-ups, and I literally do not know anyone who has ever compiled a Kotlin program. Simply extrapolating from one's own experience is fraught with peril.
bad_user 9y ago
N.B. Scala and its ecosystem is actually a lot more oriented towards FP than F# or OCaml for that matter. Its popularity also far surpasses F# and OCaml in jobs available, books, libraries or other metrics that count.
dbattaglia 9y ago
We use Scala as the primary backend language at Elastic for the Elastic Cloud SAAS & enterprise products (along with Java, Python and Go).
dominotw 9y ago
> functional programming people now gravitate more towards either F# (for .NET ecosystem) or OCaml/Reason (for the more unixy world). And the academic/research/learning crowd seems to like Haskell more What if you want functional programming without having to go to MS stack and stay on JVM. You only have have two real options clojure and scala.
SOLAR_FIELDS 9y ago
If you make heavy use of Spark there is still no better language/ecosystem than Scala.
applehire 9y ago
We use Scala heavily. If kafka/Spark/Monoids/Semigroups/Cats/Algebird makes any sense to you and you are looking for a job, send your resume to mansur.ashraf@apple.com
aphexairlines 9y ago
Here are a bunch of job postings at Amazon citing that they use Scala: https://www.linkedin.com/jobs/amazon-scala-jobs/?country=gb https://www.linkedin.com/jobs/amazon-scala-jobs/?country=gb https://www.indeed.co.uk/cmp/Amazon.com/jobs?q=scala&l=London%2C+Greater+London https://www.indeed.co.uk/cmp/Amazon.com/jobs?q=scala&l=Londo... https://www.amazon.jobs/en/search?base_query=scala&loc_query= https://www.amazon.jobs/en/search?base_query=scala&loc_query...

by nnq 9y ago

kprybol 9y ago

Scala is used pretty heavily in the big data world, particularly if you are working with Spark.

joshlemer 9y ago
Not to mention, Flink, Akka, Apache Beam (via the Scio library or the Java api).
- kprybol 9y ago
  Kafka too
  - dominotw 9y ago
    Kafka switched to java 100%, afaik.
bitL 9y ago
Python is still king there, even DataBricks market it that way. Scala is for advanced stuff that matters.
- RBerenguel 9y ago
  Python is king in analysis, but for big data engineering, most of the building blocks (as mentioned elsewhere in this thread, Akka, Kafka, Flink, Spark, etc) are written in Scala
  - sandGorgon 9y ago
    It doesn't matter. Because pyspark is still the go-to language. In fact with spark 2.3 python UDF, the performance gap has also reduced. https://mindfulmachines.io/blog/2018/4/3/spark-rdds-and-dataframes https://mindfulmachines.io/blog/2018/4/3/spark-rdds-and-data...
    - RBerenguel 9y ago
      Being able to pass data through Arrow is a big improvement, but there's also a lot of serialisation going on you pay in Python. Also, if you want to do anything in the fancy areas (like, write your own optimisation rule for the SparkSQL optimiser) it's Scala. Even something simple as writing a custom aggregator is impossible in Python (at least it was in 2.2, haven't checked in 2.3 or "current" 2.4)
    - kprybol 9y ago
      It depends on the use case. Our work primarily revolves around extending Spark with custom pipelines, models, ensembles, etc. to be deployed into our production systems (petabyte scale). Scala was really the only way to go for us.
      - sandGorgon 9y ago
        I can understand performance difference, but I have not generally seen a difference in building custom pipelines and ensembles .. although I grant I'm not at your scale yet. What kind of specific pipelines did you have trouble in pyspark ?
        RBerenguel 9y ago
        Although we decided to start using Scala specifically because PySpark was not as performant (2.0 is not so far ago), a reasonable use case I keep always in mind is aggregation (and in general any API which is still not solid/experimental/under work). Python bindings are always the last to be available (because all groundwork is being done in Scala). We have a relatively large scale process that takes advantage of custom-built aggregation methods on top of groupedDatasets, where we can pack a good deal of logic in the merge and reduce steps of aggregation. We could replicate this in Python using reducers, but aggregating makes more sense semantically, which makes the code is easier to understand. Also, the testing facilities for Spark code under Scala are a bit more advanced than under Python (they are not super-great, but are better), even without considering that being strongly typed makes a whole kind of errors impossible, right out of the compiler. I very, very rarely think of using PySpark (and I have way more experience with Python than with Scala) when working with Spark. In a kitchen setting, it would be like having to prepare a cake and having to choose between a fork and a whisker. I can get it done with the fork, but I'll do a better and faster job with the whisker.
- wenc 9y ago
  Scala is still primarily used for data engineering workloads due to the fact it is a JVM language. (There's Java too, but no one wants to write Java code) PySpark is often used for data science experimentation, but is not as frequently found in production pipelines due to the serialization/deserialization overhead between Python and the JVM. In recent years this problem is less pronounced due to the introduction of Spark dataframes which obviates the performance differences between PySpark and Scala Spark, but for UDFs, Scala Spark is still faster. A newer development that may change all this is the introduction (in Spark 2.3) of Apache Arrow, a in-memory column store engine which lets Python UDFs work with the in-memory object without serializing/deserializing. This is very exciting as this lets Python get closer to the performance of JVM languages. I've played around with it on Spark 2.3 -- the Arrow interface works but still not quite production-ready but I expect it will only get better. Many folks are making strategic bets on Arrow technology due to the AI/GPU craze (and an in-memory standard enables multiple parties to build GPU-based analytics [1]), so there is tremendous momentum there. At some point I expect the relative importance of Scala on Spark will decrease with respect to Python. (even though Spark APIs are Scala native) [1] https://www.nextplatform.com/2017/05/09/goai-keeping-databases-analytics-machine-learning-gpu/ https://www.nextplatform.com/2017/05/09/goai-keeping-databas...

phdp 9y ago

I know quite a few finance companies use it - Morgan Stanley and Goldman Sachs come to mind.

pjmlp 9y ago

F# only has access to a subset of .NET deployment scenarios, where Scala can be used pretty much everywhere there is a JVM. So most companies actually are more keen to bet on Scala than F#. OCaml/Reason world lacks the wealth of Java libraries, which are an import away on Scala. Regarding Kotlin, so far its killer use case is targeting Android, where one is stuck with an ageing Java subset. Outside of Android, it remains to be seen how it will keep up with the Java improvements coming up every 6 months now.

stephen123 9y ago
Not only on JVM, scala can be used in the browser with scala.js or without a JVM with scala native.
- GiorgioG 9y ago
  Yep - you're right... F# definitely can't be used in the browser: http://fable.io/ http://fable.io/
GiorgioG 9y ago
> F# only has access to a subset of .NET deployment scenarios And what subset would that be? The only place you can't use F# IIRC is UWP application which is likely not the deciding factor in choosing F# or Scala.
- pjmlp 9y ago
  Anything GUI related, and their respective tooling support on Visual Studio, which it never got. It is still playing catch up with VS 2017 tooling for VB.NET and C#.
  - GiorgioG 9y ago
    If you're waiting for feature parity with C# in VS, you'll never get it. There are millions of C# developers and F# devs are counted in the tens of thousands (sadly) so there's a reason for that. I don't believe most organizations considering adopting F# or Scala are considering them for GUI development so I'm not sure why you'd rule out F# because of that

boubiyeah 9y ago

Scala has an insanely productive and efficient streaming ecosystem with the likes of Kafka and Akka streams. You can use those with other languages but it's not nearly as nice.

sidlls 9y ago
Why isn't it nearly as nice? Is there something particular about Scala such that Kafka or Akka are best implemented in it? Could you give some concrete examples (e.g. compare it with Java, C, C++, Rust, Haskell)?
- chamakits 9y ago
  I think the parent comment was talking more about using those libraries with Scala, not necessarily implementing them. Can't speak too much about the others, but I recently looked into using Akka with Java. Akka itself had a good deal of documentation for Java users. However, other related products (look tools for monitoring Akka, etc) were clearly treating Java as a second class citizen. When there was documentation, it would be outdated, and unmaintained. I ran into this over and over again, which weighted heavily on my decision to not use Akka. I'm not quite sure if there is anything about the language that would have made it particularly challenging (doubt it), but it definitely seems like the community built around it is more Scala centric. And that in itself makes it challenging to use with anything else.
  - sidlls 9y ago
    I took the comment to mean that Scala is the best choice for having implemented, e.g. Kafka, and that other languages somehow aren't suited for it, and asked the question with that in mind.
- lmm 9y ago
  (Compared to Java, C or C++ there's all the usual ML-family goodness - first-class functions, algebraic data types with pattern matching, type inference). Scala has a for/yield construct similar to Haskell's "do notation", which is the perfect way to work with async code - it strikes the right balance of avoiding an unreadable callback pyramid of doom, but still your async yield points visible (the difference between <- and = is about as concise as it gets, but still very visible). And having HKT and allowing you to express concepts like monads means that a whole library ecosystem can build up of standard operatons operations (e.g. traverse, cataM) that work on async futures and also on other contextual/effecty types (e.g. audit logging, database transactions). That's the big advantage it has over Rust and most other competitors, and it particularly shows in things like Kafka/Akka that need to work with async a lot, but also in large programming projects generally. (What it shows up as in practice is that you can do everything in "plain old code" - all the things that need reflection/interceptors/macros/metaclasses/agents/... in other languages are just libraries in Scala - and that makes development so much easier and bugs so much rarer) Haskell has all that - the trouble with Haskell is that "there's no way to get to there from here". I was able to go from writing Java on Friday to writing Scala on Monday - not great Scala, but working Scala, and I was just as productive as I was in Java. I couldn't've done that with Haskell.

sgeb 9y ago

Here's one datapoint: AutoScout24 (the vehicle vertical of Scout24, a Europe-based digital marketplace) has fully committed to Scala a couple of years ago as part of a re-platforming (microservices on AWS). It's still the recommended default language. Not a "big" company by global standards, but not small either, with around 120 engineers.

joslin01 9y ago

Twitter created a website "Scala School" awhile ago, so I'm guessing they didn't jump ship yet (but could be wrong). https://twitter.github.io/scala_school/ https://twitter.github.io/scala_school/

joshlemer 9y ago
Twitter is still very much heavily committed to Scala.

Bizarro 9y ago

Yeah, I don't think Scala 3 changes anything with regards to perception and/or adoption of Scala. That ship has pretty much sailed. Martin Odersky seems like a really good language designer. I took a look at the Scala 3 languages features, and a more radical departure from historic Scala probably would have helped Scala.

lmm 9y ago
Three or four years ago the biggest criticism of Scala was that releases were breaking backwards compatibility too much. On this very thread you'll find people worrying that Scala 3 will create a Python 2/3-style endless awkward transition. At this stage it's a mature language with a big established ecosystem, it can't afford to break too radically. (Did you have specific breaking changes in mind? The most important changes I'd want to make to the language - having a syntax for guaranteed-safe equality comparison and guaranteed-safe pattern matching - could be done in a backwards compatible way, and the only other thing I can think that I'd like would be Idris-style totality checking which could also be backwards compatible.)
- Bizarro 9y ago
  I wouldn't have called it Scala 3. And if I had the big brain of Martin Odersky, 5 years ago I would've had started thinking about an intermediate step in programming productivity, correctness - instead of doing Doty. I don't follow Scala that much anymore, but it seems that it's mostly a reworking of the internal consistency of Scala from an implementation perspective. At least that was the gist of it I got from watching Odersky give a talk about it a couple years ago.
  - lmm 9y ago
    > 5 years ago I would've had started thinking about an intermediate step in programming productivity, correctness - instead of doing Doty. Wait what? In the last post you were complaining it was too close to existing scala, now you're saying it's too big a step? Again, what is the change you're actually advocating?

afroboy 9y ago

Twitter now is running its Scala code on GraalVM.

polskibus 9y ago
Woah, can you provide a reference? I'm curious about the rationale and performance implications.
- ErunamoJAZZ 9y ago
  https://twitter.com/graalvm/status/986587968667291648 https://twitter.com/graalvm/status/986587968667291648
  - polskibus 9y ago
    Did you find any benchmarks with hard numbers regarding performance improvement?

misja111 9y ago

> Just curious, what companies (or kinds of companies) are betting on Scala nowadays? LinkedIn, Twitter, The Guardian, Morgan Stanley, Barclays, Zalando. Generally speaking, Scala is used a lot by companies involved with Big Data (because they use Spark).

sulam 9y ago
I’m not sure if your info is dated or mine is, but all the systems development at LNKD that was being done in Scala is now done in Java. There is some legacy Scala still but as of a couple years ago they decided no new development. Perhaps they are still using Scala in a data science context, since it is hard to avoid there, these days.
- sidlls 9y ago
  It's trivial to avoid it in data science, with even modest to large (although not "Big") data in the few TB range. My data science team's entire stack is basically Python, with a smidge of Java and Rust for infrastructure (soa, pipelines) development.
  - sulam 9y ago
    I almost added the clarifying statement "on the JVM" but got lazy. Yes, Python is the obvious choice for data science until you hit a certain scale. It's not even clear to me that most companies doing data science in Scala have that scale -- they're just using tools and libraries companies at that scale have open sourced. You could call it cargo culting, but I think it's more nuanced than that. I think engineers can be separated into two camps roughly: those who are passionate about the language(s) they use, and those who are simply trying to get the result they need and don't care what language they use to get it. A lot of data science engineers naturally fall into the second bucket, so using Scala because a library they want to use is written in it comes naturally, even if they could get the job done in Python (possibly with a bit more wheel reinvention).
    - sidlls 9y ago

noelwelsh 9y ago

Interesting that you see it that way. I see Haskell and Purescript usage growing, and I don't know many people interested in O'Caml / Reason. Not trying to say you're wrong---just commenting on the effect of our respective filter bubbles. As for Scala usage. I'm a Scala consultant, so I'm biased, but I'm seeing a lot of adoption. Most finance companies and most media companies are using Scala.

joneholland 9y ago

Expedia is a big time scala shop, but new stuff is trending towards Kotlin.

dudul 9y ago

F# and OCaml? What source do you use to say that? Scala is way more popular in the industry that these 2 combined.

CookWithMe 9y ago

I'm working since 3 years at a growing startup that has been hiring Scala Engineers at least every half year since I started. In the last ~18 months, the amount of CVs we're getting has been constantly increasing. Part of that is probably that our startups hiring matures, but it's also the kind of CV that is changing. I'd say that 3 years ago, there was an 80% chance that the applicant was highly self-motivated to learn Scala in their freetime, and tried/did introduce it at his/her current workplace. Today, there is an 80% chance that the applicant either "had to" learn it in their current workplace, or learned Scala when switching jobs. (Don't get me wrong, they're still motivated, and they took the chance when it was there!) So there is a switch from the Early Adopters to the Early Majority (where the Early Majority now has worked 1 or 2 years with Scala at their current job, and is confident enough to look for a new one). One driving force was definitely Spark, but there are a lot of Enterprise apps, unrelated to ML (usually with higher traffic requirements). The sort that would have most likely been done with Java or C# 5 years ago. It seems a lot of Enterprises introduce Scala when they try to break up their (Java) monolith into microservices. So it seems that Scala has been carving out it's place in backend/microservices with scalability requirements, and is eating part of Javas cake there.

saosebastiao 9y ago

I still think Scala is a better better-jav than Kotlin, but Kotlin captures that market better because of the much lower learning overhead. I'm much more of an ML-style programmer, but the thing keeping me away from OCaml is the lack of a multicore runtime. Not only is Scala's concurrency mature and performant, implicits (like having an ExecutionContext) make it extremely easy to use. Even if OCaml released it's multicore runtime today, it would take years to get to the maturity of the Scala ecosystem. Plus, occasionally I run into areas where OOP really blows away functional programming. It's nice to not have to learn a new language for those occasional times.

yawaramin 9y ago
OCaml fully supports OOP (the 'O' in OCaml).

JackFr 9y ago

> It seems that the people wanting "better Java" all decided they like Kotlin I've been using Java and Scala exclusively for the past 10 years at two large NY banks and two fintech start-ups, and I literally do not know anyone who has ever compiled a Kotlin program. Simply extrapolating from one's own experience is fraught with peril.

bad_user 9y ago

N.B. Scala and its ecosystem is actually a lot more oriented towards FP than F# or OCaml for that matter. Its popularity also far surpasses F# and OCaml in jobs available, books, libraries or other metrics that count.

dbattaglia 9y ago

We use Scala as the primary backend language at Elastic for the Elastic Cloud SAAS & enterprise products (along with Java, Python and Go).

dominotw 9y ago

> functional programming people now gravitate more towards either F# (for .NET ecosystem) or OCaml/Reason (for the more unixy world). And the academic/research/learning crowd seems to like Haskell more What if you want functional programming without having to go to MS stack and stay on JVM. You only have have two real options clojure and scala.

SOLAR_FIELDS 9y ago

If you make heavy use of Spark there is still no better language/ecosystem than Scala.

applehire 9y ago

We use Scala heavily. If kafka/Spark/Monoids/Semigroups/Cats/Algebird makes any sense to you and you are looking for a job, send your resume to mansur.ashraf@apple.com

aphexairlines 9y ago

Here are a bunch of job postings at Amazon citing that they use Scala: https://www.linkedin.com/jobs/amazon-scala-jobs/?country=gb https://www.linkedin.com/jobs/amazon-scala-jobs/?country=gb https://www.indeed.co.uk/cmp/Amazon.com/jobs?q=scala&l=London%2C+Greater+London https://www.indeed.co.uk/cmp/Amazon.com/jobs?q=scala&l=Londo... https://www.amazon.jobs/en/search?base_query=scala&loc_query= https://www.amazon.jobs/en/search?base_query=scala&loc_query...