9 ms·

Strategic Scala Style: Principle of Least Power

118 points by rspivak 11y ago

kailuowang 11y ago
The fun part of developing in scala is that it supports multiple completely different styles and paradigms, each has its own strength and weakness. You can pick the most suitable one for the functionality you are working on. On other hand it also make learning more complex. To be most effective, you need to learn these styles and paradigms and the subset of the language features you limit yourself to when in that style. Otherwise, scala is definitely one of the languages that is most capable of producing messy programs.
- leonroy 11y ago
  > The fun part of developing in scala is that it supports multiple completely different styles and paradigms. Putting on my management hat that thought fills me with horror. For example pull requests become a nightmare when you have extremely experienced Scala devs using every trick in the book and producing code which no one else on the team understands except those with similar levels of Scala knowledge. Honestly, it's greek for the rest of us and furthermore it seems each Scala expert has their own way of implementing the same functionality. I enjoy programming in Scala, I write code in it (when I have to) but in all honesty as cool as it is we find ourselves needing to stick with Python, Java and JavaScript to ensure we have some sort of consistency between new hires and experienced devs.
  - justinhj 11y ago
    This is a very good point. I work on a team with 7 Scala developers of varied experience. When someone commits code it is reviewed. If the reviewer doesn't understand the code we will talk it over as a group and either learn the new technique or decide that the complexity is not worth the benefit and maybe even rewrite the commit. We encourage people to talk about new techniques before implementing so that rewrites are rare.
  - lihaoyi 11y ago
    I agree with this 100%. Having too many ways of doing things, and no way to pick between them, results in everyone doing things every which way, all different. That's why this document exists at all. We can definitely make things better, but to do so requires putting together guidelines, proposals, and consensus (things developers are historically pretty poor at), rather than just cranking out code and complaining (both of which developers excel at, especially fueled by coffee)
    - incepted 11y ago
      But we went through that with C++ already: all companies used to publish their own guidelines of what C++ features were allowed and not allowed to use. Of course, all these subsets were different. And of course, if you work on open source project, it's just impossible to be productive in such a language without knowing all of it.
lihaoyi 11y ago
Neat to see this appear here. I wrote this. Ask me anything
- kailuowang 11y ago
  Not relevant to your post, but I kept getting asked by data engineers who are interested picking up scala (thanks to spark): how shall I learn scala? Do you happen to have any advice for them?
  - lihaoyi 11y ago
    Find something you want to do, try to do it, find out what you don't know, and learn it. You could try learning from a book or a course, but personally I've never found a better way of learning "what I need to learn for X" than trying to do X and then googling all the things I don't know. Sure, you'll forget a whole bunch of that stuff, but if it's at all important you'll find yourself re-learning it again and again until it sticks.
  - audi100quattro 11y ago
    Scala by example and the coursera courses by Martin Odersky have helped me, but like Li said, there is no substitute to having something in mind and getting your hands dirty using Akka, Play or Spark and Googling what you don't know. http://www.scala-lang.org/docu/files/ScalaByExample.pdf http://www.scala-lang.org/docu/files/ScalaByExample.pdf
- Negative1 11y ago
  No specific questions about your blog post but want to echo the sentiment that Scala is incredibly easy for refactoring. With Python (or even C++) it can be quite the chore, but as you said, building simple and using the compiler to guide you has great benefits. I'm currently going through the process right now with an AWS database backup utility I've been writing. Since you're here, on another subject, I've been toying with using Scala.js with Duktape to integrate Scala into a C++ project as a scripting language. Have you seen or heard anyone who has successfully done this or any pointers you may have? p.s. Thank you for all you've contributed to the Scala community. Your posts are always enlightening and your projects incredibly forward thinking!
chickenbane 11y ago
An excellent read, Haoyi is a great Scala engineer. Additionally, if you love using the Scala REPL, he has a entertaining and compelling video here: https://youtu.be/dP5tkmWAhjg https://youtu.be/dP5tkmWAhjg
spdionis 11y ago
> Given a choice of languages, choose the least powerful language capable of solving your problem Then no one would ever use scala?
- lmm 11y ago
  Scala allows you to restrict yourself to a much greater extent than many other languages. You could say use assembly for everything - there's nothing that assembly can't do, so in a sense assembly is more powerful than any other language. For me the reasons I use Scala rather than Java/Python/etc. are the same reasons I would use Java/Python/etc. rather than assembly. To write code with the same guarantees as the core of my system would be theoretically possible in a "simpler" language, but it would be incredibly verbose: it would take longer to write and be much harder to maintain (because you'd have to keep a lot more in your head to understand each piece of functionality). If I weakened the guarantees (i.e. didn't keep track of as many possible side effects) I'd have a higher defect rate, especially when refactoring. That's the small heart of a few projects, but I see a lot of merit in using the same language for your whole codebase: http://www.teamten.com/lawrence/writings/java-for-everything.html http://www.teamten.com/lawrence/writings/java-for-everything... . And Scala is actually a pretty nice language for one-off throwaway scripts where you don't care about tracking any effects or similar as well.
  - TeMPOraL 11y ago
    > To write code with the same guarantees as the core of my system would be theoretically possible in a "simpler" language, but it would be incredibly verbose: it would take longer to write and be much harder to maintain (because you'd have to keep a lot more in your head to understand each piece of functionality). I could use the same reasoning to justify writing my code in Common Lisp instead. The thing about Turing-complete languages is that the only real dimmension of power is their expressiveness - i.e. how much boilerplate do you have to write to implement something (and therefore how much of that cruft other people have to deal with when trying to understand your code). I therefore find this principle as applied to programming languages to be illogical. It makes sense in non-Turing-complete languages though (configurations, DSLs) - because there by "least power" one means power in the computational sense. Accidentally making a language too powerful when it's not needed can lead to security issues.
davnn 11y ago
Guys please have a quick look at Concepts, Techniques and Models of Computer Programming, it's incredibly valuable. Basically the main point of the post is the principle of least expressiveness. It really makes sense after you know more about multi-paradigm programming. Disclaimer: just a fan of the book :-)
tome 11y ago
This is full of great advice. After a brief skim I think I agree with everything except the advice on mutability. Haoyi says "if you are actually modeling something which changes over time, using mutable [variables] is fine". I think this is absolutely wrong. You should even use immutable variables to model things which change over time (after all, almost everything does). Only use mutable variables when performance means you can't get away without them. Other than that the guide is pretty much how I would program (as a Haskell developer, rather than a Scala one).
- lmm 11y ago
  Note that although many prominent Scala people program in a very Haskelly style, mutable variables are significantly more idiomatic/first-class in Scala than they are in Haskell.
  - mafribe 11y ago
    I agree. The fact that Scala is proper multi-paradigm, and has a first-class functional sublanguage as well as a first-class imperative sublanguage is part of its great appeal. I have mentored quite a few people over the last couple of years in Scala. In my experience, going from imperative Java, C, Python straight to Scalaz/Typelevel/higher-kinded monad style pure functional programming never works, and drives Scala-beginners away. What works extremely well is to start with Scala as a nice Java without semicolons, and then, as they become more proficient in the OO/imperative Scala fragment, gradually to introduce more advanced features, starting with case classes. The reputation Scala has for being too difficult comes from certain parts of the Scala elite trying to convince everybody, including beginners, that higher-kinded purely functional Scala is the only way to use the language. Nothing wrong with this style of programming per se and it's great that people are pushing the envelope, but the conceptual gap is too big for a language learner hailing from the imperative world (which is most of them).
  - tome 11y ago
    Oh I'm sure that's true, but as a recovering Python programmer I'm uncomfortable with justifications of mutability under any except the most extreme circumstances. ("In our codebase we avoid writing side effecting code", my colleagues said, whilst filling the codebase with side effecting code.)
jonathanpoulter 11y ago
Some useful advice, even to someone not writing in Scala, especially sections like published interfaces. > When defining an interface to a package someone else will use... > 1. The simplest interface to a package is static method with standard types > 2. Next, static methods that take/return custom types, presumably with methods > 3. Next, is needing the user to instantiate some classes > 4. Lastly, is needing the user to inherit from some classes
atemerev 11y ago
tl;dr Scala is powerful. With great power comes great responsibility. Constrain yourself. :)
incepted 11y ago
It's refreshing to see some down-to-earth pragmatism in Scala land for a change. Thanks, Li-Haoyi.
optimuspaul 11y ago
How do you define power? What makes a solution or language more powerful than another?

118 points by rspivak 11y ago

The fun part of developing in scala is that it supports multiple completely different styles and paradigms, each has its own strength and weakness. You can pick the most suitable one for the functionality you are working on. On other hand it also make learning more complex. To be most effective, you need to learn these styles and paradigms and the subset of the language features you limit yourself to when in that style. Otherwise, scala is definitely one of the languages that is most capable of producing messy programs.

leonroy 11y ago
> The fun part of developing in scala is that it supports multiple completely different styles and paradigms. Putting on my management hat that thought fills me with horror. For example pull requests become a nightmare when you have extremely experienced Scala devs using every trick in the book and producing code which no one else on the team understands except those with similar levels of Scala knowledge. Honestly, it's greek for the rest of us and furthermore it seems each Scala expert has their own way of implementing the same functionality. I enjoy programming in Scala, I write code in it (when I have to) but in all honesty as cool as it is we find ourselves needing to stick with Python, Java and JavaScript to ensure we have some sort of consistency between new hires and experienced devs.
- justinhj 11y ago
  This is a very good point. I work on a team with 7 Scala developers of varied experience. When someone commits code it is reviewed. If the reviewer doesn't understand the code we will talk it over as a group and either learn the new technique or decide that the complexity is not worth the benefit and maybe even rewrite the commit. We encourage people to talk about new techniques before implementing so that rewrites are rare.
- lihaoyi 11y ago
  I agree with this 100%. Having too many ways of doing things, and no way to pick between them, results in everyone doing things every which way, all different. That's why this document exists at all. We can definitely make things better, but to do so requires putting together guidelines, proposals, and consensus (things developers are historically pretty poor at), rather than just cranking out code and complaining (both of which developers excel at, especially fueled by coffee)
  - incepted 11y ago
    But we went through that with C++ already: all companies used to publish their own guidelines of what C++ features were allowed and not allowed to use. Of course, all these subsets were different. And of course, if you work on open source project, it's just impossible to be productive in such a language without knowing all of it.

lihaoyi 11y ago

Neat to see this appear here. I wrote this. Ask me anything

kailuowang 11y ago
Not relevant to your post, but I kept getting asked by data engineers who are interested picking up scala (thanks to spark): how shall I learn scala? Do you happen to have any advice for them?
- lihaoyi 11y ago
  Find something you want to do, try to do it, find out what you don't know, and learn it. You could try learning from a book or a course, but personally I've never found a better way of learning "what I need to learn for X" than trying to do X and then googling all the things I don't know. Sure, you'll forget a whole bunch of that stuff, but if it's at all important you'll find yourself re-learning it again and again until it sticks.
- audi100quattro 11y ago
  Scala by example and the coursera courses by Martin Odersky have helped me, but like Li said, there is no substitute to having something in mind and getting your hands dirty using Akka, Play or Spark and Googling what you don't know. http://www.scala-lang.org/docu/files/ScalaByExample.pdf http://www.scala-lang.org/docu/files/ScalaByExample.pdf
Negative1 11y ago
No specific questions about your blog post but want to echo the sentiment that Scala is incredibly easy for refactoring. With Python (or even C++) it can be quite the chore, but as you said, building simple and using the compiler to guide you has great benefits. I'm currently going through the process right now with an AWS database backup utility I've been writing. Since you're here, on another subject, I've been toying with using Scala.js with Duktape to integrate Scala into a C++ project as a scripting language. Have you seen or heard anyone who has successfully done this or any pointers you may have? p.s. Thank you for all you've contributed to the Scala community. Your posts are always enlightening and your projects incredibly forward thinking!

chickenbane 11y ago

An excellent read, Haoyi is a great Scala engineer. Additionally, if you love using the Scala REPL, he has a entertaining and compelling video here: https://youtu.be/dP5tkmWAhjg https://youtu.be/dP5tkmWAhjg

spdionis 11y ago

> Given a choice of languages, choose the least powerful language capable of solving your problem Then no one would ever use scala?

lmm 11y ago
Scala allows you to restrict yourself to a much greater extent than many other languages. You could say use assembly for everything - there's nothing that assembly can't do, so in a sense assembly is more powerful than any other language. For me the reasons I use Scala rather than Java/Python/etc. are the same reasons I would use Java/Python/etc. rather than assembly. To write code with the same guarantees as the core of my system would be theoretically possible in a "simpler" language, but it would be incredibly verbose: it would take longer to write and be much harder to maintain (because you'd have to keep a lot more in your head to understand each piece of functionality). If I weakened the guarantees (i.e. didn't keep track of as many possible side effects) I'd have a higher defect rate, especially when refactoring. That's the small heart of a few projects, but I see a lot of merit in using the same language for your whole codebase: http://www.teamten.com/lawrence/writings/java-for-everything.html http://www.teamten.com/lawrence/writings/java-for-everything... . And Scala is actually a pretty nice language for one-off throwaway scripts where you don't care about tracking any effects or similar as well.
- TeMPOraL 11y ago
  > To write code with the same guarantees as the core of my system would be theoretically possible in a "simpler" language, but it would be incredibly verbose: it would take longer to write and be much harder to maintain (because you'd have to keep a lot more in your head to understand each piece of functionality). I could use the same reasoning to justify writing my code in Common Lisp instead. The thing about Turing-complete languages is that the only real dimmension of power is their expressiveness - i.e. how much boilerplate do you have to write to implement something (and therefore how much of that cruft other people have to deal with when trying to understand your code). I therefore find this principle as applied to programming languages to be illogical. It makes sense in non-Turing-complete languages though (configurations, DSLs) - because there by "least power" one means power in the computational sense. Accidentally making a language too powerful when it's not needed can lead to security issues.

davnn 11y ago

Guys please have a quick look at Concepts, Techniques and Models of Computer Programming, it's incredibly valuable. Basically the main point of the post is the principle of least expressiveness. It really makes sense after you know more about multi-paradigm programming. Disclaimer: just a fan of the book :-)

tome 11y ago

This is full of great advice. After a brief skim I think I agree with everything except the advice on mutability. Haoyi says "if you are actually modeling something which changes over time, using mutable [variables] is fine". I think this is absolutely wrong. You should even use immutable variables to model things which change over time (after all, almost everything does). Only use mutable variables when performance means you can't get away without them. Other than that the guide is pretty much how I would program (as a Haskell developer, rather than a Scala one).

lmm 11y ago
Note that although many prominent Scala people program in a very Haskelly style, mutable variables are significantly more idiomatic/first-class in Scala than they are in Haskell.
- mafribe 11y ago
  I agree. The fact that Scala is proper multi-paradigm, and has a first-class functional sublanguage as well as a first-class imperative sublanguage is part of its great appeal. I have mentored quite a few people over the last couple of years in Scala. In my experience, going from imperative Java, C, Python straight to Scalaz/Typelevel/higher-kinded monad style pure functional programming never works, and drives Scala-beginners away. What works extremely well is to start with Scala as a nice Java without semicolons, and then, as they become more proficient in the OO/imperative Scala fragment, gradually to introduce more advanced features, starting with case classes. The reputation Scala has for being too difficult comes from certain parts of the Scala elite trying to convince everybody, including beginners, that higher-kinded purely functional Scala is the only way to use the language. Nothing wrong with this style of programming per se and it's great that people are pushing the envelope, but the conceptual gap is too big for a language learner hailing from the imperative world (which is most of them).
- tome 11y ago
  Oh I'm sure that's true, but as a recovering Python programmer I'm uncomfortable with justifications of mutability under any except the most extreme circumstances. ("In our codebase we avoid writing side effecting code", my colleagues said, whilst filling the codebase with side effecting code.)

jonathanpoulter 11y ago

Some useful advice, even to someone not writing in Scala, especially sections like published interfaces. > When defining an interface to a package someone else will use... > 1. The simplest interface to a package is static method with standard types > 2. Next, static methods that take/return custom types, presumably with methods > 3. Next, is needing the user to instantiate some classes > 4. Lastly, is needing the user to inherit from some classes

atemerev 11y ago

tl;dr Scala is powerful. With great power comes great responsibility. Constrain yourself. :)

incepted 11y ago

It's refreshing to see some down-to-earth pragmatism in Scala land for a change. Thanks, Li-Haoyi.

optimuspaul 11y ago

How do you define power? What makes a solution or language more powerful than another?