6 ms·

Language War - Scala versus Python

29 points by zubinmehta 14y ago

voidlogic 14y ago
No discussion of run-time characteristics? I realize for a some start-ups this is not the most important metric, but these guys sound like they might be compute bound. And faster language can mean cheaper/less hardware. Scala vs Python: http://benchmarksgame.alioth.debian.org/u64q/benchmark.php?test=all&lang=scala&lang2=python3 http://benchmarksgame.alioth.debian.org/u64q/benchmark.php?t... Also since he mentioned Haskell at first, Haskell vs Scala: is also interesting: http://benchmarksgame.alioth.debian.org/u64q/benchmark.php?test=all&lang=ghc&lang2=scala http://benchmarksgame.alioth.debian.org/u64q/benchmark.php?t... Haskell vs Python is just for lolz: http://benchmarksgame.alioth.debian.org/u64q/benchmark.php?test=all&lang=ghc&lang2=python3 http://benchmarksgame.alioth.debian.org/u64q/benchmark.php?t...
- cover_drive 14y ago
  Thanks! Great articles. We are not compute-bound for the near-term, so it's not our most important consideration. As the business evolves, we will embark on a fresh language war where compute performance is likely to be a key factor.
- lake99 14y ago
  Note that in each case, Python has the smallest code. Generally, but not always, this means that people can address tasks quicker in Python.
  - tomp 14y ago
    For doing anything serious in Python, you will need unit tests. In my experience, they double the amount of code. Haskell and Scala have static typing instead (and therefore need much less unit tests).
- fijal 14y ago
  Seriously, computer language shootout is just harmful. Those benchmarks are bad (but it's unclear whether cross-language comparisons can get much better), implementations are worse. And on top of that it does not include opitmized VMs like PyPy or LuaJIT (in fact, it does not include PyPy because we complained at some point).
  - igouy 14y ago
    >> because we complained at some point << That's not true, and you already know that's not true. http://news.ycombinator.com/item?id=4599431 http://news.ycombinator.com/item?id=4599431 Please stop making this pathetic accusation.
    - fijal 14y ago
      Oh, yes sorry, it was completely coincidental. It was correlated in time though.
      - igouy 14y ago
        >> It was correlated in time though. << Post hoc ergo propter hoc is a well known fallacy. Lest anyone forget that your repeated complaints don't withstand scrutiny -- http://news.ycombinator.com/item?id=4598737 http://news.ycombinator.com/item?id=4598737
  - voidlogic 14y ago
    All benchmarks come with the implicit disclaimer: The best benchmark is always your application. All benchmarks are flawed, use your judgement and determine how flawed a benchmark is; Any flaws are relative to your application similarity to what the benchmark tests. An imperfect tool is not a useless tool, so long as you are smart about how you use it. This is probably relevant too: http://benchmarksgame.alioth.debian.org/dont-jump-to-conclusions.php http://benchmarksgame.alioth.debian.org/dont-jump-to-conclus...
    - igouy 14y ago
leothekim 14y ago
"And who the hell does Scala? (Actually, Twitter does!)" Foursquare also uses Scala, as does LinkedIn. Here's a page of organizations using it: http://www.scala-lang.org/node/1658 http://www.scala-lang.org/node/1658
- cover_drive 14y ago
  Good to see Scala's popularity. I mentioned only Twitter because I admire them :) -Ashwin
- melling 14y ago
  This was done a few years ago. I'd love to hear what people have to say about Scala now that code bases have grown. Anyone working with a couple hundred thousand lines of Scala? For instance, I know compilation performance was always frustrating for developers. What's it like for teams of people dealing with a large code base?
  - leothekim 14y ago
    We are working with Scala at that scale at Foursquare, and compilation times are definitely a headache. What helps is ensuring your dependencies are acyclic so you're able to do smaller incremental compiles when you make changes. To get us on the path of a DAG-ified codebase, we've been using a build tool developed by Twitter called "pants": https://github.com/twitter/commons https://github.com/twitter/commons which has some similarities to Google's Blaze build tool.
wheaties 14y ago
Novus Partners also does Scala. We've even got a few Python guys coding in Scala. It's just a language with the JVM ecosystem to back it up. Python has it's own ecosystem and for numerics, access to LAPACK and UBLAS is pretty awesome, I must admit. That said, both can exist fairly comfortably in a company and can be used for different purposes. I like Python for prototyping, algorithm validation, and just to hack on. I like Scala for damned near everything else. (Also, Adam is going to be open sourcing what is essentially a Scala clone of Pandas but statically typed and with comparable performance.)
drucken 14y ago
The article's basis of the comparison for building an entire business seems to be very shallow. Surely other considerations are more important, such as, - encapsulation/domain partitioning - messaging - libraries and library maturity - native code interoperability (likely critical for this application) - concurrency - performance - JVM platform - etc. Maybe, as a result of this type of analysis, Python-like language is more suitable in some domains and a JVM language in others. In my opinion, for the requirements quoted, with a deep and performant mathematical framework involved, I cannot envisage how Scala/JVM could win any "war" for the core of the business.
- cover_drive 14y ago
  A few more considerations will be posted in Part 2 of the post. I appreciate your pointers - as the business grows and takes shape, a fresh evaluation will be required where some of the considerations you mention will be taken into account.
- userulluipeste 14y ago
  This kind of comparison reminds me about something similar about Git vs Mercurial: http://importantshock.wordpress.com/2008/08/07/git-vs-mercurial/ http://importantshock.wordpress.com/2008/08/07/git-vs-mercur... It looks more like an emotional kind of story than one about practical decisions.
  - cover_drive 14y ago
    Indeed! At the early stages of a startup, most decisions (including technical ones) have a strong emotional/instinctive flavor. When I worked at large corporations, the decisions were almost always devoid of emotions.
beggi 14y ago
I haven't tried Scala but since no one has mentioned it: Clojure is a pleasure to work with if you want a functional language in JVM. I have no idea which and how many startups use Clojure, besides Datomic :) It would be interesting to know.
- dwelch2344 14y ago
  I'd like to agree here. IMO up-and-coming Clojure is an up-and-coming competitor to Scala. As a Lisp variant for the JVM, I'm a major fan of it. Unfortunately I can't share any personal experience as the core of my work is in the Enterprise so I haven't had the right opportunity to present itself. But if you're coming from a mathematical background you should check it out
nnq 14y ago
> Typecheck in Python People actually use it? It seems abandoned and the link to homepage broken: https://pypi.python.org/pypi/typecheck https://pypi.python.org/pypi/typecheck. I'm asking because I'd really like the idea of starting a project the dynamic typing way and then bolting on a (semi)-static type system on top once more new programmers join the game, preferably being able to completely disable it on production machines for best performance.
- shared4you 14y ago
  Have you checked out Cython[1]? It's used quite often in Numpy. Basically, you add some type annotations like, int, double, etc. for some speed-up. There is a quick tutorial as well [2]. [1]: http://cython.org/ http://cython.org/ [2]: http://wiki.cython.org/tutorials/numpy http://wiki.cython.org/tutorials/numpy
  - nnq 14y ago
    I know, it great for the speed-up. But for just making things more manageable it's not worth the added complication of build/compile. I know, my problem is actually solved by "properly written tests" :) ...for I still long for some optional drop-in static typing.
z3phyr 14y ago
Haskell all the way!
- cover_drive 14y ago
  :)
wyqueshocec 14y ago
> Besides, dynamic typing scares the hell out of me Funny. I feel exactly the opposite. Static typing is false security. I program in JS. My coworkers in Java. They have so many bugs that make it to prod because they assume that just because it compiles it must therefore be safe.
- bsg75 14y ago
  Are there no tests beyond simple compilation?
- cover_drive 14y ago
  I must admit that JS scares me more than anything else. Would love to hear about defenses you employ in JS.
- eropple 14y ago
  I prefer static typing because those assurances reduce the number of things I have to manage--they mean I must keep less state in my head. I do not assume that it guarantees things that it does not. Attributing your coworkers' "if it builds, it works" assumptions to static typing strikes me as incorrect. Static typing, at least in Java, ensures only a (relatively) small set of conditions are true; I don't think you can really blame that for increased/additional assumptions on the part of sub-par programmers.
  - stcredzero 14y ago
    Either way, you end up keeping lots of things in your head. With dynamic langs, you do have to think about types and use them conservatively. With languages like Java -- the same. No toolset is going to work well unless the dev is on top of things and good at using the tool well. I've found that debugging in a dynamic environment is great for managing types in a well architected system. It falls down in a poorly architected one, in ways that static typing would not ave allowed. However, in such systems, "it compiles so it must be correct," programmers just get stuck in deeper snow.
    - 14y ago
tikhonj 14y ago
I think you shouldn't have let people talk you out of using Haskell. It's a very nice language and has some advantages for this sort of project (at least based on your cursory description). However, its advantages are really not the important part. Rather, I just wish you wouldn't dismiss it immediately as a crazy choice. It's no more crazy than any other less-popular language. It has a reputation for being impractical, but this reputation is rather unfair especially in the light of recent very practical developments like simpler concurrency, an improved IO manager, very good web frameworks and a fair about of strong libraries both for very specific domains and general productivity. The main disadvantage is that many people find it hard to learn. However, this is a function (heh) of functional programming rather than the language itself. It's actually a simpler language than Scala in many ways because it tries to do one thing well--for example, it has no sub-typing, so you do not have to ever worry about covariance and contravariance. Now, my point here is not that you should always use Haskell, just that you should seriously consider it. Too many people dismiss it out of hand almost as a joke when it is anything but.
- cover_drive 14y ago
  Thanks for your support. Like I mentioned, I will circle back to Haskell as the business develops. If I was the only programmer in my startup, I would likely go with Haskell for important components of our software. But I have concerns about hiring a large team that would thrive in Haskell. One never knows - I'm keeping an open mind :)
  - tikhonj 14y ago
    Hmm, I'm not convinced that hiring a team of good developers for Scala is all that much harder than hiring good Haskell programmers. Haskell has a disproportionate number of very capable people interested in it, and from what I've heard acts as both a way to attract top developers and as something of a filter (at least in encouraging applicants to be more self-selected). I think these effects cancel out Haskell's relative lack of popularity unless you expect to be hiring at the enterprise Java level--something like literally thousands of developers. Anyhow, keeping an open mind is the most important thing, and you clearly have no problems there. I just wish others would follow suit in that regard.
Flow 14y ago
Did you guys consider Google Go? I really like the language and its environment after coding in it for a bit.
- shared4you 14y ago
  I think the reason was the math libraries. There aren't mature libraries yet for numerical computing. Python has Numpy/Scipy, but am not sure of Scala (I don't use).
tapan_pandita 14y ago
Python has real good, battle tested and actively developed libraries for math, scientific computing and analytics. Most of these are written in C and hence give great performance in addition to the flexibility of python. Some great ones are: Numpy/Scipy, LAPACK, Pandas (lots more for machine learning as well). Also, I think the REPL is no match for IPython :). For your specific domain (math and analytics) python could be a great fit. Of course the lack of type systems and no compile step can be scary, but for a startup, which needs to move fast, python gives you the power of quick iteration. Just to round this off, from the zen of python: "practicality beats purity."
- zubinmehta 14y ago
  +django

29 points by zubinmehta 14y ago

No discussion of run-time characteristics? I realize for a some start-ups this is not the most important metric, but these guys sound like they might be compute bound. And faster language can mean cheaper/less hardware. Scala vs Python: http://benchmarksgame.alioth.debian.org/u64q/benchmark.php?test=all&lang=scala&lang2=python3 http://benchmarksgame.alioth.debian.org/u64q/benchmark.php?t... Also since he mentioned Haskell at first, Haskell vs Scala: is also interesting: http://benchmarksgame.alioth.debian.org/u64q/benchmark.php?test=all&lang=ghc&lang2=scala http://benchmarksgame.alioth.debian.org/u64q/benchmark.php?t... Haskell vs Python is just for lolz: http://benchmarksgame.alioth.debian.org/u64q/benchmark.php?test=all&lang=ghc&lang2=python3 http://benchmarksgame.alioth.debian.org/u64q/benchmark.php?t...

cover_drive 14y ago
Thanks! Great articles. We are not compute-bound for the near-term, so it's not our most important consideration. As the business evolves, we will embark on a fresh language war where compute performance is likely to be a key factor.
lake99 14y ago
Note that in each case, Python has the smallest code. Generally, but not always, this means that people can address tasks quicker in Python.
- tomp 14y ago
  For doing anything serious in Python, you will need unit tests. In my experience, they double the amount of code. Haskell and Scala have static typing instead (and therefore need much less unit tests).
fijal 14y ago
Seriously, computer language shootout is just harmful. Those benchmarks are bad (but it's unclear whether cross-language comparisons can get much better), implementations are worse. And on top of that it does not include opitmized VMs like PyPy or LuaJIT (in fact, it does not include PyPy because we complained at some point).
- igouy 14y ago
  >> because we complained at some point << That's not true, and you already know that's not true. http://news.ycombinator.com/item?id=4599431 http://news.ycombinator.com/item?id=4599431 Please stop making this pathetic accusation.
  - fijal 14y ago
    Oh, yes sorry, it was completely coincidental. It was correlated in time though.
    - igouy 14y ago
      >> It was correlated in time though. << Post hoc ergo propter hoc is a well known fallacy. Lest anyone forget that your repeated complaints don't withstand scrutiny -- http://news.ycombinator.com/item?id=4598737 http://news.ycombinator.com/item?id=4598737
- voidlogic 14y ago
  All benchmarks come with the implicit disclaimer: The best benchmark is always your application. All benchmarks are flawed, use your judgement and determine how flawed a benchmark is; Any flaws are relative to your application similarity to what the benchmark tests. An imperfect tool is not a useless tool, so long as you are smart about how you use it. This is probably relevant too: http://benchmarksgame.alioth.debian.org/dont-jump-to-conclusions.php http://benchmarksgame.alioth.debian.org/dont-jump-to-conclus...
  - igouy 14y ago

leothekim 14y ago

"And who the hell does Scala? (Actually, Twitter does!)" Foursquare also uses Scala, as does LinkedIn. Here's a page of organizations using it: http://www.scala-lang.org/node/1658 http://www.scala-lang.org/node/1658

cover_drive 14y ago
Good to see Scala's popularity. I mentioned only Twitter because I admire them :) -Ashwin
melling 14y ago
This was done a few years ago. I'd love to hear what people have to say about Scala now that code bases have grown. Anyone working with a couple hundred thousand lines of Scala? For instance, I know compilation performance was always frustrating for developers. What's it like for teams of people dealing with a large code base?
- leothekim 14y ago
  We are working with Scala at that scale at Foursquare, and compilation times are definitely a headache. What helps is ensuring your dependencies are acyclic so you're able to do smaller incremental compiles when you make changes. To get us on the path of a DAG-ified codebase, we've been using a build tool developed by Twitter called "pants": https://github.com/twitter/commons https://github.com/twitter/commons which has some similarities to Google's Blaze build tool.

wheaties 14y ago

Novus Partners also does Scala. We've even got a few Python guys coding in Scala. It's just a language with the JVM ecosystem to back it up. Python has it's own ecosystem and for numerics, access to LAPACK and UBLAS is pretty awesome, I must admit. That said, both can exist fairly comfortably in a company and can be used for different purposes. I like Python for prototyping, algorithm validation, and just to hack on. I like Scala for damned near everything else. (Also, Adam is going to be open sourcing what is essentially a Scala clone of Pandas but statically typed and with comparable performance.)

drucken 14y ago

The article's basis of the comparison for building an entire business seems to be very shallow. Surely other considerations are more important, such as, - encapsulation/domain partitioning - messaging - libraries and library maturity - native code interoperability (likely critical for this application) - concurrency - performance - JVM platform - etc. Maybe, as a result of this type of analysis, Python-like language is more suitable in some domains and a JVM language in others. In my opinion, for the requirements quoted, with a deep and performant mathematical framework involved, I cannot envisage how Scala/JVM could win any "war" for the core of the business.

cover_drive 14y ago
A few more considerations will be posted in Part 2 of the post. I appreciate your pointers - as the business grows and takes shape, a fresh evaluation will be required where some of the considerations you mention will be taken into account.
userulluipeste 14y ago
This kind of comparison reminds me about something similar about Git vs Mercurial: http://importantshock.wordpress.com/2008/08/07/git-vs-mercurial/ http://importantshock.wordpress.com/2008/08/07/git-vs-mercur... It looks more like an emotional kind of story than one about practical decisions.
- cover_drive 14y ago
  Indeed! At the early stages of a startup, most decisions (including technical ones) have a strong emotional/instinctive flavor. When I worked at large corporations, the decisions were almost always devoid of emotions.

beggi 14y ago

I haven't tried Scala but since no one has mentioned it: Clojure is a pleasure to work with if you want a functional language in JVM. I have no idea which and how many startups use Clojure, besides Datomic :) It would be interesting to know.

dwelch2344 14y ago
I'd like to agree here. IMO up-and-coming Clojure is an up-and-coming competitor to Scala. As a Lisp variant for the JVM, I'm a major fan of it. Unfortunately I can't share any personal experience as the core of my work is in the Enterprise so I haven't had the right opportunity to present itself. But if you're coming from a mathematical background you should check it out

nnq 14y ago

> Typecheck in Python People actually use it? It seems abandoned and the link to homepage broken: https://pypi.python.org/pypi/typecheck https://pypi.python.org/pypi/typecheck. I'm asking because I'd really like the idea of starting a project the dynamic typing way and then bolting on a (semi)-static type system on top once more new programmers join the game, preferably being able to completely disable it on production machines for best performance.

shared4you 14y ago
Have you checked out Cython[1]? It's used quite often in Numpy. Basically, you add some type annotations like, int, double, etc. for some speed-up. There is a quick tutorial as well [2]. [1]: http://cython.org/ http://cython.org/ [2]: http://wiki.cython.org/tutorials/numpy http://wiki.cython.org/tutorials/numpy
- nnq 14y ago
  I know, it great for the speed-up. But for just making things more manageable it's not worth the added complication of build/compile. I know, my problem is actually solved by "properly written tests" :) ...for I still long for some optional drop-in static typing.

z3phyr 14y ago

Haskell all the way!

cover_drive 14y ago
:)

wyqueshocec 14y ago

> Besides, dynamic typing scares the hell out of me Funny. I feel exactly the opposite. Static typing is false security. I program in JS. My coworkers in Java. They have so many bugs that make it to prod because they assume that just because it compiles it must therefore be safe.

bsg75 14y ago
Are there no tests beyond simple compilation?
cover_drive 14y ago
I must admit that JS scares me more than anything else. Would love to hear about defenses you employ in JS.
eropple 14y ago
I prefer static typing because those assurances reduce the number of things I have to manage--they mean I must keep less state in my head. I do not assume that it guarantees things that it does not. Attributing your coworkers' "if it builds, it works" assumptions to static typing strikes me as incorrect. Static typing, at least in Java, ensures only a (relatively) small set of conditions are true; I don't think you can really blame that for increased/additional assumptions on the part of sub-par programmers.
- stcredzero 14y ago
  Either way, you end up keeping lots of things in your head. With dynamic langs, you do have to think about types and use them conservatively. With languages like Java -- the same. No toolset is going to work well unless the dev is on top of things and good at using the tool well. I've found that debugging in a dynamic environment is great for managing types in a well architected system. It falls down in a poorly architected one, in ways that static typing would not ave allowed. However, in such systems, "it compiles so it must be correct," programmers just get stuck in deeper snow.
  - 14y ago

tikhonj 14y ago

I think you shouldn't have let people talk you out of using Haskell. It's a very nice language and has some advantages for this sort of project (at least based on your cursory description). However, its advantages are really not the important part. Rather, I just wish you wouldn't dismiss it immediately as a crazy choice. It's no more crazy than any other less-popular language. It has a reputation for being impractical, but this reputation is rather unfair especially in the light of recent very practical developments like simpler concurrency, an improved IO manager, very good web frameworks and a fair about of strong libraries both for very specific domains and general productivity. The main disadvantage is that many people find it hard to learn. However, this is a function (heh) of functional programming rather than the language itself. It's actually a simpler language than Scala in many ways because it tries to do one thing well--for example, it has no sub-typing, so you do not have to ever worry about covariance and contravariance. Now, my point here is not that you should always use Haskell, just that you should seriously consider it. Too many people dismiss it out of hand almost as a joke when it is anything but.

cover_drive 14y ago
Thanks for your support. Like I mentioned, I will circle back to Haskell as the business develops. If I was the only programmer in my startup, I would likely go with Haskell for important components of our software. But I have concerns about hiring a large team that would thrive in Haskell. One never knows - I'm keeping an open mind :)
- tikhonj 14y ago
  Hmm, I'm not convinced that hiring a team of good developers for Scala is all that much harder than hiring good Haskell programmers. Haskell has a disproportionate number of very capable people interested in it, and from what I've heard acts as both a way to attract top developers and as something of a filter (at least in encouraging applicants to be more self-selected). I think these effects cancel out Haskell's relative lack of popularity unless you expect to be hiring at the enterprise Java level--something like literally thousands of developers. Anyhow, keeping an open mind is the most important thing, and you clearly have no problems there. I just wish others would follow suit in that regard.

Flow 14y ago

Did you guys consider Google Go? I really like the language and its environment after coding in it for a bit.

shared4you 14y ago
I think the reason was the math libraries. There aren't mature libraries yet for numerical computing. Python has Numpy/Scipy, but am not sure of Scala (I don't use).

tapan_pandita 14y ago

Python has real good, battle tested and actively developed libraries for math, scientific computing and analytics. Most of these are written in C and hence give great performance in addition to the flexibility of python. Some great ones are: Numpy/Scipy, LAPACK, Pandas (lots more for machine learning as well). Also, I think the REPL is no match for IPython :). For your specific domain (math and analytics) python could be a great fit. Of course the lack of type systems and no compile step can be scary, but for a startup, which needs to move fast, python gives you the power of quick iteration. Just to round this off, from the zen of python: "practicality beats purity."

zubinmehta 14y ago
+django