13 ms·

Homogenization of scientific computing – Python is eating other languages’ lunch

193 points by reactor 13y ago

mrfusion 13y ago
My only question mark from this is matplotlib. I tried it five or six years ago and it seemed clunky to use and install. And worst I couldn't seem to just throw up a plot, I recall there being a lot of settings required. And the plots didn't look good by default you had to fool with fonts, font sizes, etc. Does anyone know if it's improved a lot since then? Otherwise I'm not seeing how it could hold a candle to R's plotting abilities and ease of use.
- bsg75 13y ago
  R may still have an advantage when it comes to plotting simplicity. On the Python side, matplotlib is still a bit of a pain, but has improved. Also look at ggplot.py (alpha-ish?) and Bokeh from ContinuumIO
  - synparb 13y ago
    There is also Seaborn: http://stanford.edu/~mwaskom/software/seaborn/index.html http://stanford.edu/~mwaskom/software/seaborn/index.html and Vincent: https://github.com/wrobstory/vincent https://github.com/wrobstory/vincent Also the code examples given on AstroML to work well for figuring out how to make publication quality figures in Matplotlib: http://www.astroml.org/book_figures/ http://www.astroml.org/book_figures/
  - plafl 13y ago
    What is simpler than this? x = arange(0, 2*pi, 1e-2) plot(x, sin(x))
    - leephillips 13y ago
      This: set xrange [0:2*pi] plot sin(x) EDIT: Yes, this is gnuplot, not R.
      - plafl 13y ago
        However, that is gnuplot (I think), not R code. I have not been clear with my question, since I was talking about this parent comment: R may still have an advantage when it comes to plotting simplicity
    - bsg75 13y ago
      For things that only need a one or two line command set, Python and R are probably similar. Once things get a bit more complex, matplotlib may not seem as "friendly" to those used to ggplot. But this about perspective. Coming from a Python background, I would rather stay in Python and work with matplotlib, seaborn, or even ggplot.py, than try to work my data management code into the R model.
- hharrison 13y ago
  Yep, ggplot is the one thing that I keep coming back to R for, plus the odd statistical model I can't find in statsmodels - which is rarer and rarer. One of the many awesome features of IPython - the interactive python shell and notebook, is that you can call code blocks in R just by prefacing with %%R. So my plotting habits are usually first to try the python port of ggplot, and if that can't handle my situation I just jump into R without having to switch windows or do any complicated data transfer. It's worth mentioning that matplotlib is designed to mimic Matlab's plotting API, so for people coming from Matlab there's very little change, plus there's all the benefits of the other plotting libraries others have mentioned.
- Fomite 13y ago
  I vastly prefer R's plotting capabilities, even if I don't generally use ggplot2.
- xioxox 13y ago
  You could try my veusz plotting GUI / plotting package as an alternative to matplotlib: http://home.gna.org/veusz/ http://home.gna.org/veusz/ I think the output looks nicer than matplotlib by default, and you can have a nice GUI and scriptable interface.
jliechti1 13y ago
I have a Python background and recently signed for the Coursera course on R that just started (https://www.coursera.org/course/compdata https://www.coursera.org/course/compdata) because I wanted to get a small taste of R and see how it differed from Python's scientific computing stack. So right now I'm not far enough in the learning curve to see all the benefits R provides. Is it worth investing time in R now if I'm already pretty familiar with a good amount of the Python ecosystem? Or, would it make more sense to continue on in Python?
- micro_cam 13y ago
  If you are serious about data analysis you should probably at least read R (and maybe Matlab) as lots of algorithms were released and only exist in one of those languages. You could get by with a more general statistics course that happened to use R.
- xerula 13y ago
  I would say it depends on precisely what scientific work you need to do. E.g. for phylogenetic statistics there are some nice R packages that bundle simulation techniques and measures that are so far not implemented as conveniently, or at all, in Python. So you need to explore what packages/libraries are out there that fulfill your needs (and also consider how much time/skill/interest you have to code your own packages/libraries where needed). R is still really popular for stats/prototyping/data viz and a useful language to have up your sleeve.
- rz2k 13y ago
  I think that is a great course, and definitely worth your time since it covers the material concisely and at a quick pace.
zwieback 13y ago
Too bad the first part of the post title was edited out of the HN title. I think outside of scientific computing the picture is a little more nuanced.
- StefanKarpinski 13y ago
  Even inside of scientific computing, the picture is quite a bit more nuanced. This post is basically about a the author's personal migration to Python as a user of other people's scientific programming packages. In doing interviews with people inside of companies, there's fairly little actual use of Python for scientific computing – lots of Python for data preparation, but R and Matlab (not to mention Simulink) still dominate for the actual scientific part. And of course, there's the bizarre blind spot that the SciPy community has to the fact that they are really doing scientific computing in C – literally every single package you use that's scalable and performant is actually written in C. This is true of R and Matlab too, of course.
  - synparb 13y ago
    Although I would say in academia I've seen a rapid expansion of Python as both a Matlab replacement and for doing non-HPC work, at least in the field I work in (computational chemistry and biomolecular simulation).
  - hharrison 13y ago
    Yeah, I'm a Python convert like the author, though coming mostly from Matlab rather than R, and everyone in my field reacts with surprise when I tell them I prefer Python. They're open-minded, and I'm hoping to convert a few myself, but I don't think the mass migration has happened yet. Regarding your second comment- you're correct of course, but what makes this a "blind spot"? After all, if the user is writing code in Python, they're doing scientific computing in Python, regardless of what the Python library calls behind the scenes. In my experience, a lot of people doing scientific computing--particularly those more interested in the science than the computing--could care less about what's going on behind the curtain. Any moment they have to think about implementation is a moment not thinking about science and therefore a waste of time. So it's actually a benefit for an ecosystem to hide the underlying mechanics--calling it a "bizarre blind spot" seems to imply they're doing something wrong.
cwmma 13y ago
In GIS at least I'm noticing a divided migration with one camp standardizing to python and the other to JavaScript.
frik 13y ago
For scientific computing with some R or Matlab/Octave background, I suggest the new language Julia: http://julialang.org/ http://julialang.org/ Scientific computing Python community commonly use version 2.x there is the migration step to version 3.x ahead..
- sentenza 13y ago
  For me, Python has everything I need, among which there are many things that R or Matlab have not. If I should summarize what makes Python so suited for the things I do (and did when I was still doing research) it's the following: Python is an easy to use scripting language that can be integrated with number-crunching C/C++ code and for which a scientific standard library _with a vibrant community_ exists. Also, I haven't written a piece of 2.x code in half a year, which is of course only possible because scipy and matplotlib are 3.x ready.
  - pessimizer 13y ago
    C is directly callable from Julia.
  - dangayle 13y ago
    >> Also, I haven't written a piece of 2.x code in half a year, which is of course only possible because scipy and matplotlib are 3.x ready. Whoa. I didn't know people like you existed in the wild. Someone needs to contact the authorities and let them know that you exist.
deleted 13y ago
[deleted]
- StefanKarpinski 13y ago
  Edit: in response to "why isn't there a movement around using JavaScript for scientific computing", which I thought was an excellent question (which has crossed my mind on occasion). Typed arrays and real support for integers are crucial features for scientific computing. Although JavaScript recently got some support for typed arrays, they are pretty awkward to use and there still isn't any support for 64-bit integers, let alone even larger types (128-bit). There's also no support for true multidimensional arrays, meaning you're left to simulate them for yourself like in C. Oh, and let's not forget how awful basic things like equality are in JavaScript. Given how awkward numbers, typed arrays and multidimensional arrays are in JS and how essential all of these things are for scientific computing, I think you can see why this hasn't happened.
  - Eiwatah4 13y ago
    You don't have to simulate multidimensional arrays in C. It's just easier than wrapping your head around the weird syntax required to pass them around: http://pastebin.com/JTjQMfxr http://pastebin.com/JTjQMfxr
    - StefanKarpinski 13y ago
      That's all well and good until the dimensions of the array are dynamic. Which they are in all real code.
      - Eiwatah4 13y ago
        C has variable sized arrays. http://pastebin.com/BEgDNAhu http://pastebin.com/BEgDNAhu That's no longer backward-compatible to C89, though.
        StefanKarpinski 13y ago
JonSkeptic 13y ago
I know that I use python because of how easy it is to code. I can focus wholly, totally on the logic of my code without ever worrying about if I misplaced a semi-colon or left out some weird punctuation. Python frees me to code and not worry about things that get in the way of coding. That's why it's eating other language's lunches, the freedom is almost intoxicating.
- area51org 13y ago
  It's ironic that you say that, given that if you don't get the whitespace correct, you'll have a syntax error. That's one of the big reason Python rubs me the wrong way: white space is semantic.
  - pyre 13y ago
    Getting the indentation right should be the least of your worries if you have a good editor (and don't do something like mix spaces and tabs, which I think everyone is in general agreement with across all languages). When was the last time that you manually typed out 4 (or 2, or 8, etc) spaces to indent a line of code vs. just hitting tab and letting the editor handle inserting those spaces (or the editor automatically indenting when you hit enter on the previous line)? As an aside, all of the people that I've met in person that get red in the face over the idea of white space being semantic are the sort of people that write code like this: sub function1 { return map { $_[2]->do_something($_) } @{shift->(@_)[0]} } I'm sorry, but I can't get worked up about not being able to write code like that. Note: the above code is a reasonable approximation of actual code I encountered by an actual person that would get visibly upset about Python's semantic white space. P.S. The two '$_'s in the map block actually refer to two different variables, and this is one of the reasons I remember that bit of code. It makes no sense to mix usage like that because it becomes confusing.
    - frou_dh 13y ago
      > Getting the indentation right should be the least of your worries if you have a good editor (and don't do something like mix spaces and tabs, which I think everyone is in general agreement with across all languages). Nope. Tabs are for indentation and spaces are for alignment. It's precisely because of non-good (or maybe non-smart) editors that people can't be bothered acknowledging or practicing this distinction and end up using spaces for both.
Fomite 13y ago
As much as I like scikit-learn and pandas, Python likely won't be replacing my R code for quite some time, and I'll continue to hop between the two of then. R is, first and foremost, a language for statistical analysis, and that's really where it shines. Python is getting better (it used to be "you want to do...what?"), but it doesn't have nearly the package infrastructure R does for advanced statistics. It is, for most statistical computing tasks not even on the radar for a number of my colleagues.
norswap 13y ago
Python now has useful libraries, let's all rejoice! (And the part about it eating other's language lunch is unsubstantiated.)
gajomi 13y ago
>The combination of NumPy/SciPy, MatPlotLib, pandas and statmodels had effectively replaced R for me, and I hadn’t even noticed I am surprised that he "hadn't noticed" the switch from plots in R to MatPlotLib. I am a long time MatPlotLib user and I STILL find myself noticing all the time just how painful is can be (irregular data model, weird function names, the insanity that it the documentation). Then I go to the page and feel guilty because the guy who started the project (which I am using for free) died and all the finished plots look so beautiful.
fit2rule 13y ago
I find Lua more interesting than Python. It has all the simplicity, all of the power, none of the indentation, and is quite a nice portable tool. That said, I do wonder at times what it is about Lua that makes so many people not-interested in it, when .. from my naive point of view .. its an almost perfect language for rapid development. I don't have that feeling about Python, quite so much ..
- theorique 13y ago
  +1 I'm a user of both, fan of both, but the lightness of Lua is a big plus. The main downside, IMO, is that the Lua user base is smaller so there's more need to roll your own solutions for things that Python already has several libraries for.
  - CmonDev 13y ago
    "roll your own solutions" - that's part of fun!
    - collyw 13y ago
      its also time consuming, and often buggier than pre-rolled tested solutions.
    - dagw 13y ago
      I agree. However the people paying me would rather I spent my time solving their problems.
    - theorique 13y ago
      For a personal project, sure, but if it's a work project that has to get to the goal line along with ten other things, right NOW!, then it's nice to have some drop-in libraries that "just work".
- freehunter 13y ago
  I love, love, love Lua. I use it for everything. That being said, there's a lot I'd like to use it for that I can't. I'm not a great programmer by a longshot, I'm a hacker in the most traditional of senses (ie, not a hacker who builds billion-dollar wildly successful startups. I write one-off programs to solve a need or to automate a task). Lua doesn't have a lot of features. This is great because it's small and simple to learn, but it makes some jobs harder. As I'm not a great programmer, there are features that people could write themselves, but that's beyond my skill level. I'm constantly pushing tasks to the OS level, which makes my code one step above a Bash script. Other times I'll push something to C, which turns friendly code into a death trap. Lua needs some love and care from the community and it would be perfect. Python has that love and care, but it's still a mess (IMO) at the core language.
Xcelerate 13y ago
I do scientific computing, and Python is one language I never actually got around to learning for some reason. However, as a long-time hobby, I do have an interest in programming languages so I like exploring things like Haskell, Clojure, Lisp, etc. One language I'm really excited about for scientific computing though is Julia. From a language-design perspective, it's beautiful. It was actually thought out rather than kludged together. I've been trying to gradually use it more and more for my research, but the only problem I've found so far is the large mental context-switch I make going from my usual languages to Julia. It's hard to tell what Julia code will be the most performant because there's many ways of doing the same task. I saw someone in the comments on this page mention that you can hand-tune the LLVM generated output within the REPL itself. I imagine this would be very useful if I can get around to learning it (anyone know a good tutorial?)
- michaelochurch 13y ago
  What are your thoughts on Clojure's suitability as a go-to language for scientific computing (except in lower-level, high-performance scenarios that might recommend Julia)? I think it has serious potential in this field. It's not there yet, but it's getting there, and the core.matrix standardization helps a lot.
  - collyw 13y ago
    Its functional. Looking at the way bioinformaticians code, that will make it too complicated for many of them. SQL seems too much for many of them.
  - jamesjporter 13y ago
    I agree with collyw; Lisp is too much for most scientists, who just care about getting their research done and can't be bother to learn all this weird FP/paren stuff [1]. Supplying an obvious, familiar syntax that looks like math written on paper for, e.g., matrix operations, is critical. This is actually one of the core tensions in Julia development imho: balancing having a sane, well designed language (from a programmer's perspective) vs. having a tool that allows scientists to quickly and easily crank out results. [1]: Note; I am not insinuating that Lisp is bad, I like it personally. Just relaying the response you will get from most practicing scientists who are not trained as programmers.
jofer 13y ago
I do all of my scientific computing in python these days. However, I think it's interesting to compare popularity using stackoverflow (which isn't a great metric, as most scientists aren't aware that it exists): Semi-useless Stackoverflow Popularity Metric -------------------------------------------- Searching for questions tagged "[r]": * 45,119 questions Searching for questions tagged "[matlab] or [simulink]": * 27,044 questions Searching for questions tagged "[numpy] or [scipy] or [pandas] or [matplotlib]": * 18,745 questions Searching for questions tagged "[julia-lang]": * 95 questions Searching for questions tagged "[python]": * 255,603 Sure, python isn't as widely used for scientific computing as R or matlab (as evidenced by the third item above), but there's a lot to be said for using a very widely-used language for scientific computing. This is doubly true once you branch out from the "core" scientific code. Building a deployable desktop application is a lot easier in python than in matlab (Done it, partly through java. Don't want to again.) or R (Never tried. Might be easier than I think.).
- collyw 13y ago
  I saw a post on Perl loosing ground to Python based on Stack Overflow posts. Then I remembered, Perl Monks is far better than Stack Overflow for Perl. I would far rather see the discussions that are encouraged there, than the "closed as not constructive" crap on Stack Overflow. (Though to be fair I do see Python getting used more and more, and hear less of Perl).
  - jofer 13y ago
    Very true. Similarly, there probably are a lot more matlab questions on MatlabCentral than on SO. However, stackoverflow is at least easily searchable and widely used. It's certainly not a purely random sample, though. (On a side note, I do feel like the excessive closing of questions on SO has gotten a bit better recently. That's just my opinion, though.)
- chillingeffect 13y ago
  Matlab has its own community. There have been 25,000 answers in the last 30 days. [1]. It has 100,000 users. [2] That stackoverflow metric is not great. It is semi-useless. [1] http://www.mathworks.com/matlabcentral/answers/activities http://www.mathworks.com/matlabcentral/answers/activities [2] http://www.mathworks.com/matlabcentral/about/answers/ http://www.mathworks.com/matlabcentral/about/answers/
plg 13y ago
Python so slow!!
- baldfat 13y ago
  What does this have to do with Scientific Computing? I don't think there is anyone would say that in the realm of Scientific Computing there is a problem with slow in python.
  - 0x001E84EE 13y ago
    I would argue that many Scientific Computing problems involve large amounts of data and/or computation.
    - baldfat 13y ago
      Have you ever worked with MatLab or other "Scientific Computing Languages"? They are like 10 times slower then Python. I am guessing you are against dynamic languages?
- plg 13y ago
  and the spacing thing... what a nightmare for copy/paste
  - plg 13y ago
    to those downvoting the parent to this comment ... honestly, do you disagree? If so I must be missing some trick, and if so I would love to hear what it is.
    - dllthomas 13y ago
      What editor(s) do you use?
      - plg 13y ago
        sublime text emacs textmate problem is typically copying and pasting from an editor into a terminal window running python... usually for code blocks that have not just one but at least two (or more) levels of indentation iPython and pasting with %cpaste usually helps but what if I want plain Python not iPython? anyway it's a problem I don't have with any other language and so that's why I'm complaining about it
ghaff 13y ago
Folks may be interested in this piece by Stephen O'Grady of RedMonk: http://redmonk.com/sogrady/2013/11/26/python-r/ http://redmonk.com/sogrady/2013/11/26/python-r/ He looks at the contention that Python is killing R--based on various data sources--and ultimately concludes: "While the original argument is certainly defensible, then, I find it ultimately unpersuasive. The evidence isn’t there, yet at least, to convince me that R is being replaced by Python on a volume basis. With key packages like ggplot2 being ported, however, it will be interesting to watch for any future shift."
sushirain 13y ago
Please add (2013) to the title, since it's not new.
julienchastang 13y ago
I mostly agree with this article, but we are not there yet. I work with scientists who love the IPython Notebook technology. Some claim the IP[y]: Notebook to be the best thing since the Mosaic web browser and the most important development in scientific computing in a decade. I tend to agree, it is a revolutionary technology and the idea of executable papers is tantalizing. But there are also big problems. In particular, setting up a Python environment with all the necessary libraries is a real pain in the neck even with technologies like pip. For a fee, companies like Enthought are making good progress at taking the pain away (though what happens when you have awkward custom dependencies?). Cloud solutions for preconfigured IP[y]: Notebook servers is another exciting possibility, but not ideal if you work with big data where you want your data local to your Python environment. Also, as I understand, taking advantage of multicore parallelism is not trivial because of the Python Global Interpreter Lock. I have also worked in JVM environments where parallel computing is becoming significantly easier and I don't see that happening in Python anytime soon. I would love to be proven wrong, of course.
- dkersten 13y ago
  Re: setting up environment, I love Anaconda for that reason. Wget the installer, run it, all done - you have a fully featured Python environment ready to go, including Numpy, Scipy, Scikit-learn and much more. Running IPython and the IPython Notebook is then trivial. If you need anything else, you can use their own Conda package manager, or you can just use pip as usual.
  - julienchastang 13y ago
    @dkersten What happens in the scenario where you have other Python distributions on your system? Does Anaconda keep things nicely compartmentalized like a virtualenv?
    - msellout 13y ago
      Yes
    - dkersten 13y ago
sprash 13y ago
Title should have been: "Homogenization of scientific computing – Python 2.x is eating other languages’ lunch" Nobody cares about Python 3 in the scientific community.
guildR 13y ago
RCloud might be useful to look at. It implements Ipython Notebooks for R, with added collaboration based on a github backend.

193 points by reactor 13y ago

My only question mark from this is matplotlib. I tried it five or six years ago and it seemed clunky to use and install. And worst I couldn't seem to just throw up a plot, I recall there being a lot of settings required. And the plots didn't look good by default you had to fool with fonts, font sizes, etc. Does anyone know if it's improved a lot since then? Otherwise I'm not seeing how it could hold a candle to R's plotting abilities and ease of use.

bsg75 13y ago
R may still have an advantage when it comes to plotting simplicity. On the Python side, matplotlib is still a bit of a pain, but has improved. Also look at ggplot.py (alpha-ish?) and Bokeh from ContinuumIO
- synparb 13y ago
  There is also Seaborn: http://stanford.edu/~mwaskom/software/seaborn/index.html http://stanford.edu/~mwaskom/software/seaborn/index.html and Vincent: https://github.com/wrobstory/vincent https://github.com/wrobstory/vincent Also the code examples given on AstroML to work well for figuring out how to make publication quality figures in Matplotlib: http://www.astroml.org/book_figures/ http://www.astroml.org/book_figures/
- plafl 13y ago
  What is simpler than this? x = arange(0, 2*pi, 1e-2) plot(x, sin(x))
  - leephillips 13y ago
    This: set xrange [0:2*pi] plot sin(x) EDIT: Yes, this is gnuplot, not R.
    - plafl 13y ago
      However, that is gnuplot (I think), not R code. I have not been clear with my question, since I was talking about this parent comment: R may still have an advantage when it comes to plotting simplicity
  - bsg75 13y ago
    For things that only need a one or two line command set, Python and R are probably similar. Once things get a bit more complex, matplotlib may not seem as "friendly" to those used to ggplot. But this about perspective. Coming from a Python background, I would rather stay in Python and work with matplotlib, seaborn, or even ggplot.py, than try to work my data management code into the R model.
hharrison 13y ago
Yep, ggplot is the one thing that I keep coming back to R for, plus the odd statistical model I can't find in statsmodels - which is rarer and rarer. One of the many awesome features of IPython - the interactive python shell and notebook, is that you can call code blocks in R just by prefacing with %%R. So my plotting habits are usually first to try the python port of ggplot, and if that can't handle my situation I just jump into R without having to switch windows or do any complicated data transfer. It's worth mentioning that matplotlib is designed to mimic Matlab's plotting API, so for people coming from Matlab there's very little change, plus there's all the benefits of the other plotting libraries others have mentioned.
Fomite 13y ago
I vastly prefer R's plotting capabilities, even if I don't generally use ggplot2.
xioxox 13y ago
You could try my veusz plotting GUI / plotting package as an alternative to matplotlib: http://home.gna.org/veusz/ http://home.gna.org/veusz/ I think the output looks nicer than matplotlib by default, and you can have a nice GUI and scriptable interface.

jliechti1 13y ago

I have a Python background and recently signed for the Coursera course on R that just started (https://www.coursera.org/course/compdata https://www.coursera.org/course/compdata) because I wanted to get a small taste of R and see how it differed from Python's scientific computing stack. So right now I'm not far enough in the learning curve to see all the benefits R provides. Is it worth investing time in R now if I'm already pretty familiar with a good amount of the Python ecosystem? Or, would it make more sense to continue on in Python?

micro_cam 13y ago
If you are serious about data analysis you should probably at least read R (and maybe Matlab) as lots of algorithms were released and only exist in one of those languages. You could get by with a more general statistics course that happened to use R.
xerula 13y ago
I would say it depends on precisely what scientific work you need to do. E.g. for phylogenetic statistics there are some nice R packages that bundle simulation techniques and measures that are so far not implemented as conveniently, or at all, in Python. So you need to explore what packages/libraries are out there that fulfill your needs (and also consider how much time/skill/interest you have to code your own packages/libraries where needed). R is still really popular for stats/prototyping/data viz and a useful language to have up your sleeve.
rz2k 13y ago
I think that is a great course, and definitely worth your time since it covers the material concisely and at a quick pace.

zwieback 13y ago

Too bad the first part of the post title was edited out of the HN title. I think outside of scientific computing the picture is a little more nuanced.

StefanKarpinski 13y ago
Even inside of scientific computing, the picture is quite a bit more nuanced. This post is basically about a the author's personal migration to Python as a user of other people's scientific programming packages. In doing interviews with people inside of companies, there's fairly little actual use of Python for scientific computing – lots of Python for data preparation, but R and Matlab (not to mention Simulink) still dominate for the actual scientific part. And of course, there's the bizarre blind spot that the SciPy community has to the fact that they are really doing scientific computing in C – literally every single package you use that's scalable and performant is actually written in C. This is true of R and Matlab too, of course.
- synparb 13y ago
  Although I would say in academia I've seen a rapid expansion of Python as both a Matlab replacement and for doing non-HPC work, at least in the field I work in (computational chemistry and biomolecular simulation).
- hharrison 13y ago
  Yeah, I'm a Python convert like the author, though coming mostly from Matlab rather than R, and everyone in my field reacts with surprise when I tell them I prefer Python. They're open-minded, and I'm hoping to convert a few myself, but I don't think the mass migration has happened yet. Regarding your second comment- you're correct of course, but what makes this a "blind spot"? After all, if the user is writing code in Python, they're doing scientific computing in Python, regardless of what the Python library calls behind the scenes. In my experience, a lot of people doing scientific computing--particularly those more interested in the science than the computing--could care less about what's going on behind the curtain. Any moment they have to think about implementation is a moment not thinking about science and therefore a waste of time. So it's actually a benefit for an ecosystem to hide the underlying mechanics--calling it a "bizarre blind spot" seems to imply they're doing something wrong.

cwmma 13y ago

In GIS at least I'm noticing a divided migration with one camp standardizing to python and the other to JavaScript.

frik 13y ago

For scientific computing with some R or Matlab/Octave background, I suggest the new language Julia: http://julialang.org/ http://julialang.org/ Scientific computing Python community commonly use version 2.x there is the migration step to version 3.x ahead..

sentenza 13y ago
For me, Python has everything I need, among which there are many things that R or Matlab have not. If I should summarize what makes Python so suited for the things I do (and did when I was still doing research) it's the following: Python is an easy to use scripting language that can be integrated with number-crunching C/C++ code and for which a scientific standard library _with a vibrant community_ exists. Also, I haven't written a piece of 2.x code in half a year, which is of course only possible because scipy and matplotlib are 3.x ready.
- pessimizer 13y ago
  C is directly callable from Julia.
- dangayle 13y ago
  >> Also, I haven't written a piece of 2.x code in half a year, which is of course only possible because scipy and matplotlib are 3.x ready. Whoa. I didn't know people like you existed in the wild. Someone needs to contact the authorities and let them know that you exist.

deleted 13y ago

[deleted]

StefanKarpinski 13y ago
Edit: in response to "why isn't there a movement around using JavaScript for scientific computing", which I thought was an excellent question (which has crossed my mind on occasion). Typed arrays and real support for integers are crucial features for scientific computing. Although JavaScript recently got some support for typed arrays, they are pretty awkward to use and there still isn't any support for 64-bit integers, let alone even larger types (128-bit). There's also no support for true multidimensional arrays, meaning you're left to simulate them for yourself like in C. Oh, and let's not forget how awful basic things like equality are in JavaScript. Given how awkward numbers, typed arrays and multidimensional arrays are in JS and how essential all of these things are for scientific computing, I think you can see why this hasn't happened.
- Eiwatah4 13y ago
  You don't have to simulate multidimensional arrays in C. It's just easier than wrapping your head around the weird syntax required to pass them around: http://pastebin.com/JTjQMfxr http://pastebin.com/JTjQMfxr
  - StefanKarpinski 13y ago
    That's all well and good until the dimensions of the array are dynamic. Which they are in all real code.
    - Eiwatah4 13y ago
      C has variable sized arrays. http://pastebin.com/BEgDNAhu http://pastebin.com/BEgDNAhu That's no longer backward-compatible to C89, though.
      - StefanKarpinski 13y ago

JonSkeptic 13y ago

I know that I use python because of how easy it is to code. I can focus wholly, totally on the logic of my code without ever worrying about if I misplaced a semi-colon or left out some weird punctuation. Python frees me to code and not worry about things that get in the way of coding. That's why it's eating other language's lunches, the freedom is almost intoxicating.

area51org 13y ago
It's ironic that you say that, given that if you don't get the whitespace correct, you'll have a syntax error. That's one of the big reason Python rubs me the wrong way: white space is semantic.
- pyre 13y ago
  Getting the indentation right should be the least of your worries if you have a good editor (and don't do something like mix spaces and tabs, which I think everyone is in general agreement with across all languages). When was the last time that you manually typed out 4 (or 2, or 8, etc) spaces to indent a line of code vs. just hitting tab and letting the editor handle inserting those spaces (or the editor automatically indenting when you hit enter on the previous line)? As an aside, all of the people that I've met in person that get red in the face over the idea of white space being semantic are the sort of people that write code like this: sub function1 { return map { $_[2]->do_something($_) } @{shift->(@_)[0]} } I'm sorry, but I can't get worked up about not being able to write code like that. Note: the above code is a reasonable approximation of actual code I encountered by an actual person that would get visibly upset about Python's semantic white space. P.S. The two '$_'s in the map block actually refer to two different variables, and this is one of the reasons I remember that bit of code. It makes no sense to mix usage like that because it becomes confusing.
  - frou_dh 13y ago
    > Getting the indentation right should be the least of your worries if you have a good editor (and don't do something like mix spaces and tabs, which I think everyone is in general agreement with across all languages). Nope. Tabs are for indentation and spaces are for alignment. It's precisely because of non-good (or maybe non-smart) editors that people can't be bothered acknowledging or practicing this distinction and end up using spaces for both.

Fomite 13y ago

As much as I like scikit-learn and pandas, Python likely won't be replacing my R code for quite some time, and I'll continue to hop between the two of then. R is, first and foremost, a language for statistical analysis, and that's really where it shines. Python is getting better (it used to be "you want to do...what?"), but it doesn't have nearly the package infrastructure R does for advanced statistics. It is, for most statistical computing tasks not even on the radar for a number of my colleagues.

norswap 13y ago

Python now has useful libraries, let's all rejoice! (And the part about it eating other's language lunch is unsubstantiated.)

gajomi 13y ago

>The combination of NumPy/SciPy, MatPlotLib, pandas and statmodels had effectively replaced R for me, and I hadn’t even noticed I am surprised that he "hadn't noticed" the switch from plots in R to MatPlotLib. I am a long time MatPlotLib user and I STILL find myself noticing all the time just how painful is can be (irregular data model, weird function names, the insanity that it the documentation). Then I go to the page and feel guilty because the guy who started the project (which I am using for free) died and all the finished plots look so beautiful.

fit2rule 13y ago

I find Lua more interesting than Python. It has all the simplicity, all of the power, none of the indentation, and is quite a nice portable tool. That said, I do wonder at times what it is about Lua that makes so many people not-interested in it, when .. from my naive point of view .. its an almost perfect language for rapid development. I don't have that feeling about Python, quite so much ..

theorique 13y ago
+1 I'm a user of both, fan of both, but the lightness of Lua is a big plus. The main downside, IMO, is that the Lua user base is smaller so there's more need to roll your own solutions for things that Python already has several libraries for.
- CmonDev 13y ago
  "roll your own solutions" - that's part of fun!
  - collyw 13y ago
    its also time consuming, and often buggier than pre-rolled tested solutions.
  - dagw 13y ago
    I agree. However the people paying me would rather I spent my time solving their problems.
  - theorique 13y ago
    For a personal project, sure, but if it's a work project that has to get to the goal line along with ten other things, right NOW!, then it's nice to have some drop-in libraries that "just work".
freehunter 13y ago
I love, love, love Lua. I use it for everything. That being said, there's a lot I'd like to use it for that I can't. I'm not a great programmer by a longshot, I'm a hacker in the most traditional of senses (ie, not a hacker who builds billion-dollar wildly successful startups. I write one-off programs to solve a need or to automate a task). Lua doesn't have a lot of features. This is great because it's small and simple to learn, but it makes some jobs harder. As I'm not a great programmer, there are features that people could write themselves, but that's beyond my skill level. I'm constantly pushing tasks to the OS level, which makes my code one step above a Bash script. Other times I'll push something to C, which turns friendly code into a death trap. Lua needs some love and care from the community and it would be perfect. Python has that love and care, but it's still a mess (IMO) at the core language.

Xcelerate 13y ago

I do scientific computing, and Python is one language I never actually got around to learning for some reason. However, as a long-time hobby, I do have an interest in programming languages so I like exploring things like Haskell, Clojure, Lisp, etc. One language I'm really excited about for scientific computing though is Julia. From a language-design perspective, it's beautiful. It was actually thought out rather than kludged together. I've been trying to gradually use it more and more for my research, but the only problem I've found so far is the large mental context-switch I make going from my usual languages to Julia. It's hard to tell what Julia code will be the most performant because there's many ways of doing the same task. I saw someone in the comments on this page mention that you can hand-tune the LLVM generated output within the REPL itself. I imagine this would be very useful if I can get around to learning it (anyone know a good tutorial?)

michaelochurch 13y ago
What are your thoughts on Clojure's suitability as a go-to language for scientific computing (except in lower-level, high-performance scenarios that might recommend Julia)? I think it has serious potential in this field. It's not there yet, but it's getting there, and the core.matrix standardization helps a lot.
- collyw 13y ago
  Its functional. Looking at the way bioinformaticians code, that will make it too complicated for many of them. SQL seems too much for many of them.
- jamesjporter 13y ago
  I agree with collyw; Lisp is too much for most scientists, who just care about getting their research done and can't be bother to learn all this weird FP/paren stuff [1]. Supplying an obvious, familiar syntax that looks like math written on paper for, e.g., matrix operations, is critical. This is actually one of the core tensions in Julia development imho: balancing having a sane, well designed language (from a programmer's perspective) vs. having a tool that allows scientists to quickly and easily crank out results. [1]: Note; I am not insinuating that Lisp is bad, I like it personally. Just relaying the response you will get from most practicing scientists who are not trained as programmers.

jofer 13y ago

I do all of my scientific computing in python these days. However, I think it's interesting to compare popularity using stackoverflow (which isn't a great metric, as most scientists aren't aware that it exists): Semi-useless Stackoverflow Popularity Metric -------------------------------------------- Searching for questions tagged "[r]": * 45,119 questions Searching for questions tagged "[matlab] or [simulink]": * 27,044 questions Searching for questions tagged "[numpy] or [scipy] or [pandas] or [matplotlib]": * 18,745 questions Searching for questions tagged "[julia-lang]": * 95 questions Searching for questions tagged "[python]": * 255,603 Sure, python isn't as widely used for scientific computing as R or matlab (as evidenced by the third item above), but there's a lot to be said for using a very widely-used language for scientific computing. This is doubly true once you branch out from the "core" scientific code. Building a deployable desktop application is a lot easier in python than in matlab (Done it, partly through java. Don't want to again.) or R (Never tried. Might be easier than I think.).

collyw 13y ago
I saw a post on Perl loosing ground to Python based on Stack Overflow posts. Then I remembered, Perl Monks is far better than Stack Overflow for Perl. I would far rather see the discussions that are encouraged there, than the "closed as not constructive" crap on Stack Overflow. (Though to be fair I do see Python getting used more and more, and hear less of Perl).
- jofer 13y ago
  Very true. Similarly, there probably are a lot more matlab questions on MatlabCentral than on SO. However, stackoverflow is at least easily searchable and widely used. It's certainly not a purely random sample, though. (On a side note, I do feel like the excessive closing of questions on SO has gotten a bit better recently. That's just my opinion, though.)
chillingeffect 13y ago
Matlab has its own community. There have been 25,000 answers in the last 30 days. [1]. It has 100,000 users. [2] That stackoverflow metric is not great. It is semi-useless. [1] http://www.mathworks.com/matlabcentral/answers/activities http://www.mathworks.com/matlabcentral/answers/activities [2] http://www.mathworks.com/matlabcentral/about/answers/ http://www.mathworks.com/matlabcentral/about/answers/

plg 13y ago

Python so slow!!

baldfat 13y ago
What does this have to do with Scientific Computing? I don't think there is anyone would say that in the realm of Scientific Computing there is a problem with slow in python.
- 0x001E84EE 13y ago
  I would argue that many Scientific Computing problems involve large amounts of data and/or computation.
  - baldfat 13y ago
    Have you ever worked with MatLab or other "Scientific Computing Languages"? They are like 10 times slower then Python. I am guessing you are against dynamic languages?
plg 13y ago
and the spacing thing... what a nightmare for copy/paste
- plg 13y ago
  to those downvoting the parent to this comment ... honestly, do you disagree? If so I must be missing some trick, and if so I would love to hear what it is.
  - dllthomas 13y ago
    What editor(s) do you use?
    - plg 13y ago
      sublime text emacs textmate problem is typically copying and pasting from an editor into a terminal window running python... usually for code blocks that have not just one but at least two (or more) levels of indentation iPython and pasting with %cpaste usually helps but what if I want plain Python not iPython? anyway it's a problem I don't have with any other language and so that's why I'm complaining about it

ghaff 13y ago

Folks may be interested in this piece by Stephen O'Grady of RedMonk: http://redmonk.com/sogrady/2013/11/26/python-r/ http://redmonk.com/sogrady/2013/11/26/python-r/ He looks at the contention that Python is killing R--based on various data sources--and ultimately concludes: "While the original argument is certainly defensible, then, I find it ultimately unpersuasive. The evidence isn’t there, yet at least, to convince me that R is being replaced by Python on a volume basis. With key packages like ggplot2 being ported, however, it will be interesting to watch for any future shift."

sushirain 13y ago

Please add (2013) to the title, since it's not new.

julienchastang 13y ago

I mostly agree with this article, but we are not there yet. I work with scientists who love the IPython Notebook technology. Some claim the IP[y]: Notebook to be the best thing since the Mosaic web browser and the most important development in scientific computing in a decade. I tend to agree, it is a revolutionary technology and the idea of executable papers is tantalizing. But there are also big problems. In particular, setting up a Python environment with all the necessary libraries is a real pain in the neck even with technologies like pip. For a fee, companies like Enthought are making good progress at taking the pain away (though what happens when you have awkward custom dependencies?). Cloud solutions for preconfigured IP[y]: Notebook servers is another exciting possibility, but not ideal if you work with big data where you want your data local to your Python environment. Also, as I understand, taking advantage of multicore parallelism is not trivial because of the Python Global Interpreter Lock. I have also worked in JVM environments where parallel computing is becoming significantly easier and I don't see that happening in Python anytime soon. I would love to be proven wrong, of course.

dkersten 13y ago
Re: setting up environment, I love Anaconda for that reason. Wget the installer, run it, all done - you have a fully featured Python environment ready to go, including Numpy, Scipy, Scikit-learn and much more. Running IPython and the IPython Notebook is then trivial. If you need anything else, you can use their own Conda package manager, or you can just use pip as usual.
- julienchastang 13y ago
  @dkersten What happens in the scenario where you have other Python distributions on your system? Does Anaconda keep things nicely compartmentalized like a virtualenv?
  - msellout 13y ago
    Yes
  - dkersten 13y ago

sprash 13y ago

Title should have been: "Homogenization of scientific computing – Python 2.x is eating other languages’ lunch" Nobody cares about Python 3 in the scientific community.

guildR 13y ago

RCloud might be useful to look at. It implements Ipython Notebooks for R, with added collaboration based on a github backend.