8 ms·

Why Python rocks for research

210 points by agconway 16y ago

bwooceli 16y ago
I learned Python on the fly specifically for research. I used Django to build out an enterprise reporting/analytics system to support a customer experience (survey) program for the cost of time (huzzah open source). We had bids on this project upwards of 80k. We generate ~100k surveys / month and are able to get targeted, meaningful, automated insights directly to front-line management. Python FTW.
- shill 16y ago
  Python + Django FTW.
RodgerTheGreat 16y ago
This article makes some fairly convincing arguments that Python is a more flexible tool than Matlab or Perl, but I can't help but come away with the sense that the author hasn't tried many other languages. There are an awful lot of languages that provide iterators, a powerful set of data structures, extensive libraries and facilities for structuring and maintaining large codebases. .Net languages (maybe F# would be good for this?), Java or most of the emerging languages for the JVM stack, Ruby (which is generally considered to be "different but equivalent" to Python), and so forth.
- vog 16y ago
  > There are an awful lot of languages that provide ... Still, how many of them have a fast interactive interpreter ("command line") with a decent usability? How many of those provide good libraries for numerical as well as symbolic math? With an API that is easy to write, to understand and to extend? Python may not be the only language with those qualities, but there aren't many languages (and ecosystems around them) which can compete on all those areas. Python seems to be one one the few "best fits" for scientific applications.
  - Dn_Ab 16y ago
    There are Ocaml, F# and Clojure with some combination of great tools, speed (clojure addressed this recently I think but i only have visual experience with clojure), light syntax, books and documentation, repl,excellent platform, wide libarary choice and or decent interop with C. Also F# is doing some really cool stuff to do with datasets awareness in the language. Haskell seems a perfect fit for mathematical use and while I haven't used it in a couple years, I would hesitate to suggest it due to a lack of mature library options, difficulty of FFI and perhaps a steep initial curve. Scala is a good language for an entire application but provides too much scaffolding for scientific applications. R is fairly widely used but is also itself very quirky.
    - thesz 16y ago
      I am with you on Haskell. It counts on Freedom, Readability, Documentation System (including lhs2tex which will turn your "integrate f 0 a" into \Int_0^a{f dx}), High-level vs low-level, Standard library (including hackage/cabal), Data structures, Module system, Calling syntax, Default arguments (currying), Multiple programming paradigms (there was a saying that Haskell is the best imperative language). It partially counts on most other points. Myself, I choose Haskell for my research project, as it was best language on (expressiveness times safety) scale. Strong type system certainly helps sweeping out errors.
    - wynand 16y ago
      I'm a long time Python programmer and I am currently using it for scientific programming (for my PhD). I've shopped around a lot for alternatives to Python and looked at Ocaml, F#, Boo, Clojure, Common Lisp, Haskell and Scala. There are a couple of things still keeping me with Python: * The REPL, especially IPython, * Numpy & Scipy, * Networkx & igraph, * jpype for fairly seamless JVM integration (this way, I can interact with Cytoscape), rpy for fairly seamless R integration, * ZODB, * IPython's parallel processing framework. The .net environment probably comes the closest to providing everything here, but the REPLs need a lot of work and the graph libraries have more complicated interfaces which make them a pain to use on the command line. Python is not without its warts but when I recently had to spend time with Matlab again after a few years, I was reminded of just how nice the Python ecosystem is in comparison. Update: I should add that if you rely on one of Matlab's toolboxes, you might not find any decent alternatives outside of Matlab. You can always use a bridge like MLabWrap to access Matlab from Python.
- mechanical_fish 16y ago
  I only scanned the article very briefly, but my impression is that the important comparison is vs. Matlab. The other players aren't really in the author's game. It's a question of the use case and the community and the library support. In theory, .Net could displace Matlab or Python as the canonical platform for scientific researchers. And in theory Python could displace PHP as the canonical platform for classic CRUD web apps. In practice neither is likely to happen, no matter how much we might or might not wish it to.
  - hogu 16y ago
    I've been a Matlab user for 10 yrs, recently switched to python because of the increased flexibility. never going back to Matlab ever again. Python math and science libs can be a little rough around the edges, but the flexibility more than makes up for it for me.
    - samd 16y ago
      I'm a Matlab user myself, but I try to use Ruby whenever possible if for nothing else but to just get away from all the god damned matrices. I didn't realize Python had similar capabilities, I look forward to trying it out. But does Python have "Index exceeds matrix dimensions." errors? I just can't imagine life without seeing a few of those every day. That said, the workspace is really handy, is there some equivalent GUI in Python?
      - infinite8s 16y ago
        Check out EPD: http://www.enthought.com/products/ http://www.enthought.com/products/, particularly the ETS framework. It contains almost all the popular scientific/numeric libraries in python and is free for academic use (and much of it is open-source). (disclaimer - i work for Enthought).
      - timwiseman 16y ago
        As infinite8s points out, EPD from enthought is really quite nice. Personally, I rather like Python XY ( http://www.pythonxy.com/ http://www.pythonxy.com/ )though. It is totally free, open source, and for small little scripts I am a huge fan of the Spyder IDE. It lacks some of the features of bulkier IDEs so I also use Eclipse from time to time. But Spyder is light and much faster than Eclipse with every feature I would want when working on small projects that can be contained in just one or two files.
- wwortiz 16y ago
  Do they all have equivalent libraries to python's scipy and matplotlib? I think that is why the author could move to python as these provide a pretty large subset of what matlab has and makes the transition less problematic. Java probably does and .NET may have something like this but I don't know of them or their amount of documentation.
  - kenjackson 16y ago
    .NET has various things in this space, but I think a lot are commercial products. .NET is not used as much in academia, so most things are targeted at professionals. With that said, you can probably get free/cheap licenses for many/most of commercial products. But I think Python probably has the most active community as its been used as glue for a long time in this space.
- bad_user 16y ago
  SciPy, NumPy, Matplotlib, Cython, Sphinx, PIL. The article should've also mentioned NLTK. IMHO, languages matter less than the available libraries, and in my experience only Java matches the depth of the Python ecosystem. That Python is a nice language to work with, that's just a bonus.
  - ekidd 16y ago
    I agree about Python's libraries. Normally, I'm a Ruby programmer, because I like Ruby's syntax and the Rack/Rails/Sinatra web stack. But if I need to do scientific or linguistic programming, Python is absolutely amazing. SciPy, NumPy, Matplotlib and NLTK support a rich and deep ecosystem. And it's vastly nicer programming environment than Matlab and Octave. (Of course, GNU R is also pretty useful if you're doing pure statistics.)
    - adw 16y ago
      Which is why we use Python. We're a web app (Django) doing big data aggregation/processing (Numpy).
    - ajmurmann 16y ago
      Same here. I used to do a lot of Python programming, but moved to Ruby because of Rails. When I had to work on a machine learning paper, I spent weeks looking for a equivalent to NumPy or Matplotlib in the Ruby world, but nothing comes even close!
  - osdf 16y ago
    Also, (basic) GPU operations with cudamat (or even finer, gnumpy) at your fingertips.
- variety 16y ago
  actually, he doesn't make any assertions that Python is more flexible than Perl (which would be rather doubtful), only that it is more readable (which, as a perlista, I'm sad to say is probably true). but I also get the sense that this is the first time he's seriously delved into a dynamic programming language. much of what he's saying about Python is exactly what bioinformaticists were saying about Perl in the late 90s / early naughts.
  - expeditious 16y ago
    The thing about readability of Perl vs. Python ... I dunno. I don't think it's the syntax itself that causes Perl to have some readability issues. Stuff like `$foo =~ s/bar/baz/g` or `join '-', split( q{ }, $a_str)` is very readable. Context and corner cases maybe makes readability suffer. For example, what does `m///g` do in scalar vs. list context? I don't recall.
    - variety 16y ago
      you might say that Perl's lack of (obvious, first-pass) readability is a two-headed dog from hell: - it's a (very) good thing, in that there's -a- -lot- of expressive power hiding in those corner cases (esp list v. scalar context). in fact perl is nearly unique in the degree to which it embraces, rather than seeks to quash the potential for nuance and flexibility to be found in the various eddies and whirlpools that lie "between" non-whitespace elements of code. this is something that goes to the deepest intellectual roots of the language (to be more like the way the human brain thinks, rather than the way machines think). - it's a (very) bad thing, in that the same "power" for expressiveness just represents little more than an endless stream of banana peels to most first time users (understandably sending many of them running into the arms of the other major dynamic programming languages). and that it just makes it too damn easy to write (barely) functional, pothole-laden frontline scripts (such that Perl is partly responsible for "scripting" being such a dirty word, in many quarters). another aspect people don't like to talk about its nearly coprophilic fondness for not just nuanced and context-sensitive, but intentionally obtuse syntax (the impossible to remember "special" variables such as $[, $;, $' etc being probably the worst examples). this also (sadly) has a lot to do with the community's deference to the aesthetics of its original authors (and is also plenty understandable, given the extent to which unabashed ugliness -- as personified by makefiles, shell languages and macro-laden C and C++ -- ruled the day at the time). unfortunately it also blinds a lot of Perl folk to the fact that beloved muse just happens to look awful darn cluttery (or worse) to many reasonably intelligent people who come from more "modern" programming backgrounds (or who at least came onto the scene comparatively recently).
- hyperbovine 16y ago
  It's all about Numpy. Mayavi, IPython, mlab & friends are great for when I need to plot things or look at data, but Numpy is the workhorse I keep coming back to day in and day out. And also, the thing I wish for most when I have to use other languages. The combination of the speed of C and the elegance of, well, anything that is not C, is hard to beat. Once you get down the basics of array broadcasting, types, etc., it's possible to do some amazingly elegant things in Numpy, and quickly too. The numpy library has seemingly every array function I have ever wanted. If I were Matlab I'd be scared :-)
- crocowhile 16y ago
  None of the languages you mention is easy enough to be grasped in a week from someone who never did programming in their life. Believe me: most of the people who suddenly has to do data analysis for their phd or postdoc projects hardly know how to use excel, so easy of use is essential. Also, you want a non compiled language. Most of the time you do interactive programming and change parameters on the fly, according of the result of the analysis. Finally, matplotlib, one of python's most complete graphic library, is a breeze to use. Making graphs in an interactive way with java or .net is simply impossible. I am a neuroscientist and most people in the field use Matlab. I use python (in fact I use ONLY open source software, by choice). It's amazing how many advantages python gave me on my daily life.
agconway 16y ago
Python rocks, but Python + R + bash rocks way harder for research
- levesque 16y ago
  R is definitely powerful and a good part of any scientific data analysis toolkit. I use python, ipython, matplotlib, numpy and R. I call my R scripts directly from python using rpy.
  - wildanimal 16y ago
    Agreed. I use Python for heavy shell-scripting and text-processing (though R surprisingly does have respectable facilities for all but the most overwhelming of these tasks) and R for the rest of the analysis. I've thought about switching to NumPy/SciPy as it's part of Python to integrate everything, but R's data frame, factors, and reshape, plyr, and lattice packages makes you think very differently about how to approach the data - and hard to go back to lower-level manipulation of arrays; not to mention all the stats/graphics packages which are very easy to install and apply. And documentation of its functions is superb.
apl 16y ago
In the end, it's all about the ecosystem. Perl wins for bioinformatics because there are boatloads of scientists already using it, with all the neat libraries and resources that brings. Equally, Python wins for, say, prototyping in robotics because of libraries, support and so on. There's nothing intrinsically science-apt about Python/Perl, but Ruby and friends can't compete when it comes to the programming environment; that's what counts.
- cageface 16y ago
  As a language I prefer Ruby but the Python ecosystem for this kind of thing is definitely a huge advantage. You can actually do quite a bit with Ruby + GSL but it's still not really competitive.
  - apl 16y ago
    Yep, same with me. I do prefer Ruby, but it's just not feasible at the moment. Languages that get behind in a certain area enter a terribly vicious circle: There's no scientific community backing Ruby, so why would I develop something that'd make Ruby more competitive?
    - djacobs 16y ago
      I'm in the same boat and it seems like several others as well. I'm fine moving to Python for the moment, but I wonder how the Ruby community will ever know if there's enough of a demand for scientific tools to merit their development?
      - phren0logy 16y ago
        I prefer Ruby generally, but honestly Ruby and Python are sufficiently similar that I'd rather smart programmers put their time to good use doing something other than reimplementing Python's science libs. I think that making some tools available in a totally different language (maybe something functional) would be much more useful, because it would allow for a very different approach to the problem if needed. In a perfect world, we could also have the option of using light wrappers around OpenCL matrix libraries, and push the linear algebra to GPUs that eat matrices for breakfast.
ogrisel 16y ago
And the corollary: Why do researchers never respect the PEP8 when they write python code? Yes I am a bit overreacting since the blog post is very well written and I actually agree 100% with the content. But please people: respect the PEP8 [1]. It makes your readers feel at home while reading your code. It is very important if you want to get new contributors to your project. See [2] for instance. [1] http://www.python.org/dev/peps/pep-0008/ http://www.python.org/dev/peps/pep-0008/ [2] http://www.dataists.com/2010/10/whats-the-use-of-sharing-code-nobody-can-read/ http://www.dataists.com/2010/10/whats-the-use-of-sharing-cod...
- hogu 16y ago
  I wasn't aware of pep8 when I started, most science people arrive at python from a different path. What I mean is, for a long time I knew much more about numpy than about python itself. there are some things in pep8 that are bad for science, the spaces around operations, and also the 80 chars to a line... scientific expressions are often long and complicated, yes you can do it while adhering to pep8, but its kind of a PITA
  - ogrisel 16y ago
    The 80 chars limit has a justification: expressions that don't fit on 80 chars (or two lines using parens) are not readable anyway. In such a case temporary variables with meaningful names would both help respect the 80 chars constraint and make the expression easier to understand by the reader. Furthermore having 80 chars is great to have vertically split editors with the code on one panel and the tests or the documentation on the other panel.
    - Goladus 16y ago
      > The 80 chars limit has a justification: expressions that don't fit on 80 chars (or two lines using parens) are not readable anyway I disagree. There are many cases, especially after 2-3 levels of indentation, where 80 characters is an unreasonably narrow space. I don't have a strong preference for reading code within 80 characters. And I'd much rather comments use 80 characters plus indentation rather than worry about whether I've got my screens vertically split. > Furthermore having 80 chars is great to have vertically split editors with the code on one panel and the tests or the documentation on the other panel. Having lines here or there that go beyond 80 characters doesn't completely prevent you from doing this, and having an entire statement on a single line of code makes line-based tools like grep or kill-line more effective. Limiting to 80 characters is a good idea, but it's easy to see that there are a few significant tradeoffs, and that someone trying to get something done is not going to want to bother.
sintesoro 16y ago
Python is good, you could also consider Maxima. A single example: f(x):= x^2+3x+7; Maxima provides: Symbolic computation, blas and laplack integration for numeric algebra, 500 pages manual in several languages, a complete library for statistics, differential equation, calculus, series. Graphics with matplotllib. Also maxima language is not much complicate that python: for i in range(10):print ii versus for i:0 thru 9 do print ii; [i2 for i in range(10)] versus makelist(i*2,i,0,9) But Matlab libraries are greater than python and maxima.
- dagw 16y ago
  Python has sage and sympy for doing symbolic math. Although admittedly they're quite primitive compared to maple and mathematica (haven't used Maxima, so I can't really compare)
  - koenigdavidmj 16y ago
    Sage actually includes Maxima, for ultimate convenience.
- kwantam 16y ago
  Maxima rocks for symbolic math. I prefer it to Mathematica, which is saying a lot. In contrast, octave always feels like "almost-Matlab" and I still prefer the latter. Also see wxMaxima, which will (among many other things) produce LaTeX for you.
  - nurbl 16y ago
    If you haven't already, you might also want to check out Sage (http://www.sagemath.org/ http://www.sagemath.org/) It uses python to "glue" together other free math tools (e.g. Maxima) into a unified system with a nice interface.
woodson 16y ago
Often languages are chosen based on already existing tools used in a specific research project. That's why it's good to be able to quickly pick up new languages. For example scripting languages integrated in some frameworks, like Scheme in the Festival speech synthesis system. In the end, this often results in projects involving things like Python, Scheme, bash, R and a bit of tcl ;-).
rdouble 16y ago
I've wasted most of my professional life tweaking various unix software to make it work. However, the typical scientific python setup proved to be too frustrating to install on OSX. The recommended solution is to just buy the Enthought distro. If I'm paying for software anyway, why is Enthought better than Matlab?
- wgrover 16y ago
  I've never had any problems putting a research-grade python setup on OSX - just use the .dmg installer files files available for python (2.6.X for compatibility), numpy, scipy, and matplotlib. But I agree that the Enthought Python Distro is also a good alternative (if a little bloated for my needs), and it's also free for academics.
  - osdf 16y ago
    For a basic setup on OS X, I found scipy superpack to be an excellent choice: http://stronginference.com/scipy-superpack/ http://stronginference.com/scipy-superpack/
    - gkelly 16y ago
      I've found macports provides a ton of python libraries, like numpy, scipy, and matplotlib, all easily installed from the command line.
  - wildanimal 16y ago
    When I tried to use easy_install with the EPD it tried to take me to their repository, for which we don't have appropriate permissions? So I sucked it up and installed some from .dmg's and others from source (matplotlib was difficult). And I found out just because it's Python doesn't mean it's portable - for instance NumPy/SciPy is hardly ever installed on other machines (it's not trivial, as is written here many times), and I find that administrators won't update their Python installation so I've had to re-write bits of my code to accommodate Python 2.3-2.4 at times! But all said, it's a nice tool and hopefully these installation/ version issues will work itself out.
miloshasan 16y ago
Python is awfully close to being a superior (and free) replacement for Matlab, but there are a few annoyances that keep preventing me from switching forever. Unfortunately, these are mostly not bugs but bad design that is believed to be correct by the core developers, so it is unlikely to ever change: - Matrices are a pain. The r_[] and c_[] operators could be a reasonable replacement for Matlab's elegant matrix construction syntax, but they do not work as expected (as smart hstack and vstack), instead doing something completely different and inconsistent for vectors and matrices. - Tensors are a bigger pain. Matlab has a very well-defined semantics for operations like permute and reshape; in NumPy these operations sometimes create just a view, at other times they reshuffle the memory contents. I know the idea was to "protect" the user from having to know the memory layout of data, but this idea is bad. - Ipython is great in every way except when it comes to reloading parts of your program. After any tiny change to your code, the only safe thing to do is to quit ipython and start it again. All the other options (run, reset, reload...) make some secret and wrong assumptions on what you want to reload. In contrast, this works flawlessly in Matlab.
- deleted 16y ago
  [deleted]
ansgri 16y ago
One more powerful combo is R+Java+C. UI, integration and massive data processing in Java, R for prototyping, plotting and model fitting, and C if you have numerical simulations.
maurits 16y ago
I feel this article is somewhat unbalanced in its single minded rejoice for a certain tool/environment. So in the same spirit here come a couple of reasons not to switch from Matlab to Python, all stemming from my experience when I decided to try to switch from Matlab to Python/C - installing all these packages on (any) system is painful. Different versions don't play together or don't work (yet) on some platform and or architecture. This stems from my own experience of getting a version of python to work with numpy, scipy, matplotlib, opencv and PIL on a windows, mac, and linux machine. No 100 percent success yet on any platform. - central and consistent documentation. Even for very simple cases, I got a bit of a headache. I encounter a python print statement for the first time that obviously differs somewhat from its c printf cousin. I google "python print syntax" only to find that the first xx hits, including the official documentation, do not cover the full specification of this statement. I fear the moment I might actually need detailed information on something less trivial. - Numerical integration is more accurate in Matlab. - Visualization capabilities of matlab are more powerful. But who knows, perhaps there is yet another package floating around :-) - Matlab may not have advanced data-structures, but it is a rapid prototyping tool, for testing ideas. If I need to write an actual application, I will use a tool and language geared for that task.
- xiongchiamiov 16y ago
  > central and consistent documentation. Even for very simple cases, I got a bit of a headache. I encounter a python print statement for the first time that obviously differs somewhat from its c printf cousin. I google "python print syntax" only to find that the first xx hits, including the official documentation, do not cover the full specification of this statement. I fear the moment I might actually need detailed information on something less trivial. http://docs.python.org/reference/simple_stmts.html#the-print-statement http://docs.python.org/reference/simple_stmts.html#the-print... lists eveything about `print` - it takes stuff in, converts 'em to strings, and sticks in on stdout. It doesn't refer to prinf-like formatting because that's for strings in general. If you weren't aware of this, you probably should have been going through a basic Python tutorial, rather than just jumping into the middle of things. Python's documentation is the best I've encountered so far, and I find good docs to be an important value in the community, as well. I guess YMMV, though.
chbarts 16y ago
As usual, Common Lisp has many of the same features. The only thing it doesn't really have now is the same ecosystem as Python.
b_emery 16y ago
> In MatLab everything is flat – all functions are declared in the global namespace. However, this discourages code reusability by making the programmer do extra work keeping disparate program components separate. In other words, without a hierarchical structure to the program, it’s difficult to extract and reuse specific functionality. I completely disagree. Reusable Matlab code has been my holy grail for the last couple of years. The key is to break out specific functionality as subfunctions. When these are abstracted and generally useful elsewhere, then they become new tools for the toolbox. The subfunctions also make great starting points for repurposing code. This layout results in much less work.
- tel 16y ago
  It's absolutely true that using more functions makes your code more maintainable. It's also absolutely true that nearly every other language on the planet does this better than Matlab.
  - snth 16y ago
    Yes; in particular, Matlab's restriction of ONE outside-callable function per file is a huge pain.
woodson 16y ago
For those interested in scientific frameworks and existing packages: Scientific track of EuroSciPy 2010 [1]. Everything from seismology to visual programming. http://www.euroscipy.org/track/870 http://www.euroscipy.org/track/870

210 points by agconway 16y ago

I learned Python on the fly specifically for research. I used Django to build out an enterprise reporting/analytics system to support a customer experience (survey) program for the cost of time (huzzah open source). We had bids on this project upwards of 80k. We generate ~100k surveys / month and are able to get targeted, meaningful, automated insights directly to front-line management. Python FTW.

shill 16y ago
Python + Django FTW.

RodgerTheGreat 16y ago

This article makes some fairly convincing arguments that Python is a more flexible tool than Matlab or Perl, but I can't help but come away with the sense that the author hasn't tried many other languages. There are an awful lot of languages that provide iterators, a powerful set of data structures, extensive libraries and facilities for structuring and maintaining large codebases. .Net languages (maybe F# would be good for this?), Java or most of the emerging languages for the JVM stack, Ruby (which is generally considered to be "different but equivalent" to Python), and so forth.

vog 16y ago
> There are an awful lot of languages that provide ... Still, how many of them have a fast interactive interpreter ("command line") with a decent usability? How many of those provide good libraries for numerical as well as symbolic math? With an API that is easy to write, to understand and to extend? Python may not be the only language with those qualities, but there aren't many languages (and ecosystems around them) which can compete on all those areas. Python seems to be one one the few "best fits" for scientific applications.
- Dn_Ab 16y ago
  There are Ocaml, F# and Clojure with some combination of great tools, speed (clojure addressed this recently I think but i only have visual experience with clojure), light syntax, books and documentation, repl,excellent platform, wide libarary choice and or decent interop with C. Also F# is doing some really cool stuff to do with datasets awareness in the language. Haskell seems a perfect fit for mathematical use and while I haven't used it in a couple years, I would hesitate to suggest it due to a lack of mature library options, difficulty of FFI and perhaps a steep initial curve. Scala is a good language for an entire application but provides too much scaffolding for scientific applications. R is fairly widely used but is also itself very quirky.
  - thesz 16y ago
    I am with you on Haskell. It counts on Freedom, Readability, Documentation System (including lhs2tex which will turn your "integrate f 0 a" into \Int_0^a{f dx}), High-level vs low-level, Standard library (including hackage/cabal), Data structures, Module system, Calling syntax, Default arguments (currying), Multiple programming paradigms (there was a saying that Haskell is the best imperative language). It partially counts on most other points. Myself, I choose Haskell for my research project, as it was best language on (expressiveness times safety) scale. Strong type system certainly helps sweeping out errors.
  - wynand 16y ago
    I'm a long time Python programmer and I am currently using it for scientific programming (for my PhD). I've shopped around a lot for alternatives to Python and looked at Ocaml, F#, Boo, Clojure, Common Lisp, Haskell and Scala. There are a couple of things still keeping me with Python: * The REPL, especially IPython, * Numpy & Scipy, * Networkx & igraph, * jpype for fairly seamless JVM integration (this way, I can interact with Cytoscape), rpy for fairly seamless R integration, * ZODB, * IPython's parallel processing framework. The .net environment probably comes the closest to providing everything here, but the REPLs need a lot of work and the graph libraries have more complicated interfaces which make them a pain to use on the command line. Python is not without its warts but when I recently had to spend time with Matlab again after a few years, I was reminded of just how nice the Python ecosystem is in comparison. Update: I should add that if you rely on one of Matlab's toolboxes, you might not find any decent alternatives outside of Matlab. You can always use a bridge like MLabWrap to access Matlab from Python.
mechanical_fish 16y ago
I only scanned the article very briefly, but my impression is that the important comparison is vs. Matlab. The other players aren't really in the author's game. It's a question of the use case and the community and the library support. In theory, .Net could displace Matlab or Python as the canonical platform for scientific researchers. And in theory Python could displace PHP as the canonical platform for classic CRUD web apps. In practice neither is likely to happen, no matter how much we might or might not wish it to.
- hogu 16y ago
  I've been a Matlab user for 10 yrs, recently switched to python because of the increased flexibility. never going back to Matlab ever again. Python math and science libs can be a little rough around the edges, but the flexibility more than makes up for it for me.
  - samd 16y ago
    I'm a Matlab user myself, but I try to use Ruby whenever possible if for nothing else but to just get away from all the god damned matrices. I didn't realize Python had similar capabilities, I look forward to trying it out. But does Python have "Index exceeds matrix dimensions." errors? I just can't imagine life without seeing a few of those every day. That said, the workspace is really handy, is there some equivalent GUI in Python?
    - infinite8s 16y ago
      Check out EPD: http://www.enthought.com/products/ http://www.enthought.com/products/, particularly the ETS framework. It contains almost all the popular scientific/numeric libraries in python and is free for academic use (and much of it is open-source). (disclaimer - i work for Enthought).
    - timwiseman 16y ago
      As infinite8s points out, EPD from enthought is really quite nice. Personally, I rather like Python XY ( http://www.pythonxy.com/ http://www.pythonxy.com/ )though. It is totally free, open source, and for small little scripts I am a huge fan of the Spyder IDE. It lacks some of the features of bulkier IDEs so I also use Eclipse from time to time. But Spyder is light and much faster than Eclipse with every feature I would want when working on small projects that can be contained in just one or two files.
wwortiz 16y ago
Do they all have equivalent libraries to python's scipy and matplotlib? I think that is why the author could move to python as these provide a pretty large subset of what matlab has and makes the transition less problematic. Java probably does and .NET may have something like this but I don't know of them or their amount of documentation.
- kenjackson 16y ago
  .NET has various things in this space, but I think a lot are commercial products. .NET is not used as much in academia, so most things are targeted at professionals. With that said, you can probably get free/cheap licenses for many/most of commercial products. But I think Python probably has the most active community as its been used as glue for a long time in this space.
bad_user 16y ago
SciPy, NumPy, Matplotlib, Cython, Sphinx, PIL. The article should've also mentioned NLTK. IMHO, languages matter less than the available libraries, and in my experience only Java matches the depth of the Python ecosystem. That Python is a nice language to work with, that's just a bonus.
- ekidd 16y ago
  I agree about Python's libraries. Normally, I'm a Ruby programmer, because I like Ruby's syntax and the Rack/Rails/Sinatra web stack. But if I need to do scientific or linguistic programming, Python is absolutely amazing. SciPy, NumPy, Matplotlib and NLTK support a rich and deep ecosystem. And it's vastly nicer programming environment than Matlab and Octave. (Of course, GNU R is also pretty useful if you're doing pure statistics.)
  - adw 16y ago
    Which is why we use Python. We're a web app (Django) doing big data aggregation/processing (Numpy).
  - ajmurmann 16y ago
    Same here. I used to do a lot of Python programming, but moved to Ruby because of Rails. When I had to work on a machine learning paper, I spent weeks looking for a equivalent to NumPy or Matplotlib in the Ruby world, but nothing comes even close!
- osdf 16y ago
  Also, (basic) GPU operations with cudamat (or even finer, gnumpy) at your fingertips.
variety 16y ago
actually, he doesn't make any assertions that Python is more flexible than Perl (which would be rather doubtful), only that it is more readable (which, as a perlista, I'm sad to say is probably true). but I also get the sense that this is the first time he's seriously delved into a dynamic programming language. much of what he's saying about Python is exactly what bioinformaticists were saying about Perl in the late 90s / early naughts.
- expeditious 16y ago
  The thing about readability of Perl vs. Python ... I dunno. I don't think it's the syntax itself that causes Perl to have some readability issues. Stuff like `$foo =~ s/bar/baz/g` or `join '-', split( q{ }, $a_str)` is very readable. Context and corner cases maybe makes readability suffer. For example, what does `m///g` do in scalar vs. list context? I don't recall.
  - variety 16y ago
    you might say that Perl's lack of (obvious, first-pass) readability is a two-headed dog from hell: - it's a (very) good thing, in that there's -a- -lot- of expressive power hiding in those corner cases (esp list v. scalar context). in fact perl is nearly unique in the degree to which it embraces, rather than seeks to quash the potential for nuance and flexibility to be found in the various eddies and whirlpools that lie "between" non-whitespace elements of code. this is something that goes to the deepest intellectual roots of the language (to be more like the way the human brain thinks, rather than the way machines think). - it's a (very) bad thing, in that the same "power" for expressiveness just represents little more than an endless stream of banana peels to most first time users (understandably sending many of them running into the arms of the other major dynamic programming languages). and that it just makes it too damn easy to write (barely) functional, pothole-laden frontline scripts (such that Perl is partly responsible for "scripting" being such a dirty word, in many quarters). another aspect people don't like to talk about its nearly coprophilic fondness for not just nuanced and context-sensitive, but intentionally obtuse syntax (the impossible to remember "special" variables such as $[, $;, $' etc being probably the worst examples). this also (sadly) has a lot to do with the community's deference to the aesthetics of its original authors (and is also plenty understandable, given the extent to which unabashed ugliness -- as personified by makefiles, shell languages and macro-laden C and C++ -- ruled the day at the time). unfortunately it also blinds a lot of Perl folk to the fact that beloved muse just happens to look awful darn cluttery (or worse) to many reasonably intelligent people who come from more "modern" programming backgrounds (or who at least came onto the scene comparatively recently).
hyperbovine 16y ago
It's all about Numpy. Mayavi, IPython, mlab & friends are great for when I need to plot things or look at data, but Numpy is the workhorse I keep coming back to day in and day out. And also, the thing I wish for most when I have to use other languages. The combination of the speed of C and the elegance of, well, anything that is not C, is hard to beat. Once you get down the basics of array broadcasting, types, etc., it's possible to do some amazingly elegant things in Numpy, and quickly too. The numpy library has seemingly every array function I have ever wanted. If I were Matlab I'd be scared :-)
crocowhile 16y ago
None of the languages you mention is easy enough to be grasped in a week from someone who never did programming in their life. Believe me: most of the people who suddenly has to do data analysis for their phd or postdoc projects hardly know how to use excel, so easy of use is essential. Also, you want a non compiled language. Most of the time you do interactive programming and change parameters on the fly, according of the result of the analysis. Finally, matplotlib, one of python's most complete graphic library, is a breeze to use. Making graphs in an interactive way with java or .net is simply impossible. I am a neuroscientist and most people in the field use Matlab. I use python (in fact I use ONLY open source software, by choice). It's amazing how many advantages python gave me on my daily life.

agconway 16y ago

Python rocks, but Python + R + bash rocks way harder for research

levesque 16y ago
R is definitely powerful and a good part of any scientific data analysis toolkit. I use python, ipython, matplotlib, numpy and R. I call my R scripts directly from python using rpy.
- wildanimal 16y ago
  Agreed. I use Python for heavy shell-scripting and text-processing (though R surprisingly does have respectable facilities for all but the most overwhelming of these tasks) and R for the rest of the analysis. I've thought about switching to NumPy/SciPy as it's part of Python to integrate everything, but R's data frame, factors, and reshape, plyr, and lattice packages makes you think very differently about how to approach the data - and hard to go back to lower-level manipulation of arrays; not to mention all the stats/graphics packages which are very easy to install and apply. And documentation of its functions is superb.

apl 16y ago

In the end, it's all about the ecosystem. Perl wins for bioinformatics because there are boatloads of scientists already using it, with all the neat libraries and resources that brings. Equally, Python wins for, say, prototyping in robotics because of libraries, support and so on. There's nothing intrinsically science-apt about Python/Perl, but Ruby and friends can't compete when it comes to the programming environment; that's what counts.

cageface 16y ago
As a language I prefer Ruby but the Python ecosystem for this kind of thing is definitely a huge advantage. You can actually do quite a bit with Ruby + GSL but it's still not really competitive.
- apl 16y ago
  Yep, same with me. I do prefer Ruby, but it's just not feasible at the moment. Languages that get behind in a certain area enter a terribly vicious circle: There's no scientific community backing Ruby, so why would I develop something that'd make Ruby more competitive?
  - djacobs 16y ago
    I'm in the same boat and it seems like several others as well. I'm fine moving to Python for the moment, but I wonder how the Ruby community will ever know if there's enough of a demand for scientific tools to merit their development?
    - phren0logy 16y ago
      I prefer Ruby generally, but honestly Ruby and Python are sufficiently similar that I'd rather smart programmers put their time to good use doing something other than reimplementing Python's science libs. I think that making some tools available in a totally different language (maybe something functional) would be much more useful, because it would allow for a very different approach to the problem if needed. In a perfect world, we could also have the option of using light wrappers around OpenCL matrix libraries, and push the linear algebra to GPUs that eat matrices for breakfast.

ogrisel 16y ago

And the corollary: Why do researchers never respect the PEP8 when they write python code? Yes I am a bit overreacting since the blog post is very well written and I actually agree 100% with the content. But please people: respect the PEP8 [1]. It makes your readers feel at home while reading your code. It is very important if you want to get new contributors to your project. See [2] for instance. [1] http://www.python.org/dev/peps/pep-0008/ http://www.python.org/dev/peps/pep-0008/ [2] http://www.dataists.com/2010/10/whats-the-use-of-sharing-code-nobody-can-read/ http://www.dataists.com/2010/10/whats-the-use-of-sharing-cod...

hogu 16y ago
I wasn't aware of pep8 when I started, most science people arrive at python from a different path. What I mean is, for a long time I knew much more about numpy than about python itself. there are some things in pep8 that are bad for science, the spaces around operations, and also the 80 chars to a line... scientific expressions are often long and complicated, yes you can do it while adhering to pep8, but its kind of a PITA
- ogrisel 16y ago
  The 80 chars limit has a justification: expressions that don't fit on 80 chars (or two lines using parens) are not readable anyway. In such a case temporary variables with meaningful names would both help respect the 80 chars constraint and make the expression easier to understand by the reader. Furthermore having 80 chars is great to have vertically split editors with the code on one panel and the tests or the documentation on the other panel.
  - Goladus 16y ago
    > The 80 chars limit has a justification: expressions that don't fit on 80 chars (or two lines using parens) are not readable anyway I disagree. There are many cases, especially after 2-3 levels of indentation, where 80 characters is an unreasonably narrow space. I don't have a strong preference for reading code within 80 characters. And I'd much rather comments use 80 characters plus indentation rather than worry about whether I've got my screens vertically split. > Furthermore having 80 chars is great to have vertically split editors with the code on one panel and the tests or the documentation on the other panel. Having lines here or there that go beyond 80 characters doesn't completely prevent you from doing this, and having an entire statement on a single line of code makes line-based tools like grep or kill-line more effective. Limiting to 80 characters is a good idea, but it's easy to see that there are a few significant tradeoffs, and that someone trying to get something done is not going to want to bother.

sintesoro 16y ago

Python is good, you could also consider Maxima. A single example: f(x):= x^2+3x+7; Maxima provides: Symbolic computation, blas and laplack integration for numeric algebra, 500 pages manual in several languages, a complete library for statistics, differential equation, calculus, series. Graphics with matplotllib. Also maxima language is not much complicate that python: for i in range(10):print ii versus for i:0 thru 9 do print ii; [i2 for i in range(10)] versus makelist(i*2,i,0,9) But Matlab libraries are greater than python and maxima.

dagw 16y ago
Python has sage and sympy for doing symbolic math. Although admittedly they're quite primitive compared to maple and mathematica (haven't used Maxima, so I can't really compare)
- koenigdavidmj 16y ago
  Sage actually includes Maxima, for ultimate convenience.
kwantam 16y ago
Maxima rocks for symbolic math. I prefer it to Mathematica, which is saying a lot. In contrast, octave always feels like "almost-Matlab" and I still prefer the latter. Also see wxMaxima, which will (among many other things) produce LaTeX for you.
- nurbl 16y ago
  If you haven't already, you might also want to check out Sage (http://www.sagemath.org/ http://www.sagemath.org/) It uses python to "glue" together other free math tools (e.g. Maxima) into a unified system with a nice interface.

woodson 16y ago

Often languages are chosen based on already existing tools used in a specific research project. That's why it's good to be able to quickly pick up new languages. For example scripting languages integrated in some frameworks, like Scheme in the Festival speech synthesis system. In the end, this often results in projects involving things like Python, Scheme, bash, R and a bit of tcl ;-).

rdouble 16y ago

I've wasted most of my professional life tweaking various unix software to make it work. However, the typical scientific python setup proved to be too frustrating to install on OSX. The recommended solution is to just buy the Enthought distro. If I'm paying for software anyway, why is Enthought better than Matlab?

wgrover 16y ago
I've never had any problems putting a research-grade python setup on OSX - just use the .dmg installer files files available for python (2.6.X for compatibility), numpy, scipy, and matplotlib. But I agree that the Enthought Python Distro is also a good alternative (if a little bloated for my needs), and it's also free for academics.
- osdf 16y ago
  For a basic setup on OS X, I found scipy superpack to be an excellent choice: http://stronginference.com/scipy-superpack/ http://stronginference.com/scipy-superpack/
  - gkelly 16y ago
    I've found macports provides a ton of python libraries, like numpy, scipy, and matplotlib, all easily installed from the command line.
- wildanimal 16y ago
  When I tried to use easy_install with the EPD it tried to take me to their repository, for which we don't have appropriate permissions? So I sucked it up and installed some from .dmg's and others from source (matplotlib was difficult). And I found out just because it's Python doesn't mean it's portable - for instance NumPy/SciPy is hardly ever installed on other machines (it's not trivial, as is written here many times), and I find that administrators won't update their Python installation so I've had to re-write bits of my code to accommodate Python 2.3-2.4 at times! But all said, it's a nice tool and hopefully these installation/ version issues will work itself out.

miloshasan 16y ago

Python is awfully close to being a superior (and free) replacement for Matlab, but there are a few annoyances that keep preventing me from switching forever. Unfortunately, these are mostly not bugs but bad design that is believed to be correct by the core developers, so it is unlikely to ever change: - Matrices are a pain. The r_[] and c_[] operators could be a reasonable replacement for Matlab's elegant matrix construction syntax, but they do not work as expected (as smart hstack and vstack), instead doing something completely different and inconsistent for vectors and matrices. - Tensors are a bigger pain. Matlab has a very well-defined semantics for operations like permute and reshape; in NumPy these operations sometimes create just a view, at other times they reshuffle the memory contents. I know the idea was to "protect" the user from having to know the memory layout of data, but this idea is bad. - Ipython is great in every way except when it comes to reloading parts of your program. After any tiny change to your code, the only safe thing to do is to quit ipython and start it again. All the other options (run, reset, reload...) make some secret and wrong assumptions on what you want to reload. In contrast, this works flawlessly in Matlab.

deleted 16y ago
[deleted]

ansgri 16y ago

One more powerful combo is R+Java+C. UI, integration and massive data processing in Java, R for prototyping, plotting and model fitting, and C if you have numerical simulations.

maurits 16y ago

I feel this article is somewhat unbalanced in its single minded rejoice for a certain tool/environment. So in the same spirit here come a couple of reasons not to switch from Matlab to Python, all stemming from my experience when I decided to try to switch from Matlab to Python/C - installing all these packages on (any) system is painful. Different versions don't play together or don't work (yet) on some platform and or architecture. This stems from my own experience of getting a version of python to work with numpy, scipy, matplotlib, opencv and PIL on a windows, mac, and linux machine. No 100 percent success yet on any platform. - central and consistent documentation. Even for very simple cases, I got a bit of a headache. I encounter a python print statement for the first time that obviously differs somewhat from its c printf cousin. I google "python print syntax" only to find that the first xx hits, including the official documentation, do not cover the full specification of this statement. I fear the moment I might actually need detailed information on something less trivial. - Numerical integration is more accurate in Matlab. - Visualization capabilities of matlab are more powerful. But who knows, perhaps there is yet another package floating around :-) - Matlab may not have advanced data-structures, but it is a rapid prototyping tool, for testing ideas. If I need to write an actual application, I will use a tool and language geared for that task.

xiongchiamiov 16y ago
> central and consistent documentation. Even for very simple cases, I got a bit of a headache. I encounter a python print statement for the first time that obviously differs somewhat from its c printf cousin. I google "python print syntax" only to find that the first xx hits, including the official documentation, do not cover the full specification of this statement. I fear the moment I might actually need detailed information on something less trivial. http://docs.python.org/reference/simple_stmts.html#the-print-statement http://docs.python.org/reference/simple_stmts.html#the-print... lists eveything about `print` - it takes stuff in, converts 'em to strings, and sticks in on stdout. It doesn't refer to prinf-like formatting because that's for strings in general. If you weren't aware of this, you probably should have been going through a basic Python tutorial, rather than just jumping into the middle of things. Python's documentation is the best I've encountered so far, and I find good docs to be an important value in the community, as well. I guess YMMV, though.

chbarts 16y ago

As usual, Common Lisp has many of the same features. The only thing it doesn't really have now is the same ecosystem as Python.

b_emery 16y ago

> In MatLab everything is flat – all functions are declared in the global namespace. However, this discourages code reusability by making the programmer do extra work keeping disparate program components separate. In other words, without a hierarchical structure to the program, it’s difficult to extract and reuse specific functionality. I completely disagree. Reusable Matlab code has been my holy grail for the last couple of years. The key is to break out specific functionality as subfunctions. When these are abstracted and generally useful elsewhere, then they become new tools for the toolbox. The subfunctions also make great starting points for repurposing code. This layout results in much less work.

tel 16y ago
It's absolutely true that using more functions makes your code more maintainable. It's also absolutely true that nearly every other language on the planet does this better than Matlab.
- snth 16y ago
  Yes; in particular, Matlab's restriction of ONE outside-callable function per file is a huge pain.

woodson 16y ago

For those interested in scientific frameworks and existing packages: Scientific track of EuroSciPy 2010 [1]. Everything from seismology to visual programming. http://www.euroscipy.org/track/870 http://www.euroscipy.org/track/870