10 ms·

Python, Machine Learning, and Language Wars. A Highly Subjective Point of View

167 points by Lofkin 11y ago

zzleeper 11y ago
Quite interesting post. I feel that a lot of the numerical Pythonistas are in the same spot: They tolerate most languages, but find R's syntax a bit unnatural, Matlab lacking when trying to go beyond pure matrix stuff, and are waiting to see if Julia picks up (which it seems to be from what I can tell)
- GFK_of_xmaspast 11y ago
  One of the hats I wear is 'does numerics in python'. R is fine when you're playing to its strengths, matlab is an abomination, and I have absolutely no interest in julia.
  - sgt101 11y ago
    Why no interest? If it is something that could make you more productive and make your life better then shouldn't you take an interest? Or do you mean "I have looked closely at Julia and it's no good because..." If so the because bit will be of interest to the Julia community (it's 0.4 now so lots of distance to go in its development before it becomes stable)
    - GFK_of_xmaspast 11y ago
      I took a look at Julia a couple years ago. As far as I could see, the only reason to consider it over python is the speed improvements, and if I'm in a situation where python isn't fast enough, I've already got c, c++, and java to reach for. (Also at the time the library support was lacking (I just checked and the graphs package, which I needed at the time, is still pretty minimal)).
Carou 11y ago
https://i.instagram.com/prin_mcgui/ https://i.instagram.com/prin_mcgui/
Carou 11y ago
https://i.instagram.com/prin_mcgui/ https://i.instagram.com/prin_mcgui/
idunning 11y ago
As someone who almost exclusively uses Julia for their day-to-day work (and side projects), I think most of the author's thoughts about Julia are correct. I think the language is great, and using it makes my life better. There are some packages that are actually better than any of their equivalents in other languages, in my opinion. On the other hand, I've also got a higher tolerance for things not being perfect, I can figure things out for myself (and luckily have the time do so), and I'm willing to code it up if it doesn't already exist (to a point). Naturally, that is not true for most people, and thats fine. The author isn't willing to take the risk that Julia won't "survive", which is fair. Its definitely not complete yet, but its getting there. I am confident that it will survive (and thrive) though, and continue growing the not-insubstantial community. I have a feeling the author will find their way to Julia-land eventually, in a couple of years or so.
- rasbt 11y ago
  Thanks for the comment (I am the author of this article). > I have a feeling the author will find their way to Julia-land eventually, in a couple of years or so. I have a strong feeling that this will eventually happen :). In an ideal, less busy, world, I would love to use Julia alongside to explore and battle-test it further. Or even develop useful packages, libraries, and functions for it. The truth is, I am currently lacking the time to do that :(. I mean, Python works for me, and I am currently more into the scientific problem solving so that I don't have the time :(. When I say that Python works for me I mean that I am currently happy since it can do everything for me I need, however, this doesn't mean that Julia couldn't do certain things better ;). Anyways, I really like your comment. I am wondering if you would be okay with it if I include it in a "Other people's experiences and opinions" section at the bottom. I think this would be extremely helpful for people who are new to the "data science field" -- my article is strongly biased towards Python as you noticed :P
  - idunning 11y ago
    Sure, go ahead!
  - dnautics 11y ago
rm999 11y ago
I switched from mostly using R to Python about a year ago for gluing together my data pipeline (from data source all the way to production models and frontends/visualizations). It hasn't really impacted what I'm capable of doing or my productivity, except the standard extra googling that comes in the first couple years I use any language. The main reason I went for Python is purely practical: it's a language people outside my team will respect and deal with. It makes it easier for me to collaborate in many different ways: share tools with other teams, transfer ownership of my code, get help when I need it, etc. Data science at some companies has the reputation of "hack something together and throw it over the wall for someone else to deal with". In my experience R only furthers this reputation. Which is too bad, it's really great at what it does.
- rasbt 11y ago
  I went through the very same process :). I really like your comment, you highlight something that I forgot to mention in this clarity "It makes it easier for me to collaborate in many different ways: share tools with other teams, transfer ownership of my code, get help when I need". Would you mind if I add it as to a "other people's experiences" section at the bottom of the article?
  - rm999 11y ago
    Definitely feel free to add that!
- geomark 11y ago
  The part about sharing makes a lot of sense since Python use is so wide spread. The throwing-over-the-wall effect isn't a language specific issue, more of a work culture issue. Seems to me if you practice "literate programming" with R markdown you can greatly improve the sharing aspect and reduce the throw-it-over-the-wall issue.
  - srean 11y ago
    Totally agree about it being a cultural thing, have first hand experience at some of the usual suspects. I have a far more cynical label for it: deliver your turd (typically formed in MATLAB) for someone else to polish. It is surprising how common this is in some places and groups. I mention groups because when the group moves from one place to another it brings that turd polishing culture along.
deleted 11y ago
[deleted]
sampo 11y ago
Andrew Ng said in the Coursera Machine learning class that according to his experience, students implement the course homework faster in Octave/Matlab than in Python. But yes, the point of that course is to implement and play around with small numerical algorithms, whereas the linked blog is about someone who mainly calls existing machine learning libraries from Python. Ref. https://news.ycombinator.com/item?id=4485877 https://news.ycombinator.com/item?id=4485877
- rasbt 11y ago
  Interesting! It's hard for a single person to tell, because you can only start one way digging into machine learning. However, from a teacher's perspective, this is a useful observation. I also started with Matlab since it was the language that was used in my classes. However, I think this is also a little bit context dependent: For someone who has never programmed before, Matlab may be more intuitive. I think in an ideal world, you should let your students choose what language they want to use to solve the problem :).
  - jordigh 11y ago
    For someone who has never programmed before, Matlab may be more intuitive. Yes, this is Matlab's target audience: programmers who will not call themselves "programmers". In recent versions, they have tried even harder to hide the code away from the user, by trying to make everything work by clicking on buttons. I have heard from many Matlab users call themselves "not a programmer". They don't feel like writing software is what they're doing when they're using Matlab.
    - noobermin 11y ago
      On balance, I've met people who claim they are intermediate programmers and use matlab only, and are freaked to shit when they see higher order functions in other languages, or even the idea of passing a "function", like, say, a pointer to a function, into another function, even though function pointers have existed since C.
misiti3780 11y ago
Octave/Matlab are "great" but good luck trying to integrate them into a production web application. Since you cant really do that - avoid using them unless you are fine with implementing the same algorithm twice. Matlab licenses cost money also, and the toolboxes cost additional money. R is useful because there are a lot of resources as it has been along for so long and is used by a large portion of the stats community. It also has a lot of useful libraries that have not been ported over to other languages yet (ggmap!!!). But you still still run into the same problem that you cannot integrate R into a production web application. I am pretty sure Hadoop streaming does not support R,Octave, or Matlab either
- jordigh 11y ago
  Octave/Matlab are "great" but good luck trying to integrate them into a production web application What problems are you facing with Octave? It has, in fact, been integrated into a couple of production web applications I know of: https://octave.im/ https://octave.im/ http://octave-online.net/ http://octave-online.net/ https://www.rollapp.com/app/octave https://www.rollapp.com/app/octave I have promised a while ago to improve its Python integration so that Python and Octave can be in the same process (there are lots of advantages to that kind of tight integration instead of relying on parsing output through pipes). Perhaps that could help you?
  - misiti3780 11y ago
    wow - i have never seen this. thanks for the links! (i take back my octave comment). that would help me
- RA_Fisher 11y ago
  I'd like to kindly challenge the notion that you can't integrate R into a web application. I've started using R to power jobs that are used by a large web application. The R packages httr or RCurl make it pretty easy to make http requests -- (enabling me to send things to a web server to be consumed into a database and run by back-end code). It's also possible to prepare data in R and then send to a space like S3 with a System("s3cmd sync some-data s3://some-data") call. I've also been using Python a good bit lately. I don't see either has having a universal advantage for a data pipeline.
DrNuke 11y ago
I love the hacking approach in the post: a tool is only a tool to do something valuable and not the goal itself. The Python ecosystem is the right tool at the right time, nowadays, because of the data science explosion and the need to interact very quickly with non-specialists.
Adam_O 11y ago
From the perspective of a student, most of the good online analytics/data analysis/stats courses use R, so it is hard to get away from it while learning the material. Once you get the base concepts down, switching to python shouldn't be hard. I think most people still prefer ggplot2 for visualization though. Whenever I use R I feel like a statistician, I can feel that 'cold rigor' emanating from the language. But in the end I think it is advantageous to wield both languages. Also I really see Jupyter as a new standard for communication. Your narrative and supporting code all in one place, ready for sharing.
- rasbt 11y ago
  Yes, I think you are right. Out of curiosity, when I browsed over Coursera's course catalog, most data science related material seems to be taught in Matlab or R (however, there are also others, e.g., Klein's Linear Algebra class in Python). Personally, I think that instructors shouldn't enforce a language requirement. I believe for big platforms such as coursera it shouldn't be to hard to run an respective interpreter to check the code/answer uploaded by a student.
  - digitalzombie 11y ago
    Most classes have to teach the subject and how to program. Programming is beginning to be an essential skills so they have to choose a language to teach. Also those classes that chose R, from my experiences, are non CS classes, the professor are from other discipline. They just want a tool that solve their need quick. An example is the Princeton's Stat class, the professor is a humanity major. The class gave us tons of data and we had to do ANOVA and such and we needed a computer to crunch so number can't do by hand. So he chose R which he uses a lot.
- IndianAstronaut 11y ago
  Jupyter is amazing. It is great to see workflow and makes it easy to graph and find trends in the data, as well as mistakes in a data flow.
thanatropism 11y ago
One thing missing here: Matlab syntax is actually very close to modern Fortran. At least twice I've written Fortran code (for Monte Carlo simulations; different contexts) by overwriting Matlab code adding types / general verbosity / fixing the syntax of do-loops / etc.
- jordigh 11y ago
  They've been trying to Javaify the Matlab syntax for close to two decades now. They're moving towards making everything an object like in Java. They're getting pretty close to that.
  - sgt101 11y ago
    One of the lessons of Julia for me is that everything is an object is a problem, not a boon. I think organizing code in modules with sets of related types and functions manipulating those types allows more natural and modular decompositions for reuse.
- xixi77 11y ago
  This is a very good point actually, I have done the same a few times. Of course this works well until you get to code heavily dependent on toolbox functions. Which leads to another advantage of Matlab (at least as long as you don't have to pay for it) -- the documentation (including toolbox documentation) is way ahead of any competition. Whoever came up with the R package documentation standard has done that language a great disservice :( IMO, the fact that every R package includes a huge pdf with alphabetical listing of functions and data sets is actively harmful: without it, perhaps more package authors would at least feel compelled to write a 2-page readme.txt (just like various matlab package writers do), and that would have been actually useful.
geomark 11y ago
I just completed the Coursera data science track which took me from a complete R newbie to being at least somewhat proficient. Having previously used Python for a quite a bit of web programming, I disliked R at first except for its power in statistical programming. But I've since discovered a number of great R packages that make it a pleasure to use for things I would normally turn to Python for. Like I recently discovered the rvest package for webscraping. Data visualizations with R seem vastly superior, unless I am missing something with Python (highly likely). And putting up a slick statistics app is easy with shiny or RStudio Presenter. But R can't really scale to a large production app, isn't that right? So I feel I need to keep working with both Python and R. Added: That's a nice list Lofkin. Thanks. Also, in the article he says that Python syntax feels more natural, which I also felt. But then I started to use things like the magrittr and dplyr packages in R which gives you nice things like pipes and that feeling starts to ebb.
- Lofkin 11y ago
  For stats plotting in python: https://github.com/mwaskom/seaborn https://github.com/mwaskom/seaborn https://github.com/yhat/ggplot https://github.com/yhat/ggplot For stats plotting and web apps in python: https://github.com/bokeh/bokeh https://github.com/bokeh/bokeh For calling r libraries in python: https://pypi.python.org/pypi/rpy2 https://pypi.python.org/pypi/rpy2 For out of core datasets in python: https://github.com/blaze/dask https://github.com/blaze/dask https://github.com/blaze/blaze https://github.com/blaze/blaze
  - rasbt 11y ago
    Nice collection, let me add one more item to this list: Seaborn: statistical data visualization: http://stanford.edu/~mwaskom/software/seaborn/ http://stanford.edu/~mwaskom/software/seaborn/
- dswalter 11y ago
  You bring up a good point in favor of R: Hadley Wickham and the rest of the RStudio people. Packages like {ggplot2,rvest,dplyr,devtools, etc.} are basically creating a sub-language for R. I use both at the moment, but I echo the OP's ideas that R's target audience is statisticians, where Python's target audience is broader and includes statisticians and computer scientists. And Python's syntax is nicer to work with. That's why it's become the primary glue language. That said, the overhead for learning Python as your first data science language is a bit problematic for me, as you basically have to learn Python followed by Python's data science tools (pandas, matplotlib, etc). whereas with R, you're learning the language and the data science tools at the same time, even if they're a bit idiosyncratic.
JuliaLang 11y ago
Julia love!
Lofkin 11y ago
Personally I'm tempted to make the switch to Julia, but slow higher order functions, high churn in the core data infrastructure and no Pymc 3 are keeping me on pydata for a bit longer. I have numba to hold me over.
a_bonobo 11y ago
>I think it [Perl] is still quite common in the bioinformatics field though!? That's true - many day-to-day tasks in bioinformatics are more or less plain-text parsing [1], and Perl excels in parsing text and quickly using regular expressions. "My" generation of bioinformaticians doing data cleanup and analysis (20-30) uses Python, sometimes because plotting is nicer, the language is easier to get into, it's more commonly taught in universities, or other reasons - people older than that normally use Perl. Both BioPython and BioPerl are extremely useful. [1] Relevant quote from Robert Edgar: "Biology = strcomp()" from https://robertedgar.wordpress.com/2010/05/04/an-unemployed-gentleman-scholar/ https://robertedgar.wordpress.com/2010/05/04/an-unemployed-g...
- rasbt 11y ago
  Thanks for the insights! Also here, this comment would make an interesting addition to a "Feedback" section at the end of the article to give people a broader view on this topic. May I have your permission to post your comment below the article?
  - a_bonobo 11y ago
    Of course you have my permission :)
- dalke 11y ago
  (Minor typo corrections; "Biopython", not "BioPython" and "strcmp" not "strcomp". I co-founded Biopython. We chose to avoid CamelCase.)
  - a_bonobo 11y ago
    Sorry about that, that's what happens in caffeine-less typing - I sadly can't fix it at this point anymore, edit is disabled now
leni536 11y ago
And there is nothing wrong with C++. For linear algebra I use the armadillo library and it's really a nice wrapper around LAPACK and BLAS (and fast!). For some reason scientists are somewhat afraid of C++. For some reason you "have to" prototype in an "easier" language. Sure, you can't use C++ as a calculator as opposed to interpreted languages, but I see people being stuck with their computations at the prototyping language and eventually not bringing it to a faster platform. Point being: C++ is not hard for scientific calculations.
- rasbt 11y ago
  I agree with you. However, note that many people who are using Python for writing scientific code make use of C/C++ in one way or the other (aside from NumPy, SciPy, and Theano). For example, many people write the "most intensive" computations down in C/C++/Cython if they call those functions frequently -- Python becomes a wrapper. One example that pops into my mind is khmer (https://github.com/dib-lab/khmer https://github.com/dib-lab/khmer)
  - RA_Fisher 11y ago
    I believe S (ancestor of R) started as C glue at Bell Labs. I've heard it said a few times that R is slow, but that doesn't really make much sense if your bottleneck routines are R calls to C++.
  - srean 11y ago
    This is a common refrain: drop down to C, C++, Fortran for the computation intensive parts. It works, but only to a degree. The inefficiencies lie in the vectorization semantics of the host language(s) that leads to extra copies and extra levels of indirection. So this dual language mode of operation typically does not approach what one could have obtained had one disposed the baggage entirely, except for I/O. Usually in the quest for better speed, that is what remains, as one moves progressively larger portions of the application in the C, C++, Fortran part of the code. A reason I like Julia is that I can largely avoid this dual language annoyance, and enjoy the succinctness of pithy vectorized expressions using https://github.com/lindahua/Devectorize.jl https://github.com/lindahua/Devectorize.jl
dafrankenstein2 11y ago
.NET's F# is also good..though maybe not a better alternative

167 points by Lofkin 11y ago

Quite interesting post. I feel that a lot of the numerical Pythonistas are in the same spot: They tolerate most languages, but find R's syntax a bit unnatural, Matlab lacking when trying to go beyond pure matrix stuff, and are waiting to see if Julia picks up (which it seems to be from what I can tell)

GFK_of_xmaspast 11y ago
One of the hats I wear is 'does numerics in python'. R is fine when you're playing to its strengths, matlab is an abomination, and I have absolutely no interest in julia.
- sgt101 11y ago
  Why no interest? If it is something that could make you more productive and make your life better then shouldn't you take an interest? Or do you mean "I have looked closely at Julia and it's no good because..." If so the because bit will be of interest to the Julia community (it's 0.4 now so lots of distance to go in its development before it becomes stable)
  - GFK_of_xmaspast 11y ago
    I took a look at Julia a couple years ago. As far as I could see, the only reason to consider it over python is the speed improvements, and if I'm in a situation where python isn't fast enough, I've already got c, c++, and java to reach for. (Also at the time the library support was lacking (I just checked and the graphs package, which I needed at the time, is still pretty minimal)).

Carou 11y ago

https://i.instagram.com/prin_mcgui/ https://i.instagram.com/prin_mcgui/

Carou 11y ago

https://i.instagram.com/prin_mcgui/ https://i.instagram.com/prin_mcgui/

idunning 11y ago

As someone who almost exclusively uses Julia for their day-to-day work (and side projects), I think most of the author's thoughts about Julia are correct. I think the language is great, and using it makes my life better. There are some packages that are actually better than any of their equivalents in other languages, in my opinion. On the other hand, I've also got a higher tolerance for things not being perfect, I can figure things out for myself (and luckily have the time do so), and I'm willing to code it up if it doesn't already exist (to a point). Naturally, that is not true for most people, and thats fine. The author isn't willing to take the risk that Julia won't "survive", which is fair. Its definitely not complete yet, but its getting there. I am confident that it will survive (and thrive) though, and continue growing the not-insubstantial community. I have a feeling the author will find their way to Julia-land eventually, in a couple of years or so.

rasbt 11y ago
Thanks for the comment (I am the author of this article). > I have a feeling the author will find their way to Julia-land eventually, in a couple of years or so. I have a strong feeling that this will eventually happen :). In an ideal, less busy, world, I would love to use Julia alongside to explore and battle-test it further. Or even develop useful packages, libraries, and functions for it. The truth is, I am currently lacking the time to do that :(. I mean, Python works for me, and I am currently more into the scientific problem solving so that I don't have the time :(. When I say that Python works for me I mean that I am currently happy since it can do everything for me I need, however, this doesn't mean that Julia couldn't do certain things better ;). Anyways, I really like your comment. I am wondering if you would be okay with it if I include it in a "Other people's experiences and opinions" section at the bottom. I think this would be extremely helpful for people who are new to the "data science field" -- my article is strongly biased towards Python as you noticed :P
- idunning 11y ago
  Sure, go ahead!
- dnautics 11y ago

rm999 11y ago

I switched from mostly using R to Python about a year ago for gluing together my data pipeline (from data source all the way to production models and frontends/visualizations). It hasn't really impacted what I'm capable of doing or my productivity, except the standard extra googling that comes in the first couple years I use any language. The main reason I went for Python is purely practical: it's a language people outside my team will respect and deal with. It makes it easier for me to collaborate in many different ways: share tools with other teams, transfer ownership of my code, get help when I need it, etc. Data science at some companies has the reputation of "hack something together and throw it over the wall for someone else to deal with". In my experience R only furthers this reputation. Which is too bad, it's really great at what it does.

rasbt 11y ago
I went through the very same process :). I really like your comment, you highlight something that I forgot to mention in this clarity "It makes it easier for me to collaborate in many different ways: share tools with other teams, transfer ownership of my code, get help when I need". Would you mind if I add it as to a "other people's experiences" section at the bottom of the article?
- rm999 11y ago
  Definitely feel free to add that!
geomark 11y ago
The part about sharing makes a lot of sense since Python use is so wide spread. The throwing-over-the-wall effect isn't a language specific issue, more of a work culture issue. Seems to me if you practice "literate programming" with R markdown you can greatly improve the sharing aspect and reduce the throw-it-over-the-wall issue.
- srean 11y ago
  Totally agree about it being a cultural thing, have first hand experience at some of the usual suspects. I have a far more cynical label for it: deliver your turd (typically formed in MATLAB) for someone else to polish. It is surprising how common this is in some places and groups. I mention groups because when the group moves from one place to another it brings that turd polishing culture along.

deleted 11y ago

[deleted]

sampo 11y ago

Andrew Ng said in the Coursera Machine learning class that according to his experience, students implement the course homework faster in Octave/Matlab than in Python. But yes, the point of that course is to implement and play around with small numerical algorithms, whereas the linked blog is about someone who mainly calls existing machine learning libraries from Python. Ref. https://news.ycombinator.com/item?id=4485877 https://news.ycombinator.com/item?id=4485877

rasbt 11y ago
Interesting! It's hard for a single person to tell, because you can only start one way digging into machine learning. However, from a teacher's perspective, this is a useful observation. I also started with Matlab since it was the language that was used in my classes. However, I think this is also a little bit context dependent: For someone who has never programmed before, Matlab may be more intuitive. I think in an ideal world, you should let your students choose what language they want to use to solve the problem :).
- jordigh 11y ago
  For someone who has never programmed before, Matlab may be more intuitive. Yes, this is Matlab's target audience: programmers who will not call themselves "programmers". In recent versions, they have tried even harder to hide the code away from the user, by trying to make everything work by clicking on buttons. I have heard from many Matlab users call themselves "not a programmer". They don't feel like writing software is what they're doing when they're using Matlab.
  - noobermin 11y ago
    On balance, I've met people who claim they are intermediate programmers and use matlab only, and are freaked to shit when they see higher order functions in other languages, or even the idea of passing a "function", like, say, a pointer to a function, into another function, even though function pointers have existed since C.

misiti3780 11y ago

Octave/Matlab are "great" but good luck trying to integrate them into a production web application. Since you cant really do that - avoid using them unless you are fine with implementing the same algorithm twice. Matlab licenses cost money also, and the toolboxes cost additional money. R is useful because there are a lot of resources as it has been along for so long and is used by a large portion of the stats community. It also has a lot of useful libraries that have not been ported over to other languages yet (ggmap!!!). But you still still run into the same problem that you cannot integrate R into a production web application. I am pretty sure Hadoop streaming does not support R,Octave, or Matlab either

jordigh 11y ago
Octave/Matlab are "great" but good luck trying to integrate them into a production web application What problems are you facing with Octave? It has, in fact, been integrated into a couple of production web applications I know of: https://octave.im/ https://octave.im/ http://octave-online.net/ http://octave-online.net/ https://www.rollapp.com/app/octave https://www.rollapp.com/app/octave I have promised a while ago to improve its Python integration so that Python and Octave can be in the same process (there are lots of advantages to that kind of tight integration instead of relying on parsing output through pipes). Perhaps that could help you?
- misiti3780 11y ago
  wow - i have never seen this. thanks for the links! (i take back my octave comment). that would help me
RA_Fisher 11y ago
I'd like to kindly challenge the notion that you can't integrate R into a web application. I've started using R to power jobs that are used by a large web application. The R packages httr or RCurl make it pretty easy to make http requests -- (enabling me to send things to a web server to be consumed into a database and run by back-end code). It's also possible to prepare data in R and then send to a space like S3 with a System("s3cmd sync some-data s3://some-data") call. I've also been using Python a good bit lately. I don't see either has having a universal advantage for a data pipeline.

DrNuke 11y ago

I love the hacking approach in the post: a tool is only a tool to do something valuable and not the goal itself. The Python ecosystem is the right tool at the right time, nowadays, because of the data science explosion and the need to interact very quickly with non-specialists.

Adam_O 11y ago

From the perspective of a student, most of the good online analytics/data analysis/stats courses use R, so it is hard to get away from it while learning the material. Once you get the base concepts down, switching to python shouldn't be hard. I think most people still prefer ggplot2 for visualization though. Whenever I use R I feel like a statistician, I can feel that 'cold rigor' emanating from the language. But in the end I think it is advantageous to wield both languages. Also I really see Jupyter as a new standard for communication. Your narrative and supporting code all in one place, ready for sharing.

rasbt 11y ago
Yes, I think you are right. Out of curiosity, when I browsed over Coursera's course catalog, most data science related material seems to be taught in Matlab or R (however, there are also others, e.g., Klein's Linear Algebra class in Python). Personally, I think that instructors shouldn't enforce a language requirement. I believe for big platforms such as coursera it shouldn't be to hard to run an respective interpreter to check the code/answer uploaded by a student.
- digitalzombie 11y ago
  Most classes have to teach the subject and how to program. Programming is beginning to be an essential skills so they have to choose a language to teach. Also those classes that chose R, from my experiences, are non CS classes, the professor are from other discipline. They just want a tool that solve their need quick. An example is the Princeton's Stat class, the professor is a humanity major. The class gave us tons of data and we had to do ANOVA and such and we needed a computer to crunch so number can't do by hand. So he chose R which he uses a lot.
IndianAstronaut 11y ago
Jupyter is amazing. It is great to see workflow and makes it easy to graph and find trends in the data, as well as mistakes in a data flow.

thanatropism 11y ago

One thing missing here: Matlab syntax is actually very close to modern Fortran. At least twice I've written Fortran code (for Monte Carlo simulations; different contexts) by overwriting Matlab code adding types / general verbosity / fixing the syntax of do-loops / etc.

jordigh 11y ago
They've been trying to Javaify the Matlab syntax for close to two decades now. They're moving towards making everything an object like in Java. They're getting pretty close to that.
- sgt101 11y ago
  One of the lessons of Julia for me is that everything is an object is a problem, not a boon. I think organizing code in modules with sets of related types and functions manipulating those types allows more natural and modular decompositions for reuse.
xixi77 11y ago
This is a very good point actually, I have done the same a few times. Of course this works well until you get to code heavily dependent on toolbox functions. Which leads to another advantage of Matlab (at least as long as you don't have to pay for it) -- the documentation (including toolbox documentation) is way ahead of any competition. Whoever came up with the R package documentation standard has done that language a great disservice :( IMO, the fact that every R package includes a huge pdf with alphabetical listing of functions and data sets is actively harmful: without it, perhaps more package authors would at least feel compelled to write a 2-page readme.txt (just like various matlab package writers do), and that would have been actually useful.

geomark 11y ago

I just completed the Coursera data science track which took me from a complete R newbie to being at least somewhat proficient. Having previously used Python for a quite a bit of web programming, I disliked R at first except for its power in statistical programming. But I've since discovered a number of great R packages that make it a pleasure to use for things I would normally turn to Python for. Like I recently discovered the rvest package for webscraping. Data visualizations with R seem vastly superior, unless I am missing something with Python (highly likely). And putting up a slick statistics app is easy with shiny or RStudio Presenter. But R can't really scale to a large production app, isn't that right? So I feel I need to keep working with both Python and R. Added: That's a nice list Lofkin. Thanks. Also, in the article he says that Python syntax feels more natural, which I also felt. But then I started to use things like the magrittr and dplyr packages in R which gives you nice things like pipes and that feeling starts to ebb.

Lofkin 11y ago
For stats plotting in python: https://github.com/mwaskom/seaborn https://github.com/mwaskom/seaborn https://github.com/yhat/ggplot https://github.com/yhat/ggplot For stats plotting and web apps in python: https://github.com/bokeh/bokeh https://github.com/bokeh/bokeh For calling r libraries in python: https://pypi.python.org/pypi/rpy2 https://pypi.python.org/pypi/rpy2 For out of core datasets in python: https://github.com/blaze/dask https://github.com/blaze/dask https://github.com/blaze/blaze https://github.com/blaze/blaze
- rasbt 11y ago
  Nice collection, let me add one more item to this list: Seaborn: statistical data visualization: http://stanford.edu/~mwaskom/software/seaborn/ http://stanford.edu/~mwaskom/software/seaborn/
dswalter 11y ago
You bring up a good point in favor of R: Hadley Wickham and the rest of the RStudio people. Packages like {ggplot2,rvest,dplyr,devtools, etc.} are basically creating a sub-language for R. I use both at the moment, but I echo the OP's ideas that R's target audience is statisticians, where Python's target audience is broader and includes statisticians and computer scientists. And Python's syntax is nicer to work with. That's why it's become the primary glue language. That said, the overhead for learning Python as your first data science language is a bit problematic for me, as you basically have to learn Python followed by Python's data science tools (pandas, matplotlib, etc). whereas with R, you're learning the language and the data science tools at the same time, even if they're a bit idiosyncratic.

JuliaLang 11y ago

Julia love!

Lofkin 11y ago

Personally I'm tempted to make the switch to Julia, but slow higher order functions, high churn in the core data infrastructure and no Pymc 3 are keeping me on pydata for a bit longer. I have numba to hold me over.

a_bonobo 11y ago

>I think it [Perl] is still quite common in the bioinformatics field though!? That's true - many day-to-day tasks in bioinformatics are more or less plain-text parsing [1], and Perl excels in parsing text and quickly using regular expressions. "My" generation of bioinformaticians doing data cleanup and analysis (20-30) uses Python, sometimes because plotting is nicer, the language is easier to get into, it's more commonly taught in universities, or other reasons - people older than that normally use Perl. Both BioPython and BioPerl are extremely useful. [1] Relevant quote from Robert Edgar: "Biology = strcomp()" from https://robertedgar.wordpress.com/2010/05/04/an-unemployed-gentleman-scholar/ https://robertedgar.wordpress.com/2010/05/04/an-unemployed-g...

rasbt 11y ago
Thanks for the insights! Also here, this comment would make an interesting addition to a "Feedback" section at the end of the article to give people a broader view on this topic. May I have your permission to post your comment below the article?
- a_bonobo 11y ago
  Of course you have my permission :)
dalke 11y ago
(Minor typo corrections; "Biopython", not "BioPython" and "strcmp" not "strcomp". I co-founded Biopython. We chose to avoid CamelCase.)
- a_bonobo 11y ago
  Sorry about that, that's what happens in caffeine-less typing - I sadly can't fix it at this point anymore, edit is disabled now

leni536 11y ago

And there is nothing wrong with C++. For linear algebra I use the armadillo library and it's really a nice wrapper around LAPACK and BLAS (and fast!). For some reason scientists are somewhat afraid of C++. For some reason you "have to" prototype in an "easier" language. Sure, you can't use C++ as a calculator as opposed to interpreted languages, but I see people being stuck with their computations at the prototyping language and eventually not bringing it to a faster platform. Point being: C++ is not hard for scientific calculations.

rasbt 11y ago
I agree with you. However, note that many people who are using Python for writing scientific code make use of C/C++ in one way or the other (aside from NumPy, SciPy, and Theano). For example, many people write the "most intensive" computations down in C/C++/Cython if they call those functions frequently -- Python becomes a wrapper. One example that pops into my mind is khmer (https://github.com/dib-lab/khmer https://github.com/dib-lab/khmer)
- RA_Fisher 11y ago
  I believe S (ancestor of R) started as C glue at Bell Labs. I've heard it said a few times that R is slow, but that doesn't really make much sense if your bottleneck routines are R calls to C++.
- srean 11y ago
  This is a common refrain: drop down to C, C++, Fortran for the computation intensive parts. It works, but only to a degree. The inefficiencies lie in the vectorization semantics of the host language(s) that leads to extra copies and extra levels of indirection. So this dual language mode of operation typically does not approach what one could have obtained had one disposed the baggage entirely, except for I/O. Usually in the quest for better speed, that is what remains, as one moves progressively larger portions of the application in the C, C++, Fortran part of the code. A reason I like Julia is that I can largely avoid this dual language annoyance, and enjoy the succinctness of pithy vectorized expressions using https://github.com/lindahua/Devectorize.jl https://github.com/lindahua/Devectorize.jl

dafrankenstein2 11y ago

.NET's F# is also good..though maybe not a better alternative