9 ms·
R vs. Python for Data Science [video]
- photoJ 10y agoDoesn't seem like a huge debate for me. R is written in a way that encourages thinking like a statistician. Python environment enables simpler ways to "productionalize" your code and makes it easier to "think like an engineer". Both can be powerful, and each has some corner cases the other doesn't. If you know why you need to use one use it. If you don't, choose Python if your more of a coder and R if your more of a mathematician. Then switch when you get bored! :)
- jfaucett 10y agoI agree with a lot of what you said. I do a lot of maths and development, and for me it boils down to using R when doing analysis, anything statistics based, and general prototyping, then I switch to python when I have to build an app, even though for some problem domains i.e. Image Processing or NLP I will just start out in python because I prefer the libs. As a side note on julia. The only reasons I haven't started using it on the daily - and I've tried it and liked much of what I've seen - is that the module system doesn't let you build modular production code and apps like Python does and R is just so good for what I use it for there's no reason to look for a replacement.
- photoJ 10y agoI am always amazed when I work in R, how some ideas are just effortless, and you can ignored the syntax and just think about the concept. I really enjoy that.
- Jcol1 10y ago"the module system doesn't let you build modular production code and apps like Python does" Can you elaborate on this?
- haddr 10y agoThis! In my opinion this is where the difference lies: with python you can easily build bigger systems with more flexibility, where your data science code is an important piece. R shines in statistics and you can also produce an end-to-end system, but given that you fit well the R ecosystem and its constraints.
- photoJ 10y agoI'd like to know more about end-to-end systems in R. I'm not familiar with serious production environments using such tools. Any pointers?
- haddr 10y agoBy end-to-end i mean that you can: (1) read data from SQL or NoSQL DB, excel file, csv, JSON through some REST service, etc. (2) process it in R (3) display it to the user through some interactive web interface e.g. using Shiny It is relatively easy if your app fit the whole loop. I would recommend with some Shiny tutorial to get you started: http://shiny.rstudio.com/tutorial/ http://shiny.rstudio.com/tutorial/ But if you need e.g. some stream processing or more complex guis, then R might be not enough I guess.
- photoJ 10y agoCool! Yeah, I know of that workflow. I've seen some neat stuff done with that. Thanks for the follow up!
- gvd 10y agoUse Scala or some other statically typed language for building bigger stuff. Python might be nice for the first 3 lines but you will be convinced once you need to do some serious refactoring.
- tnecniv 10y agoWhat benefits does R have other than vectors as native citizens? I mostly use MATLAB (ugh).
- cschmidt 10y agoThe biggest benefit of R is a huge set of statistical libraries. If there is any sort of stats function you might want, it is available in R.
- glial 10y agoIn my mind one of the primary benefits of R over Matlab are the 'data frame' data structure and the ecosystem that's built up around that, for slicing/dicing/plotting/modeling/etc. Also, having factors as first-class citizens is super nice.
- confounded 10y agoThe other benefit R has is the incredibly productive and thoughtful Hadley Wickham. He's completely changed the way I write R over the last five years, and I'm a great deal more productive as a result (as is my code).
- makeset 10y agoR's Scheme-like lexical scoping, lazy evaluation, and reflective abilities make it very extensible if you know what you are doing.
- deleted 10y ago[deleted]
- transfire 10y agoJulia?
- photoJ 10y agoYeah, not to mention all the DL frameworks.
- Recurecur 10y agoI think Julia will eventually eclipse both R and Python as a math/science language - and possibly even a general purpose language.
- stewbrew 10y agoI'd rather bet on scala or f#.
- tnecniv 10y agoPremature, but scheduled to hit 1.0 in ~1yr. Really promising right now, though.
- patrick_99 10y agoI've always used R for analysis, but using Python opens up the world of PySpark and more scalable ML environments. I think long term Python will become more prevalent in Data science for this reason.
- danso 10y agoAs someone who teaches introductory-level programming in the data science/visualization domain, I think R is undoubtedly easier for getting non-programmers from CSV/Excel data to great visualizations via ggplot2. In fact, I keep R up-to-date on my own machine for when I need to do visualizations that I can't easily hack in Matplotlib+seaborn. But I stick to teaching Python because my priority is that students become general-purpose programmer/hackers, and data science is a very small part of what they might use programming for in their careers. Python's pandas is definitely not as elegant as R's tidyverse, where the conventions and data structures are more baked into the language. But numpy+pandas is manageable enough for novices (and just fine for experienced programmers). It's a small tradeoff for all the things you can do in Python, whether it is build and deploy a web app with Flask/Django/etc, or integrate with AWS via boto, or more specific things like manipulate video with moviepy. That said, obviously not every stats-focused person wants to or needs to become an all-purpose engineer. I think R has basically become the lingua franca for folks in poli sci and similar departments who want to do statistical analysis, and it seems to be a huge step up from doing things in SPSS or Excel.
- SubiculumCode 10y agoR does some things really well, and there are some advanced methids available ,but let's be clear, some functions in SPSS are quite advanced, for example SPSS mixed models are more advanced than lme4.
- bobbert_barker 10y agoWhat is more advanced about SPSS mixed models (not accusatory, just ignorant)? It looks like there are more covariance options, but is the estimation/implementation better/more robust or faster?
- SubiculumCode 10y agoWell I may be ignorant also, but as far as I can tell lme4/lmer doesnt have ANY options to specify the variance-covariance matrix while SPSS does. Also, when the number of within subject levels are few, I often find that using the classic repeated measures approach, but estimating using ML/REML (to handle random missing data) on a specified (or unstructured) covariance-variance matrix to be more powerful and/or more likely to converge while accounting for important dependences than mixed models using random effects...SPSS does this kind of analysis easily, but I havent found any equivalent packave in R.
- earino 10y agoHello everyone, I'm the guy in the video! SPOILER ALERT If you watch you will notice I do a really sneaky thing where it's possible that I'm comparing the two in order to show the folly of saying that it's R vs Python... :) If you have any questions, I'll do my best to be around! Cheers.
- minimaxir 10y agoHere's a similar discussion between the merits of R vs. Python months ago: https://news.ycombinator.com/item?id=11867268 https://news.ycombinator.com/item?id=11867268 My comment from the thread: At the end of the day, for machine learning applications, your data is in a tabular format. (in Python, a pandas data frame) Yes, Python has a few tricks like list comprehensions for speeding up data processing into that analyzable form. R has a few tricks for processing tabular data as well. (e.g. dplyr). There are tradeoffs and the skill is finding which works best. Using a single programming language is a bad philosophy even for non-statistical developers.
- lottin 10y agoLearn both and decide for yourself.
- IndianAstronaut 10y agoThe most sensible approach. Anyone can jump back and forth between the two without much problem since a lot of the syntax is so similar. It is better to have more tools in your arsenal than less.
- hcarvalhoalves 10y agoR (specially w/ R Studio) is effectively a better Excel. The problem it creates is that someone, somewhere, will eventually present some projection they put together quickly in R, and that will set expectations on stakeholders that know nothing about software to start demanding that solution to scale indefinitely, compute in realtime, or be used inside any kind of production system really. It doesn't mean the same can't happen w/ Python, but it at least offers some migration path to more scalable / hardened solutions if you're careful.
- photoJ 10y agoI don't know any research statisticians that produce code for excel. R is THE place to see their most recent advances. And sadly despite the algo's being ported to python, sometimes R still has the "best" implementations.
- hcarvalhoalves 10y ago> I don't know any research statisticians that produce code for excel. Never said that. What I said is that R, nowadays, creates the same kinds of issues / attrition Excel created on companies back in the 90's - you end up w/ someone, in some corner of the company, creating solutions that can't scale on top of it. In comparison, if the original work is in Python, you usually have more alternatives when this prototype lands on the hands of a software eng. That is the main, and only difference that matters, IMO. As a tool for research, I agree it's completely fine. Also, I too feel the pain of some algos, sadly, being available only on R (had to write my own wrappers to estimate some models on top of R script and export the coefficients to be used in Python's sklearn).
- photoJ 10y agoSure, I can see that coming from your perspective. Matlab can do things that are similar, a researcher creates a model that is totally detached from the production environment, works beautifully and leaves ALOT of engineering required of the production team. I wonder how probabilistic programing will fit over the next 10(?) years. I'm not a fan of "I programed this, now you implement it" dev cycle and I that will decrease as Data Science matures. Sadly you are at the friction point for now.
- baldfat 10y agoHuge Debate = Click Bait Spoiler R and Python(Pandas) are both great tools R = Domain Language Python = General Purpose Some things in R are worth it for many people.
- dpitkin 10y agoGreat comment, once you have lived the difference between R and Python error messages you will unlock the next level of the data science maze.
- joelgrus 10y agoSo "political detox week" lasted like an hour?
- lochland 10y agoThe introduction to the article is the second best introduction to any article ever. The first, which is in a totally unrelated field, is: "THEY do a lot of things wrong in the United States, but they do a lot of things right, too. The NFL draft is at, or at least near, the top of the good list." (see http://www.afl.com.au/news/2013-04-30/uncle-sams-draft-week-extravaganza http://www.afl.com.au/news/2013-04-30/uncle-sams-draft-week-...)
- thearn4 10y agoEh, I don't expect this to be as heated as the "R or SAS" debates that many statisticians have...
- cannonpr 10y agoI work at a data science company that uses both, in a DevOps capacity. We actually have a fairly hard policy against R in production or in client deliverable code. That's mostly because of the difficulty in 'productionizing' R code, and the relative immaturity of several common R libraries. However that having been said, nearly all of our junior Data scientists from a non software engineering background do pick up R faster and feel more comfortable in it, so we mostly use it for rapid prototyping.
- dpitkin 10y agoI am a big R user and evangelist when possible. I agree with your company and team production policy at the same time and the specific "immaturity" is the utterly incomprehensible error messages that I get from R on a daily basis. I think this is a core R team opportunity or choice if they want to transition from a stats domain-specific-language to a general-purpose language.
- stewbrew 10y agoWhat kind of incomprehensible error messages do you get from R?
- rabboRubble 10y agoTry installing the RODBC package on Mac OS X el Capitan. Dollars to donuts it won't go smoothly. Let me know how long it takes you to successfully install it.
- stewbrew 10y agoTo be fair, these are error messages from one nonstandard library (which works great under Windows BTW) on an OS that doesn't natively use ODBC (does it?) -- and I guess some another 3rd party shared lib not under the control of the authors of RODBC is involved.
- 10y ago
- asimjalis 10y agoIncanter/Clojure?
- peatmoss 10y agoHow's the health of the ecosystem doing? Is it being actively developed these days? Last github activity I see was 5 months ago.
- iaw 10y agoAny time I see a domniondatalab.com article I discount it severely. Historically they've been less useful and more biased towards selling their services. Just because it's a company's blog doesn't make the info bad, my experience with that particular blog makes me hesitant.
- replies_to_all 10y agoR you faggots.
- sctb 10y agoNo comments like this, please. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- sandGorgon 10y agoPandas with the Rodeo ide is pretty much R. R's advantage is singular and simple - its not that the language is better,.. but rather that it has existed in its niche for much longer, so it has a much larger set if libraries that exist in the stats space. in other words, CRAN.
- yomritoyj 10y agoR's unusual calling convention makes it very easy to write little DSLs and access program text at runtime. This I think makes for much more expressive APIs like base R's formulas or dplyr's data manipulation facilities. In all other languages I know, formulas or data queries would need to be (quasi-)quoted or terms appearing in them would have to be declared to have a special type, spoiling the cleanness of the code
- gravelc 10y agoI don't know about too many others in the bioinformatics field, but I reckon I write about 80% of my code in Python and 20% in R. R for stats and plotting; Python for wrangling data. Having said that, they're the languages I learned in my studies (along with java) and I haven't had the time and motivation to move beyond them since.
- narutouzumaki 10y agoFor some reasons the video stops playing for me after about a minute in in Chrome (same in incognito)
- gorbachev 10y agoClickbait title. I think the consensus is to let the data scientists use whatever tools they feel most productive with whatever that tool is.