6 ms·
I have to come to love R (for what I use it it for), but reading this makes me realize how unusual my R-workflow must be, because most of the 'advantages' of Ju
by chimeracoder 15y ago
I have to come to love R (for what I use it it for), but reading this makes me realize how unusual my R-workflow must be, because most of the 'advantages' of Julia over R don't really come up in my daily workflow anymore - it seems that's likely because I've adapted to the shortcomings of R and have twisted other tools to my needs. I'll add Julia to my list of languages to check out in more detail, because perhaps Julia could replace my need for this rather esoteric workflow that I've developed out of sheer necessity.
I use Python (NumPy/SciPy) for most of the data preprocessing, and perhaps that's why. I used to do this in R, and I realized that it's just a lot easier to get done in Python (and it ends up being faster anyway). The problem is that Python/NumPy/SciPy still doesn't lend itself quite as well as R does to certain aspects of the statistician's use case. It's possible that things have changed since the last time I evaluated the two, but I still find it easier to prototype various models in R, even if I do all of the preqrequisite data munging in a different environment.
I understand that R, like Perl, is 'blessed' (pun intended) with two different, incompatible type systems - in fact, this is the reason I avoid using R's type system, and whenever I'm advising newcomers, I always recommend the same. I don't write statistical packages, so this doesn't come up, but when I find myself needing to write a method in R, I ask myself if this would actually be done more easily another way instead. Generally, I find the answer is 'yes, yes it would'.
I really do think the problem is the type system. The kind of type system that lends itself well to data manipulation is not the same type system that lends itself well to model manipulation - when I think about it, I've unconsciously segregated my workflow into two parts, doing everything naturally done with Python's type system in Python, and likewise for R. Maybe that's just the way that I happen to approach data manipulation, but I think it's non-coincidental. R's relative homoiconicity (compared to Python) makes it really nice for some things, but there are other warts with its typing that are just too annoying to work around, when a python shell is just a few keystrokes away.
I guess the answer is (as always!) to use a purely homoiconic Lisp dialect, so you get the best of both worlds but that's asking a lot of statisticians.
I really have come love R for what it does do, though. Of all all the statistical software packages I've seen (comparable: SAS, SPSS, Stata, MATLAB), it's far and away the best (and the GNU license makes it very, very attractive to broke students looking to avoid the still-absurdly-priced student licenses for the alternatives). That said, I still sigh every time I realize that I'm essentially gluing together two separate runtime environments for something that should really be easily integrated. I do what I do now because it ends up being faster than using either Python or R for everything, but it still strikes me as weird that a language so perfect for munging data (Python) can still be so awkward for analyzing it, and vice versa.
- TalGalili 15y agoHi chimeracoder, I am very curious to better understand how you find python better for the data pre-processing stage. I use only R, and would love to know what "I am missing" here. Any simple example will also make it easier to understand. Thanks!
- probably 15y agoWell explained by chimeracoder. Data-table centric operations are much more natural in R while sequential objects (lists, tuples, and strings) are quickly manipulated in Python (there are more string/regex methods there). I am a heavy Python user, but when I use Numpy/Scipy I don't feel like I'm using Python much anymore so at that point I either switch to R (or Fortran)... though I'm quite optimistic that at some point the pandas DataFrame can become my default storage structure from which I can parse out R tasks through Rpy, SQLite, HDF5, or possibly Reddis. matplotlib is very verbose though; I almost prefer Matlab's graphics model... though less so than R's basic and lattice graphics.
- keithflower 15y agoI tend to use Python and Num/SciPy for as much as I can, then reach for RPy: http://rpy.sourceforge.net/ http://rpy.sourceforge.net/
- chimeracoder 15y agoHaha, that's really interesting - I can't believe I've never seen that. I'll see if this may solve my problems....
- drunkpotato 15y agoThat is very interesting. I use Python to pre-process data for Matlab, and have been giving serious thought lately to learning R for its free license and easy(?) integration with Hadoop. Can you briefly comment on the advantages or R over Matlab aside from licensing?
- TalGalili 15y ago