7 ms·

PyThor- Python meets R

113 points by nipun_batra 11y ago

pfh 11y ago
For when you really need a less purely functional language, when you want your data to mutate, and your code not to. When the possibility any function might be a macro is too frightening, but the possibility any function might mangle its inputs is fine. Embrace the madness of R, my friend.
amelius 11y ago
I just can't wrap my head around R. I'd love it if there was an automatic translator from R to Python. Even though the Python code would be more cluttered, it would probably be more easy to read :)
- baldfat 11y ago
  R has a lot of choices. I find that the Hadley Universe of libraries of ggplot2, ggvis, dplyr, tidyr, stringr, readr and others with the piping of %>% to be the easiest code for reading and getting things done. R has been transformed in the last 5 years but still gets a bad rap which I don't feel it deserves. R is really more a functional language which most people don't recognize and when they learn the standard core of R with Lisp inspired ideas it really throws them for a loop. I ended up learning Racket (after trying to teach myself Haskell 3 times) and I can say I now "get it," but using the new libraries has really made that point mute. I love R and am excited to see all the support from Microsoft and other large companies jumping on board. This does look like a decent way to move to Python though. If someone was to do that I hope they use Rodeo IDE.
  - nipun_batra 11y ago
    I'll second that R is really awesome! Being a computer scientist, it opens up for me a lot of statistical packages.
    - s_q_b 11y ago
      R is the only real game in town for free software for scientific computing these days. From a political perspective R was in the right place at the right time. It was a decent high level language that could handle matrix processing gracefully. Scipy/Numpy weren't ready for production yet. The others were Matlab, SAS, and Stata, all of which R makes look like APL. I'm glad the field has a more healthy open source landscape with worthy competitors in the form of Julia and Numpy/Scipy. R's biggest sin is failure to force people to use functional paradigms by providing juuuust enough imperative sugar to make average Joe programmer feel at home. R is a functional language, and that's the principle under which is should be taught. That said, R also has a large number of main technical strengths: 1. Fast basic statistics within the REPL so you can test hunches quickly. 2. Cutting edge algorithms that often aren't implemented anywhere else. 3. Hugely strong engineering packages for civil, environmental, defense, aerospace and basically any IRL engineering field you can think of. R comes free, with these advantages and many more for the average lab tech.
      - mziel 11y ago
        And lists + dataframes as first class citizens. Everything (including dataframes) is pretty much a list. Most R packages nowadays have dataframes in the centre of implementation. Compare it to other languages, lots of code is needed to convert the data from one format to another, because there's no underlying data structure (Python is much better than the rest with regards to that, but still not as good as R).
      - baldfat 11y ago
        > R's biggest sin is failure to force people to use functional paradigms by providing juuuust enough imperative sugar to make average Joe programmer feel at home. R is a functional language, and that's the principle under which is should be taught. 100% in agreement. That states what I have discovered after learning Functional programming. The sad thing is people don't know that it is functional PAST first class functions. http://link.springer.com/chapter/10.1007%2F978-3-642-40447-4_12 http://link.springer.com/chapter/10.1007%2F978-3-642-40447-4...
      - semi-extrinsic 11y ago
  - jzwinck 11y ago
    You mention that R doesn't deserve the bad rap it had 5 years ago. Less than two years ago I learned the hard way that R reference counting had only three possible counter values. 0, 1, and 2+. So if you took a second reference to something then deleted it, that object's reference count would stay at 2+ forever. Then if you modified it, Copy On Write would kick in, even though there was only one reference living. This was the source of crippling inefficiency in production code. I asked about it back then and the reply was yeah, R does that, hopefully it won't soon. Broken Copy On Write was not acceptable to me in 2014.
  - bachmeier 11y ago
    > R is really more a functional language which most people don't recognize and when they learn the standard core of R with Lisp inspired ideas it really throws them for a loop. Just to clarify, R was initially a dialect of Scheme. On the other hand, with its frustrating silent type changes, you can see its S roots in the same place that gave us C and C++. That's quite a combination in terms of learning curve.
    - lispm 11y ago
      There is Lisp-stuff in R not coming from Scheme. Like object system inspired by CLOS, FEXPRs from very old Lisps, ...
- nipun_batra 11y ago
  I wonder if a fully automatic translator would ever be feasible. The current approach of handwriting the wrapper code for the packages I'm interested in seems to work well. I guess with a bit of community support, the recipe set can become richer.
- lottin 11y ago
  R is all about data structures. Everything is built from vectors and lists. Arrays are vectors with a 'dimension' attribute. Data-frames are lists of vectors of the same length. And so on. And factors, which again are a kind of vector, are the primary tool for partitioning the data in groups, so you can have 'ragged' arrays. When you understand how all these work together you get the hang of R.
  - jzwinck 11y ago
    Almost all of what you said applies equally to Python when using NumPy, Pandas, and SciPy. In R, a Factor is also the bizarre result you get if you load a flat file incorrectly. Lots of things in R proceed without stopping on errors, and you end up with weird data that isn't really usable but still lets your program continue.
    - nipun_batra 11y ago
      Language wars again :) I think both languages have their own strengths. I come from a programming background and took to Python. However, I often come up in situations when there are R implementations to some advanced statistical routines and none exist for Python. I'm sure vice versa would also be true. So, this is an attempt from that angle :)
    - lottin 11y ago
      It's true, however NumPy, Pandas, and SciPy are external libraries. I think it's a slight disadvantage. In R all these fancy data structures are built into the language and are used everywhere in a natural way. Whereas Numpy feels a little bit like an appendage or like a language inside a language. That said I don't dislike Python/Numpy and I think it definitely has its uses.
    - amelius 11y ago
      > Lots of things in R proceed without stopping on errors This alone would make me run away very fast from R.
      - 11y ago
- sandGorgon 11y ago
  the biggest concern I have heard from friends who work in stats and mathematics academia is that it is much, MUCH easier to publish a package in CRAN than to PyPi. I think most users of Python are that... "users". But R is used by creators of algorithms who find it very easy to create and distribute code. In fact a lot of them talk about how difficult it is to install packages in python, while any R script has "install.packages" right in it. I'm willing to bet if Pandas/Numpy/Jupyter built a CRAN-PY and made the packaging system similar to R, the adoption would be much different.
  - nipun_batra 11y ago
    I do agree that stats people love R. This is one of the reasons I wrote this article. So that I can leverage all the good work they've gone. It is way better to write a wrapper than to reimplement everything from scratch. I'm coming from academia where I have a lot of systems work. So, I tend to be Python first user. But, I do want to utilise all the packages that are exclusively available in R. Packaging is improving in Python afaik. People like Anaconda!
    - sandGorgon 11y ago
      its not packaging per se - conda and gem are brilliant, but its intended for software creators and not math acads. install.package may not be a brilliant piece of software, but its perfect for people who develop stats algorithms and just publish to cran. This is the package submission page to CRAN - https://cran.r-project.org/submit.html https://cran.r-project.org/submit.html This is the equivalent for Pypi - https://pypi.python.org/pypi?%3Aaction=submit_form https://pypi.python.org/pypi?%3Aaction=submit_form
      - stdbrouw 11y ago
        Submitting a package is the least important part of distributing it. For CRAN you can use the submit form you link to, for Python it's `python setup.py sdist upload` and you're done. CRAN in fact has much stricter standards for packages than Python -- you need a vignette, you need tests etc. -- which is why new R packages are usually on GitHub long before they make it to CRAN, whereas PyPI is happy to accept alpha and beta releases without documentation and without tests.
deleted 11y ago
[deleted]
spot 11y ago
Beaker Notebook provides a different model for working in Python and R. Instead of wrapping one language in the other, you get a notebook where cells can be either Python or R, fully native, and they can communicate with a shared object. The result is much simpler and more natural IMO: https://pub.beakernotebook.com/#/publications/56648fcc-2e8e-41a6-aa4a-1249ee39023c?fullscreen=true https://pub.beakernotebook.com/#/publications/56648fcc-2e8e-... Beaker has a feature "autotranslation" that converts the data between R, Python, JavaScript, Julia, Scala, Clojure, and many other languages, completely automatically. Learn more at http://BeakerNotebook.com http://BeakerNotebook.com
- nipun_batra 11y ago
  Very interesting! Thanks for the comment! I do see the advantages of beaker. I also see the advantages other way around :)
- baldfat 11y ago
  Jupyter Notebooks (Formerly IPython Notebooks) http://jupyter.org/ http://jupyter.org/ is kernel agnostic and can work with all the other languages also. I use it for R and Python myself. (For data work I still prefer RStudio or Python's Rodeo) Breaker uses the same jkernel as IPython and backend of Jupyter.
  - spot 11y ago
    that's right, but Jupyter notebooks have just one language each, they don't facilitate polyglot programming.
    - captaindiego 11y ago
      Their are some "cell magics" for running different languages in individual blocks in a Jupyter notebook. I gave it a try in passing with javascript a while back and I remember there was a way to share variables through as well. You can see the available magics here: https://ipython.org/ipython-doc/3/interactive/magics.html#cellmagic-javascript https://ipython.org/ipython-doc/3/interactive/magics.html#ce... Unfortunately it looks like there is none for R or Julia, but perhaps they are installed by those specific packages, or their is some other way I don't know of.
phillipamann 11y ago
I thought this was interesting but the resulting code was so ugly and awkward that I would refrain from using it. I would just prototype in R and convert to Python rather than using both in this sort of Frankenstein way.
- baldfat 11y ago
  Just as a point of interest: Why would you convert a R program into a Python program? I can't think of any benefit in terms of speed or reporting?
  - huac 11y ago
    Speed and scale (and reporting too, I guess) are very important considerations for data analysis.
    - baldfat 11y ago
      But I wouldn't see Python as being speedier or better at scale and certainly not reporting. What are your speed up and scale advantages in Python over R. (Seriously interested) Going R to Clojure or Julia seems like a gain but Python.
      - phillipamann 11y ago
        Sorry for delay. I think it's just better interoperability with many other systems. R has very hacked on interoperability in my opinion but I am not an expert.
- Fomite 11y ago
  This assumes you're willing to convert someone's cutting edge statistical research package in R to Python just because the code is 'ugly and awkward'
mikeskim 11y ago
it is probably easier to just wrap R functions as a subprocess in python. you would have to write some R code to do this which might involve writing to a flat file. i do this all the time in R wrapping up something like rgf and calling from within R. i have wrapped up python and called it from R using system. the same idea works for python as well. you could probably write some kind of general wrapper that would work for most of R's ML type functions.

113 points by nipun_batra 11y ago

pfh 11y ago

For when you really need a less purely functional language, when you want your data to mutate, and your code not to. When the possibility any function might be a macro is too frightening, but the possibility any function might mangle its inputs is fine. Embrace the madness of R, my friend.

amelius 11y ago

I just can't wrap my head around R. I'd love it if there was an automatic translator from R to Python. Even though the Python code would be more cluttered, it would probably be more easy to read :)

baldfat 11y ago
R has a lot of choices. I find that the Hadley Universe of libraries of ggplot2, ggvis, dplyr, tidyr, stringr, readr and others with the piping of %>% to be the easiest code for reading and getting things done. R has been transformed in the last 5 years but still gets a bad rap which I don't feel it deserves. R is really more a functional language which most people don't recognize and when they learn the standard core of R with Lisp inspired ideas it really throws them for a loop. I ended up learning Racket (after trying to teach myself Haskell 3 times) and I can say I now "get it," but using the new libraries has really made that point mute. I love R and am excited to see all the support from Microsoft and other large companies jumping on board. This does look like a decent way to move to Python though. If someone was to do that I hope they use Rodeo IDE.
- nipun_batra 11y ago
  I'll second that R is really awesome! Being a computer scientist, it opens up for me a lot of statistical packages.
  - s_q_b 11y ago
    R is the only real game in town for free software for scientific computing these days. From a political perspective R was in the right place at the right time. It was a decent high level language that could handle matrix processing gracefully. Scipy/Numpy weren't ready for production yet. The others were Matlab, SAS, and Stata, all of which R makes look like APL. I'm glad the field has a more healthy open source landscape with worthy competitors in the form of Julia and Numpy/Scipy. R's biggest sin is failure to force people to use functional paradigms by providing juuuust enough imperative sugar to make average Joe programmer feel at home. R is a functional language, and that's the principle under which is should be taught. That said, R also has a large number of main technical strengths: 1. Fast basic statistics within the REPL so you can test hunches quickly. 2. Cutting edge algorithms that often aren't implemented anywhere else. 3. Hugely strong engineering packages for civil, environmental, defense, aerospace and basically any IRL engineering field you can think of. R comes free, with these advantages and many more for the average lab tech.
    - mziel 11y ago
      And lists + dataframes as first class citizens. Everything (including dataframes) is pretty much a list. Most R packages nowadays have dataframes in the centre of implementation. Compare it to other languages, lots of code is needed to convert the data from one format to another, because there's no underlying data structure (Python is much better than the rest with regards to that, but still not as good as R).
    - baldfat 11y ago
      > R's biggest sin is failure to force people to use functional paradigms by providing juuuust enough imperative sugar to make average Joe programmer feel at home. R is a functional language, and that's the principle under which is should be taught. 100% in agreement. That states what I have discovered after learning Functional programming. The sad thing is people don't know that it is functional PAST first class functions. http://link.springer.com/chapter/10.1007%2F978-3-642-40447-4_12 http://link.springer.com/chapter/10.1007%2F978-3-642-40447-4...
    - semi-extrinsic 11y ago
- jzwinck 11y ago
  You mention that R doesn't deserve the bad rap it had 5 years ago. Less than two years ago I learned the hard way that R reference counting had only three possible counter values. 0, 1, and 2+. So if you took a second reference to something then deleted it, that object's reference count would stay at 2+ forever. Then if you modified it, Copy On Write would kick in, even though there was only one reference living. This was the source of crippling inefficiency in production code. I asked about it back then and the reply was yeah, R does that, hopefully it won't soon. Broken Copy On Write was not acceptable to me in 2014.
- bachmeier 11y ago
  > R is really more a functional language which most people don't recognize and when they learn the standard core of R with Lisp inspired ideas it really throws them for a loop. Just to clarify, R was initially a dialect of Scheme. On the other hand, with its frustrating silent type changes, you can see its S roots in the same place that gave us C and C++. That's quite a combination in terms of learning curve.
  - lispm 11y ago
    There is Lisp-stuff in R not coming from Scheme. Like object system inspired by CLOS, FEXPRs from very old Lisps, ...
nipun_batra 11y ago
I wonder if a fully automatic translator would ever be feasible. The current approach of handwriting the wrapper code for the packages I'm interested in seems to work well. I guess with a bit of community support, the recipe set can become richer.
lottin 11y ago
R is all about data structures. Everything is built from vectors and lists. Arrays are vectors with a 'dimension' attribute. Data-frames are lists of vectors of the same length. And so on. And factors, which again are a kind of vector, are the primary tool for partitioning the data in groups, so you can have 'ragged' arrays. When you understand how all these work together you get the hang of R.
- jzwinck 11y ago
  Almost all of what you said applies equally to Python when using NumPy, Pandas, and SciPy. In R, a Factor is also the bizarre result you get if you load a flat file incorrectly. Lots of things in R proceed without stopping on errors, and you end up with weird data that isn't really usable but still lets your program continue.
  - nipun_batra 11y ago
    Language wars again :) I think both languages have their own strengths. I come from a programming background and took to Python. However, I often come up in situations when there are R implementations to some advanced statistical routines and none exist for Python. I'm sure vice versa would also be true. So, this is an attempt from that angle :)
  - lottin 11y ago
    It's true, however NumPy, Pandas, and SciPy are external libraries. I think it's a slight disadvantage. In R all these fancy data structures are built into the language and are used everywhere in a natural way. Whereas Numpy feels a little bit like an appendage or like a language inside a language. That said I don't dislike Python/Numpy and I think it definitely has its uses.
  - amelius 11y ago
    > Lots of things in R proceed without stopping on errors This alone would make me run away very fast from R.
    - 11y ago
sandGorgon 11y ago
the biggest concern I have heard from friends who work in stats and mathematics academia is that it is much, MUCH easier to publish a package in CRAN than to PyPi. I think most users of Python are that... "users". But R is used by creators of algorithms who find it very easy to create and distribute code. In fact a lot of them talk about how difficult it is to install packages in python, while any R script has "install.packages" right in it. I'm willing to bet if Pandas/Numpy/Jupyter built a CRAN-PY and made the packaging system similar to R, the adoption would be much different.
- nipun_batra 11y ago
  I do agree that stats people love R. This is one of the reasons I wrote this article. So that I can leverage all the good work they've gone. It is way better to write a wrapper than to reimplement everything from scratch. I'm coming from academia where I have a lot of systems work. So, I tend to be Python first user. But, I do want to utilise all the packages that are exclusively available in R. Packaging is improving in Python afaik. People like Anaconda!
  - sandGorgon 11y ago
    its not packaging per se - conda and gem are brilliant, but its intended for software creators and not math acads. install.package may not be a brilliant piece of software, but its perfect for people who develop stats algorithms and just publish to cran. This is the package submission page to CRAN - https://cran.r-project.org/submit.html https://cran.r-project.org/submit.html This is the equivalent for Pypi - https://pypi.python.org/pypi?%3Aaction=submit_form https://pypi.python.org/pypi?%3Aaction=submit_form
    - stdbrouw 11y ago
      Submitting a package is the least important part of distributing it. For CRAN you can use the submit form you link to, for Python it's `python setup.py sdist upload` and you're done. CRAN in fact has much stricter standards for packages than Python -- you need a vignette, you need tests etc. -- which is why new R packages are usually on GitHub long before they make it to CRAN, whereas PyPI is happy to accept alpha and beta releases without documentation and without tests.

deleted 11y ago

[deleted]

spot 11y ago

Beaker Notebook provides a different model for working in Python and R. Instead of wrapping one language in the other, you get a notebook where cells can be either Python or R, fully native, and they can communicate with a shared object. The result is much simpler and more natural IMO: https://pub.beakernotebook.com/#/publications/56648fcc-2e8e-41a6-aa4a-1249ee39023c?fullscreen=true https://pub.beakernotebook.com/#/publications/56648fcc-2e8e-... Beaker has a feature "autotranslation" that converts the data between R, Python, JavaScript, Julia, Scala, Clojure, and many other languages, completely automatically. Learn more at http://BeakerNotebook.com http://BeakerNotebook.com

nipun_batra 11y ago
Very interesting! Thanks for the comment! I do see the advantages of beaker. I also see the advantages other way around :)
baldfat 11y ago
Jupyter Notebooks (Formerly IPython Notebooks) http://jupyter.org/ http://jupyter.org/ is kernel agnostic and can work with all the other languages also. I use it for R and Python myself. (For data work I still prefer RStudio or Python's Rodeo) Breaker uses the same jkernel as IPython and backend of Jupyter.
- spot 11y ago
  that's right, but Jupyter notebooks have just one language each, they don't facilitate polyglot programming.
  - captaindiego 11y ago
    Their are some "cell magics" for running different languages in individual blocks in a Jupyter notebook. I gave it a try in passing with javascript a while back and I remember there was a way to share variables through as well. You can see the available magics here: https://ipython.org/ipython-doc/3/interactive/magics.html#cellmagic-javascript https://ipython.org/ipython-doc/3/interactive/magics.html#ce... Unfortunately it looks like there is none for R or Julia, but perhaps they are installed by those specific packages, or their is some other way I don't know of.

phillipamann 11y ago

I thought this was interesting but the resulting code was so ugly and awkward that I would refrain from using it. I would just prototype in R and convert to Python rather than using both in this sort of Frankenstein way.

baldfat 11y ago
Just as a point of interest: Why would you convert a R program into a Python program? I can't think of any benefit in terms of speed or reporting?
- huac 11y ago
  Speed and scale (and reporting too, I guess) are very important considerations for data analysis.
  - baldfat 11y ago
    But I wouldn't see Python as being speedier or better at scale and certainly not reporting. What are your speed up and scale advantages in Python over R. (Seriously interested) Going R to Clojure or Julia seems like a gain but Python.
    - phillipamann 11y ago
      Sorry for delay. I think it's just better interoperability with many other systems. R has very hacked on interoperability in my opinion but I am not an expert.
Fomite 11y ago
This assumes you're willing to convert someone's cutting edge statistical research package in R to Python just because the code is 'ugly and awkward'

mikeskim 11y ago

it is probably easier to just wrap R functions as a subprocess in python. you would have to write some R code to do this which might involve writing to a flat file. i do this all the time in R wrapping up something like rgf and calling from within R. i have wrapped up python and called it from R using system. the same idea works for python as well. you could probably write some kind of general wrapper that would work for most of R's ML type functions.