5 ms·

I cannot understand why I would use Python over R. R is designed from the ground up for massive amounts of data processing at speed and with ease. Even if Pytho

I cannot understand why I would use Python over R. R is designed from the ground up for massive amounts of data processing at speed and with ease. Even if Python continues accreting computational functionality, it will never be as fast or as efficient as R. Improving Python for something R is designed to do seems to me to be a huge waste of time: familiarity should not be the driving force behind replicating R's functionality. That's just so wrong.

kuzehanka 8y ago
> R is designed from the ground up for massive amounts of data processing at speed What? The R ecosystem doesn't provide meaningful out of core capabilities, nevermind the ability to handle anything approaching 'massive amounts of data'. -- Would sure love to know why an agenda-less factual comment is getting downvoted.
- javierluraschi 8y ago
  In my experience, R is really fast since I t was designed to store data in columnar format which we now all know is best for data analysis. So, in most cases, scaling up computation is quite easy. To scale out, you can use Apache Spark with R, the interface I’ve worked on, sparklyr is quite easy to use and allows you to scale out computation. Just to give you an example of what’s possible, I was playing around yesterday with a ray tracing prototype someone is building and scaled it out in Spark, see https://twitter.com/javierluraschi/status/1120557693721354240?s=21 https://twitter.com/javierluraschi/status/112055769372135424... — it’s a misconception that R is slow or can’t scale.
  - kuzehanka 8y ago
    You can plug any compute kernel you want into spark, that's not a pro or con of R. Column stores are standard in any analytics pipeline today. They make up Python's Pandas, R's dplyr, and Java's DataFrame. How or why does R stand out for 'massive amounts of data'? R does not have have meaningful out of core compute offerings that compare with something like Dask. R does not at all have cluster compute offerings that compare to Dask Distributed. If you want to know what real performance looks like, check out Python's cudf which will shortly fully match the Pandas api. That raytracing example you linked would run at interactive rates with cudf, I really don't see any basis for perf arguments in R's favour, and 'massive data' arguments are laughable here. Whatever advantages R has, perf or scalability are definitely not amongst them.
    - tylermw 8y ago
      > That raytracing example you linked would run at interactive rates with cudf, I really don't see any basis for perf arguments in R's favour, and 'massive data' arguments are laughable here. I don't see how the "GPU DataFrames" provided in cuDF would enhance a raytracer in any way.
      - bicubic 8y ago
        You don’t see how a gpu accelerated numeric array would speed up ray tracing?
        tylermw 8y ago
        The bottlenecks for raytracing are primarily in scene traversal/intersection testing--which does not benefit from a GPU-accelerated array structure.
        bicubic 8y ago
        You are literally as wrong as it is possible to be wrong. http://hoxxep.github.io/webgl-ray-tracing-demo/ http://hoxxep.github.io/webgl-ray-tracing-demo/ https://devblogs.nvidia.com/accelerated-ray-tracing-cuda/ https://devblogs.nvidia.com/accelerated-ray-tracing-cuda/
        tylermw 8y ago
        No, I'm not. I'm well aware of CUDA being used to accelerate raytracing. That cannot be accomplished by simply providing a GPU-accelerated data frame structure, as cuDF provides.
    - Annatar 8y ago
      You are arguing for Python and speed in the same breath? If you want portable speed, you better "warm up a chair" and master Fortran. Bonus: modern Fortran is a joy to develop in, far more fun than Python. And you get to compile to machine code, either for a processor or a GPU.

kuzehanka 8y ago

> R is designed from the ground up for massive amounts of data processing at speed What? The R ecosystem doesn't provide meaningful out of core capabilities, nevermind the ability to handle anything approaching 'massive amounts of data'. -- Would sure love to know why an agenda-less factual comment is getting downvoted.

javierluraschi 8y ago
In my experience, R is really fast since I t was designed to store data in columnar format which we now all know is best for data analysis. So, in most cases, scaling up computation is quite easy. To scale out, you can use Apache Spark with R, the interface I’ve worked on, sparklyr is quite easy to use and allows you to scale out computation. Just to give you an example of what’s possible, I was playing around yesterday with a ray tracing prototype someone is building and scaled it out in Spark, see https://twitter.com/javierluraschi/status/1120557693721354240?s=21 https://twitter.com/javierluraschi/status/112055769372135424... — it’s a misconception that R is slow or can’t scale.
- kuzehanka 8y ago
  You can plug any compute kernel you want into spark, that's not a pro or con of R. Column stores are standard in any analytics pipeline today. They make up Python's Pandas, R's dplyr, and Java's DataFrame. How or why does R stand out for 'massive amounts of data'? R does not have have meaningful out of core compute offerings that compare with something like Dask. R does not at all have cluster compute offerings that compare to Dask Distributed. If you want to know what real performance looks like, check out Python's cudf which will shortly fully match the Pandas api. That raytracing example you linked would run at interactive rates with cudf, I really don't see any basis for perf arguments in R's favour, and 'massive data' arguments are laughable here. Whatever advantages R has, perf or scalability are definitely not amongst them.
  - tylermw 8y ago
    > That raytracing example you linked would run at interactive rates with cudf, I really don't see any basis for perf arguments in R's favour, and 'massive data' arguments are laughable here. I don't see how the "GPU DataFrames" provided in cuDF would enhance a raytracer in any way.
    - bicubic 8y ago
      You don’t see how a gpu accelerated numeric array would speed up ray tracing?
      - tylermw 8y ago
        The bottlenecks for raytracing are primarily in scene traversal/intersection testing--which does not benefit from a GPU-accelerated array structure.
        bicubic 8y ago
        You are literally as wrong as it is possible to be wrong. http://hoxxep.github.io/webgl-ray-tracing-demo/ http://hoxxep.github.io/webgl-ray-tracing-demo/ https://devblogs.nvidia.com/accelerated-ray-tracing-cuda/ https://devblogs.nvidia.com/accelerated-ray-tracing-cuda/
        tylermw 8y ago
        No, I'm not. I'm well aware of CUDA being used to accelerate raytracing. That cannot be accomplished by simply providing a GPU-accelerated data frame structure, as cuDF provides.
  - Annatar 8y ago
    You are arguing for Python and speed in the same breath? If you want portable speed, you better "warm up a chair" and master Fortran. Bonus: modern Fortran is a joy to develop in, far more fun than Python. And you get to compile to machine code, either for a processor or a GPU.