5 ms·

Julia claims performance closer to Fortran or C, with the expressiveness of a high-level scripting language. It seems to have enough traction that it's probably

Julia claims performance closer to Fortran or C, with the expressiveness of a high-level scripting language. It seems to have enough traction that it's probably the way of the future. R quirks ... the <- syntax, the ~ formula syntax, looping through a dataframe is horrifically slow when you can't vectorize, multiple legacy object semantics, poor parallel/multiprocessing support, poor support for datasets that don't fit in memory ... upside is any statistical method probably has a decent implementation in R python - most comprehensive ecosystem and libraries beyond statistics, e.g. Web, numerical and scientific computing, machine learning (Tensorflow), NLP, generally a good language to learn to program in, pretty easy and forgiving while also being reasonably expressive, performant, offering functional as well as object oriented features/styles. the good folks at plotly are working on a shiny equivalent (dash) but it's not out yet. Django + matplotlib or bokeh or some client-side graphics like plotly.js is potentially powerful but not really as integrated as shiny.

kprybol 10y ago
Julia's biggest hurdle is the lack of well functioning DataFrames (or the current fork, DataTables). Tons of issues around nullable arrays, etc. have really slowed progress. I do think it's got a ton of upside, but I've found that reimplementing my R or Python scripts in Julia to be too much of a hassle. Costs of reimplemention greatly outweigh the not insignificant gains in speed. Also check out this article on updates to R 3.4. R tends to be fast enough for most work (I use it regularly on one-off analysis or things that won't ever make it farther than ad-hoc reporting/findings but can't imagine using it in production systems). The listed changes should go a long way towards making R just fast(er) enough for dealing with larger datasets (doesn't help with datasets larger than memory though). For large datasets all the momentum seems to be moving towards Spark (sparklyr is RStudio's SparkR integration. Very much a beta but getting better by the day). On the Python front Dask is awesome for out of memory computation that has no equivalent in R.
- mindcrime 10y ago
  For large datasets all the momentum seems to be moving towards Spark (sparklyr is RStudio's SparkR integration. Worst case, you can always use MPI with R and run on a Beowulf cluster. Of course that might not help if you want to use a function from a library, and the library itself expects everything to be in memory on one node, but at least it gives you another option for parallelization.
  - kprybol 10y ago
    Absolutely, though as you mention, removing the ability to use packages and the necessity of writing statistical code that properly accounts for data being spread out across multiple nodes would likely be out of the reach of your everyday/typical R user. An open sourced alternative to Revolution R/Microsoft R Server's out of core processing backend + distributed analtyics packages would be a huge addition to the R language.
- kprybol 10y ago
  Realized I never posted the link about R 3.4 that I referenced. https://cdn.ampproject.org/c/s/www.r-bloggers.com/performance-improvements-coming-to-r-3-4-0/amp/ https://cdn.ampproject.org/c/s/www.r-bloggers.com/performanc...
- wakkaflokka 10y ago
  How have I not heard of Dask? Would have saved me a lot of pain when trying to deal with oom datasets in my Luigi pipeline.
phillc73 10y ago
I guess I never found the <- syntax odd, because that's all I know. Not using = for assignment kind of makes sense to me, as equals has mathematical connotations. I always thought there were plenty of options for parallel and high(er) performance computing with R.[1] Data sets which don't entirely fit in memory can be stored pretty much anywhere and queried. Simplest would be an SQLite database. Formats like Feather[2] and fst[3] are also really useful. There are multiple R interfaces for Redis.[4] R has a library to interact with the Tensorflow API[5] and many other machine learning services. "Multiple legacy object semantics" doesn't really mean a lot to me. Knowing nothing else, that's just the way R is, plus I don't think there are that many really strange ways of doing things. Mostly there's a discernable pattern across functions. Sounding like a fanboy sure, but as I said I just don't know anything else and while Julia sounds nice (speed, way of the future etc) it doesn't have the breadth of library support, stability or community that R does yet and I don't see a compelling reason for me personally to spend time learning Python. Not saying R is best to the exclusion of all else, but for data analysis, generalised scripting and some web based reports/reactive apps, I can't find a compelling reason to switch, and I guess don't really understand the criticism of the language. [1] https://cran.r-project.org/web/views/HighPerformanceComputing.html https://cran.r-project.org/web/views/HighPerformanceComputin... [2] https://github.com/wesm/feather https://github.com/wesm/feather [3] https://github.com/fstpackage/fst https://github.com/fstpackage/fst [4] https://github.com/antirez/redis-doc/pull/798/files https://github.com/antirez/redis-doc/pull/798/files [5] https://github.com/rstudio/tensorflow/blob/master/README.md https://github.com/rstudio/tensorflow/blob/master/README.md
- haihaibye 10y ago
  >> that's all I know >> I can't find a compelling reason to switch See The Blub paradox: http://www.paulgraham.com/avg.html http://www.paulgraham.com/avg.html You should learn a few languages even if you don't need them, just for the mind expansion and learning what is necessary and what is your culture's conventions. It's like travelling to a different country, beware the patriot who has never left his home country!
- RockyMcNuts 10y ago
  If you like it and it does what you need, that's great! If/when you learn python, you'll get it. Python is expressive, readable, structured in a way R isn't...Shiny is all right but no one in their right mind would build a large general purpose web app in R.

kprybol 10y ago

Julia's biggest hurdle is the lack of well functioning DataFrames (or the current fork, DataTables). Tons of issues around nullable arrays, etc. have really slowed progress. I do think it's got a ton of upside, but I've found that reimplementing my R or Python scripts in Julia to be too much of a hassle. Costs of reimplemention greatly outweigh the not insignificant gains in speed. Also check out this article on updates to R 3.4. R tends to be fast enough for most work (I use it regularly on one-off analysis or things that won't ever make it farther than ad-hoc reporting/findings but can't imagine using it in production systems). The listed changes should go a long way towards making R just fast(er) enough for dealing with larger datasets (doesn't help with datasets larger than memory though). For large datasets all the momentum seems to be moving towards Spark (sparklyr is RStudio's SparkR integration. Very much a beta but getting better by the day). On the Python front Dask is awesome for out of memory computation that has no equivalent in R.

mindcrime 10y ago
For large datasets all the momentum seems to be moving towards Spark (sparklyr is RStudio's SparkR integration. Worst case, you can always use MPI with R and run on a Beowulf cluster. Of course that might not help if you want to use a function from a library, and the library itself expects everything to be in memory on one node, but at least it gives you another option for parallelization.
- kprybol 10y ago
  Absolutely, though as you mention, removing the ability to use packages and the necessity of writing statistical code that properly accounts for data being spread out across multiple nodes would likely be out of the reach of your everyday/typical R user. An open sourced alternative to Revolution R/Microsoft R Server's out of core processing backend + distributed analtyics packages would be a huge addition to the R language.
kprybol 10y ago
Realized I never posted the link about R 3.4 that I referenced. https://cdn.ampproject.org/c/s/www.r-bloggers.com/performance-improvements-coming-to-r-3-4-0/amp/ https://cdn.ampproject.org/c/s/www.r-bloggers.com/performanc...
wakkaflokka 10y ago
How have I not heard of Dask? Would have saved me a lot of pain when trying to deal with oom datasets in my Luigi pipeline.

phillc73 10y ago

I guess I never found the <- syntax odd, because that's all I know. Not using = for assignment kind of makes sense to me, as equals has mathematical connotations. I always thought there were plenty of options for parallel and high(er) performance computing with R.[1] Data sets which don't entirely fit in memory can be stored pretty much anywhere and queried. Simplest would be an SQLite database. Formats like Feather[2] and fst[3] are also really useful. There are multiple R interfaces for Redis.[4] R has a library to interact with the Tensorflow API[5] and many other machine learning services. "Multiple legacy object semantics" doesn't really mean a lot to me. Knowing nothing else, that's just the way R is, plus I don't think there are that many really strange ways of doing things. Mostly there's a discernable pattern across functions. Sounding like a fanboy sure, but as I said I just don't know anything else and while Julia sounds nice (speed, way of the future etc) it doesn't have the breadth of library support, stability or community that R does yet and I don't see a compelling reason for me personally to spend time learning Python. Not saying R is best to the exclusion of all else, but for data analysis, generalised scripting and some web based reports/reactive apps, I can't find a compelling reason to switch, and I guess don't really understand the criticism of the language. [1] https://cran.r-project.org/web/views/HighPerformanceComputing.html https://cran.r-project.org/web/views/HighPerformanceComputin... [2] https://github.com/wesm/feather https://github.com/wesm/feather [3] https://github.com/fstpackage/fst https://github.com/fstpackage/fst [4] https://github.com/antirez/redis-doc/pull/798/files https://github.com/antirez/redis-doc/pull/798/files [5] https://github.com/rstudio/tensorflow/blob/master/README.md https://github.com/rstudio/tensorflow/blob/master/README.md

haihaibye 10y ago
>> that's all I know >> I can't find a compelling reason to switch See The Blub paradox: http://www.paulgraham.com/avg.html http://www.paulgraham.com/avg.html You should learn a few languages even if you don't need them, just for the mind expansion and learning what is necessary and what is your culture's conventions. It's like travelling to a different country, beware the patriot who has never left his home country!
RockyMcNuts 10y ago
If you like it and it does what you need, that's great! If/when you learn python, you'll get it. Python is expressive, readable, structured in a way R isn't...Shiny is all right but no one in their right mind would build a large general purpose web app in R.