8 ms·

It is good that it starts with ecosystem. It is IMHO the biggest practical argument for or against any language (along with, what is related, community). R, la

by stared 8y ago

It is good that it starts with ecosystem. It is IMHO the biggest practical argument for or against any language (along with, what is related, community). R, language-wise, is awful, but thanks to its community it is very popular in statistics and some parts of machine learning (albeit it is getting less popular than Python, in the last years, vide https://www.kdnuggets.com/2018/05/poll-tools-analytics-data-science-machine-learning-results.html https://www.kdnuggets.com/2018/05/poll-tools-analytics-data-...). Python has pros and cons, but is de facto standard for data science due to its ecosystem. Haskell... well, show me a working Haskell data science project, and I may change my mind. So far I hear about Haskell from people wanting to make things clean and abstract (and as a self-development tool), rather than done. Julia may be more interesting (and it offers types!) plus much more focus om performance for iterative operations (which Haskell is lacking). Though, each time I tried it, I had to go back to Python - again, due to maturity of its data science ecosystem.

ellisv 8y ago
The best language is the one that gets the job done (or one you can use to get the job done). I've used R for a long time and agree it is terrible (as a language) but the community has learned from other languages and gotten much better in the past few years. I'm quite optimistic about Julia but agree the ecosystem is still maturing.
eigenspace 8y ago
> Julia may be more interesting (and it offers types!) plus much more focus om performance for iterative operations (which Haskell is lacking). Though, each time I tried it, I had to go back to Python - again, due to maturity of its data science ecosystem. I definitely agree that the Julia ecosystem is still maturing and not everything is as developed, however I've often found that the lack of maturity is less of an issue in Julia because the flexability of the language through generics, parametric types and macros makes a lot of the infrastructure needed in Python unessessary in Julia. Obviously your milage may vary in this regard though.
- nextos 8y ago
  Same here, Julia's lack of a very broad package ecosystem is less of an issue because i) it's very easy to interface with R, Python and C; ii) types and multiple dispatch plus basic libraries and efficient code generation make it quick to develop good number crunching code. Furthermore, the ecosystem is growing quickly, and it already has some unique math & stats packages not found elsewhere. Real gems. See https://discourse.julialang.org/t/what-package-s-are-state-of-the-art-or-attract-you-to-julia-and-make-you-stay-there-not-easily-replicateable-in-e-g-python-r-matlab/11294 https://discourse.julialang.org/t/what-package-s-are-state-o...
perturbation 8y ago
I am a data scientist (and use both R and Python regularly). I would take the plot in your link with a grain of salt - RapidMiner is not frequently used (at all) with any of my friends / coworkers, though maybe I'm just in a bubble. I like R a lot the more I use it. Tidyverse libraries (especially purrr + dplyr + ggplot2) make R a joy to use. I would argue that R is ahead of Python in terms of libraries for everything but deep learning (which is, after all, only part of ML).
- claytonjy 8y ago
  couldn't agree more; I think working in R is a vastly different experience now than it was 5+ years ago, and has changed (for the better) much more quickly than Python has. I'm primarily thinking of the tidyverse here; the three you mention are so much more intuitive than loops + pandas + matplotlib. With Max Kuhn's new tidymodels stuff, I think R has a real shot at providing a nice alternative to sci-kit learn, though there's a lot of catching-up to do. For the deep learning stuff, Keras in R is about as nice as Keras in python, but I'm not holding my breath for pytorch-like workflows in R.
  - perturbation 8y ago
    MLR and caret do a lot in the 'unified API for models' approach, though (IMO) they're not as streamlined / mature as sklearn, especially if trying to do modeling using sparse matrices (they mostly expect dense matrices as input).
    - claytonjy 8y ago
      I haven't played with MLR; how do you like it, esp. compared to caret? I never really caught on to caret as it felt rigid and clunky and non-idiomatic, but I've been using Max's newer rsample (setup CV) + recipes (like sklearn pipeline) + yardstick (metrics) packages to good effect lately. parsnip, which will handle the core model-fitting, seems promising but is too early to use yet. I don't expect sparse matrix support to get any better, as the core model functions would have to be rewritten entirely to avoid rehydrating them, which AFAIK nobody is seriously working on :(
nerdponx 8y ago
R, language-wise, is awful Yes and no. Data frames are built-in to the standard library, which also happens to contain a lot of stats functions that are data-frame-aware. The formula API is really convenient. The language distinguishes between NA and NaN. Vectors and matrices are the core data structure. The language is almost fully homoiconic and can be hacked and introspected more or less arbitrarily. The standard plotting library gives direct, near-pixel-level control and a lot of high-level convenience tools (albeit with bad defaults). There's a lot to be said for that. If you think of R as a statistics DSL, it's an excellent tool. If you think of it as a general-purpose programming language, a high-performance computing language, or just about anything else, you will come away disappointed. That's not what it's meant for.
T-R 8y ago
I agree in principle, but this is exactly why we've moved our data pipeline from Python to Haskell. The Python ecosystem has this concept of what is "pythonic", which, while it results in more code around the internet looking familiar, it largely resolves to "write it out yourself by hand". The result is that there are often no library functions for these things, even in libraries where you'd expect them to be. Need to coalesce an Optional, or need a switch statement? Write out the conditional. Need zipWith, flatMap, or mapMaybe? Write out the list comprehension. Need a combinator, or something curried? Write it out. We spent an obnoxious amount of time fighting with bugs from this, or weird edge cases in pandas and numpy, that would show up halfway through training, or not at all until we measured results. Heisenbugs from generators were also a huge issue. We ultimately decided the situation wasn't tenable for the quick iteration cycles we needed, and moved all the data munging to Haskell. Almost everything we need for data munging is in base, usually in prelude. If it isn't, it's in lens. Changes and refactors are quick, and no more surprises halfway through training.
- kjeetgill 8y ago
  I like using zip/map/reduce in my data munge phases as much as anybody and I think python can become a ball of mud too fast sometimes. But your statement is just so out there to me: > it largely resolves to "write it out yourself by hand" me: Oh, I always felt the ecosystem was thorough between numpy, scipy, and bindings to most any database or C code like zookeeper, openCV, redis, etc. but I guess you needed something in a more specialized domain? > Need to coalesce an Optional, or need a switch statement? Write out the conditional. Need zipWith, flatMap, or mapMaybe? Write out the list comprehension. Need a combinator, or something curried? Write it out. ... I tried to be charitable, and maybe coming from a language with all of these things these are glaring omissions, but I had trouble no rolling my eyes. Whatever itertools doesn't have can be done in 2-3 lines. A little repetitive? sure. But so onerous it factors into language decisions? bizarre.
  - T-R 8y ago
    It's not about it being hard to write them out; everything that needs to get written out is surface area for bugs. One missed edge case could just mean that a node of your input layer gets zeroed for some of your rows where it shouldn't, and all you see is no correlation where you expect some. The little bugs that creep in writing this simple boilerplate kill days of work. One person writes unzip as zip(*arr), another writes it as two separate assignments with the list comprehension written out. It's a tiny bit of code; they both look like perfectly fine unzipping code, and they pass tests, but if you pass one of those code blocks a generator instead of a list - say, someone swaps a list comprehension for map - half your data disappears. No errors, just a node that shows no correlation. Since the results are all getting serialized anyway, the amount of work to just have that part of the pipeline in a language with some guard rails against that kind of thing is pretty minimal.

by stared 8y ago

ellisv 8y ago

The best language is the one that gets the job done (or one you can use to get the job done). I've used R for a long time and agree it is terrible (as a language) but the community has learned from other languages and gotten much better in the past few years. I'm quite optimistic about Julia but agree the ecosystem is still maturing.

eigenspace 8y ago

> Julia may be more interesting (and it offers types!) plus much more focus om performance for iterative operations (which Haskell is lacking). Though, each time I tried it, I had to go back to Python - again, due to maturity of its data science ecosystem. I definitely agree that the Julia ecosystem is still maturing and not everything is as developed, however I've often found that the lack of maturity is less of an issue in Julia because the flexability of the language through generics, parametric types and macros makes a lot of the infrastructure needed in Python unessessary in Julia. Obviously your milage may vary in this regard though.

nextos 8y ago
Same here, Julia's lack of a very broad package ecosystem is less of an issue because i) it's very easy to interface with R, Python and C; ii) types and multiple dispatch plus basic libraries and efficient code generation make it quick to develop good number crunching code. Furthermore, the ecosystem is growing quickly, and it already has some unique math & stats packages not found elsewhere. Real gems. See https://discourse.julialang.org/t/what-package-s-are-state-of-the-art-or-attract-you-to-julia-and-make-you-stay-there-not-easily-replicateable-in-e-g-python-r-matlab/11294 https://discourse.julialang.org/t/what-package-s-are-state-o...

perturbation 8y ago

I am a data scientist (and use both R and Python regularly). I would take the plot in your link with a grain of salt - RapidMiner is not frequently used (at all) with any of my friends / coworkers, though maybe I'm just in a bubble. I like R a lot the more I use it. Tidyverse libraries (especially purrr + dplyr + ggplot2) make R a joy to use. I would argue that R is ahead of Python in terms of libraries for everything but deep learning (which is, after all, only part of ML).

claytonjy 8y ago
couldn't agree more; I think working in R is a vastly different experience now than it was 5+ years ago, and has changed (for the better) much more quickly than Python has. I'm primarily thinking of the tidyverse here; the three you mention are so much more intuitive than loops + pandas + matplotlib. With Max Kuhn's new tidymodels stuff, I think R has a real shot at providing a nice alternative to sci-kit learn, though there's a lot of catching-up to do. For the deep learning stuff, Keras in R is about as nice as Keras in python, but I'm not holding my breath for pytorch-like workflows in R.
- perturbation 8y ago
  MLR and caret do a lot in the 'unified API for models' approach, though (IMO) they're not as streamlined / mature as sklearn, especially if trying to do modeling using sparse matrices (they mostly expect dense matrices as input).
  - claytonjy 8y ago
    I haven't played with MLR; how do you like it, esp. compared to caret? I never really caught on to caret as it felt rigid and clunky and non-idiomatic, but I've been using Max's newer rsample (setup CV) + recipes (like sklearn pipeline) + yardstick (metrics) packages to good effect lately. parsnip, which will handle the core model-fitting, seems promising but is too early to use yet. I don't expect sparse matrix support to get any better, as the core model functions would have to be rewritten entirely to avoid rehydrating them, which AFAIK nobody is seriously working on :(

nerdponx 8y ago

R, language-wise, is awful Yes and no. Data frames are built-in to the standard library, which also happens to contain a lot of stats functions that are data-frame-aware. The formula API is really convenient. The language distinguishes between NA and NaN. Vectors and matrices are the core data structure. The language is almost fully homoiconic and can be hacked and introspected more or less arbitrarily. The standard plotting library gives direct, near-pixel-level control and a lot of high-level convenience tools (albeit with bad defaults). There's a lot to be said for that. If you think of R as a statistics DSL, it's an excellent tool. If you think of it as a general-purpose programming language, a high-performance computing language, or just about anything else, you will come away disappointed. That's not what it's meant for.

T-R 8y ago

I agree in principle, but this is exactly why we've moved our data pipeline from Python to Haskell. The Python ecosystem has this concept of what is "pythonic", which, while it results in more code around the internet looking familiar, it largely resolves to "write it out yourself by hand". The result is that there are often no library functions for these things, even in libraries where you'd expect them to be. Need to coalesce an Optional, or need a switch statement? Write out the conditional. Need zipWith, flatMap, or mapMaybe? Write out the list comprehension. Need a combinator, or something curried? Write it out. We spent an obnoxious amount of time fighting with bugs from this, or weird edge cases in pandas and numpy, that would show up halfway through training, or not at all until we measured results. Heisenbugs from generators were also a huge issue. We ultimately decided the situation wasn't tenable for the quick iteration cycles we needed, and moved all the data munging to Haskell. Almost everything we need for data munging is in base, usually in prelude. If it isn't, it's in lens. Changes and refactors are quick, and no more surprises halfway through training.

kjeetgill 8y ago
I like using zip/map/reduce in my data munge phases as much as anybody and I think python can become a ball of mud too fast sometimes. But your statement is just so out there to me: > it largely resolves to "write it out yourself by hand" me: Oh, I always felt the ecosystem was thorough between numpy, scipy, and bindings to most any database or C code like zookeeper, openCV, redis, etc. but I guess you needed something in a more specialized domain? > Need to coalesce an Optional, or need a switch statement? Write out the conditional. Need zipWith, flatMap, or mapMaybe? Write out the list comprehension. Need a combinator, or something curried? Write it out. ... I tried to be charitable, and maybe coming from a language with all of these things these are glaring omissions, but I had trouble no rolling my eyes. Whatever itertools doesn't have can be done in 2-3 lines. A little repetitive? sure. But so onerous it factors into language decisions? bizarre.
- T-R 8y ago
  It's not about it being hard to write them out; everything that needs to get written out is surface area for bugs. One missed edge case could just mean that a node of your input layer gets zeroed for some of your rows where it shouldn't, and all you see is no correlation where you expect some. The little bugs that creep in writing this simple boilerplate kill days of work. One person writes unzip as zip(*arr), another writes it as two separate assignments with the list comprehension written out. It's a tiny bit of code; they both look like perfectly fine unzipping code, and they pass tests, but if you pass one of those code blocks a generator instead of a list - say, someone swaps a list comprehension for map - half your data disappears. No errors, just a node that shows no correlation. Since the results are all getting serialized anyway, the amount of work to just have that part of the pipeline in a language with some guard rails against that kind of thing is pretty minimal.