4 ms·

The bulk of people trying to get stuff done in a data / statistical / ML context are heavily biased toward not having to write their own tools. I fancy myself a

The bulk of people trying to get stuff done in a data / statistical / ML context are heavily biased toward not having to write their own tools. I fancy myself a reasonably good programmer, and reasonably competent at applied statistics, but if I had to, say, implement a way to estimate my own GLMs, I’d quickly go crazy. A lot of research and data science (at least as far as the computing is concerned) is plumbing just like a lot of web development is writing CRUD apps. Part of the success of R is that statistical researchers (i.e. the tool makers) implement clever new methods in R. Someone else has done the lit review comparing that fancy new method to other related methods. Someone else has validated the math. Someone else uploaded the code to CRAN. Python has historically lagged R in the stats space because statisticians weren’t building tooling for it. Even fairly common models that have long been in R are still missing from the Python ecosystem. But something started to change a few years back when the buzz about Bayesian methods started. Most MCMC based estimations of Bayesian models became more of a computational problem than a stats problem. Python started to not be laughably deficient in Bayesian methods. And then we started having an explosion of deep learning whatnot, and nobody knew what the hell they were doing, and so everyone just used the frameworks gifted from big companies like Google—and they released stuff with Python APIs. So I think that what language you use for data science is still driven by ecosystem, and also your ecosystem’s comparative advantage in implementing the methods you care about. All else being equal, I’d love it if we had better languages for data science. Julia may get there—it’s a reasonably nice language that both Pythonistas and R...barians(?) find inoffensive. Given the strength of R’s tidyverse in implementing useful DSLs, I’d love to see the Racket community blow our minds with the language-oriented programming that they’re about, but ultimately mind-blowing will take a back seat to ecosystem.

deleted 8y ago
[deleted]
werphesti 8y ago
I've been using R for 20 years, and I think the history does matter. R is an open source clone of S, which had some traction with rigorous statisticians via Bell Labs. So it had stats at its core. R was just a version of S that had less overhead and was easier to access. The explosive growth of R really is more about the explosive growth of stats and data analysis than anything else. R and S were sort of growing within the stats community anyway, and then when stats took off, so did R. I think it's open source quality and the fact that it is more similar to other programming platforms than SAS or Stata helped too as stats branched out into computer science. Lisp is an interesting comparison with lessons for Haskell. XLispStat was a competitor to R very early on but died out relatively quickly. I always was sad about that, because I loved lisp, and it was great having stats embedded in a broader language, but the reaction was uniformly the same: that lisp was just too weird, too hard to read, and too hard to program in. Lisp has diminished in importance in computer science more broadly, but didn't die out in the same way XLispStat did in stats. Haskell is suffering similar issues. The lack of ecosystem is partly because it's coming from the outside in, rather from the inside out, but part of it is because it is just perceived as odd. I love functional programming but am increasingly becoming convinced that any language that pushes too hard on one paradigm is going to lose out to one that is less pure. As great as functional approaches are, sometimes it's just easier to think and organize procedurally, and this is increasingly true as you get closer to the metal. The real elephant in the room is the poor performance of the languages currently dominating data science, whether that be R or Python. The LLVM basically made it possible to right a conceptually clean language that also exhibits good performance, so we don't have to choose between expressive and performant languages so much, unless you're talking about embedded systems or low-level systems programming. Although many people don't want to program a GLM (and maybe shouldn't be for integrity's sake) there are many times when going down to the likelihood function and optimization level for an unusual case shouldn't result in a huge performance hit. You shouldn't have to change to C or even Rust for something that conceptually isn't that much lower level. Things like Julia and Nim really make this possible, and is where things will probably eventually head (even if I'm not sure it will be either of those). I also wonder if we'll see things like Rust taking off via higher-level extensions such as Lia or Gluon. My guess is that if "functional" languages will take off with stats, it will be through something like OCaml (especially if that gets its distributed/parallelism worked out quickly enough).
- peatmoss 8y ago
  Thank you for the thoughtful reply. The impedance mismatch when needing to write performant code from the R/Python universe is real. And that Julia has managed Pretty Good performance while simultaneously being a pretty decent language is huge. XLispStat was before my (stats) time, sadly. Being part of a more general purpose language ecosystem is one of the selling points of Python, though from a performance and expressiveness standpoint Python is strictly inferior to Lisp in just about every way. Except weird. To be honest, I’m more fussed about good support for functional style than I am about enforced functional purity. So in that sense, maybe you’re right that the next big data / stats language will be multi paradigm.

deleted 8y ago

[deleted]

werphesti 8y ago

I've been using R for 20 years, and I think the history does matter. R is an open source clone of S, which had some traction with rigorous statisticians via Bell Labs. So it had stats at its core. R was just a version of S that had less overhead and was easier to access. The explosive growth of R really is more about the explosive growth of stats and data analysis than anything else. R and S were sort of growing within the stats community anyway, and then when stats took off, so did R. I think it's open source quality and the fact that it is more similar to other programming platforms than SAS or Stata helped too as stats branched out into computer science. Lisp is an interesting comparison with lessons for Haskell. XLispStat was a competitor to R very early on but died out relatively quickly. I always was sad about that, because I loved lisp, and it was great having stats embedded in a broader language, but the reaction was uniformly the same: that lisp was just too weird, too hard to read, and too hard to program in. Lisp has diminished in importance in computer science more broadly, but didn't die out in the same way XLispStat did in stats. Haskell is suffering similar issues. The lack of ecosystem is partly because it's coming from the outside in, rather from the inside out, but part of it is because it is just perceived as odd. I love functional programming but am increasingly becoming convinced that any language that pushes too hard on one paradigm is going to lose out to one that is less pure. As great as functional approaches are, sometimes it's just easier to think and organize procedurally, and this is increasingly true as you get closer to the metal. The real elephant in the room is the poor performance of the languages currently dominating data science, whether that be R or Python. The LLVM basically made it possible to right a conceptually clean language that also exhibits good performance, so we don't have to choose between expressive and performant languages so much, unless you're talking about embedded systems or low-level systems programming. Although many people don't want to program a GLM (and maybe shouldn't be for integrity's sake) there are many times when going down to the likelihood function and optimization level for an unusual case shouldn't result in a huge performance hit. You shouldn't have to change to C or even Rust for something that conceptually isn't that much lower level. Things like Julia and Nim really make this possible, and is where things will probably eventually head (even if I'm not sure it will be either of those). I also wonder if we'll see things like Rust taking off via higher-level extensions such as Lia or Gluon. My guess is that if "functional" languages will take off with stats, it will be through something like OCaml (especially if that gets its distributed/parallelism worked out quickly enough).

peatmoss 8y ago
Thank you for the thoughtful reply. The impedance mismatch when needing to write performant code from the R/Python universe is real. And that Julia has managed Pretty Good performance while simultaneously being a pretty decent language is huge. XLispStat was before my (stats) time, sadly. Being part of a more general purpose language ecosystem is one of the selling points of Python, though from a performance and expressiveness standpoint Python is strictly inferior to Lisp in just about every way. Except weird. To be honest, I’m more fussed about good support for functional style than I am about enforced functional purity. So in that sense, maybe you’re right that the next big data / stats language will be multi paradigm.