4 ms·

numpy exists (somewhat indirectly) because matlab cost $$$. and now numpy is a far larger use of matlab's concepts than matlab will ever be. I haven't seen an

by dekhn 3y ago

numpy exists (somewhat indirectly) because matlab cost $$$. and now numpy is a far larger use of matlab's concepts than matlab will ever be. I haven't seen anything an "A-type" data scientist needs in R/SAS/SPSS that was intrinsic to the language and couldn't be ported to Python. I don't want a "data/scientific" language, or a "web" language, or a "UI" language- I want one language that explicitly supports all the usecases well-enough.

qsort 3y ago
> I haven't seen anything an "A-type" data scientist needs in R/SAS/SPSS that was intrinsic to the language Not intrinsic to the language but in terms of the ecosystem, for a purely A-type data science workload R is significantly better. dplyr + ggplot vs. pandas + matplotlib isn't even remotely close. Now, obviously in the real world nothing is ever purely model work, which is why Python more than makes up for the difference. > I don't want a "data/scientific" language, or a "web" language, or a "UI" language- I want one language that explicitly supports all the usecases well-enough. Me neither. *taps sign* >> The difference is that Python doesn't entirely suck as a general-purpose language.
- CuriouslyC 3y ago
  R is great when your data is already together in one place and homogenous/processed. In that instance the workflow using dplyr/ggplot is better. R the language is hot garbage though, and the libraries available for doing random things that aren't working with certain data structures, doing math or plotting are generally much worse than their python equivalents. That is the reason python dominates data engineering, and why it's pushing R out everywhere but academia.
  - dekhn 3y ago
    R will still have a solid place in industrial pharma for a long time (even as python growths, R growth will continue. R is pretty tightly intertwined with the process of processing and submitting data from clinical trials to the FDA, much to SAS's chagrin. Personally I think we have to accept that R, Python, Java, C++, and Go are going to be with us for the rest of our lives. I would expect PHP, Ruby, and Perl to go away much faster. Rust is still in question.
    - eep_social 3y ago
      Rails is probably going to keep Ruby relevant for a lot longer than Perl. I might go as far as to predict that Ruby will be the new COBOL or FORTRAN - widely used but neither cheap nor easy to find someone to modify it.
    - fbdab103 3y ago
      Are you from the future? R is making progress to replacing SAS in pharma, but has a long ways to go. Here is an article from last year[0] patting themselves on the back for making a R submission to the FDA. There are oodles of existing processes where data has to be formatted just so because that is how SAS does it. Nobody wants to rip up and re-validate that code until they must. [0] https://www.r-consortium.org/blog/2022/03/16/update-successful-r-based-test-package-submitted-to-fda https://www.r-consortium.org/blog/2022/03/16/update-successf...
      - dekhn 3y ago
        The trope about SAS being required is pretty old... people trotted that out when I last worked in pharma ~15 years ago. The FDA specifically released an article saying that R is perfectly acceptable for submission https://blog.revolutionanalytics.com/2012/06/fda-r-ok.html https://blog.revolutionanalytics.com/2012/06/fda-r-ok.html Please don't spread the SAS FUD. I work in Pharma and talk to the statisticians all the time.
        fbdab103 3y ago
        I currently work in pharma, and I can point to teams of SAS programmers. There are definitely R efforts, and I have no doubt it is the eventual future, but it is not yet here.
- dekhn 3y ago
  If there is anything in R that is better, it could be ported. It's kind of different from language intrinsics that can't be easily ported. In fact I think there's a great argument for writing a standardized data processing and visual representation layer in C++, and then making it usable from all the languages. It would be nice if this was true for the notebook layer.
  - disgruntledphd2 3y ago
    It's the domain knowledge that's difficult. Like the example up thread about the lack of generalised additive models. The version in R was written originally by the developers of the method. There's no Python version because no one with sufficient expertise has ported it. Don't get me wrong, I know mostly write Python because it's a better general purpose language but there are big big tradeoffs.
fluidcruft 3y ago
Matlab has a more interesting dispatch model than python though. (Mathematica has even better dispatch but I haven't used it in almost 20 years at this point) Julia's model is even better, but I just never have time to dig into it deeply enough to get to the zen of Julia. One day when I have time to understand Julia I'd like to, but it's hard to compete with the reliability of python just always working.
- dekhn 3y ago
  What is dispatch in this context? Function call polymorphism? Aside: I wasn't a big fan of the Mathematica language so I wrote "PyML" a looooong time ago- it turned unevaluated Python expressions into Mathematica, transmitted them via MathLink to be evaluated the Mathematica kernel, and then the results were send back and converted to a Python expression. This was long before sympy. It never went anywhere (I found ways to simply not depend on Mathematica).
  - fluidcruft 3y ago
    Yes, function call polymorphism. Especially combined with Mathematica's piecewise syntax. I didn't really use it for numerics, it was more for Lisp/Prolog type things. I was always impressed at how short and readable the Mathematica code ended up being. I used to use Mathematica quite a bit to generate C++ functions. But it's a very different use case than Matlab. One of the things I like about python is it isn't too bad for sliding between lispy/functional modes for some things and more lapacky things elsewhere. Matlab has a better syntax for the lapack parts but trying to do anything functional gets really annoying and very, very slow. But still I would say Mathematica has the best pure math syntax for those sorts of things. I don't have any experience with sympy, I should probably check it out.
    - dekhn 3y ago
      Mathematica is nothing less than a work of art. I dedicated a fair amount of time to become proficient, but it never solved the use case I had (find derivatives for complicated functions- this was before autodiff was common). Unfortunately, Wolfram himself has made a number of decisions that greatly limited its applicability and so it will always be niche.
BlueTemplar 3y ago
Well, why didn't the free alternatives to Matlab like Octave or Scilab win out then ?
- gilcot 3y ago
  I think people were not ready to switch from some heavy established software. Also by that time, costless was seen as less qualitative (certainly because organisations are happy with licences and are always looking for support contract ?) Coming later, when the market(?) is ready, made R more successful. I think people used to compare to established standard and detractors are always there to point out missing stuffs, giving impression that Octave or Scilab are behind. Coming with a radical different language, R was condemned to quickly succeed or perish, and anyway avoid direct and frontal comparisons.
  - pwdisswordfishc 3y ago
    But R is itself a reimplementation of S.
- fbdab103 3y ago
  Same reason people use Excel and not LibreOffice. Until you can promise 100% compatibility, users are going to run into something that breaks vs the original platform. For real business processes, the licensing cost is not worth sweating about the problem for parity.
bigger_cheese 3y ago
>I haven't seen anything an "A-type" data scientist needs in R/SAS/SPSS that was intrinsic to the language and couldn't be ported to Python. I don't know what "A-Type" is so I might be misunderstanding you. For SAS at least in my Org what keeps it entrenched is Enterprise Guide and the associated environment. Having a drag and drop GUI to do the data extraction and manipulation and the analytical tasks makes it very easy for non experts to be incredibly productive. The people that use it here are Engineers (Non-software variety) and Finance accountant types. These people would not be productive writing python code (or R) but still need to use something heavier then excel for data analysis. Over the years we have chipped away at parts of SAS with things like Power BI and Azure ML Studio but I don't see python playing in the same space.
mr_toad 3y ago
> and couldn't be ported to Python “Couldn’t be” is not the same as “has been”. If R has a package that Python doesn’t, I’m not going to port it to Python, I’m just going to use R.

by dekhn 3y ago

qsort 3y ago

> I haven't seen anything an "A-type" data scientist needs in R/SAS/SPSS that was intrinsic to the language Not intrinsic to the language but in terms of the ecosystem, for a purely A-type data science workload R is significantly better. dplyr + ggplot vs. pandas + matplotlib isn't even remotely close. Now, obviously in the real world nothing is ever purely model work, which is why Python more than makes up for the difference. > I don't want a "data/scientific" language, or a "web" language, or a "UI" language- I want one language that explicitly supports all the usecases well-enough. Me neither. *taps sign* >> The difference is that Python doesn't entirely suck as a general-purpose language.

CuriouslyC 3y ago
R is great when your data is already together in one place and homogenous/processed. In that instance the workflow using dplyr/ggplot is better. R the language is hot garbage though, and the libraries available for doing random things that aren't working with certain data structures, doing math or plotting are generally much worse than their python equivalents. That is the reason python dominates data engineering, and why it's pushing R out everywhere but academia.
- dekhn 3y ago
  R will still have a solid place in industrial pharma for a long time (even as python growths, R growth will continue. R is pretty tightly intertwined with the process of processing and submitting data from clinical trials to the FDA, much to SAS's chagrin. Personally I think we have to accept that R, Python, Java, C++, and Go are going to be with us for the rest of our lives. I would expect PHP, Ruby, and Perl to go away much faster. Rust is still in question.
  - eep_social 3y ago
    Rails is probably going to keep Ruby relevant for a lot longer than Perl. I might go as far as to predict that Ruby will be the new COBOL or FORTRAN - widely used but neither cheap nor easy to find someone to modify it.
  - fbdab103 3y ago
    Are you from the future? R is making progress to replacing SAS in pharma, but has a long ways to go. Here is an article from last year[0] patting themselves on the back for making a R submission to the FDA. There are oodles of existing processes where data has to be formatted just so because that is how SAS does it. Nobody wants to rip up and re-validate that code until they must. [0] https://www.r-consortium.org/blog/2022/03/16/update-successful-r-based-test-package-submitted-to-fda https://www.r-consortium.org/blog/2022/03/16/update-successf...
    - dekhn 3y ago
      The trope about SAS being required is pretty old... people trotted that out when I last worked in pharma ~15 years ago. The FDA specifically released an article saying that R is perfectly acceptable for submission https://blog.revolutionanalytics.com/2012/06/fda-r-ok.html https://blog.revolutionanalytics.com/2012/06/fda-r-ok.html Please don't spread the SAS FUD. I work in Pharma and talk to the statisticians all the time.
      - fbdab103 3y ago
        I currently work in pharma, and I can point to teams of SAS programmers. There are definitely R efforts, and I have no doubt it is the eventual future, but it is not yet here.
dekhn 3y ago
If there is anything in R that is better, it could be ported. It's kind of different from language intrinsics that can't be easily ported. In fact I think there's a great argument for writing a standardized data processing and visual representation layer in C++, and then making it usable from all the languages. It would be nice if this was true for the notebook layer.
- disgruntledphd2 3y ago
  It's the domain knowledge that's difficult. Like the example up thread about the lack of generalised additive models. The version in R was written originally by the developers of the method. There's no Python version because no one with sufficient expertise has ported it. Don't get me wrong, I know mostly write Python because it's a better general purpose language but there are big big tradeoffs.

fluidcruft 3y ago

Matlab has a more interesting dispatch model than python though. (Mathematica has even better dispatch but I haven't used it in almost 20 years at this point) Julia's model is even better, but I just never have time to dig into it deeply enough to get to the zen of Julia. One day when I have time to understand Julia I'd like to, but it's hard to compete with the reliability of python just always working.

dekhn 3y ago
What is dispatch in this context? Function call polymorphism? Aside: I wasn't a big fan of the Mathematica language so I wrote "PyML" a looooong time ago- it turned unevaluated Python expressions into Mathematica, transmitted them via MathLink to be evaluated the Mathematica kernel, and then the results were send back and converted to a Python expression. This was long before sympy. It never went anywhere (I found ways to simply not depend on Mathematica).
- fluidcruft 3y ago
  Yes, function call polymorphism. Especially combined with Mathematica's piecewise syntax. I didn't really use it for numerics, it was more for Lisp/Prolog type things. I was always impressed at how short and readable the Mathematica code ended up being. I used to use Mathematica quite a bit to generate C++ functions. But it's a very different use case than Matlab. One of the things I like about python is it isn't too bad for sliding between lispy/functional modes for some things and more lapacky things elsewhere. Matlab has a better syntax for the lapack parts but trying to do anything functional gets really annoying and very, very slow. But still I would say Mathematica has the best pure math syntax for those sorts of things. I don't have any experience with sympy, I should probably check it out.
  - dekhn 3y ago
    Mathematica is nothing less than a work of art. I dedicated a fair amount of time to become proficient, but it never solved the use case I had (find derivatives for complicated functions- this was before autodiff was common). Unfortunately, Wolfram himself has made a number of decisions that greatly limited its applicability and so it will always be niche.

BlueTemplar 3y ago

Well, why didn't the free alternatives to Matlab like Octave or Scilab win out then ?

gilcot 3y ago
I think people were not ready to switch from some heavy established software. Also by that time, costless was seen as less qualitative (certainly because organisations are happy with licences and are always looking for support contract ?) Coming later, when the market(?) is ready, made R more successful. I think people used to compare to established standard and detractors are always there to point out missing stuffs, giving impression that Octave or Scilab are behind. Coming with a radical different language, R was condemned to quickly succeed or perish, and anyway avoid direct and frontal comparisons.
- pwdisswordfishc 3y ago
  But R is itself a reimplementation of S.
fbdab103 3y ago
Same reason people use Excel and not LibreOffice. Until you can promise 100% compatibility, users are going to run into something that breaks vs the original platform. For real business processes, the licensing cost is not worth sweating about the problem for parity.

bigger_cheese 3y ago

>I haven't seen anything an "A-type" data scientist needs in R/SAS/SPSS that was intrinsic to the language and couldn't be ported to Python. I don't know what "A-Type" is so I might be misunderstanding you. For SAS at least in my Org what keeps it entrenched is Enterprise Guide and the associated environment. Having a drag and drop GUI to do the data extraction and manipulation and the analytical tasks makes it very easy for non experts to be incredibly productive. The people that use it here are Engineers (Non-software variety) and Finance accountant types. These people would not be productive writing python code (or R) but still need to use something heavier then excel for data analysis. Over the years we have chipped away at parts of SAS with things like Power BI and Azure ML Studio but I don't see python playing in the same space.

mr_toad 3y ago

> and couldn't be ported to Python “Couldn’t be” is not the same as “has been”. If R has a package that Python doesn’t, I’m not going to port it to Python, I’m just going to use R.