7 ms·

Numpy is certainly amazing, but there are tons of competitors in the data/scientific space, which pure "A-type" data scientists tend to prefer: R, SSPS, Matlab.

by qsort 3y ago

Numpy is certainly amazing, but there are tons of competitors in the data/scientific space, which pure "A-type" data scientists tend to prefer: R, SSPS, Matlab... The difference is that Python doesn't entirely suck as a general-purpose language. Sure, you might have better options, but it's still reasonable to write almost anything in Python. Other scripting languages like Ruby, JS and Lua are probably a little bit better when evaluated strictly on their merits as programming languages, but they lack the ecosystem. In other words, it might be inelegant, slow, a mess, whatever, but the concept behind it is basically correct.

TuringNYC 3y ago
>> R, SSPS, Matlab... I'd immediately throw out "competitors" that cost hundreds of dollars to use. https://www.mathworks.com/pricing-licensing.html https://www.mathworks.com/pricing-licensing.html Academic pricing is $275/yr and normal is almost a thousand a year. Then you pay for modules on top.
- ImaCake 3y ago
  Yes, and this is why R has become so dominant in the data and statistics spaces. R is free, and has gained so much momentum that it often has packages that no other language has. Yesterday I found out that there was just no way to easily reproduce mgcv::gam in python. There are loads of similar examples. As someone who likes both R and Python, and had the misfortune to learn Stata; I am very glad to see R winning over these expensive dinosaurs. Maybe one day it will even displace matlab!
  - a_bonobo 3y ago
    R even has packages to read in NanoSIMS data! [1] There are perhaps a hundred NanoSIMS machines in the world as they're extremely expensive [2]. Yet we have an R package on CRAN to analyse their data! [1] https://cran.r-project.org/web/packages/lans2r/vignettes/lans2r.html https://cran.r-project.org/web/packages/lans2r/vignettes/lan... [2] This 2017 reviews says there are 42 machines: https://www.osti.gov/servlets/purl/1398172 https://www.osti.gov/servlets/purl/1398172
deadmutex 3y ago
Matlab costs $$. Matlab was ahead in perf for a long time, not sure now. E.g. they used to JIT code, etc. Comparing python to R, python feels like a much more mature language. R always seemed only useful for small scale projects, e.g. writing some script to process data with < 20 lines of code. Python codebases just scale much better.
- kstrauser 3y ago
  It also has the nice feature of being good at multiple things. If you get a model running in Python, and decide you want to be able to run the code via an API, put it behind Flask and ta-da!, now it's a network service. It's not the best way to write analysis code, and it's not the best way to write a web service, but it's probably the best way to write analysis code that runs on a web service. I picked web services as a random example, not one I'm particularly passionate about. But as a general trend, if you want to combine 2 unrelated domains, Python probably has you covered better or more easily than just about any other language.
  - fjkdlsjflkds 3y ago
    > If you get a model running in Python, and decide you want to be able to run the code via an API, put it behind Flask and ta-da!, now it's a network service. You could say the same about a model (or any function) running in R. Want to instantly convert it to a web service? Use plumbr or RestRserve packages for that. Want to make a quick web application around it? Use the shiny package for that. Just because Python is good at X, doesn't mean other languages are necessarily worse at it.
local_person 3y ago
[dead]
dekhn 3y ago
numpy exists (somewhat indirectly) because matlab cost $$$. and now numpy is a far larger use of matlab's concepts than matlab will ever be. I haven't seen anything an "A-type" data scientist needs in R/SAS/SPSS that was intrinsic to the language and couldn't be ported to Python. I don't want a "data/scientific" language, or a "web" language, or a "UI" language- I want one language that explicitly supports all the usecases well-enough.
- qsort 3y ago
  > I haven't seen anything an "A-type" data scientist needs in R/SAS/SPSS that was intrinsic to the language Not intrinsic to the language but in terms of the ecosystem, for a purely A-type data science workload R is significantly better. dplyr + ggplot vs. pandas + matplotlib isn't even remotely close. Now, obviously in the real world nothing is ever purely model work, which is why Python more than makes up for the difference. > I don't want a "data/scientific" language, or a "web" language, or a "UI" language- I want one language that explicitly supports all the usecases well-enough. Me neither. *taps sign* >> The difference is that Python doesn't entirely suck as a general-purpose language.
  - CuriouslyC 3y ago
    R is great when your data is already together in one place and homogenous/processed. In that instance the workflow using dplyr/ggplot is better. R the language is hot garbage though, and the libraries available for doing random things that aren't working with certain data structures, doing math or plotting are generally much worse than their python equivalents. That is the reason python dominates data engineering, and why it's pushing R out everywhere but academia.
    - dekhn 3y ago
      R will still have a solid place in industrial pharma for a long time (even as python growths, R growth will continue. R is pretty tightly intertwined with the process of processing and submitting data from clinical trials to the FDA, much to SAS's chagrin. Personally I think we have to accept that R, Python, Java, C++, and Go are going to be with us for the rest of our lives. I would expect PHP, Ruby, and Perl to go away much faster. Rust is still in question.
CuriouslyC 3y ago
Some industries are stuck on matlab (jet propulsion, for example) but in general people from sciences are either Python or R, and SAS/SPSS are only really used in places like health care or pharma where there's a lot of regulation in place, because the software is designed around that regulation, unlike Python and R.
- dimask 3y ago
  It is also still quite dominant in neuroscientific fields, as there is a lot of software and code there from the past, but it is changing little by little.
karmasimida 3y ago
Anyone who used Matlab to process text before 2015 wouldn’t consider it anywhere closer as a competitor to Python Same goes for R/SASS Python has no competitor in terms of the whole package, it is a full featured language while being particularly good at data
nl 3y ago
This is a good point. I've done a fair bit of R and there is no doubt that as a pure data science tool it's usually quicker to produce something than Python. But I prefer Python because inevitably any data science project ends up having other "bits" that aren't pure data science, and Python's super string general purpose computing libraries are hard to beat.
twobitshifter 3y ago
I would just add that for scientific scripting Julia is usually nicer to program in than the combo of numpy and Python. Julia is fast enough on its own that you can do the matrix calculations directly and it supports real mathematic matrix operators. It’s also a great type based design that makes using packages magical.
- gaganyaan 3y ago
  I think Julia has a hard uphill battle against the "Python is the second best language for anything" effect. Julia looks pretty cool to me, but I already know Python. I'll probably learn more about it in my spare time, but I'm also never going to be able to convince my coworkers to try it out, because they also know Python and that's good enough for them. And thus Python's dominance continues.
  - ziotom78 3y ago
    I was able to convince my coworkers to stop working on a Python project (a simulator for a scientific instrument) once we all realised that we would have not been able to keep using plain Python because of performance. When the alternatives where (1) to force everybody to learn either C or Fortran, or (2) to rewrite everything in Julia, they asked me to provide examples of both solutions. After I showed them a few different approaches (pybind, cbind, f2py...), there was full consensus for Julia. We moved what we implemented so far (it was not to much, luckily), and so far nobody has regretted the choice. The problem in using two languages is that you do not have just to learn a new language besides Python. You have also to figure out how to properly interface them. (f2py was considered the easiest solution, but we feared that the 1- vs 0-based array indexing would have likely caused many bugs.)
    - targafarian 3y ago
      Did you evaluate Cython? I'm not anti-Julia, but I like that my Cython code is useable out of the box from Python, with no wrapping, and then users can continue to use their Jupiter + Python scripting workflows with performant bespoke modules complemented by the full Python ecosystem. Someday I'll do a project in Julia. But for some such projects, Rust seems fully guaranteed to be performant while Julia might or might not be, so I might still lean towards Rust (unless one of the high quality packages of Julia removes a lot of development time, which is a decent possibility).
nextlevelwizard 3y ago
>Python doesn't entirely suck as a general-purpose language. I always get caught off guard by comments like these. In my mind Python doesn't at all suck as a general purpose language. Only real argument is execution speed, but most people aren't actually writing code where Python's "slowness" matters and if you actually need that speed you can write that part in C and still use it from your Python application.
LeanderK 3y ago
> which pure "A-type" data scientists tend to prefer: R, SSPS, Matlab I very much disagree and I would say it's the opposite. The only competitor for data-scientists is R, especially if you're doing stats-heavy analysis (opposed to an ML-heavy analysis). SSPS is in my experience used mainly by someone without a data-scientist background, e.g. psychology, similarly to matlab, where the only users I know are engineers. I would bet that the overwhelming amount of data-scientists use python, even if it's just to set up the data-pipeline. It's just the better general programming language and data-scientist are not analysing all the time but have to get the data in shape and available first.

by qsort 3y ago

TuringNYC 3y ago

>> R, SSPS, Matlab... I'd immediately throw out "competitors" that cost hundreds of dollars to use. https://www.mathworks.com/pricing-licensing.html https://www.mathworks.com/pricing-licensing.html Academic pricing is $275/yr and normal is almost a thousand a year. Then you pay for modules on top.

ImaCake 3y ago
Yes, and this is why R has become so dominant in the data and statistics spaces. R is free, and has gained so much momentum that it often has packages that no other language has. Yesterday I found out that there was just no way to easily reproduce mgcv::gam in python. There are loads of similar examples. As someone who likes both R and Python, and had the misfortune to learn Stata; I am very glad to see R winning over these expensive dinosaurs. Maybe one day it will even displace matlab!
- a_bonobo 3y ago
  R even has packages to read in NanoSIMS data! [1] There are perhaps a hundred NanoSIMS machines in the world as they're extremely expensive [2]. Yet we have an R package on CRAN to analyse their data! [1] https://cran.r-project.org/web/packages/lans2r/vignettes/lans2r.html https://cran.r-project.org/web/packages/lans2r/vignettes/lan... [2] This 2017 reviews says there are 42 machines: https://www.osti.gov/servlets/purl/1398172 https://www.osti.gov/servlets/purl/1398172

deadmutex 3y ago

Matlab costs $$. Matlab was ahead in perf for a long time, not sure now. E.g. they used to JIT code, etc. Comparing python to R, python feels like a much more mature language. R always seemed only useful for small scale projects, e.g. writing some script to process data with < 20 lines of code. Python codebases just scale much better.

kstrauser 3y ago
It also has the nice feature of being good at multiple things. If you get a model running in Python, and decide you want to be able to run the code via an API, put it behind Flask and ta-da!, now it's a network service. It's not the best way to write analysis code, and it's not the best way to write a web service, but it's probably the best way to write analysis code that runs on a web service. I picked web services as a random example, not one I'm particularly passionate about. But as a general trend, if you want to combine 2 unrelated domains, Python probably has you covered better or more easily than just about any other language.
- fjkdlsjflkds 3y ago
  > If you get a model running in Python, and decide you want to be able to run the code via an API, put it behind Flask and ta-da!, now it's a network service. You could say the same about a model (or any function) running in R. Want to instantly convert it to a web service? Use plumbr or RestRserve packages for that. Want to make a quick web application around it? Use the shiny package for that. Just because Python is good at X, doesn't mean other languages are necessarily worse at it.

local_person 3y ago

[dead]

dekhn 3y ago

numpy exists (somewhat indirectly) because matlab cost $$$. and now numpy is a far larger use of matlab's concepts than matlab will ever be. I haven't seen anything an "A-type" data scientist needs in R/SAS/SPSS that was intrinsic to the language and couldn't be ported to Python. I don't want a "data/scientific" language, or a "web" language, or a "UI" language- I want one language that explicitly supports all the usecases well-enough.

qsort 3y ago
> I haven't seen anything an "A-type" data scientist needs in R/SAS/SPSS that was intrinsic to the language Not intrinsic to the language but in terms of the ecosystem, for a purely A-type data science workload R is significantly better. dplyr + ggplot vs. pandas + matplotlib isn't even remotely close. Now, obviously in the real world nothing is ever purely model work, which is why Python more than makes up for the difference. > I don't want a "data/scientific" language, or a "web" language, or a "UI" language- I want one language that explicitly supports all the usecases well-enough. Me neither. *taps sign* >> The difference is that Python doesn't entirely suck as a general-purpose language.
- CuriouslyC 3y ago
  R is great when your data is already together in one place and homogenous/processed. In that instance the workflow using dplyr/ggplot is better. R the language is hot garbage though, and the libraries available for doing random things that aren't working with certain data structures, doing math or plotting are generally much worse than their python equivalents. That is the reason python dominates data engineering, and why it's pushing R out everywhere but academia.
  - dekhn 3y ago
    R will still have a solid place in industrial pharma for a long time (even as python growths, R growth will continue. R is pretty tightly intertwined with the process of processing and submitting data from clinical trials to the FDA, much to SAS's chagrin. Personally I think we have to accept that R, Python, Java, C++, and Go are going to be with us for the rest of our lives. I would expect PHP, Ruby, and Perl to go away much faster. Rust is still in question.

CuriouslyC 3y ago

Some industries are stuck on matlab (jet propulsion, for example) but in general people from sciences are either Python or R, and SAS/SPSS are only really used in places like health care or pharma where there's a lot of regulation in place, because the software is designed around that regulation, unlike Python and R.

dimask 3y ago
It is also still quite dominant in neuroscientific fields, as there is a lot of software and code there from the past, but it is changing little by little.

karmasimida 3y ago

Anyone who used Matlab to process text before 2015 wouldn’t consider it anywhere closer as a competitor to Python Same goes for R/SASS Python has no competitor in terms of the whole package, it is a full featured language while being particularly good at data

nl 3y ago

This is a good point. I've done a fair bit of R and there is no doubt that as a pure data science tool it's usually quicker to produce something than Python. But I prefer Python because inevitably any data science project ends up having other "bits" that aren't pure data science, and Python's super string general purpose computing libraries are hard to beat.

twobitshifter 3y ago

I would just add that for scientific scripting Julia is usually nicer to program in than the combo of numpy and Python. Julia is fast enough on its own that you can do the matrix calculations directly and it supports real mathematic matrix operators. It’s also a great type based design that makes using packages magical.

gaganyaan 3y ago
I think Julia has a hard uphill battle against the "Python is the second best language for anything" effect. Julia looks pretty cool to me, but I already know Python. I'll probably learn more about it in my spare time, but I'm also never going to be able to convince my coworkers to try it out, because they also know Python and that's good enough for them. And thus Python's dominance continues.
- ziotom78 3y ago
  I was able to convince my coworkers to stop working on a Python project (a simulator for a scientific instrument) once we all realised that we would have not been able to keep using plain Python because of performance. When the alternatives where (1) to force everybody to learn either C or Fortran, or (2) to rewrite everything in Julia, they asked me to provide examples of both solutions. After I showed them a few different approaches (pybind, cbind, f2py...), there was full consensus for Julia. We moved what we implemented so far (it was not to much, luckily), and so far nobody has regretted the choice. The problem in using two languages is that you do not have just to learn a new language besides Python. You have also to figure out how to properly interface them. (f2py was considered the easiest solution, but we feared that the 1- vs 0-based array indexing would have likely caused many bugs.)
  - targafarian 3y ago
    Did you evaluate Cython? I'm not anti-Julia, but I like that my Cython code is useable out of the box from Python, with no wrapping, and then users can continue to use their Jupiter + Python scripting workflows with performant bespoke modules complemented by the full Python ecosystem. Someday I'll do a project in Julia. But for some such projects, Rust seems fully guaranteed to be performant while Julia might or might not be, so I might still lean towards Rust (unless one of the high quality packages of Julia removes a lot of development time, which is a decent possibility).

nextlevelwizard 3y ago

>Python doesn't entirely suck as a general-purpose language. I always get caught off guard by comments like these. In my mind Python doesn't at all suck as a general purpose language. Only real argument is execution speed, but most people aren't actually writing code where Python's "slowness" matters and if you actually need that speed you can write that part in C and still use it from your Python application.

LeanderK 3y ago

> which pure "A-type" data scientists tend to prefer: R, SSPS, Matlab I very much disagree and I would say it's the opposite. The only competitor for data-scientists is R, especially if you're doing stats-heavy analysis (opposed to an ML-heavy analysis). SSPS is in my experience used mainly by someone without a data-scientist background, e.g. psychology, similarly to matlab, where the only users I know are engineers. I would bet that the overwhelming amount of data-scientists use python, even if it's just to set up the data-pipeline. It's just the better general programming language and data-scientist are not analysing all the time but have to get the data in shape and available first.