4 ms·

The “luck” Python ran into was that some people came together to write NumPy. The alternatives at the time were Matlab and R. Matlab is not a great general prog

by oivey 4y ago

The “luck” Python ran into was that some people came together to write NumPy. The alternatives at the time were Matlab and R. Matlab is not a great general programming language and is expensive. R has esoteric syntax and also isn’t a great general purpose language. This shift didn’t happen so long ago that it can’t be remembered. It certainly didn’t happen because it was installed by default by some Linux distros. For better or worse (and honestly, probably often for better), people writing scientific Python care about solving a particular numerical problem. Code quality often matters less than just getting the right answer. Knowledge of a language or code quality is rarely the hard or important part. Scientists generally spend time learning math and software engineers spend time learning good design practices. It shouldn’t be a great surprise what each group becomes good at. Time is finite.

z9znz 4y ago
> Knowledge of a language or code quality is rarely the hard or important part. That is true. The problem is, that code in Python of questionable quality (but real utility) is now turned into production code; and it is done so by building Python stuff around it. So now the insignificant implementation detail which was the language choice is now driving the entire project. Also, a PhD data scientist/mathematician can get hired for a big salary at a company (especially fintech), and their code will drive critical business decisions. A regular software engineer is mandated to build a structure around that code to make it production worthy. In my experience there is no traction to say, "Great, you've learned how to answer this tough question; now let's implement your solution in a real language."
- analog31 4y ago
  To me, this seems like the wrong way of doing it. I'm a scientist, and I code in Python, but my code is never directly incorporated into the production code base. (Similar story for the portions of my work involving electronics, mechanics, etc). A decent generalization is that once I solve a problem, and demonstrate a result using Python, the actual amount of code that needs to be understood and translated into the production language is usually a few tens of lines at most. On the production side, the choice of language is just a tiny part of the entire process. They have frameworks, revision control systems, coding standards, review processes, the whole nine yards. Just using a "better" language isn't even an important step towards writing production code. If I wrote in the software department's favored language, they would still have to turn it into production code within their procedures. The fact that a few lines of Python represents hundreds of lines of production code makes it more productive for me to think my way through problems and express the principles underlying the solution to the coders. I see no reason why the tool that's used for discovering a solution has to be the same tool as the one used for putting the solution into production. Laboratory chemists don't use industrial manufacturing processes when trying to make a milligram of a new compound.
  - z9znz 4y ago
    If you run a company, then you might make better choices (or at least allow for proper discovery of what approach to take to building a large system). But there seems to be an accepted rule that "if you're doing data science, you are a Python shop". And if you are trying to pitch a data science company to investors, they know Python for data science. New companies start all the time using a technology because "that's what you use", without knowing why or why not. In fact, (and not that I idolize him, which I really do not), Paul Graham's http://www.paulgraham.com/avg.html http://www.paulgraham.com/avg.html essay talks about this. Most companies follow whatever is the accepted practice for doing things. It may "work", but it definitely may not be optimal. And the companies that choose a different path based on more consideration or more awareness may have an easier time. I mean after all, there was a time when everyone wrote C (because C was so much better than COBOL or whatever). And believe it or not, there was a time when it was impossible to convince a company to try Java. Or as the old saying went, "nobody got fired for buying IBM". After all, it was the safe path. Substitute Oracle, or C, or C++, or Java, or Python, or PHP, or Windows (there was a time when people didn't trust Linux and couldn't imagine why you would ever try to run your business software on Linux machines). Your reasoning makes sense, but in practice in the software world, it is not as well planned or thought out as you might expect.
    - pbronez 4y ago
      For machine learning companies, Julia is probably the dark horse language that could give you a competitive advantage. Like always, the problem with using something more niche is the ecosystem strength. You need to train people more, build more tooling yourself. Still, there is performance to be had if you work for it.
      - workingon 4y ago
        ChrisRackauckas 4y ago
        Please post a concrete example of one that you have ran into.
- oivey 4y ago
  Data scientists are hired for a certain set of skills, and software engineers are hired for a different set. That’s the reality of specialization. I don’t often see data scientists complaining that software engineers are bad at statistics. I’m not sure why so many software engineers feel the need to complain that data scientists are bad at software engineering. At my job I work both sides of the fence. Sometimes I write research code in Python/Julia/Matlab. Sometimes I write production code in C++/CUDA. The choice of production language is primarily due to performance needs rather than code quality, though. I’ll repeat myself but slightly differently: the main reason Python is used over, say, Ruby is because Ruby lacks a comparatively good library for working with arrays. It subsequently didn’t develop all the ecosystem for ML and scientific computing (SciPy, Matplotlib, sklearn, skimage, Torch, etc). It’s basically never going to catch up. And when it comes down to it, Ruby is basically as ill-suited as Python, Clojure, Elixir, and most other languages when it comes to doing something like automatic differentiation on a GPU. And again, even so, Python is still the best choice among those because so many resources have been poured into PyTorch, Matplotlib, NumPy, etc.
  - z9znz 4y ago
    I'm not attacking data scientists. I have no objection to how they solve their problems. My point is that their choice of Python (because of the libraries they use, or the training they got in school?) bleeds out and causes Python to be used in places where it doesn't fit so well. If the libraries you mention were written for Ruby (and they absolutely could have been... probably with fewer defects and fewer lines of code), then we wouldn't be having this discussion. There is no use of any of those libraries that someone could demonstrate which could not be equally or better demonstrated in one of the other languages you mention. Therefore, the only argument for using Python is that it's what those libraries are written in. In fact, with a little effort an API could be written and wrapped around those libraries to make them available to other languages, decoupling the bigger projects from the library-specific logic. This is how it should be done at any reasonable company anyway. The data science wizards build their specific systems in whatever language they need, and that gets built into a minimal service. But things often start highly coupled, meaning the special code in Python gets business logic and persistence stuff built around it also in Python.
dalke 4y ago
Yes, it was some luck ... and a lot of hard work. But it wasn't only NumPy that helped. I was around in the 1990s when Python started making waves in the scientific computing and research community. Python wasn't installed by default (we mostly used IRIX back then). Python was excellent for "steering" low-level computations, to use the buzzword at the time. That is, if you had a large body of existing C, Fortran, whatever code, you could turn them into Python extensions, let Python glue the components together, and use a Python program/script to organize the high-level data- and control-flow. Swig was a popular tool to help automate that process. Swig 1.0 had support for Tcl, Perl, Guile and Python, back in 1996. I used Swig to interface to an existing cheminformatics toolkit, and published my experience at https://www.drdobbs.com/cpp/making-c-extensions-more-pythonic/184404439 https://www.drdobbs.com/cpp/making-c-extensions-more-pythoni... . Python made it very easy to convert the implicit object model of the underlying C toolkit into an explicit OO model in Python. Python's reference-counted garbage collection was also a good fit to the C library. Together these helped simplify scientific research work quite a bit. And that's without NumPy. Even now I do a lot of non-numerical scientific software development, and rarely have need for NumPy. While I often use Python to steer components written in C or C++.
- kristjansson 4y ago
  This is the only reasonable comment in this whole thread. Python's was growth was basically organic, NumPy/SciPy/etc. are later extensions of that organic adoption.

by oivey 4y ago

z9znz 4y ago

> Knowledge of a language or code quality is rarely the hard or important part. That is true. The problem is, that code in Python of questionable quality (but real utility) is now turned into production code; and it is done so by building Python stuff around it. So now the insignificant implementation detail which was the language choice is now driving the entire project. Also, a PhD data scientist/mathematician can get hired for a big salary at a company (especially fintech), and their code will drive critical business decisions. A regular software engineer is mandated to build a structure around that code to make it production worthy. In my experience there is no traction to say, "Great, you've learned how to answer this tough question; now let's implement your solution in a real language."

analog31 4y ago
To me, this seems like the wrong way of doing it. I'm a scientist, and I code in Python, but my code is never directly incorporated into the production code base. (Similar story for the portions of my work involving electronics, mechanics, etc). A decent generalization is that once I solve a problem, and demonstrate a result using Python, the actual amount of code that needs to be understood and translated into the production language is usually a few tens of lines at most. On the production side, the choice of language is just a tiny part of the entire process. They have frameworks, revision control systems, coding standards, review processes, the whole nine yards. Just using a "better" language isn't even an important step towards writing production code. If I wrote in the software department's favored language, they would still have to turn it into production code within their procedures. The fact that a few lines of Python represents hundreds of lines of production code makes it more productive for me to think my way through problems and express the principles underlying the solution to the coders. I see no reason why the tool that's used for discovering a solution has to be the same tool as the one used for putting the solution into production. Laboratory chemists don't use industrial manufacturing processes when trying to make a milligram of a new compound.
- z9znz 4y ago
  If you run a company, then you might make better choices (or at least allow for proper discovery of what approach to take to building a large system). But there seems to be an accepted rule that "if you're doing data science, you are a Python shop". And if you are trying to pitch a data science company to investors, they know Python for data science. New companies start all the time using a technology because "that's what you use", without knowing why or why not. In fact, (and not that I idolize him, which I really do not), Paul Graham's http://www.paulgraham.com/avg.html http://www.paulgraham.com/avg.html essay talks about this. Most companies follow whatever is the accepted practice for doing things. It may "work", but it definitely may not be optimal. And the companies that choose a different path based on more consideration or more awareness may have an easier time. I mean after all, there was a time when everyone wrote C (because C was so much better than COBOL or whatever). And believe it or not, there was a time when it was impossible to convince a company to try Java. Or as the old saying went, "nobody got fired for buying IBM". After all, it was the safe path. Substitute Oracle, or C, or C++, or Java, or Python, or PHP, or Windows (there was a time when people didn't trust Linux and couldn't imagine why you would ever try to run your business software on Linux machines). Your reasoning makes sense, but in practice in the software world, it is not as well planned or thought out as you might expect.
  - pbronez 4y ago
    For machine learning companies, Julia is probably the dark horse language that could give you a competitive advantage. Like always, the problem with using something more niche is the ecosystem strength. You need to train people more, build more tooling yourself. Still, there is performance to be had if you work for it.
    - workingon 4y ago
      - ChrisRackauckas 4y ago
        Please post a concrete example of one that you have ran into.
oivey 4y ago
Data scientists are hired for a certain set of skills, and software engineers are hired for a different set. That’s the reality of specialization. I don’t often see data scientists complaining that software engineers are bad at statistics. I’m not sure why so many software engineers feel the need to complain that data scientists are bad at software engineering. At my job I work both sides of the fence. Sometimes I write research code in Python/Julia/Matlab. Sometimes I write production code in C++/CUDA. The choice of production language is primarily due to performance needs rather than code quality, though. I’ll repeat myself but slightly differently: the main reason Python is used over, say, Ruby is because Ruby lacks a comparatively good library for working with arrays. It subsequently didn’t develop all the ecosystem for ML and scientific computing (SciPy, Matplotlib, sklearn, skimage, Torch, etc). It’s basically never going to catch up. And when it comes down to it, Ruby is basically as ill-suited as Python, Clojure, Elixir, and most other languages when it comes to doing something like automatic differentiation on a GPU. And again, even so, Python is still the best choice among those because so many resources have been poured into PyTorch, Matplotlib, NumPy, etc.
- z9znz 4y ago
  I'm not attacking data scientists. I have no objection to how they solve their problems. My point is that their choice of Python (because of the libraries they use, or the training they got in school?) bleeds out and causes Python to be used in places where it doesn't fit so well. If the libraries you mention were written for Ruby (and they absolutely could have been... probably with fewer defects and fewer lines of code), then we wouldn't be having this discussion. There is no use of any of those libraries that someone could demonstrate which could not be equally or better demonstrated in one of the other languages you mention. Therefore, the only argument for using Python is that it's what those libraries are written in. In fact, with a little effort an API could be written and wrapped around those libraries to make them available to other languages, decoupling the bigger projects from the library-specific logic. This is how it should be done at any reasonable company anyway. The data science wizards build their specific systems in whatever language they need, and that gets built into a minimal service. But things often start highly coupled, meaning the special code in Python gets business logic and persistence stuff built around it also in Python.

dalke 4y ago

Yes, it was some luck ... and a lot of hard work. But it wasn't only NumPy that helped. I was around in the 1990s when Python started making waves in the scientific computing and research community. Python wasn't installed by default (we mostly used IRIX back then). Python was excellent for "steering" low-level computations, to use the buzzword at the time. That is, if you had a large body of existing C, Fortran, whatever code, you could turn them into Python extensions, let Python glue the components together, and use a Python program/script to organize the high-level data- and control-flow. Swig was a popular tool to help automate that process. Swig 1.0 had support for Tcl, Perl, Guile and Python, back in 1996. I used Swig to interface to an existing cheminformatics toolkit, and published my experience at https://www.drdobbs.com/cpp/making-c-extensions-more-pythonic/184404439 https://www.drdobbs.com/cpp/making-c-extensions-more-pythoni... . Python made it very easy to convert the implicit object model of the underlying C toolkit into an explicit OO model in Python. Python's reference-counted garbage collection was also a good fit to the C library. Together these helped simplify scientific research work quite a bit. And that's without NumPy. Even now I do a lot of non-numerical scientific software development, and rarely have need for NumPy. While I often use Python to steer components written in C or C++.

kristjansson 4y ago
This is the only reasonable comment in this whole thread. Python's was growth was basically organic, NumPy/SciPy/etc. are later extensions of that organic adoption.