19 ms·

The counterintuitive rise of Python in scientific computing (2020)

220 points by leonry 5y ago

jrm4 5y ago
Yet another hardcore programmy type discovers that usability is infinity more important that how many clock-cycles you save. Programming languages aren't for computers, they're for people.
- da39a3ee 5y ago
  matplotlib doesn't score highly on usability. jupyter notebooks encourage disorganized, unprincipled programming; chaotic re-running of cells in the face of global mutable state; and prevent budding programmers from learning to use version control because the JSON format was designed without version control in mind.
  - mistrial9 5y ago
    .. and get more new users each year than four generations of PCs combined
  - monkeybutton 5y ago
    Being able to hack out code to explore and experiment with data while not having to reload and reprocess data (thanks to that global mutable state!) saves a hell of a lot of time in the long run.
  - wiz21c 5y ago
    For Jupyter, it depends on the workflow. Especially with data sciences. In data science, you spend a lot of time playing with the data, testing things, drawing charts, computing, etc. When you do that, the cost of starting a python interpreter, loading the imports, loading the (usually big) data becomes a real pain 'cos you iterate like hell. Working in a REPL becomes really important. But even more, working with Jupyter allows you to work out a very detailed explanation of your thought process, describing your ideas, your experiments. Being able to mix code and explanations is really important (and reminiscent of literate programming). You got the same kind of flow with R. As data scientist, I'm concerned about data, statistics, maths, understanding the problem (instead of the solution). I don't care about code. Once I get my data understanding right then comes the time of turning all of that into a software that can be used. Before that, Jupyter really gives a productivity boost. For the code part, yep, you need other principles where Jupyter may not be suitable.
    - patrick451 5y ago
      It's interesting, I never feel like I get these exploratory benefits from jupyter notebooks. I just end up feeling like one hand and half my brain is tied behind my back. I'm most productive iterating in a similar way to what you describe, but in an ipython terminal, running a script and libraries that I'm iterating on in a real editor. If there are expensive computations that I want' to check point, I just save and load them as pickle files.
      - wiz21c 5y ago
        really interesting. I may have overlooked IPython a bit (I just thought Jupyter was its improved version). For the moment, maybe like you, I prerpocess the data (which takes minutes) into numpy array which then take seconds to load. But once I add imports, everything takes about 5 or 6 seconds to load everything I need. So Jupyter remains a good idea. Moreover, I love (and actually need) to mix math and text, so markdown+latex maths is really a great combo. I dont' know if one can do that in IPython, I'll sure look!
      - kzrdude 5y ago
        I have to say I think a jupyter notebook format is a 10x improvement in productivity over ipython. It's just so much easier to work with - and a step more reproducible too, my scribbles are all there saved in the notebook, at least!
  - patrick451 5y ago
    > matplotlib doesn't score highly on usability. The one place matplotlib sucks is any kind of interactivity. But other than that, matplotlib has the best, most intuitive interface of all the python plotting libraries I've tried. It's also one of the few libraries that doesn't rely on generating html for a webbrowser, which makes for a miserable workflow. I still think Matlab's plotting is untouched by open source options.
    - taeric 5y ago
      The charts that R produces are typically better looking. But...R. :(
    - cinntaile 5y ago
      Matplotlib hides a lot of complexity if you ask me. As soon as you do something in a different way than intended you're off searching stackoverflow for a post that did something similar to what you want. Then you tweak it a little and hope it works.
  - analog31 5y ago
    I've programmed in a number of languages over the past 40+ years, starting with BASIC, and every one of them encourages sloppy coding. The good discipline always has to be taught, learned, and willingly practiced. The closest I came to a language designed for teaching good practices was Pascal. I find it easier to read and understand bad code written in Python, than good code written in the C family languages.
    - da39a3ee 5y ago
      Yes, what I was saying was that writing Python code in files is a better and more educational way to program than writing Python code in a Jupyter Notebook. It wasn't a criticism of Python.
      - analog31 5y ago
        I use Jupyter a lot, but have a personal rule to do "restart kernel and run all cells" once in a while, to scare up any kind of hidden state or out-of-order execution problems. For instance, if I'm about to leave a notebook for a while, I'll make sure it runs without error from top to bottom. In that sense, I'm making it work like Python code in a file. The advantage of code in files is that I can use all of the slick code analysis tools that will warn me about my mistakes. I wish there were something that would let those tools go through the code in a Python notebook from top to bottom.
  - kzrdude 5y ago
    papermill is good and ploomber is a thing to watch. Ploomber makes it systematic - store notebooks as .py (py:percent files for example), parameterize them with papermill and execute as a batch job. One can view the resulting jupyter notebooks as .ipynb later and produce reports as html if wanted. It's really good already, and better if ploomber gets more development. The whole reason it works is because it's easy to open the .py notebook and work on it, interactively, in jupyter. The main idea - jupytext for .py notebooks and papermill for parameters & execution - that's already "stable" and easy for anyone to use for their own purposes.
    - edublancas 5y ago
      (ploomber maintainer here) Any feedback for us? What can we do to improve Ploomber?
      - kzrdude 5y ago
        Maybe I haven't come far enough with my ploomber use to tell yet! It works nicely but I know I'll learn more and open my eyes more as I go. As a first impression, I eventually found meta.extract_upstream = False which I think is important. Reason: The code for each step should be a lego piece, a black box with inputs and outputs. That code should not itself hardcode what its predecessor in the pipeline is - you connect the pieces in pipeline.yaml. (extract_upstream = False is not by itself enough to solve this, since you also need to be able to rename inputs/outputs for a notebook to be fully reusable as a lego piece, but it's good enough for now.) I also for my own sanity need to know more about how the jupyter extension part works, how it decides to load injected-parameters or not. But maybe I could learn that somehow from docs. In general I want components that are easy to understand and plug together and less magic (but the whole jupyter ecosystem's source code feels this way to me unfortunately, lots of hard to follow abstractions passing things around). But it's developing rapidly and already very useful, thank you so much!
        edublancas 5y ago
        This is great feedback, thanks a lot! I'll ensure we display the "extract_upstream" more prominently in the docs, we've been getting this feedback a couple times now :) Re: the Jupyter extension injects the cell when the file you're opening is declared in the pipeline.yaml file. You can turn off the extension if you prefer. Feel free to join our community, this feedback helps us make Ploomber better! https://ploomber.io/community https://ploomber.io/community
- rewq4321 5y ago
  Not infinity, but yeah it's worth more than people generally think. But in the end you don't really lose many clock cycles anyway because everything actually runs in C/CUDA/etc. behind the scenes
- deleted 5y ago
  [deleted]
- Yoric 5y ago
  I personally find numpy irritating, surprising, misleading and definitely a wrong definition of "usability". YMMV.
  - Scene_Cast2 5y ago
    What are your thoughts on Pandas? As someone who uses numpy almost daily, I think that numpy is "overextended" beyond its core niche, sure. So - making it work with things outside that niche (e.g. streaming, non-rectangular data, non-uniform data, nonhomogeneous data, etc) is painful. However, 1) there's Pandas for that, and 2) I disagree with "misleading" and "surprising". What makes you think that?
    - m_mueller 5y ago
      Not GP, but I’m using pandas daily to build up a BI platform within a financial institution. Compared to Matlab and even Fortran it has some issues IMO: * why distinguish between Series and DataFrame? just give me an interface for m x n matrices or even higher dimensions. * pure vs. in-place operations. not such a big fan of having multiple versions of the same function, e.g. a more pythonic df[“my_col”] = series vs. a more functional df.assign({“my_col”: series}) ; I’d rather have everything like the latter to be able to more easily have best practices in place. That brings me to another point: if we keep everything purely functional, then python’s syntax is making things a bit awkward. Where in something like JS you could just put every function call with its dot on a new line without the need to assign, in Python this requires putting line break characters or wrapping it in round brackets. This is one place where a language with explicit assignment terminators (semicolons) are a bit cleaner to work with. All that being said scipy is still a great choice to have both system programming and numerical business logic in one language.
    - J253 5y ago
      In my opinion and experience, I think you’re right about “there’s Pandas for that” and “that” can be almost anything. It can do almost anything but making it do almost anything requires constant reference to the docs. And I find maintainability difficult. It seems like there’s 50 kwargs for every method. Sometimes things happen in place by default, other times they don’t. Compound indexes still confuse me. But I’m not a data scientist so I don’t do much ad-hoc analysis that seems typical with pandas users.
- matsemann 5y ago
  I wonder how many days have been wasted on non-programmers trying to get their Conda environment up and running or similar. Half the data science stuff isn't reproducible, not because of the science, but because getting the notebooks running with its dependencies is almost impossible.
  - lysecret 5y ago
    have you heard of docker?
  - irrational 5y ago
    Which begs the question, why has nobody built anything better?
    - taeric 5y ago
      I'd argue that they probably have. It just isn't free.
    - belval 5y ago
      Because it's a hard problem and people love hating on Python because it doesn't come with a way to handle all the compiled dependencies that work for every OS. Data science has a ton of moving library parts, it is genuinely difficult to distribute precompiled libraries for everyone when you have 2-3 actively maintained CUDA version with 2 cuDNN version for accelerators that change every 2 years. Most team fail to standardize on an environment (say Python 3.8, Ubuntu 20.04, CUDA 11.1, and cuDNN 8) and then get hung up on a dependency not building as if it's Python's fault that it does not have control of your entire OS.
      - matsemann 5y ago
        But why is it such a big problem in Python compared to other stacks? Why does all python projects end up depending on you having those exact tools of things installed locally and the planets aligned a certain way, when other stacks do not?
- taeric 5y ago
  This feels misguided, too. It begs the question that python has good usability. Anyone that has tried managing dependencies in it will know that is mostly a lie. What python had, was that it was preinstalled on many computers and then had a large cohort of users that are insisting that others use it. And mostly force proclaiming that it is easy and readable. I'll not claim that it is hard, per se. More that it is not intrinsically easier than any other dynamic language. For evidence, the main packages that are popular are often clones of packages from other environments that were not widely installed. Jupyter can be seen as free version of many scientific applications. Matlab, Mathematica, etc. Matplotlib is rather direct in it's copy. Pretty sure there are more examples.
  - iainctduncan 5y ago
    Is managing deps in Python a pain? sure is!. Is it a pain in the other contenders of easily available dynamic languages? Yup. So that's a wash. Managing deps in dynamic languages is not a simple problem, I can't say I've tried one that did it super well yet.
    - taeric 5y ago
      I'm... Not sure insulting other languages is a path to victory. Npm, as much as it annoys me, is light-years ahead of anything in python. Quicklisp is rather pleasant, now. Ruby has had gems for a long time. I grant that it is a hard problem. I am not griping that it is not solved. More that the option community has largely failed to even pick a direction. The most used dependency methods are, in the modern spirit of python, deprecated already.
      - iainctduncan 5y ago
        I'm not insulting other languages - I'm pointing out they have the same situation! Where in that post is there an insult??? I have had many dep fights with Ruby and Node too, and I would disagree that their solutions are in any way better than pip and virtualenv.
- dang 5y ago
  "Don't be snarky." "Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
  - jrm4 5y ago
    Harsh isn't the same as shallow. I don't believe snark is ALWAYS uncalled for, only that you deploy it when it's reasonably necessary. And this is a thing that people, especially HERE, get wrong a LOT. That it's unpleasant or hits people where it hurts doesn't mean it's not important, and I think it is.
    - dang 5y ago
      The issue is the effects that it has on the ecosystem. It dumbs down threads and makes people nastier. If someone is wrong and you know more than they do, you can share correct information without putting them down.
jrochkind1 5y ago
As a rubyist, it makes me sad that python ended up here rather than ruby. And I sometimes wonder why. > As the name suggests, numeric data is manipulated through this package, not in plain Python, and behind the scenes all the heavy lifting is done by C/C++ or Fortran compiled routines. So I wonder, was it easier to write C/C++ or fortran compiled extensions in python than it was in ruby?
- belval 5y ago
  I don't know how easy it is in Ruby so I cannot give you a comparison. However it is very very easy to write Python bindings for a C/C++ library with minimal work. Solutions range from "just works" like ctypes to "actually integrates with the language" like Cython. You also have automated tools for wrapping like pybind11 which does a lot of the heavy lifting for you.
- prpl 5y ago
  It was multiple things, really. I would attribute ute some of it to Swig, Perl attrition, SCons/Software Carpentry, integration with GUI libraries, good documentation, and various other efforts in the mid 2000s. A lot of those things were solving research problems simply, and Python’s use just kept expanding. Python was already taking over in many use cases by late 2000s. Ruby was known, but it didn’t have the following at multiple levels in academia like Python did
  - jrochkind1 5y ago
    You describe what happened, which I saw happen too. The question I have is why though. Right, why did python's use in scientific computing keep expanding, and not ruby's? Why was python already taing over many use cases by the 2000s, but not ruby? Why did python develop the following at multiple levels in academia, and not ruby? (Why is Perl attrition relevant, when ruby was in fact explicitly based on Perl?) That's the question, not the answer! It seems like a lot of the answer is NumPy, which makes the question -- why did NumPy happen on python, not ruby? Certainly one answer could be "nothing having to do with the features of the language, it's just a coincidence, they chose to write it in Python, if those working on numpy had chosen to use ruby instead, history would be different." But one hypothesis is that maybe NumPy wouldn't have been as easy in ruby as python. Someone else suggested the first numpy release happened before the first ruby release, so that could also be an answer.
elil17 5y ago
It’s amazing how often the authors point of “agility” arises in real world circumstances. I’m not a programmer, but I use Python a lot in my engineering job. There have been 3 times in the past month where I got an order of magnitude speed up because SciPy implements a very complex but highly efficient algorithm which I would never have had time to deploy.
- JuettnerDistrib 5y ago
  > There have been 3 times in the past month where I got an order of magnitude speed up because SciPy implements a very complex but highly efficient algorithm which I would never have had time to deploy. Yes. I feel like the author conflates the language with the package ecosystem. Pure Python is pretty horrible for scientific computing (3*[3]=[3,3,3] is about as counterproductive to scientific computations as it gets), but Numpy changes the semantics of those operations. In other words, Python has an absolutely stellar package ecosystem. There have been attempts to bring a package ecosystem to C, but it never took off. However, I do wonder how C would fare if it had.
  - oh_my_goodness 5y ago
    Yes, sure, the strength of Python is the libraries.
  - Enhex 5y ago
    there are package managers for C and C++, but people aren't told about them. https://conan.io/ https://conan.io/ https://vcpkg.io/en/index.html https://vcpkg.io/en/index.html this implies Python's advantages is not having a package manager, but better teachers or at least teaching better practices, so it isn't even language related.
iainctduncan 5y ago
If you know actual scientists, this isn't counter intuitive at all. My partner is a scientist, so now I know tons of them, and I have done a bunch of Python coding and support for scientists, have been a Python programmer (as well as other languages) since 2005-ish. I saw this coming (as did many) 15 years ago. Most scientists, and their grad students, are trying to do a whole bunch of things in their research, and programming is just one of them. Field work, experiments, data wrangling, writing papers, defending papers, teaching, etc. And most of them do not have access to budgets for programmers or when they do, it's for a limited amount of time and work, meaning they need to be able to pick up and run with whatever the programmer did. So the fact that with Python they and their grad students (who might be there for only 2 years) can be working productively, and figure out what the hell the code did when they come back to it months later, is HUGE. As in, literally blows every other consideration to smithereens. This has meant that over the last 20 years the scientific libraries in Python got mature faster than in any other language, and this in turn has had a snowball effect. And when speed is necessary, C++ extensions can be written. But honestly, most of the time speed is not the main factor. The downside of Python in my experience is that junior teams can make heinous atrocities when a project gets really big (I have had to step in as CTO to one of those messes, so much as I love Python, I must admit this is true!) But the stuff the scientists are doing is very rarely that big. It's tools programming, scripting, making utilities, data analysis and so on. Readability counts. In some fields, it counts more than anything. I've worked in about 10 languages now over the last 20 years, and Python is still the easiest to read when you come back to some old code or have to pick up code for a small job, or hand it to a beginner to extend without having them create an unreadable mess. This is what scientists need to do all the time. Re other people's comments on Python packaging and setup being hard, well honestly I've had just as much pain with Ruby or Node. The shining exception there is R, which is giving Python a run for its money in many scientific areas. R Studio has the best "hit the ground running" experience out there and is really slick for data programming.
- analog31 5y ago
  My partner's partner is a scientist too. ;-) In addition to not having budgets for programmers, we also don't know how to manage them, for instance how to communicate our needs, decide if their implementation plans make sense, or gauge their progress. Nearly half a century after The Mythical Man Month, managing software development is still generally acknowledged to be an unsolved problem. The other two obstacles are that most programmers hate the scientific work environment, with its ever-changing requirements and frequent dead ends. And, the programmers who can work on math related stuff are in the highest demand.
hulitu 5y ago
I try to love micropython. However, its UI is at ed level. It only says "Syntax error".
- the__alchemist 5y ago
  Python excels in several domains. For example, the non-speed-critical numerical computing this article is about. It's also nice for backend web development, and scripting. Embedded isn't one of its strengths, and I'm suspicious micropython was an attempt at bringing embedded programming to people who don't want to learn more than one language.
scythe 5y ago
As a physicist, having spent eight years in academia, Python did not win by beating Fortran. Nor did it beat C++. It didn't really compete with Ruby or Lisp, although Lua (Torch) was a briefly serious competitor before everyone realized that a language developed by four people, one of whom doesn't get along with the others, couldn't be responsive to users' needs. Python defeated Matlab. I know because I cheered it on. I was there. I watched my roommates and friends struggle with introductory scientific computing in Matlab and I joined the chorus that was practically begging for Python, even though I didn't really like it. I can't even begin to explain how awful it is to try to teach programming concepts in Matlab. But something like Python or Matlab had to be the choice because the schools wanted to teach programming through a language where you could just call "graph" and the computer would display a graph. Python's team, unlike Lua's, aggressively courted educational institutions by offering scientific, numerical and graphical libraries within a programming language that works like a programming language, not a glorified computer algebra system. They even added a dedicated operator for matrix multiplication. It's a great example of finding a niche and filling it: I still don't like using Python, but I can't dispute that no other language/ecosystem comes close to offering what we need to teach programming to physics students. You want to beat Python? Build a type system that can capture dimensional analysis. Warning: it won't be easy.
- rsfern 5y ago
  > You want to beat Python? Build a type system that can capture dimensional analysis. Warning: it won't be easy. Curious about your thoughts on pint and Unitful.jl — pint doesn’t really go all the way to a full type system, and Unitful.jl doesn’t work with everything (autograd is a problem still I think). But Unitful.jl is super cool. https://pint.readthedocs.io/en/stable/wrapping.html#checking-dimensionality https://pint.readthedocs.io/en/stable/wrapping.html#checking... https://painterqubits.github.io/Unitful.jl/stable/ https://painterqubits.github.io/Unitful.jl/stable/
- _aavaa_ 5y ago
  Oh god, the atrocities I have seen colleagues do with MATLAB scripts…
- analog31 5y ago
dekhn 5y ago
Counter-intuitive? I picked it because it was the closest scripting language to C (see the select and socket APIs for good examples). And it had numeric array support early-on (making it an attractive replacement for matlab).
smitty1e 5y ago
> Of course, If the best algorithm is known beforehand or the manpower is not a problem, a lower level-language is probably faster, but this is seldom the case in real life. One is wary of one-dimensional analysis of anything in a software context. Who cares if the Fortran library runs like the blue blaze, if it cannot be readily maintained?
- Bostonian 5y ago
  It is possible to write maintainable modern Fortran without gotos with small functions and subroutines. OOP with inheritance and dynamic polymorphism is possible since the Fortran 2003 standard.
wheelerof4te 5y ago
One thing that I don't like with Python's scientific libraries is how they change the overall Python syntax. There are so many ways to slice an array or a dataframe, and only a few of them are valid Python code. Keeping the language API should have been a priority, but that is a consequence of operator overloading features.
- kzrdude 5y ago
  Can you explain what you mean more in detail? Libraries can't change the syntax of the Python language, not in the formal sense. Is this about things you want to be able to express in syntax but can't? Or the other way around - stuff that uses syntax/operators but should really be methods?
  - ohyoutravel 5y ago
    Numpy syntax comes to mind. The extra commas often aren’t valid pure Python but are required for some operations on numpy arrays. I don’t know how this works under the hood, but expect it’s a state machine under the numpy ndarray looking for the extra commas and such. i.e. some_array[0:5,0] which isn’t valid pure Python notation.
    - kzrdude 5y ago
      Extra commas are "valid in pure python" in the following sense that I can demonstrate. Open ipython3 In [3]: class Test: ...: def __getitem__(self, index): ...: print(index) ...: In [4]: Test()[1, 2, 1:3, ..., :] (1, 2, slice(1, 3, None), Ellipsis, slice(None, None, None)) It's valid and we get the complicated tuple of integers, slices, ellipsis etc as printed. Numpy has existed for a long time. Its needs have been taken care of in upstream Python, to a big extent, and other libraries can use the same features.
      - ohyoutravel 5y ago
        Interesting! Neither myself nor my coworkers could get the snippet I posted working outside the context of an ndarray, so I had speculated at that time that it there was something else going on under the hood. You seem to have a much better grasp of Python than us, would you mind posting an example where the snipped I posted successfully accesses data from an array in pure Python? That way I can not only take the L, but correct the record and learn something in the process.
derbOac 5y ago
C, C++, Fortran are still used, most Python users just don't see it because it's hidden away underneath the calling function. I've been surprised by the rise of Python in some ways although not at all in others. Languages like C, C++, Fortran, and dare I say it Rust are too low-level in their raw state for numerical computing. You had the US federal government funding language competitions because of this (see: Chapel). Languages like Python and R (and before that things like Lisp) came along and gave people a taste of something different, and it's obvious what people migrated to. Part of it is timing: multivariate computational statistics (ML/data science/DL/whatever you want to call it) just sort of started taking off in computer science communities before LLVM-based languages like Julia or Nim could get a foothold. OCaml might have fit that niche but never got there because of a desire to take a different path, or take the path more slowly. So people looked for a nice expressive language, found it in Python, and buried all the messy stuff behind wrapper functions and called it a day. It was furthered along by Matlab being another comparison on the other side -- Python looks kludgy compared to modern Fortran or C, but not compared to Matlab. All that wrapper time in Python has its costs, so I suspect as limits get pushed further we'll eventually see a migration to something else like Julia or Nim, or something else not on anyone's radar. One moral to this story is that expressiveness matters. People will go out of their way to avoid talking directly to machines at a low level.
- zozbot234 5y ago
  > You had the US federal government funding language competitions because of this (see: Chapel). Wait, weren't they supposed to be using Ada for everything anyway? What's wrong with Ada?
  - tormeh 5y ago
    Different design objectives, I believe.
- Robotbeat 5y ago
  It’s because Matlab (and Mathematica, etc) is proprietary, and therefore you always have to pay the Danegeld. So we use numpy instead because it’s extensible, it uses all the super fast C/C++/FORTRAN stuff on the backend, and is fairly easy to learn. I actually still would prefer Matlab as the syntax is more compact and natural than numpy (which is like a matlabified Python), but that’s probably just due to more experience in Matlab.
photochemsyn 5y ago
Python displaced a lot of very expensive proprietary software in the biosciences arena. Ease of use was also a major factor, as many bioscientists have relatively little background in programming, but the ability to escape the world of expensive restrictive software licenses was very attractive to the scientific community, whose historical norms emphasize the open sharing of methods and results: > "A program that performs a useful task can (and, arguably, should) be distributed to other scientists, who can then integrate it with their own code. Free software licenses facilitate this type of collaboration, and explicitly encourage individuals to enhance and share their programs. This flexibility and ease of collaborating allows scientists to develop software relatively quickly, so they can spend more time integrating and mining, rather than simply processing, their data." https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1004867 https://journals.plos.org/ploscompbiol/article?id=10.1371/jo... Now there isn't any area of molecular biology and biochemistry that doesn't have a host of Python libraries available to assist researchers with tasks like designing PCR strategies or searching for nearest matches on up to x-ray crystallography of proteins.
whatever1 5y ago
Julia is the next big thing. I am always blown away by its readability and speed. But it will take years to build a library ecosystem that can rival the python one.
- a9h74j 5y ago
  Remind me if anyone might, is there any story for Julia making use of existing python libraries?
  - sundarurfriend 5y ago
    If you mean whether it's possible, PyCall.jl has existed since nearly the beginning of Julia, and PythonCall.jl [1] is a more recent package for the same core functionality - calling into Python code. [1] https://github.com/cjdoris/PythonCall.jl https://github.com/cjdoris/PythonCall.jl
- II2II 5y ago
  That is a big part of the author's point: the library ecosystem is here for Python today. While there is a heavy penalty for anything written in Python itself, it doesn't really matter since there isn't much of a penalty once the data is passed to highly optimized libraries and those libraries allow developers to select efficient algorithms rather than implementing their own algorithms (which are likely to be less efficient).
- a9h74j 5y ago
  In this thread I am seeing a number of explanations, including: Ecosystem; mind-share; readability and engineering mind-set; history/Numpy/Matlab; teachability and academic focus. There are also comments emphasizing the "dynamic" scientific environment and need to just pick up code left by others. In terms of the latter, could one apparent requirement be this: The main contact should be with top-level code which at least looks like it is interpreted -- even if through compile-with-run-combined and/or memoization? Need part of the user interface, so to speak, be to hide all intermediate artifacts, even the very thought of object code and executables? That such stuff is for, say, "module creators" not primary users?
uoaei 5y ago
Python is an API to efficient scientific computing code. It's good for that, assuming you're using old and more verbose languages. Look into Julia as a promising alternative -- the language itself is superbly fast (aside from initial compilation) and there's an impressive scicomp ecosystem to say the least, all written in native Julia. This allows for program rewriting / metaprogramming more broadly and is insanely powerful once you get a feel for it.
- laichzeit0 5y ago
  No need for any alternatives. Python as an API is good enough.
  - uoaei 5y ago
    It is really remarkable how much more expressive some languages are over others. If you are satisfied with Python for everything you do, then you are not hitting the limits of its expressiveness. But for more naturally expressive code, other languages may have huge advantages for certain applications.
deleted 5y ago
[deleted]
amelius 5y ago
New languages should always provide bindings to call into Python modules, so you get the immediate benefit of the largest ecosystem on the planet.
jackjackk0 5y ago
I recommend one of the recent videos by Dave Beazly [1]. He lived through and contributed to the raise of Python in scientific computing first hand in the 90s, and offers some interesting insights. Plus he's always quite an entertainer. [1] https://youtu.be/4RSht_aV7AU https://youtu.be/4RSht_aV7AU
fancyfredbot 5y ago
I feel like python acts like a kind of bus in scientific computing, connecting various high performance libraries and DSLs together. That said, this article's story of someone using the wrong algorithm is a bad example in my view. Python hasn't succeeded because people are more likely to use more efficient algorithms due to easier experimentation, it has succeeded because the of the size of the ecosystem and the fact such algorithms are easily available.
StreamBright 5y ago
Python is the common scripting language of C, C++, Fortran.
sega_sai 5y ago
As many already noticed, the rise of Python is not counter-intuitive at all. (I'm a scientist myself). Basically modern python offers you a spectrum from easy to understand and quick to write python programs (those will be slow), to purely glue code that connects a lot of high performance c/C++/fortran code. And many scientists will start from pure python code with the help of numpy. In many cases it will be good enough. But if needed you can always interface with other libraries, or write yourself high performance c/c++/fortran code for the most performance critical bit, and use python to glue it together. That flexibility where you can trade speed of writing the code with the speed of execution is very valuable.
- chalst 5y ago
  At this point we can say that against the two criteria of a spectrum from prototyping to heavy lifting and ease of embedding external high-performance libraries, Julia is simply better than Python. Julia does have two drawbacks of being tied to the one, rather heavy metal, implementation and lacking the wealth of libraries outside scientific computing.
  - bernulli 5y ago
    Its third, and major, drawback is that Python has a legacy in many groups.
    - chalst 5y ago
      True, although the Python-Julia interop is surprisingly good. Gradual migration of legacy code from Python to Julia might be a possibility for some of these groups. But I was really thinking about the situation for new projects when writing that comment.
      - bernulli 5y ago
        Same thing, you will want to reuse existing code, and you do not want to split knowledge in your student group. There is no such thing as 'gradual' migration: I either have to support Python in my group, or Python and Julia -- until everything is migrated. If I have a working ecosystem using Python, with students trained in Python, and all previous work in Python, there's a whole lot of opportunity cost associated with me deciding to have the next student use Julia. I'd rather have that student build on existing tools and knowledge and do something new with their time.
blunte 5y ago
Python won because people who knew math/science domains only knew Python (or it was the best they knew). And so they made libraries for Python. And it propogated like many other bad ideas based on ignorance. Python is a miserably bad language for modern times. If you know any of half a dozen other languages, then you understand. There was a good essay, from Paul Graham?, about the ladder of awareness of programming languages. Unfortunately I can't find it now. The point is, Python has won and is frankly terrible. It has inconsistent features, but it has an awkward OOP approach (in a time when OOP is finally being recognized as bad itself), as well as seriously lacking basic language features which are only appearing as of 3.9 and 3.10. Frameworks like Django and Django Rest Framework expand on these bad ideas, creating monstrosities which make the PHP code of yore look arguably decent. Sadly, I don't think there's any way to kill this. The only option is to vastly outperform the Python people and produce reliable, readable, performant solutions in half the time and beat them to market. Perhaps someday they will die off.
- travisjungroth 5y ago
  > Python won because people who knew math/science domains only knew Python. This doesn’t explain why they knew Python in the first place, a pretty critical step. It reached popularity without a platform mandate (JS, Swift) or corporate backing (Java, Go) so there’s something going on.
  - blunte 5y ago
    Python has a lot of great libraries. It's the inverse of the chicken-egg problem. Because there are now some critically important libraries (pandas, numpy), it means that is the obvious starting place if you want to hit the ground with minimal effort. I think that's totally fine for uni. But there should be a capstone level class for data/ai scientists before they can graduate which shows other languages and teaches some general best practices of software development. There are plenty of other languages which can do the same job. And honestly, the algos which are available can be recreated if they don't exist. Most of it is not "rocket science". But the greater problem is that Python itself is a poorly designed and warty language. Whether a scientist or not, choosing Python means fighting these warts. No amount of make-up can cover some of these; and plenty of other languages start with clearer foundations.
oh_my_goodness 5y ago
This article expresses the ancient Python(/Matlab) v Fortran argument beautifully ... but it's kind of shocking that the argument is still going on at all. My generation came out of school happy to use FORTRAN indirectly, via a scripting language, for rapid prototyping. That was 30 years ago.
keskival 5y ago
I don't think Python displaced Fortran in HPC as much as it displaced Matlab (and Octave) and R in scientific computing. Displacing Fortran was a side-effect of that trend, as now it wasn't about productionizing Matlab code into Fortran, but Python could do general purpose computing adequately as well.
d--b 5y ago
Yeah it’s counter-intuitive, and it’s because it does not make much sense. Slowness is one thing, but the tooling is also clearly subpar compared to languages of the same popularity, the dynamic typing makes things difficult to maintain, the 2.7 vs 3 shit show etc. etc. The very fact that many smart people have been saying for years that Python is a fairly bad tool for data analysis should at least raise some people’s eyebrows. But no, the entire field of data science has decided that it knows better… Good for them.
nanochad 5y ago
Python is what has been popular for the last 15 years. Scientists are not programing language geeks, they just use whatever is popular, viable, and established.
bernulli 5y ago
For those unfamiliar, CERFACS (Centre Européen de Recherche et de Formation Avancée en Calcul Scientifique, i.e. European center for research and advanced training in scientific computing) is a leading research institution, with two main branches: meteorology, and engineering computational fluid dynamics. I am not affiliated and can only evaluate the engineering part, their combustion modeling group is one of the best in the world.
efxhoy 5y ago
I wrote scientific python for several years at a university research project, coming from a statistics background. I wrote a forecasting tool and related plotting, simulation, ML, evaluation etc tools. The reasons for python’s success are obviously the ecosystem. Numpy is the foundation. On top we have sklearn, statsmodels, pandas, matplotlib. Before our project most work in the department was done in Stata, a proprietary language/tool that works well for some classical regression and stats work but falls apart as soon as things get complicated. Moving to python allowed us, a group of social scientists, to work on some really hard problems. Now we have boosted tree models and other tools that just can’t be used in the old tools like Stata. Python and R run the show in social science.
socialdemocrat 5y ago
The performance Python is a real problem but Python has succeeded because scientific computing really needs interactive and dynamic programming languages. You need something which lets you easily experiment with data, plot, change code in rapid iterations without constant recompilations and reloading of data. This has been recognized for some time. The compromise had been to build performance sensitive parts in C/C++ and do the experimental/iteration part in Python. But today you don’t really have to compromise anymore. We got Julia. It solves the whole problem. You get the interactivity you need combined with the performance. Of course in my his industry momentum matters. Python has built up the momentum of an oil tanker. Even if you shut off the engines it is going to keep going for many years. But Julia is the obvious end station. It does all the things HPC and scientific computing needs. But building mains share, documentation, community, polish tools etc will of course take time.
- akhmatova 5y ago
  It solves the whole problem. You get the interactivity you need combined with the performance. There are other aspects to the "whole problem" -- you also need a massive ecosystem with adoption across disparate communities (devops, web development, etc). And decades of momentum. That's why Python isn't going away anytime soon, despite its slowness and warts.
gaze 5y ago
This article has been written a hundred times. "We abandoned a fast language for a language that is slow but can use fast libraries, and so the result is fast. It's faster because the programmer discovered existing libraries that do a better job of what they were doing already." There's so many convolved factors here I don't even know where to begin, so I guess I'll just say that I'm glad Julia exists. The author glosses over many decades of programming language and compiler research -- which makes sense, because this is not their specialty. However, what I see is the field of scientific computing migrating from a dinosaur language (Fortran isn't, actually. It just is used this way) and dinosaur practices of writing everything oneself, to one of the slowest interpreted languages that happens to be the most difficult to JIT, and saying this or that about how interpreted languages are slow but library calls are fast to justify this. At the same time they're learning to build a functioning library ecosystem. Basically, grad students are learning proper programming practices and collaboration after switching to a more expressive language, they just managed to pick the slowest and most difficult to optimize one. Maybe they just managed to wipe some of the slate clean by switching away from Fortran and its culture (the culture being the bad part), and the culture of Python filled the space, creating a net positive but somewhat unfortunate situation. Just one more time -- the idea that you can call a "faster" language to do the heavy lifting is true of every language and does not justify the choice of Python in particular. The justification for Python is the momentum, and this is in my opinion the only one.
- fastball 5y ago
  Your conclusion seems a bit reductive/circular and begs the question "yes but why does Python have momentum?" Python is/was chosen because the syntax is clean and expressive and obvious if you speak English (which most people in this context do) and because although performance is almost always worse in interpreted languages, there are clear productivity benefits when doing the kind of programming that is demanded from data science / data processing / etc. Same for dynamic vs static typing and a number of other choices made by Python. Specifically, many (most?) programs are not long-term maintained projects in this space. A lot of them are just little scripts to convert one dataset into another format, or scrape specific content from somewhere, or support a scientific paper that will not get updated after publication.
sytelus 5y ago
A lot of thanks should go to Oracle. Back in the days Java was go-to language for everything. After Oracle acquired it in 2009, the only respectable languages with good numerical libraries were Python, Julia and R. Unfortunately, Julia’s marketing wasn’t strong enough and R was decisively an ugly thing to work with.
Sugimot0 5y ago
I feel like nim[0] should replace python as it matures, but I would be curious to hear others perspectives, since mine is mainly based on reading. [0]: https://nim-lang.org/ https://nim-lang.org/
teleforce 5y ago
I'm genuinely surprised that no one here is mentioning D language in addition to Nim or Julia for replacing Python. D has already beaten Fortran in speed more than 5 years back, the legendary scientific programming language that's mentioned in the article [1]. The Fortran based libraries that are overcome by the D language apparently are still being used by Python, Nim and Julia for most of their high speed processing until today. As they always said the proof is in the pudding, and compare to all alternative D language is designed to have a similar feel to Python. By default it supports GC for easier and manageable scientific programming that is very attractive for the type A data scientist that are mainly deals with analysis and exploratory programming [2]. The latest D language is now also natively support the C language (lingua franca of scientific programming) in its compiler thus can import and compile C files directly [3]. [1] Numeric age for D: Mir GLAS is faster than OpenBLAS and Eigen: http://blog.mir.dlang.io/glas/benchmark/openblas/2016/09/23/glas-gemm-benchmark.html http://blog.mir.dlang.io/glas/benchmark/openblas/2016/09/23/... [2] There are two types of data scientists — and two types of problems to solve: https://medium.com/@jamesdensmore/there-are-two-types-of-data-scientists-and-two-types-of-problems-to-solve-a149a0148e64 https://medium.com/@jamesdensmore/there-are-two-types-of-dat... [3] Adding ANSI C11 C compiler to D so it can import and compile C files directly: https://news.ycombinator.com/item?id=27102584 https://news.ycombinator.com/item?id=27102584
jurschreuder 5y ago
People always give the argument that python calls c++ libraries, but I use both Python and c++ a lot, and writing c++ directly, calling the the same libraries, is way faster.
- ctchocula 5y ago
  I suspect the reason people claim that is they are training ML models, which may take O(hours ~ days) to run anyways. In this usecase, C++ calling C++ is faster than Python calling C++ slightly in O(seconds) is outweighed by the fact that Python is more convenient for the ML practitioner trying different models.
musicale 5y ago
I gave up Matlab and never looked back. As the article notes, various numerical kernels have been wrapped as Python compiled modules/libraries, and numpy and other systems seem to work OK for many applications.
guidorice 5y ago
I am really curious how Zig lang eventually does in scientific computing. It's already speedy compiler, language server (zls), and upcoming hot code reloading feature, makes me think that reactive coding and visualization notebooks in Zig should be feasible. Although, Zig has no operator overloading, and no dynamic dispatch though, making it fundamentally pretty different than say, Julia lang. Just as an aside: for my day job, I write Python in a scientific computing (geospatial and ML).

220 points by leonry 5y ago

jrm4 5y ago

Yet another hardcore programmy type discovers that usability is infinity more important that how many clock-cycles you save. Programming languages aren't for computers, they're for people.

da39a3ee 5y ago
matplotlib doesn't score highly on usability. jupyter notebooks encourage disorganized, unprincipled programming; chaotic re-running of cells in the face of global mutable state; and prevent budding programmers from learning to use version control because the JSON format was designed without version control in mind.
- mistrial9 5y ago
  .. and get more new users each year than four generations of PCs combined
- monkeybutton 5y ago
  Being able to hack out code to explore and experiment with data while not having to reload and reprocess data (thanks to that global mutable state!) saves a hell of a lot of time in the long run.
- wiz21c 5y ago
  For Jupyter, it depends on the workflow. Especially with data sciences. In data science, you spend a lot of time playing with the data, testing things, drawing charts, computing, etc. When you do that, the cost of starting a python interpreter, loading the imports, loading the (usually big) data becomes a real pain 'cos you iterate like hell. Working in a REPL becomes really important. But even more, working with Jupyter allows you to work out a very detailed explanation of your thought process, describing your ideas, your experiments. Being able to mix code and explanations is really important (and reminiscent of literate programming). You got the same kind of flow with R. As data scientist, I'm concerned about data, statistics, maths, understanding the problem (instead of the solution). I don't care about code. Once I get my data understanding right then comes the time of turning all of that into a software that can be used. Before that, Jupyter really gives a productivity boost. For the code part, yep, you need other principles where Jupyter may not be suitable.
  - patrick451 5y ago
    It's interesting, I never feel like I get these exploratory benefits from jupyter notebooks. I just end up feeling like one hand and half my brain is tied behind my back. I'm most productive iterating in a similar way to what you describe, but in an ipython terminal, running a script and libraries that I'm iterating on in a real editor. If there are expensive computations that I want' to check point, I just save and load them as pickle files.
    - wiz21c 5y ago
      really interesting. I may have overlooked IPython a bit (I just thought Jupyter was its improved version). For the moment, maybe like you, I prerpocess the data (which takes minutes) into numpy array which then take seconds to load. But once I add imports, everything takes about 5 or 6 seconds to load everything I need. So Jupyter remains a good idea. Moreover, I love (and actually need) to mix math and text, so markdown+latex maths is really a great combo. I dont' know if one can do that in IPython, I'll sure look!
    - kzrdude 5y ago
      I have to say I think a jupyter notebook format is a 10x improvement in productivity over ipython. It's just so much easier to work with - and a step more reproducible too, my scribbles are all there saved in the notebook, at least!
- patrick451 5y ago
  > matplotlib doesn't score highly on usability. The one place matplotlib sucks is any kind of interactivity. But other than that, matplotlib has the best, most intuitive interface of all the python plotting libraries I've tried. It's also one of the few libraries that doesn't rely on generating html for a webbrowser, which makes for a miserable workflow. I still think Matlab's plotting is untouched by open source options.
  - taeric 5y ago
    The charts that R produces are typically better looking. But...R. :(
  - cinntaile 5y ago
    Matplotlib hides a lot of complexity if you ask me. As soon as you do something in a different way than intended you're off searching stackoverflow for a post that did something similar to what you want. Then you tweak it a little and hope it works.
- analog31 5y ago
  I've programmed in a number of languages over the past 40+ years, starting with BASIC, and every one of them encourages sloppy coding. The good discipline always has to be taught, learned, and willingly practiced. The closest I came to a language designed for teaching good practices was Pascal. I find it easier to read and understand bad code written in Python, than good code written in the C family languages.
  - da39a3ee 5y ago
    Yes, what I was saying was that writing Python code in files is a better and more educational way to program than writing Python code in a Jupyter Notebook. It wasn't a criticism of Python.
    - analog31 5y ago
      I use Jupyter a lot, but have a personal rule to do "restart kernel and run all cells" once in a while, to scare up any kind of hidden state or out-of-order execution problems. For instance, if I'm about to leave a notebook for a while, I'll make sure it runs without error from top to bottom. In that sense, I'm making it work like Python code in a file. The advantage of code in files is that I can use all of the slick code analysis tools that will warn me about my mistakes. I wish there were something that would let those tools go through the code in a Python notebook from top to bottom.
- kzrdude 5y ago
  papermill is good and ploomber is a thing to watch. Ploomber makes it systematic - store notebooks as .py (py:percent files for example), parameterize them with papermill and execute as a batch job. One can view the resulting jupyter notebooks as .ipynb later and produce reports as html if wanted. It's really good already, and better if ploomber gets more development. The whole reason it works is because it's easy to open the .py notebook and work on it, interactively, in jupyter. The main idea - jupytext for .py notebooks and papermill for parameters & execution - that's already "stable" and easy for anyone to use for their own purposes.
  - edublancas 5y ago
    (ploomber maintainer here) Any feedback for us? What can we do to improve Ploomber?
    - kzrdude 5y ago
      Maybe I haven't come far enough with my ploomber use to tell yet! It works nicely but I know I'll learn more and open my eyes more as I go. As a first impression, I eventually found meta.extract_upstream = False which I think is important. Reason: The code for each step should be a lego piece, a black box with inputs and outputs. That code should not itself hardcode what its predecessor in the pipeline is - you connect the pieces in pipeline.yaml. (extract_upstream = False is not by itself enough to solve this, since you also need to be able to rename inputs/outputs for a notebook to be fully reusable as a lego piece, but it's good enough for now.) I also for my own sanity need to know more about how the jupyter extension part works, how it decides to load injected-parameters or not. But maybe I could learn that somehow from docs. In general I want components that are easy to understand and plug together and less magic (but the whole jupyter ecosystem's source code feels this way to me unfortunately, lots of hard to follow abstractions passing things around). But it's developing rapidly and already very useful, thank you so much!
      - edublancas 5y ago
        This is great feedback, thanks a lot! I'll ensure we display the "extract_upstream" more prominently in the docs, we've been getting this feedback a couple times now :) Re: the Jupyter extension injects the cell when the file you're opening is declared in the pipeline.yaml file. You can turn off the extension if you prefer. Feel free to join our community, this feedback helps us make Ploomber better! https://ploomber.io/community https://ploomber.io/community
rewq4321 5y ago
Not infinity, but yeah it's worth more than people generally think. But in the end you don't really lose many clock cycles anyway because everything actually runs in C/CUDA/etc. behind the scenes
deleted 5y ago
[deleted]
Yoric 5y ago
I personally find numpy irritating, surprising, misleading and definitely a wrong definition of "usability". YMMV.
- Scene_Cast2 5y ago
  What are your thoughts on Pandas? As someone who uses numpy almost daily, I think that numpy is "overextended" beyond its core niche, sure. So - making it work with things outside that niche (e.g. streaming, non-rectangular data, non-uniform data, nonhomogeneous data, etc) is painful. However, 1) there's Pandas for that, and 2) I disagree with "misleading" and "surprising". What makes you think that?
  - m_mueller 5y ago
    Not GP, but I’m using pandas daily to build up a BI platform within a financial institution. Compared to Matlab and even Fortran it has some issues IMO: * why distinguish between Series and DataFrame? just give me an interface for m x n matrices or even higher dimensions. * pure vs. in-place operations. not such a big fan of having multiple versions of the same function, e.g. a more pythonic df[“my_col”] = series vs. a more functional df.assign({“my_col”: series}) ; I’d rather have everything like the latter to be able to more easily have best practices in place. That brings me to another point: if we keep everything purely functional, then python’s syntax is making things a bit awkward. Where in something like JS you could just put every function call with its dot on a new line without the need to assign, in Python this requires putting line break characters or wrapping it in round brackets. This is one place where a language with explicit assignment terminators (semicolons) are a bit cleaner to work with. All that being said scipy is still a great choice to have both system programming and numerical business logic in one language.
  - J253 5y ago
    In my opinion and experience, I think you’re right about “there’s Pandas for that” and “that” can be almost anything. It can do almost anything but making it do almost anything requires constant reference to the docs. And I find maintainability difficult. It seems like there’s 50 kwargs for every method. Sometimes things happen in place by default, other times they don’t. Compound indexes still confuse me. But I’m not a data scientist so I don’t do much ad-hoc analysis that seems typical with pandas users.
matsemann 5y ago
I wonder how many days have been wasted on non-programmers trying to get their Conda environment up and running or similar. Half the data science stuff isn't reproducible, not because of the science, but because getting the notebooks running with its dependencies is almost impossible.
- lysecret 5y ago
  have you heard of docker?
- irrational 5y ago
  Which begs the question, why has nobody built anything better?
  - taeric 5y ago
    I'd argue that they probably have. It just isn't free.
  - belval 5y ago
    Because it's a hard problem and people love hating on Python because it doesn't come with a way to handle all the compiled dependencies that work for every OS. Data science has a ton of moving library parts, it is genuinely difficult to distribute precompiled libraries for everyone when you have 2-3 actively maintained CUDA version with 2 cuDNN version for accelerators that change every 2 years. Most team fail to standardize on an environment (say Python 3.8, Ubuntu 20.04, CUDA 11.1, and cuDNN 8) and then get hung up on a dependency not building as if it's Python's fault that it does not have control of your entire OS.
    - matsemann 5y ago
      But why is it such a big problem in Python compared to other stacks? Why does all python projects end up depending on you having those exact tools of things installed locally and the planets aligned a certain way, when other stacks do not?
taeric 5y ago
This feels misguided, too. It begs the question that python has good usability. Anyone that has tried managing dependencies in it will know that is mostly a lie. What python had, was that it was preinstalled on many computers and then had a large cohort of users that are insisting that others use it. And mostly force proclaiming that it is easy and readable. I'll not claim that it is hard, per se. More that it is not intrinsically easier than any other dynamic language. For evidence, the main packages that are popular are often clones of packages from other environments that were not widely installed. Jupyter can be seen as free version of many scientific applications. Matlab, Mathematica, etc. Matplotlib is rather direct in it's copy. Pretty sure there are more examples.
- iainctduncan 5y ago
  Is managing deps in Python a pain? sure is!. Is it a pain in the other contenders of easily available dynamic languages? Yup. So that's a wash. Managing deps in dynamic languages is not a simple problem, I can't say I've tried one that did it super well yet.
  - taeric 5y ago
    I'm... Not sure insulting other languages is a path to victory. Npm, as much as it annoys me, is light-years ahead of anything in python. Quicklisp is rather pleasant, now. Ruby has had gems for a long time. I grant that it is a hard problem. I am not griping that it is not solved. More that the option community has largely failed to even pick a direction. The most used dependency methods are, in the modern spirit of python, deprecated already.
    - iainctduncan 5y ago
      I'm not insulting other languages - I'm pointing out they have the same situation! Where in that post is there an insult??? I have had many dep fights with Ruby and Node too, and I would disagree that their solutions are in any way better than pip and virtualenv.
dang 5y ago
"Don't be snarky." "Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- jrm4 5y ago
  Harsh isn't the same as shallow. I don't believe snark is ALWAYS uncalled for, only that you deploy it when it's reasonably necessary. And this is a thing that people, especially HERE, get wrong a LOT. That it's unpleasant or hits people where it hurts doesn't mean it's not important, and I think it is.
  - dang 5y ago
    The issue is the effects that it has on the ecosystem. It dumbs down threads and makes people nastier. If someone is wrong and you know more than they do, you can share correct information without putting them down.

jrochkind1 5y ago

As a rubyist, it makes me sad that python ended up here rather than ruby. And I sometimes wonder why. > As the name suggests, numeric data is manipulated through this package, not in plain Python, and behind the scenes all the heavy lifting is done by C/C++ or Fortran compiled routines. So I wonder, was it easier to write C/C++ or fortran compiled extensions in python than it was in ruby?

belval 5y ago
I don't know how easy it is in Ruby so I cannot give you a comparison. However it is very very easy to write Python bindings for a C/C++ library with minimal work. Solutions range from "just works" like ctypes to "actually integrates with the language" like Cython. You also have automated tools for wrapping like pybind11 which does a lot of the heavy lifting for you.
prpl 5y ago
It was multiple things, really. I would attribute ute some of it to Swig, Perl attrition, SCons/Software Carpentry, integration with GUI libraries, good documentation, and various other efforts in the mid 2000s. A lot of those things were solving research problems simply, and Python’s use just kept expanding. Python was already taking over in many use cases by late 2000s. Ruby was known, but it didn’t have the following at multiple levels in academia like Python did
- jrochkind1 5y ago
  You describe what happened, which I saw happen too. The question I have is why though. Right, why did python's use in scientific computing keep expanding, and not ruby's? Why was python already taing over many use cases by the 2000s, but not ruby? Why did python develop the following at multiple levels in academia, and not ruby? (Why is Perl attrition relevant, when ruby was in fact explicitly based on Perl?) That's the question, not the answer! It seems like a lot of the answer is NumPy, which makes the question -- why did NumPy happen on python, not ruby? Certainly one answer could be "nothing having to do with the features of the language, it's just a coincidence, they chose to write it in Python, if those working on numpy had chosen to use ruby instead, history would be different." But one hypothesis is that maybe NumPy wouldn't have been as easy in ruby as python. Someone else suggested the first numpy release happened before the first ruby release, so that could also be an answer.

elil17 5y ago

It’s amazing how often the authors point of “agility” arises in real world circumstances. I’m not a programmer, but I use Python a lot in my engineering job. There have been 3 times in the past month where I got an order of magnitude speed up because SciPy implements a very complex but highly efficient algorithm which I would never have had time to deploy.

JuettnerDistrib 5y ago
> There have been 3 times in the past month where I got an order of magnitude speed up because SciPy implements a very complex but highly efficient algorithm which I would never have had time to deploy. Yes. I feel like the author conflates the language with the package ecosystem. Pure Python is pretty horrible for scientific computing (3*[3]=[3,3,3] is about as counterproductive to scientific computations as it gets), but Numpy changes the semantics of those operations. In other words, Python has an absolutely stellar package ecosystem. There have been attempts to bring a package ecosystem to C, but it never took off. However, I do wonder how C would fare if it had.
- oh_my_goodness 5y ago
  Yes, sure, the strength of Python is the libraries.
- Enhex 5y ago
  there are package managers for C and C++, but people aren't told about them. https://conan.io/ https://conan.io/ https://vcpkg.io/en/index.html https://vcpkg.io/en/index.html this implies Python's advantages is not having a package manager, but better teachers or at least teaching better practices, so it isn't even language related.

iainctduncan 5y ago

If you know actual scientists, this isn't counter intuitive at all. My partner is a scientist, so now I know tons of them, and I have done a bunch of Python coding and support for scientists, have been a Python programmer (as well as other languages) since 2005-ish. I saw this coming (as did many) 15 years ago. Most scientists, and their grad students, are trying to do a whole bunch of things in their research, and programming is just one of them. Field work, experiments, data wrangling, writing papers, defending papers, teaching, etc. And most of them do not have access to budgets for programmers or when they do, it's for a limited amount of time and work, meaning they need to be able to pick up and run with whatever the programmer did. So the fact that with Python they and their grad students (who might be there for only 2 years) can be working productively, and figure out what the hell the code did when they come back to it months later, is HUGE. As in, literally blows every other consideration to smithereens. This has meant that over the last 20 years the scientific libraries in Python got mature faster than in any other language, and this in turn has had a snowball effect. And when speed is necessary, C++ extensions can be written. But honestly, most of the time speed is not the main factor. The downside of Python in my experience is that junior teams can make heinous atrocities when a project gets really big (I have had to step in as CTO to one of those messes, so much as I love Python, I must admit this is true!) But the stuff the scientists are doing is very rarely that big. It's tools programming, scripting, making utilities, data analysis and so on. Readability counts. In some fields, it counts more than anything. I've worked in about 10 languages now over the last 20 years, and Python is still the easiest to read when you come back to some old code or have to pick up code for a small job, or hand it to a beginner to extend without having them create an unreadable mess. This is what scientists need to do all the time. Re other people's comments on Python packaging and setup being hard, well honestly I've had just as much pain with Ruby or Node. The shining exception there is R, which is giving Python a run for its money in many scientific areas. R Studio has the best "hit the ground running" experience out there and is really slick for data programming.

analog31 5y ago
My partner's partner is a scientist too. ;-) In addition to not having budgets for programmers, we also don't know how to manage them, for instance how to communicate our needs, decide if their implementation plans make sense, or gauge their progress. Nearly half a century after The Mythical Man Month, managing software development is still generally acknowledged to be an unsolved problem. The other two obstacles are that most programmers hate the scientific work environment, with its ever-changing requirements and frequent dead ends. And, the programmers who can work on math related stuff are in the highest demand.

hulitu 5y ago

I try to love micropython. However, its UI is at ed level. It only says "Syntax error".

the__alchemist 5y ago
Python excels in several domains. For example, the non-speed-critical numerical computing this article is about. It's also nice for backend web development, and scripting. Embedded isn't one of its strengths, and I'm suspicious micropython was an attempt at bringing embedded programming to people who don't want to learn more than one language.

scythe 5y ago

As a physicist, having spent eight years in academia, Python did not win by beating Fortran. Nor did it beat C++. It didn't really compete with Ruby or Lisp, although Lua (Torch) was a briefly serious competitor before everyone realized that a language developed by four people, one of whom doesn't get along with the others, couldn't be responsive to users' needs. Python defeated Matlab. I know because I cheered it on. I was there. I watched my roommates and friends struggle with introductory scientific computing in Matlab and I joined the chorus that was practically begging for Python, even though I didn't really like it. I can't even begin to explain how awful it is to try to teach programming concepts in Matlab. But something like Python or Matlab had to be the choice because the schools wanted to teach programming through a language where you could just call "graph" and the computer would display a graph. Python's team, unlike Lua's, aggressively courted educational institutions by offering scientific, numerical and graphical libraries within a programming language that works like a programming language, not a glorified computer algebra system. They even added a dedicated operator for matrix multiplication. It's a great example of finding a niche and filling it: I still don't like using Python, but I can't dispute that no other language/ecosystem comes close to offering what we need to teach programming to physics students. You want to beat Python? Build a type system that can capture dimensional analysis. Warning: it won't be easy.

rsfern 5y ago
> You want to beat Python? Build a type system that can capture dimensional analysis. Warning: it won't be easy. Curious about your thoughts on pint and Unitful.jl — pint doesn’t really go all the way to a full type system, and Unitful.jl doesn’t work with everything (autograd is a problem still I think). But Unitful.jl is super cool. https://pint.readthedocs.io/en/stable/wrapping.html#checking-dimensionality https://pint.readthedocs.io/en/stable/wrapping.html#checking... https://painterqubits.github.io/Unitful.jl/stable/ https://painterqubits.github.io/Unitful.jl/stable/
_aavaa_ 5y ago
Oh god, the atrocities I have seen colleagues do with MATLAB scripts…
analog31 5y ago

dekhn 5y ago

Counter-intuitive? I picked it because it was the closest scripting language to C (see the select and socket APIs for good examples). And it had numeric array support early-on (making it an attractive replacement for matlab).

smitty1e 5y ago

> Of course, If the best algorithm is known beforehand or the manpower is not a problem, a lower level-language is probably faster, but this is seldom the case in real life. One is wary of one-dimensional analysis of anything in a software context. Who cares if the Fortran library runs like the blue blaze, if it cannot be readily maintained?

Bostonian 5y ago
It is possible to write maintainable modern Fortran without gotos with small functions and subroutines. OOP with inheritance and dynamic polymorphism is possible since the Fortran 2003 standard.

wheelerof4te 5y ago

One thing that I don't like with Python's scientific libraries is how they change the overall Python syntax. There are so many ways to slice an array or a dataframe, and only a few of them are valid Python code. Keeping the language API should have been a priority, but that is a consequence of operator overloading features.

kzrdude 5y ago
Can you explain what you mean more in detail? Libraries can't change the syntax of the Python language, not in the formal sense. Is this about things you want to be able to express in syntax but can't? Or the other way around - stuff that uses syntax/operators but should really be methods?
- ohyoutravel 5y ago
  Numpy syntax comes to mind. The extra commas often aren’t valid pure Python but are required for some operations on numpy arrays. I don’t know how this works under the hood, but expect it’s a state machine under the numpy ndarray looking for the extra commas and such. i.e. some_array[0:5,0] which isn’t valid pure Python notation.
  - kzrdude 5y ago
    Extra commas are "valid in pure python" in the following sense that I can demonstrate. Open ipython3 In [3]: class Test: ...: def __getitem__(self, index): ...: print(index) ...: In [4]: Test()[1, 2, 1:3, ..., :] (1, 2, slice(1, 3, None), Ellipsis, slice(None, None, None)) It's valid and we get the complicated tuple of integers, slices, ellipsis etc as printed. Numpy has existed for a long time. Its needs have been taken care of in upstream Python, to a big extent, and other libraries can use the same features.
    - ohyoutravel 5y ago
      Interesting! Neither myself nor my coworkers could get the snippet I posted working outside the context of an ndarray, so I had speculated at that time that it there was something else going on under the hood. You seem to have a much better grasp of Python than us, would you mind posting an example where the snipped I posted successfully accesses data from an array in pure Python? That way I can not only take the L, but correct the record and learn something in the process.

derbOac 5y ago

C, C++, Fortran are still used, most Python users just don't see it because it's hidden away underneath the calling function. I've been surprised by the rise of Python in some ways although not at all in others. Languages like C, C++, Fortran, and dare I say it Rust are too low-level in their raw state for numerical computing. You had the US federal government funding language competitions because of this (see: Chapel). Languages like Python and R (and before that things like Lisp) came along and gave people a taste of something different, and it's obvious what people migrated to. Part of it is timing: multivariate computational statistics (ML/data science/DL/whatever you want to call it) just sort of started taking off in computer science communities before LLVM-based languages like Julia or Nim could get a foothold. OCaml might have fit that niche but never got there because of a desire to take a different path, or take the path more slowly. So people looked for a nice expressive language, found it in Python, and buried all the messy stuff behind wrapper functions and called it a day. It was furthered along by Matlab being another comparison on the other side -- Python looks kludgy compared to modern Fortran or C, but not compared to Matlab. All that wrapper time in Python has its costs, so I suspect as limits get pushed further we'll eventually see a migration to something else like Julia or Nim, or something else not on anyone's radar. One moral to this story is that expressiveness matters. People will go out of their way to avoid talking directly to machines at a low level.

zozbot234 5y ago
> You had the US federal government funding language competitions because of this (see: Chapel). Wait, weren't they supposed to be using Ada for everything anyway? What's wrong with Ada?
- tormeh 5y ago
  Different design objectives, I believe.
Robotbeat 5y ago
It’s because Matlab (and Mathematica, etc) is proprietary, and therefore you always have to pay the Danegeld. So we use numpy instead because it’s extensible, it uses all the super fast C/C++/FORTRAN stuff on the backend, and is fairly easy to learn. I actually still would prefer Matlab as the syntax is more compact and natural than numpy (which is like a matlabified Python), but that’s probably just due to more experience in Matlab.

photochemsyn 5y ago

Python displaced a lot of very expensive proprietary software in the biosciences arena. Ease of use was also a major factor, as many bioscientists have relatively little background in programming, but the ability to escape the world of expensive restrictive software licenses was very attractive to the scientific community, whose historical norms emphasize the open sharing of methods and results: > "A program that performs a useful task can (and, arguably, should) be distributed to other scientists, who can then integrate it with their own code. Free software licenses facilitate this type of collaboration, and explicitly encourage individuals to enhance and share their programs. This flexibility and ease of collaborating allows scientists to develop software relatively quickly, so they can spend more time integrating and mining, rather than simply processing, their data." https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1004867 https://journals.plos.org/ploscompbiol/article?id=10.1371/jo... Now there isn't any area of molecular biology and biochemistry that doesn't have a host of Python libraries available to assist researchers with tasks like designing PCR strategies or searching for nearest matches on up to x-ray crystallography of proteins.

whatever1 5y ago

Julia is the next big thing. I am always blown away by its readability and speed. But it will take years to build a library ecosystem that can rival the python one.

a9h74j 5y ago
Remind me if anyone might, is there any story for Julia making use of existing python libraries?
- sundarurfriend 5y ago
  If you mean whether it's possible, PyCall.jl has existed since nearly the beginning of Julia, and PythonCall.jl [1] is a more recent package for the same core functionality - calling into Python code. [1] https://github.com/cjdoris/PythonCall.jl https://github.com/cjdoris/PythonCall.jl
II2II 5y ago
That is a big part of the author's point: the library ecosystem is here for Python today. While there is a heavy penalty for anything written in Python itself, it doesn't really matter since there isn't much of a penalty once the data is passed to highly optimized libraries and those libraries allow developers to select efficient algorithms rather than implementing their own algorithms (which are likely to be less efficient).
a9h74j 5y ago
In this thread I am seeing a number of explanations, including: Ecosystem; mind-share; readability and engineering mind-set; history/Numpy/Matlab; teachability and academic focus. There are also comments emphasizing the "dynamic" scientific environment and need to just pick up code left by others. In terms of the latter, could one apparent requirement be this: The main contact should be with top-level code which at least looks like it is interpreted -- even if through compile-with-run-combined and/or memoization? Need part of the user interface, so to speak, be to hide all intermediate artifacts, even the very thought of object code and executables? That such stuff is for, say, "module creators" not primary users?

uoaei 5y ago

Python is an API to efficient scientific computing code. It's good for that, assuming you're using old and more verbose languages. Look into Julia as a promising alternative -- the language itself is superbly fast (aside from initial compilation) and there's an impressive scicomp ecosystem to say the least, all written in native Julia. This allows for program rewriting / metaprogramming more broadly and is insanely powerful once you get a feel for it.

laichzeit0 5y ago
No need for any alternatives. Python as an API is good enough.
- uoaei 5y ago
  It is really remarkable how much more expressive some languages are over others. If you are satisfied with Python for everything you do, then you are not hitting the limits of its expressiveness. But for more naturally expressive code, other languages may have huge advantages for certain applications.

deleted 5y ago

[deleted]

amelius 5y ago

New languages should always provide bindings to call into Python modules, so you get the immediate benefit of the largest ecosystem on the planet.

jackjackk0 5y ago

I recommend one of the recent videos by Dave Beazly [1]. He lived through and contributed to the raise of Python in scientific computing first hand in the 90s, and offers some interesting insights. Plus he's always quite an entertainer. [1] https://youtu.be/4RSht_aV7AU https://youtu.be/4RSht_aV7AU

fancyfredbot 5y ago

I feel like python acts like a kind of bus in scientific computing, connecting various high performance libraries and DSLs together. That said, this article's story of someone using the wrong algorithm is a bad example in my view. Python hasn't succeeded because people are more likely to use more efficient algorithms due to easier experimentation, it has succeeded because the of the size of the ecosystem and the fact such algorithms are easily available.

StreamBright 5y ago

Python is the common scripting language of C, C++, Fortran.

sega_sai 5y ago

As many already noticed, the rise of Python is not counter-intuitive at all. (I'm a scientist myself). Basically modern python offers you a spectrum from easy to understand and quick to write python programs (those will be slow), to purely glue code that connects a lot of high performance c/C++/fortran code. And many scientists will start from pure python code with the help of numpy. In many cases it will be good enough. But if needed you can always interface with other libraries, or write yourself high performance c/c++/fortran code for the most performance critical bit, and use python to glue it together. That flexibility where you can trade speed of writing the code with the speed of execution is very valuable.

chalst 5y ago
At this point we can say that against the two criteria of a spectrum from prototyping to heavy lifting and ease of embedding external high-performance libraries, Julia is simply better than Python. Julia does have two drawbacks of being tied to the one, rather heavy metal, implementation and lacking the wealth of libraries outside scientific computing.
- bernulli 5y ago
  Its third, and major, drawback is that Python has a legacy in many groups.
  - chalst 5y ago
    True, although the Python-Julia interop is surprisingly good. Gradual migration of legacy code from Python to Julia might be a possibility for some of these groups. But I was really thinking about the situation for new projects when writing that comment.
    - bernulli 5y ago
      Same thing, you will want to reuse existing code, and you do not want to split knowledge in your student group. There is no such thing as 'gradual' migration: I either have to support Python in my group, or Python and Julia -- until everything is migrated. If I have a working ecosystem using Python, with students trained in Python, and all previous work in Python, there's a whole lot of opportunity cost associated with me deciding to have the next student use Julia. I'd rather have that student build on existing tools and knowledge and do something new with their time.

blunte 5y ago

Python won because people who knew math/science domains only knew Python (or it was the best they knew). And so they made libraries for Python. And it propogated like many other bad ideas based on ignorance. Python is a miserably bad language for modern times. If you know any of half a dozen other languages, then you understand. There was a good essay, from Paul Graham?, about the ladder of awareness of programming languages. Unfortunately I can't find it now. The point is, Python has won and is frankly terrible. It has inconsistent features, but it has an awkward OOP approach (in a time when OOP is finally being recognized as bad itself), as well as seriously lacking basic language features which are only appearing as of 3.9 and 3.10. Frameworks like Django and Django Rest Framework expand on these bad ideas, creating monstrosities which make the PHP code of yore look arguably decent. Sadly, I don't think there's any way to kill this. The only option is to vastly outperform the Python people and produce reliable, readable, performant solutions in half the time and beat them to market. Perhaps someday they will die off.

travisjungroth 5y ago
> Python won because people who knew math/science domains only knew Python. This doesn’t explain why they knew Python in the first place, a pretty critical step. It reached popularity without a platform mandate (JS, Swift) or corporate backing (Java, Go) so there’s something going on.
- blunte 5y ago
  Python has a lot of great libraries. It's the inverse of the chicken-egg problem. Because there are now some critically important libraries (pandas, numpy), it means that is the obvious starting place if you want to hit the ground with minimal effort. I think that's totally fine for uni. But there should be a capstone level class for data/ai scientists before they can graduate which shows other languages and teaches some general best practices of software development. There are plenty of other languages which can do the same job. And honestly, the algos which are available can be recreated if they don't exist. Most of it is not "rocket science". But the greater problem is that Python itself is a poorly designed and warty language. Whether a scientist or not, choosing Python means fighting these warts. No amount of make-up can cover some of these; and plenty of other languages start with clearer foundations.

oh_my_goodness 5y ago

This article expresses the ancient Python(/Matlab) v Fortran argument beautifully ... but it's kind of shocking that the argument is still going on at all. My generation came out of school happy to use FORTRAN indirectly, via a scripting language, for rapid prototyping. That was 30 years ago.

keskival 5y ago

I don't think Python displaced Fortran in HPC as much as it displaced Matlab (and Octave) and R in scientific computing. Displacing Fortran was a side-effect of that trend, as now it wasn't about productionizing Matlab code into Fortran, but Python could do general purpose computing adequately as well.

d--b 5y ago

Yeah it’s counter-intuitive, and it’s because it does not make much sense. Slowness is one thing, but the tooling is also clearly subpar compared to languages of the same popularity, the dynamic typing makes things difficult to maintain, the 2.7 vs 3 shit show etc. etc. The very fact that many smart people have been saying for years that Python is a fairly bad tool for data analysis should at least raise some people’s eyebrows. But no, the entire field of data science has decided that it knows better… Good for them.

nanochad 5y ago

Python is what has been popular for the last 15 years. Scientists are not programing language geeks, they just use whatever is popular, viable, and established.

bernulli 5y ago

For those unfamiliar, CERFACS (Centre Européen de Recherche et de Formation Avancée en Calcul Scientifique, i.e. European center for research and advanced training in scientific computing) is a leading research institution, with two main branches: meteorology, and engineering computational fluid dynamics. I am not affiliated and can only evaluate the engineering part, their combustion modeling group is one of the best in the world.

efxhoy 5y ago

I wrote scientific python for several years at a university research project, coming from a statistics background. I wrote a forecasting tool and related plotting, simulation, ML, evaluation etc tools. The reasons for python’s success are obviously the ecosystem. Numpy is the foundation. On top we have sklearn, statsmodels, pandas, matplotlib. Before our project most work in the department was done in Stata, a proprietary language/tool that works well for some classical regression and stats work but falls apart as soon as things get complicated. Moving to python allowed us, a group of social scientists, to work on some really hard problems. Now we have boosted tree models and other tools that just can’t be used in the old tools like Stata. Python and R run the show in social science.

socialdemocrat 5y ago

The performance Python is a real problem but Python has succeeded because scientific computing really needs interactive and dynamic programming languages. You need something which lets you easily experiment with data, plot, change code in rapid iterations without constant recompilations and reloading of data. This has been recognized for some time. The compromise had been to build performance sensitive parts in C/C++ and do the experimental/iteration part in Python. But today you don’t really have to compromise anymore. We got Julia. It solves the whole problem. You get the interactivity you need combined with the performance. Of course in my his industry momentum matters. Python has built up the momentum of an oil tanker. Even if you shut off the engines it is going to keep going for many years. But Julia is the obvious end station. It does all the things HPC and scientific computing needs. But building mains share, documentation, community, polish tools etc will of course take time.

akhmatova 5y ago
It solves the whole problem. You get the interactivity you need combined with the performance. There are other aspects to the "whole problem" -- you also need a massive ecosystem with adoption across disparate communities (devops, web development, etc). And decades of momentum. That's why Python isn't going away anytime soon, despite its slowness and warts.

gaze 5y ago

This article has been written a hundred times. "We abandoned a fast language for a language that is slow but can use fast libraries, and so the result is fast. It's faster because the programmer discovered existing libraries that do a better job of what they were doing already." There's so many convolved factors here I don't even know where to begin, so I guess I'll just say that I'm glad Julia exists. The author glosses over many decades of programming language and compiler research -- which makes sense, because this is not their specialty. However, what I see is the field of scientific computing migrating from a dinosaur language (Fortran isn't, actually. It just is used this way) and dinosaur practices of writing everything oneself, to one of the slowest interpreted languages that happens to be the most difficult to JIT, and saying this or that about how interpreted languages are slow but library calls are fast to justify this. At the same time they're learning to build a functioning library ecosystem. Basically, grad students are learning proper programming practices and collaboration after switching to a more expressive language, they just managed to pick the slowest and most difficult to optimize one. Maybe they just managed to wipe some of the slate clean by switching away from Fortran and its culture (the culture being the bad part), and the culture of Python filled the space, creating a net positive but somewhat unfortunate situation. Just one more time -- the idea that you can call a "faster" language to do the heavy lifting is true of every language and does not justify the choice of Python in particular. The justification for Python is the momentum, and this is in my opinion the only one.

fastball 5y ago
Your conclusion seems a bit reductive/circular and begs the question "yes but why does Python have momentum?" Python is/was chosen because the syntax is clean and expressive and obvious if you speak English (which most people in this context do) and because although performance is almost always worse in interpreted languages, there are clear productivity benefits when doing the kind of programming that is demanded from data science / data processing / etc. Same for dynamic vs static typing and a number of other choices made by Python. Specifically, many (most?) programs are not long-term maintained projects in this space. A lot of them are just little scripts to convert one dataset into another format, or scrape specific content from somewhere, or support a scientific paper that will not get updated after publication.

sytelus 5y ago

A lot of thanks should go to Oracle. Back in the days Java was go-to language for everything. After Oracle acquired it in 2009, the only respectable languages with good numerical libraries were Python, Julia and R. Unfortunately, Julia’s marketing wasn’t strong enough and R was decisively an ugly thing to work with.

Sugimot0 5y ago

I feel like nim[0] should replace python as it matures, but I would be curious to hear others perspectives, since mine is mainly based on reading. [0]: https://nim-lang.org/ https://nim-lang.org/

teleforce 5y ago

I'm genuinely surprised that no one here is mentioning D language in addition to Nim or Julia for replacing Python. D has already beaten Fortran in speed more than 5 years back, the legendary scientific programming language that's mentioned in the article [1]. The Fortran based libraries that are overcome by the D language apparently are still being used by Python, Nim and Julia for most of their high speed processing until today. As they always said the proof is in the pudding, and compare to all alternative D language is designed to have a similar feel to Python. By default it supports GC for easier and manageable scientific programming that is very attractive for the type A data scientist that are mainly deals with analysis and exploratory programming [2]. The latest D language is now also natively support the C language (lingua franca of scientific programming) in its compiler thus can import and compile C files directly [3]. [1] Numeric age for D: Mir GLAS is faster than OpenBLAS and Eigen: http://blog.mir.dlang.io/glas/benchmark/openblas/2016/09/23/glas-gemm-benchmark.html http://blog.mir.dlang.io/glas/benchmark/openblas/2016/09/23/... [2] There are two types of data scientists — and two types of problems to solve: https://medium.com/@jamesdensmore/there-are-two-types-of-data-scientists-and-two-types-of-problems-to-solve-a149a0148e64 https://medium.com/@jamesdensmore/there-are-two-types-of-dat... [3] Adding ANSI C11 C compiler to D so it can import and compile C files directly: https://news.ycombinator.com/item?id=27102584 https://news.ycombinator.com/item?id=27102584

jurschreuder 5y ago

People always give the argument that python calls c++ libraries, but I use both Python and c++ a lot, and writing c++ directly, calling the the same libraries, is way faster.

ctchocula 5y ago
I suspect the reason people claim that is they are training ML models, which may take O(hours ~ days) to run anyways. In this usecase, C++ calling C++ is faster than Python calling C++ slightly in O(seconds) is outweighed by the fact that Python is more convenient for the ML practitioner trying different models.

musicale 5y ago

I gave up Matlab and never looked back. As the article notes, various numerical kernels have been wrapped as Python compiled modules/libraries, and numpy and other systems seem to work OK for many applications.

guidorice 5y ago

I am really curious how Zig lang eventually does in scientific computing. It's already speedy compiler, language server (zls), and upcoming hot code reloading feature, makes me think that reactive coding and visualization notebooks in Zig should be feasible. Although, Zig has no operator overloading, and no dynamic dispatch though, making it fundamentally pretty different than say, Julia lang. Just as an aside: for my day job, I write Python in a scientific computing (geospatial and ML).