11 ms·

Array Programming with NumPy

290 points by hardmaru 6y ago

alextheparrot 6y ago
As someone who cut his teeth on bioinformatics before eventually just completing a full computer science degree, I was a bit worried that this article wouldn’t address the applied scientific audience well. Pleasantly surprised, though, at how well this article evangelizes NumPy to that exact community. Many labs are gaining access to or creating physical tools that create data analysis over experimental design problems. Biologists transitioning from running gels to detect the existence of a gene to running sequencing or flow cytometry to segment populations, for example. I remember seeing I think an RNA-seq experiment that tracked the full lineage of hematopoietic (blood) stem cells by a professor - my jaw was on the floor at the level of insight. One remaining step is transitioning many bioinformatics courses from applied tooling to general program design and open-sourcing. I’ve noticed a few labs have done that really well for years, but it is not often found as a field-level skillset.
- rubatuga 6y ago
  The coolest thing I read last year was about single cell rna-seq trajectory analysis of the differentiation for different blood cells.
  - dm319 6y ago
    You might like the mass cytometry reconstruction of the human haematopoietic system [1]. I've done a bit of scRNA-seq and mass cytometry, and the issue I have with scRNA-seq is the tiny dynamic range it has compared to flow/mass cytometry, which can make identifying populations much harder. Not to mention the cost! [1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3273988/figure/F2/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3273988/figure/...
teorema 6y ago
For some reason this struck me as inappropriate for the outlet. It's a nice piece as an introduction to array programming with numpy, but seemed out of place to me.
- ddavis 6y ago
  If, going forward, 5% of all papers that use NumPy to get their results actually cite this paper, it will be one of Nature's most cited papers every year.
  - lumost 6y ago
    There's an interesting trend of what content gets published in peer-reviewed journals vs. blogs/github/etc. I suspect there is an audience segment that strongly values peer reviewed pieces that are equivalent content wise to introductory material in a variety of formats. I wonder if github should add a "Review" feature to provide a similar content authoring experience.
    - clickok 6y ago
      It would be nice if citing repositories were easier-- either for generating a reference for my own code or acknowledging when I've used someone else's code in my research. There's tons of math and physics blogs that contain useful results that the author wanted to make available but didn't manage to incorporate into a paper. I wonder if there'd be any interest in a sort of GitHub for proofs? It could even use git, since (assuming consistency) isn't math just a DAG anyways (and therefore isomorphic to a neural net, as are all things).
      - throwawaygh 6y ago
        Traditionally that sort of stuff goes in tech reports, dissertations, or text books. What's missing is the dissemination piece. Somehow people will absolutely refuse to take seriously the job of citing code they use, even when their main result is obtainable by "and then I ran something from scipy/numpy/pytorch/etc."
      - westurner 6y ago
        You can get a free DOI for and archive a tag of a Git repo with FigShare or Zenodo. If you have repo2docker REES dependency scripts (requirements.txt, environment.yml, postInstall,) in your repo, a BinderHub like https://mybinder.org https://mybinder.org can build and cache a container image and launch a (free) instance in a k8s cloud. Journals haven't yet integrated with BinderHub. Putting the suggested citation and DOI URI/URL in your README and cataloging citations in an e.g. wiki page may increase the crucial frequency of citation. A Linked Data format for presenting well-formed arguments with #StructuredPremises would help to realize the potential of the web as a graph of resources which may satisfy formal inclusion criteria for #LinkedMetaAnalyses.
throrthaway 6y ago
Ah, just the right time to publish about numpy - right when everyone is moving over to Julia because of numpy's warts.
- microcow 6y ago
  I don't think that's accurate: https://trends.google.com/trends/explore?date=today%205-y&geo=US&q=%2Fm%2F0j3djl7,%2Fm%2F021plb https://trends.google.com/trends/explore?date=today%205-y&ge...
  - throrthaway 6y ago
    Relieved to know I can still trust HNers to always take hyperbolic statements literally
    - ALittleLight 6y ago
      Not always.
- hyperbovine 6y ago
  *For certain definitions of everyone.
  - konjin 6y ago
    Me on a Thursday afternoon when I'm bored.
- alanbernstein 6y ago
  I'm interested in Julia, but if numpy has warts, I suppose Julia has not yet had time to even grow its skin.
  - Fishysoup 6y ago
    Theoretically Julia is better for scientific computing, the only issue is its package ecosystem isn't as mature as Python's. But it's growing incredibly fast and there have already been libraries available for a few years that would be really impractical to write and maintain at such a level in C++ for Python. I assume that for science at least Julia will catch on a lot.
westurner 6y ago
Looks like there's a new citation for NumPy in town. "Citing packages in the SciPy ecosystem" lists the existing citations for SciPy, NumPy, scikits, and other -Py things: https://www.scipy.org/citing.html https://www.scipy.org/citing.html ( source: https://github.com/scipy/scipy.org/blob/master/www/citing.rst https://github.com/scipy/scipy.org/blob/master/www/citing.rs... ) A better way to cite requisite software might involve referencing a https://schema.org/SoftwareApplication https://schema.org/SoftwareApplication record in JSON-LD, RDFa, or Microdata; for example: https://news.ycombinator.com/item?id=24489651 https://news.ycombinator.com/item?id=24489651 But there's as of yet no way to publish JSON-LD, RDFa, or Microdata Linked Data from LaTeX with Computer Modern.
jofer 6y ago
Don't underestimate the impact this has on getting funding or even just tenure/etc recognition for working on numpy. I'm in industry these days, but coming from the academic side, it's _really_ hard to get recognized for building the underlying infrastructure that tons of people use. I've built and maintained libraries that are used in a ton of publications, but was always told my work was "utterly and completely useless". It was also always unpublishable, as methods are never publishable in my field. Numpy has (obviously) vastly more respect and impact than my work, but the general problem remains. Articles like this are a _huge_ deal for that reason. It's an immense delayed recognition for over a decade of work from a lot of folks.
- teej 6y ago
  > my work was "utterly and completely useless I can never understand the arrogance of folks who would say something like this
  - austincheney 6y ago
    I used to maintain a code beautifier/diff tool and hear the same things about how the idea was useless only to see those same people shortly there after use my tool or a close competitor. Once you see that pattern a few times you learn to ignore it in it’s entirety. It’s hard to tell why people behave like that. I presume it’s because many people have a great fear of originality and require social validation.
    - mjburgess 6y ago
      Asymmetry between complexity of principle and power. Ie., we are easily persuaded that something very complex will be very powerful (eg., a smart phone) -- but we intuitively regard something simple (eg., a hammer) as under-powered. Hard to say how well this actually holds, but I'd guess in both cases we arent really enumerating use-cases in our head, we're just using explanatory complexity as a guide to practical power. This is probably more extreme in cases where people have a specific notion of complexity in mind, eg., in academic environments where "tool A" is as simple as "tool B" if they use the same theoretical basis. ie., Tool C is worthwhile if it includes a more complex theory, as therefore it is more powerful.
1vuio0pswjnm7 6y ago
BLAS and LAPACK were originally written for FORTRAN, and FORTRAN is particularly suited to numerical computing and array programming. https://modelingguru.nasa.gov/docs/DOC-1762 https://modelingguru.nasa.gov/docs/DOC-1762
- 6gvONxR4sf7o 6y ago
  Aren't BLAS and LAPACK just for the linear algebra bits? Numpy is so much more than linear algebra.
  - bagels 6y ago
    Yes, there's a lot more to it than that. For instance, the interface and array/matrix types make vector operations really natural and efficient in python.
- pjmlp 6y ago
  And quite modern with recent language revisions, including OOP, generics, modules and first class support for C interoperability.
- sradman 6y ago
  Indeed, FORTRAN has traditionally been used for vector and matrix math and NumPY adds the equivalent functionality to Python. The simple linear algebra abstractions for manipulating these structures are fundamental to each of the popular data science platforms. MATLAB was created as an interface to LINPACK/EISPACK without having to learn FORTRAN. The importance of this comment is the emphasis on the core fundamentals shared by all the data science platforms rather than the different tradeoffs inherent in each ecosystem.
hoytech 6y ago
Let's not forget to give at least some credit to Perl Data Langauge (PDL). It pioneered a lot of these ideas 10 years before NumPy existed, and is still a pretty great tool today: http://pdl.perl.org/index.php?page=FirstSteps http://pdl.perl.org/index.php?page=FirstSteps
- ihnorton 6y ago
  NumPy’s direct predecessor Numeric was released in 1995.
  - beagle3 6y ago
    ... and then in 2001 or so, numarray came about and was supposed to be better - but instead it sort of split the community and wasted resources. Then Travis started Numpy which somehow magically was backward compatible with both numeric and numarray - and managed to get the community united again.
  - hoytech 6y ago
    I didn't know that, thanks! It looks like these ideas were being tried in perl and python roughly in parallel.
- mjburgess 6y ago
  Er... array programming and statistical programming languages pre-date both by decades. APL from 1966, I believe, is the key lang for array programming. And statistical languages like S from 1976 come to mind: https://en.wikipedia.org/wiki/S_(programming_language) https://en.wikipedia.org/wiki/S_(programming_language) At a quick glance, it seems PDL is just a variation on S.
  - ktpsns 6y ago
    R (https://en.wikipedia.org/wiki/R_(programming_language) https://en.wikipedia.org/wiki/R_(programming_language) ) is kind of the successof of S. In certain communities (not only statistics, but for instance also biogenetics), there is quite some concurrency between R and (scientific) Python for data science. For my understanding, the numpy syntax most closely resembles what would be possible in matlab. And matlab again seems to have roots from Fortran. Thanks to that, young folks nowadays can switch so easily between Fortran and Numpy, the syntax and call structures can easily be made to almost fit to each other.
enriquto 6y ago
Young programmers are slowly but steadily reaching the power that Fortran programmers had 40 years ago. That's good news!
- mumblemumble 6y ago
  That statement only feels true to me if you interpret the word "power" as an exact synonym for "performance." Which is a definition that is valid, but also just about perfect for leading someone to miss the point. Numpy approaches the performance you could get with Fortran. Mostly because its core is written in Fortran. What Numpy offers that Fortran never did, though, is leverage. The article mentions, but doesn't really do justice to, the sheer volume of interoperability that Numpy has enabled. It's not just that all these libraries were built on top of Numpy. It's also that their common Numpy substrate makes them all deeply interoperable with each other. And that works both above and below the boundary. You can swap out BLAS and LAPACK for something else - say, CUDA, or a distributed representation - and as long as the replacement also speaks Numpy's language, you can plug it into existing libraries that were originally written against Numpy. In short: Fortran gets you performance. Numpy gets you that, and also productivity. I would argue that that actually makes Numpy more powerful than what was possible with just Fortran.
  - teleforce 6y ago
    Try D language, with its numerical library you can get both productivity and performance that is even better than the OpenBLAS (Numpy and Julia library are based on) [1]. In D you get the consistency of a single unified language semantic unlike the impedance mismatched approach that is inherent in Python and Numpy programming combination. [1]http://blog.mir.dlang.io/glas/benchmark/openblas/2016/09/23/glas-gemm-benchmark.html http://blog.mir.dlang.io/glas/benchmark/openblas/2016/09/23/...
Mikhail_K 6y ago
Numpy array syntax inconsistent to the degree it can be considered broken. For instance, to take elements number 4,3,2 of an array in Python, one writes A[3:0:-1], but for the elements 3,2,1 one has to write A[2::-1], because someone decided that making A[-1] refer to the last element of an array is "intuitive", and so A[2:-1:-1] will return empty slice. Now if you want to use array comprehensions, the first case looks similar: [A[k] for k in range(3,0,-1)] but the second now has to be [A[k] for k in range(2,-1,-1)]. Further, for some reason, array and matrix are different types and one has to convert back and forth between them.
- joaogui1 6y ago
  You don't need the matrix type in Numpy, it's just syntax sugar over a 2-dimensional array
- tgb 6y ago
  The matrix class is now officially deprecated, FYI, so just use array everywhere. I guess the generic way to write that would be A[bottom:top+1][::-1]. But the blame there goes to Python, not numpy, since the same is true of lists.

290 points by hardmaru 6y ago

As someone who cut his teeth on bioinformatics before eventually just completing a full computer science degree, I was a bit worried that this article wouldn’t address the applied scientific audience well. Pleasantly surprised, though, at how well this article evangelizes NumPy to that exact community. Many labs are gaining access to or creating physical tools that create data analysis over experimental design problems. Biologists transitioning from running gels to detect the existence of a gene to running sequencing or flow cytometry to segment populations, for example. I remember seeing I think an RNA-seq experiment that tracked the full lineage of hematopoietic (blood) stem cells by a professor - my jaw was on the floor at the level of insight. One remaining step is transitioning many bioinformatics courses from applied tooling to general program design and open-sourcing. I’ve noticed a few labs have done that really well for years, but it is not often found as a field-level skillset.

rubatuga 6y ago
The coolest thing I read last year was about single cell rna-seq trajectory analysis of the differentiation for different blood cells.
- dm319 6y ago
  You might like the mass cytometry reconstruction of the human haematopoietic system [1]. I've done a bit of scRNA-seq and mass cytometry, and the issue I have with scRNA-seq is the tiny dynamic range it has compared to flow/mass cytometry, which can make identifying populations much harder. Not to mention the cost! [1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3273988/figure/F2/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3273988/figure/...

teorema 6y ago

For some reason this struck me as inappropriate for the outlet. It's a nice piece as an introduction to array programming with numpy, but seemed out of place to me.

ddavis 6y ago
If, going forward, 5% of all papers that use NumPy to get their results actually cite this paper, it will be one of Nature's most cited papers every year.
- lumost 6y ago
  There's an interesting trend of what content gets published in peer-reviewed journals vs. blogs/github/etc. I suspect there is an audience segment that strongly values peer reviewed pieces that are equivalent content wise to introductory material in a variety of formats. I wonder if github should add a "Review" feature to provide a similar content authoring experience.
  - clickok 6y ago
    It would be nice if citing repositories were easier-- either for generating a reference for my own code or acknowledging when I've used someone else's code in my research. There's tons of math and physics blogs that contain useful results that the author wanted to make available but didn't manage to incorporate into a paper. I wonder if there'd be any interest in a sort of GitHub for proofs? It could even use git, since (assuming consistency) isn't math just a DAG anyways (and therefore isomorphic to a neural net, as are all things).
    - throwawaygh 6y ago
      Traditionally that sort of stuff goes in tech reports, dissertations, or text books. What's missing is the dissemination piece. Somehow people will absolutely refuse to take seriously the job of citing code they use, even when their main result is obtainable by "and then I ran something from scipy/numpy/pytorch/etc."
    - westurner 6y ago
      You can get a free DOI for and archive a tag of a Git repo with FigShare or Zenodo. If you have repo2docker REES dependency scripts (requirements.txt, environment.yml, postInstall,) in your repo, a BinderHub like https://mybinder.org https://mybinder.org can build and cache a container image and launch a (free) instance in a k8s cloud. Journals haven't yet integrated with BinderHub. Putting the suggested citation and DOI URI/URL in your README and cataloging citations in an e.g. wiki page may increase the crucial frequency of citation. A Linked Data format for presenting well-formed arguments with #StructuredPremises would help to realize the potential of the web as a graph of resources which may satisfy formal inclusion criteria for #LinkedMetaAnalyses.

throrthaway 6y ago

Ah, just the right time to publish about numpy - right when everyone is moving over to Julia because of numpy's warts.

microcow 6y ago
I don't think that's accurate: https://trends.google.com/trends/explore?date=today%205-y&geo=US&q=%2Fm%2F0j3djl7,%2Fm%2F021plb https://trends.google.com/trends/explore?date=today%205-y&ge...
- throrthaway 6y ago
  Relieved to know I can still trust HNers to always take hyperbolic statements literally
  - ALittleLight 6y ago
    Not always.
hyperbovine 6y ago
*For certain definitions of everyone.
- konjin 6y ago
  Me on a Thursday afternoon when I'm bored.
alanbernstein 6y ago
I'm interested in Julia, but if numpy has warts, I suppose Julia has not yet had time to even grow its skin.
- Fishysoup 6y ago
  Theoretically Julia is better for scientific computing, the only issue is its package ecosystem isn't as mature as Python's. But it's growing incredibly fast and there have already been libraries available for a few years that would be really impractical to write and maintain at such a level in C++ for Python. I assume that for science at least Julia will catch on a lot.

westurner 6y ago

Looks like there's a new citation for NumPy in town. "Citing packages in the SciPy ecosystem" lists the existing citations for SciPy, NumPy, scikits, and other -Py things: https://www.scipy.org/citing.html https://www.scipy.org/citing.html ( source: https://github.com/scipy/scipy.org/blob/master/www/citing.rst https://github.com/scipy/scipy.org/blob/master/www/citing.rs... ) A better way to cite requisite software might involve referencing a https://schema.org/SoftwareApplication https://schema.org/SoftwareApplication record in JSON-LD, RDFa, or Microdata; for example: https://news.ycombinator.com/item?id=24489651 https://news.ycombinator.com/item?id=24489651 But there's as of yet no way to publish JSON-LD, RDFa, or Microdata Linked Data from LaTeX with Computer Modern.

jofer 6y ago

Don't underestimate the impact this has on getting funding or even just tenure/etc recognition for working on numpy. I'm in industry these days, but coming from the academic side, it's _really_ hard to get recognized for building the underlying infrastructure that tons of people use. I've built and maintained libraries that are used in a ton of publications, but was always told my work was "utterly and completely useless". It was also always unpublishable, as methods are never publishable in my field. Numpy has (obviously) vastly more respect and impact than my work, but the general problem remains. Articles like this are a _huge_ deal for that reason. It's an immense delayed recognition for over a decade of work from a lot of folks.

teej 6y ago
> my work was "utterly and completely useless I can never understand the arrogance of folks who would say something like this
- austincheney 6y ago
  I used to maintain a code beautifier/diff tool and hear the same things about how the idea was useless only to see those same people shortly there after use my tool or a close competitor. Once you see that pattern a few times you learn to ignore it in it’s entirety. It’s hard to tell why people behave like that. I presume it’s because many people have a great fear of originality and require social validation.
  - mjburgess 6y ago
    Asymmetry between complexity of principle and power. Ie., we are easily persuaded that something very complex will be very powerful (eg., a smart phone) -- but we intuitively regard something simple (eg., a hammer) as under-powered. Hard to say how well this actually holds, but I'd guess in both cases we arent really enumerating use-cases in our head, we're just using explanatory complexity as a guide to practical power. This is probably more extreme in cases where people have a specific notion of complexity in mind, eg., in academic environments where "tool A" is as simple as "tool B" if they use the same theoretical basis. ie., Tool C is worthwhile if it includes a more complex theory, as therefore it is more powerful.

1vuio0pswjnm7 6y ago

BLAS and LAPACK were originally written for FORTRAN, and FORTRAN is particularly suited to numerical computing and array programming. https://modelingguru.nasa.gov/docs/DOC-1762 https://modelingguru.nasa.gov/docs/DOC-1762

6gvONxR4sf7o 6y ago
Aren't BLAS and LAPACK just for the linear algebra bits? Numpy is so much more than linear algebra.
- bagels 6y ago
  Yes, there's a lot more to it than that. For instance, the interface and array/matrix types make vector operations really natural and efficient in python.
pjmlp 6y ago
And quite modern with recent language revisions, including OOP, generics, modules and first class support for C interoperability.
sradman 6y ago
Indeed, FORTRAN has traditionally been used for vector and matrix math and NumPY adds the equivalent functionality to Python. The simple linear algebra abstractions for manipulating these structures are fundamental to each of the popular data science platforms. MATLAB was created as an interface to LINPACK/EISPACK without having to learn FORTRAN. The importance of this comment is the emphasis on the core fundamentals shared by all the data science platforms rather than the different tradeoffs inherent in each ecosystem.

hoytech 6y ago

Let's not forget to give at least some credit to Perl Data Langauge (PDL). It pioneered a lot of these ideas 10 years before NumPy existed, and is still a pretty great tool today: http://pdl.perl.org/index.php?page=FirstSteps http://pdl.perl.org/index.php?page=FirstSteps

ihnorton 6y ago
NumPy’s direct predecessor Numeric was released in 1995.
- beagle3 6y ago
  ... and then in 2001 or so, numarray came about and was supposed to be better - but instead it sort of split the community and wasted resources. Then Travis started Numpy which somehow magically was backward compatible with both numeric and numarray - and managed to get the community united again.
- hoytech 6y ago
  I didn't know that, thanks! It looks like these ideas were being tried in perl and python roughly in parallel.
mjburgess 6y ago
Er... array programming and statistical programming languages pre-date both by decades. APL from 1966, I believe, is the key lang for array programming. And statistical languages like S from 1976 come to mind: https://en.wikipedia.org/wiki/S_(programming_language) https://en.wikipedia.org/wiki/S_(programming_language) At a quick glance, it seems PDL is just a variation on S.
- ktpsns 6y ago
  R (https://en.wikipedia.org/wiki/R_(programming_language) https://en.wikipedia.org/wiki/R_(programming_language) ) is kind of the successof of S. In certain communities (not only statistics, but for instance also biogenetics), there is quite some concurrency between R and (scientific) Python for data science. For my understanding, the numpy syntax most closely resembles what would be possible in matlab. And matlab again seems to have roots from Fortran. Thanks to that, young folks nowadays can switch so easily between Fortran and Numpy, the syntax and call structures can easily be made to almost fit to each other.

enriquto 6y ago

Young programmers are slowly but steadily reaching the power that Fortran programmers had 40 years ago. That's good news!

mumblemumble 6y ago
That statement only feels true to me if you interpret the word "power" as an exact synonym for "performance." Which is a definition that is valid, but also just about perfect for leading someone to miss the point. Numpy approaches the performance you could get with Fortran. Mostly because its core is written in Fortran. What Numpy offers that Fortran never did, though, is leverage. The article mentions, but doesn't really do justice to, the sheer volume of interoperability that Numpy has enabled. It's not just that all these libraries were built on top of Numpy. It's also that their common Numpy substrate makes them all deeply interoperable with each other. And that works both above and below the boundary. You can swap out BLAS and LAPACK for something else - say, CUDA, or a distributed representation - and as long as the replacement also speaks Numpy's language, you can plug it into existing libraries that were originally written against Numpy. In short: Fortran gets you performance. Numpy gets you that, and also productivity. I would argue that that actually makes Numpy more powerful than what was possible with just Fortran.
- teleforce 6y ago
  Try D language, with its numerical library you can get both productivity and performance that is even better than the OpenBLAS (Numpy and Julia library are based on) [1]. In D you get the consistency of a single unified language semantic unlike the impedance mismatched approach that is inherent in Python and Numpy programming combination. [1]http://blog.mir.dlang.io/glas/benchmark/openblas/2016/09/23/glas-gemm-benchmark.html http://blog.mir.dlang.io/glas/benchmark/openblas/2016/09/23/...

Mikhail_K 6y ago

Numpy array syntax inconsistent to the degree it can be considered broken. For instance, to take elements number 4,3,2 of an array in Python, one writes A[3:0:-1], but for the elements 3,2,1 one has to write A[2::-1], because someone decided that making A[-1] refer to the last element of an array is "intuitive", and so A[2:-1:-1] will return empty slice. Now if you want to use array comprehensions, the first case looks similar: [A[k] for k in range(3,0,-1)] but the second now has to be [A[k] for k in range(2,-1,-1)]. Further, for some reason, array and matrix are different types and one has to convert back and forth between them.

joaogui1 6y ago
You don't need the matrix type in Numpy, it's just syntax sugar over a 2-dimensional array
tgb 6y ago
The matrix class is now officially deprecated, FYI, so just use array everywhere. I guess the generic way to write that would be A[bottom:top+1][::-1]. But the blame there goes to Python, not numpy, since the same is true of lists.