10 ms·
NumPy receives first ever funding, thanks to Moore Foundation
- santaclaus 9y agoWasn't Google funding the lead dev on NumPy for a while?
- travisoliphant 9y agoNo. I'm not aware of this happening. NumPy built on Numeric which was primarily written by Jim Hugunin while he was a grad student at MIT. While I was a professor at BYU, I wrote the the core of NumPy with a lot of community input -- outside of my regular job. I sold a book "Guide to NumPy" for a while that I used to replace the grants I should have been writing and pay for a grad student to help write iterative solvers for SciPy. Chuck Harris joined the NumPy effort early and has been steadily contributing ever since without direct funding. Many others have contributed volunteer time since then. A major reason I started Continuum was to help create places where people could get paid to write open-source. I am happy to say we have been doing this for 5 years though mostly outside core NumPy itself (Numba, Dask, Bokeh, conda, etc.) We are working to support many more open source projects more generally -- and our devs have now made additional contributions to NumPy itself. We have a thriving 40 person Community Innovation team at Continuum supporting many open source projects. I expect this funding to help bring more new people to the NumPy development ecosystem. The community also started NumFOCUS at this same time to be a community-run foundation that could be a focal point for donations and support to projects including NumPy. Nathaniel Smith wrote a great proposal and put the effort into securing this funding. It is real work to secure funding. I look forward to NumPy getting better for the benefit of all because of this work.
- metalliqaz 9y agoThey've come a long way without funding. Good for them. Mathworks taking notice, I'm sure.
- thearn4 9y agoI could have sworn that Continuum had gov't funding for numpy development, but maybe that was just for Blaze?
- joejev 9y agohttps://www.continuum.io/blog/news/continuum-wins-darpa-xdata-funding https://www.continuum.io/blog/news/continuum-wins-darpa-xdat... It was blaze and bokeh. The description of blaze given in the article is pretty out of date at this point.
- travisoliphant 9y agoContinuum co-founder here and previous NumPy developer. We received funding for Bokeh and also explorations around scalable "NumPy-like" work. The efforts around Blaze produced Numba, Dask, and Datashape as well as a more limited in scope library called Blaze. While we continue to develop those things, we are working on taking the core ideas in NumPy and creating a set of lower-level C and Python libraries that could be used by pandas, xarray, Numba, dask, arrow, and potentially NumPy: https://github.com/plures https://github.com/plures. This work is not directly related to the NumPy funding.
- thearn4 9y agoThanks for the clarification, Travis, that's pretty exciting to hear.
- in9 9y ago$645020 is good for what? 4 jr developers or 3 slightly experienced developers, working full time on numpy for 2 years?
- oh_sigh 9y agoWhat's your point with this post?
- xj9 9y agofree/open/libre software takes time and skill to create. more companies should probably be paying some kind of support or at least donating since they are no doubt using it to extract some kind of value from the market.
- jacquesm 9y agoWhat if that's what it takes? And besides that, it is $645020 more than they had otherwise, that's serious dough. (Though not serious in terms of the value created with Numpy but that is the nature of open source.)
- deleted 9y ago[deleted]
- 9y ago
- chollida1 9y agoWow, I'm surprised that this is the first funding they've ever got. It wouldn't be a big stretch to say that 90% of quantitative hedge funds use Numpy in some fashion, whether its directly, or via a library that sits on top of it like pandas or tensorflow. I can't think of a more ubiquitous library in the financial space, maybe QuicFix (http://www.quickfixengine.org/ http://www.quickfixengine.org/)... Maybe numpy's problem is visibility? Possibly it does its job so well that people don't know they are using it when they use library libraries like scikit learn and Pandas?
- jacquesm 9y agoNumpy is so 'normal' I think many people believe it is part of Python.
- SwellJoe 9y agoThere's been quite a bit of private funding, through Enthought and other companies.
- travisoliphant 9y agoI don't think this is very true. There hasn't been much private funding for NumPy. People working at private companies have worked sometimes on NumPy but very rarely as part of their full-time job.
- SwellJoe 9y agoOh, hey, Travis! (This is Joe Cooper, I worked at Enthought as a contractor for a few years before you joined the company full-time; we met a few times in Austin.) You would obviously know better than I. I guess I considered employing people who were working on NumPy as "funding", but possibly not on the scale or with the focus of a specific grant. So many of the folks who do scientific computing with Python have gone through Enthought, it seems kinda like everyone has drawn a salary or contract work from there at some point. But, I guess a lot of the work at Enthought was focused on making the tools palatable to industry rather than the actual science side of things, and much of the math they're packaging came from the academic world.
- ykler 9y agoI wonder what they plan to use it for. Numpy kind of seems finished already.
- cing 9y agoThere are about 200 open "Numpy Enhancement Proposal" issues on Github, https://github.com/numpy/numpy/labels/01%20-%20Enhancement https://github.com/numpy/numpy/labels/01%20-%20Enhancement
- ris 9y agoThat's not to say there aren't things NumPy doesn't do that it could. How about lazy evaluation or even just matrix chain multiplication? Either could save a lot of computation.
- VHRanger 9y agoDynamic arrays anyone?
- shoyer 9y agoMatrix chain multiplication was added to NumPy recently, see numpy.linalg.multi_dot: https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.linalg.multi_dot.html https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/...
- ris 9y agoHooray! Still, it's not exactly transparent in a way that you could get with lazy array evaluation.
- cdavid 9y agoMany things could be improved in NumPy: * make it easier to implement and deploy custom dtypes, fix the time-related dtype * support for ragged arrays * consolidate internals, especially around ufuncs I also think some non trivial part of pandas lowest levels belong to NumPy, though I have not thought very deeply about that one: support for missing value, some kind of indexing, etc...
- ahmedfromtunis 9y agoI really wish I could help!
- arikr 9y agoWhat's stopping you?
- rectangletangle 9y agoReally surprised there wasn't already funding for this. Numpy is an amazing library, and it's basically Python's "killer app." The fact that you can seamlessly blend numerical/data science computing with more general web applications is what makes Python great.
- carreau 9y agoJust to note that if you know of anyone who is interested in working on NumPy and potentially to move to UC Berkeley then tell them they probably should contact Nathaniel – if NumPy got funding they'll likely hire developers/community manager/technical writer ... etc . UC BIDS is a fantastic place to work at, and Nathaniel is an extraordinary person to work with. I'm going to assume there is also some opportunity for remote work.
- csaid81 9y agoIt's great that the Moore Foundation provided funding for open source data science tools in Python. Good for them! That being said, I do wonder if numpy is the most appropriate recipient. In my experience with data science, the tool that would benefit the most is not numpy, but pandas. While data scientists rarely use numpy directly, every data scientist I know who uses pandas says they are constantly having to google how to do things due to a somewhat confusing and inconsistent API. I use pandas at work every day and I'm always looking stuff up, particularly when it comes to confusing multi-indexes. In contrast, I rarely use R's dplyr at work, but the API is so natural that I hardly ever need to look things up. I would love if pandas could make a full-throated commitment to a more dplyr-like API. Nothing against pandas -- I know the devs are selflessly working very hard hard. It's just that it seems there is more bang for the buck there.
- carreau 9y agoBut.. pandas uses numpy under the hood. If numpy is better and can offload some of the core functionality from pandas that will also benefit pandas right ?
- csaid81 9y agoRight, but I'm talking about the pandas API. Stuff like how easy it is to remember exactly how to do aggregations, transformations, etc.
- csaid81 9y agoHere's some specific examples: https://twitter.com/Chris_Said/status/715249097326768128 https://twitter.com/Chris_Said/status/715249097326768128 https://twitter.com/Chris_Said/status/861244045535756290 https://twitter.com/Chris_Said/status/861244045535756290 I could be wrong, but I'm pretty sure that these would be solved by pandas API design improvements, not with numpy improvements under the hood. (NB: As always, a big thanks to the developers for all their work.)
- jacquesm 9y ago> every data scientist I know who uses pandas says they are constantly having to google how to do things due to a somewhat confusing and inconsistent API. That's a design error, not necessarily something that money will fix for you. This is why you need to think really long and hard before deploying a public API, it is very hard to change those.
- digitsman 9y agoDoes anyone have a link to the text of the proposal?
- theprop 9y agoCongratulations!! Nice work...looking for lots more math libraries :-D!
- gregjw 9y agoAbout time.
- gigatexal 9y agoImagine if .1% of wall street profits from shops that use numpy were donated to the project. Or some similar scheme for the other OSS projects used for profit by large firms.
- danjoc 9y agoWhy would they do that? Numpy gave it to them. It's theirs now. "Screw you nerds" is their opinion. Don't like that? Then dual license. Copyleft and a commercial license. They can pay, break the law, or go leech off something else.
- Radim 9y agoThe funny thing is, open source projects scramble to offer the most permissive license possible. It's either BSD/MIT or go home. I've seen people shit on projects that use copyleft (GPL, even LGPL, etc). Sounds a little schizophrenic to complain about the (lack of) money at the same time? I personally think this may be connected to the "academic" origins and mindset of much OSS. Everything's "free" in that world, which can make a software's transition into the world with real economic constraints and sustainability challenges somewhat painful.
- ferdterguson 9y agoI understand the GPL, but I personally prefer BSD/MIT and I am disappointed when I come across things I like in the GPL. IMO the GPL is not really 'free'. On the other hand, the GPL wouldn't make a difference here unless they are actually distributing it and my understanding of LGPL means that they could do whatever they wanted to as long as they use NumPy and don't change it.
- TeMPOraL 9y agoI would think that's because most open source projects don't start with money in mind, so their first goal is users. And as a user I can tell you GPL or even dual-licensing frustrates me, because it limits my options. Consider: today I might be working on FOSS. But maybe tomorrow a friend asks me to help him with his (small) business, by e.g. adding a little bit of automation. Suddenly, all my knowledge and experience of GPL-ed libraries goes to waste, as I won't be able to use any of that to help my friend. Given two equivalent libraries, one on GPL and one on MIT, I'll always go for the one on MIT. MIT, BSD, etc. seem to be the libraries that give you most options (I'd say freedom, but that's not how GPL sees freedom) while still maintaining the integrity of the library itself. Those are the licenses that best and at the same time satisfy the needs of developers who are not in it for money, and users who don't want to waste their brain cycles on going through possible legal scenarios around all their actual and imagined use cases.
- visarga 9y agoAre they going to make Numpy work on GPU? There is a library called Cupy (from Chainer) that does that but not quite well enough. In fact on my attempt to swap Numpy with Cupy, my program ran slower. https://github.com/cupy/cupy https://github.com/cupy/cupy
- glup 9y agoIf you are looking for performance optimizations beyond Numpy (GPU or CPU) take a look at Numba by Continuum
- marmaduke 9y agoSlightly off topic but you can use most of the Python stack from MATLAB since version 2014b. The syntax is a little funny but it works well outside of ABI mismatches in shared libs.
- brian_herman 9y agoYou can donate here. http://www.numpy.org/ http://www.numpy.org/ https://www.flipcause.com/secure/cause_pdetails/MzUwMQ== https://www.flipcause.com/secure/cause_pdetails/MzUwMQ==