17 ms·
SciPy 1.0: fundamental algorithms for scientific computing in Python
- dragonsh 7y ago> Our policy for evolving the API over time is that new functionality can be added, while removing or changing existing functionality can only be done if the benefits exceed the (often significant) costs to users and only after giving clear deprecation warnings to those users for at least one year. In general, we encourage changes that improve clarity in the API of the library but strongly discourage breaking backward compatibility, given our position near the base of the scientific Python computing stack. This is one of the nuggets out of it. I have been using Python with scipy and more extensively with it's sci-kit library since 2014. Started with it by solving discrete traveling sales man problem for solution of sitting arrangements and was astonished with its ease of use of matrix algebra with these libraries and got hooked to it. It helped me to do scientific computing and create a user friendly user interface on web to help end-user to just focus on siting arrangement and conference Organization. Can solve it with Matlab or GNU octave, but couldn’t have developed easily the whole system with user interface in them. If Swift wanted to be used in scientific computing it needs to follow the similar policy of managing its scientific stack. Julia is good for scientific computing, but it’s UI and web development libraries are isn’t that powerful. So will see how it develops. I constantly see on HN criticism of Python and something rewritten in Rust as something revolutionary or ground breaking that it will change the whole world. Earlier this fad was with go language. Probably in time to come Rust will be used for some core library development which brings substantial benefits like scipy, not something rewritten as wrapper around C library with unsafe code which can be done with Python much better.
- pmarin 7y agoHacker News always has been driven by fashion and novelty. It's a news site after all. The main innovation of recent programming languages is the use of social media for marketing and PR by their companies to promote them.
- jolux 7y ago> The main innovation of recent programming languages is the use of social media for marketing and PR by their companies to promote them. For some languages I think yes, others no. Rust in particular has a memory management philosophy which was basically novel when it came out; it existed in research papers and such but not very many people were using those languages to do mainstream software engineering. It’s a big deal.
- pmarin 7y agoIt is not a big deal, It is good marketing.
- leksak 7y agoI follow a lot of Rust programmers on Twitter. Here's my understanding, they are busy writing Rust code. Their tweets are about the code. Their bio points out that they are fortunate to write Rust for a living. They focus on how it was _done_. Meanwhile, the other programmers I follow share what they've built as in what their code _does_ and why that makes them giddy. (C#, Python, JS,...)
- pjmlp 7y agoJust like C++ conferences vs other languages. :) In C++ conferences, the talks are about how to do data structure X in C++, on to do certain algorithms in C++, build systems, meta-programming tricks, and so forth. In other languages, the talks are about how product X was built in language Y.
- staticassertion 7y agoI worry that I may regret asking, but what qualifies as a big deal?
- pmarin 7y agoIt's a subjective matter of course but for example a feature that Go and Rust have is type checking across module boundaries which is a big deal for me but it's not something very interesting to make marketing with.
- deleted 7y ago[deleted]
- gigatexal 7y agoThat discrete traveling salesman problem of seating arrangements ... how did you model that? I have a similar problem with organizing committees where committees have members that are in other committees and so committees with overlapping members can’t meet together etc etc.
- bhl 7y agoNot exactly TSP, but something similar on solving seating arrangements: it’s with simulated annealing; it might help with formulating your optimization problem. https://news.ycombinator.com/item?id=16548012 https://news.ycombinator.com/item?id=16548012
- gigatexal 7y agoDanke!!!
- ylk 7y agoI think you could also see this as a graph coloring problem https://en.m.wikipedia.org/wiki/Graph_coloring#Scheduling https://en.m.wikipedia.org/wiki/Graph_coloring#Scheduling Edit: to elaborate, each committee would be a vertex and a committee-vertex is connected to another vertex if one or more of their members overlap. Once you’ve got such a graph you simply have to a assign a color (or number/letter/anything) to every vertex in the graph, with the requirement that no neighboring (as defined by being connected by an edge) vertexes are allowed to have the same color. You also try to use the minimum amount of colors needed. Then you can safely schedule the committees with the same colors for the same timeslots, and be sure that everybody will be able to attend.
- gigatexal 7y agoThank you! I can run with this.
- DataDaoDe 7y ago> Probably in time to come Rust will be used for some core library development which brings substantial benefits like scipy, not something rewritten as wrapper around C library with unsafe code which can be done with Python much better. I think a nice situation would be to have all the core scientific libraries and tools written in rust with clean APIs, independent of any particular scripting language but with the clear goal of providing a clean API to wrapper languages. Then any scripting language can wrap these libs and go - essentially something like a universal numpy/scipy/scikit, etc. This would mean Ruby/Python/JavaScript/Julia, etc. would all be able to get to state of the art immediately while minimizing the duplication of effort across languages and creating a standard "ML" or "Scientific" API.
- woodson 7y agoYou mean like currently BLAS/LAPACK? Written in FORTRAN, wrapped by any scripting language, e.g. Python's NumPy/SciPy, Julia's LinearAlgebra (part of stdlib, https://docs.julialang.org/en/v1/stdlib/LinearAlgebra/ https://docs.julialang.org/en/v1/stdlib/LinearAlgebra/), R, Octave, Javascript (BLASjs etc.). That's the standard API for now.
- xiaodai 7y agoI recall a couple of efforts to write a BLAS in pure Julia. https://discourse.julialang.org/t/we-can-write-an-optimized-blas-library-in-pure-julia-please-skip-op-and-jump-to-post-4/11634 https://discourse.julialang.org/t/we-can-write-an-optimized-...
- mxcrossb 7y ago> Can solve it with Matlab or GNU octave, but couldn’t have developed easily the whole system with user interface in them. This is a very key point. I still find that Matlab often has a larger set of features and better efficiency for linear algebra. But thanks to Scipy, you really can expand your mind about what kinds of systems to interface with.
- enriquto 7y agoGNU Octave is indeed faster than numpy. This may be because GNU Octave uses highly optimized solvers (like the famous modified Cholesky method). Due to scipy's license fundamentalism, they refuse to use many of these solvers because they are GPL-licensed. It is a bit sad to see how these ridiculous ideological constraints taint an otherwise excellent software package.
- beojan 7y agoMuch as I'm personally very pro-GPL, I think there are good reasons to avoid licensing fundamental parts of the SciPy stack under a copyleft license, since anything built on top would also need to be GPLed. A scipy-copyleft for such GPLed components would be nice though.
- Dowwie 7y agoRust isn't going to replace Python because it isn't an interactive language that can be used for adhoc analysis, but it will be used to write the libraries that python imports when the existing ones are no longer sufficient for people's needs. Python isn't criticized as much as it is recognized as the lesser tool for certain categories of development. That's fine because it is a general purpose language with many great uses.
- YawningAngel 7y agoWhat do you mean? There's an RFC for a proper REPL in Rust and a bunch of good-enough hacks in the meanwhile, there just isn't that much demand for one. You can absolutely write ad-hoc scripts in Rust if you want to
- Dowwie 7y agoHobby project at best-- doing it because one can rather than because one should. That in itself is worthwhile to those involved, yet it doesn't mean it will disrupt Python. You ask me as if I don't know what I'm talking about-- as if I am not well acquainted with the rust-analyzer grinding away compilation in the background, slowing down the machine and interrupting workflow? Rust's Achilles heel is compilation. It is a hard enough problem to solve that it won't ever reach a level that it can successfully run in the background of a REPL workflow and offer a user experience comparable to that of Python. The user experience is the reason that Rust will not ever present a viable alternative. Just as with Python, this is also fine. Rust isn't the right tool for every job.
- mark_l_watson 7y agoI have a question for you or anyone else who might know the answer: both Haskell and Rust have long compilation times for large systems, so why is the Haskell repl so fast and agile for interactive development? Is it just because Haskell has been around for so long that partial compilation is more refined?
- deleted 7y ago
- pjmlp 7y agoFor Swift to be taken seriously, it needs to be actually usable across macOS, GNU/Linux and Windows, and not be something that still resembles the heyday of GNUStep experience.
- warmvanilla 7y agoVery cool. So what are your go-to tools for putting a front end on top of your models?
- dragonsh 7y agoFor the seating arrangement problem in conference everything is in Python and used flask with simple bootstrap and jinja templates.
- Feuilles_Mortes 7y agoQt is cross-platform, very stable, and has sane Python bindings. No other reasonable choice (imo).
- gumby 7y agoI agree this is a great philosophy. I have to point out as well, though, that one of the major objections to C++ is that it is "too big" and has too many "dangerous" constructs, both of which are a consequence of this approach. As a developer using the language this is great: I can write using clean modern approaches* and still use older packages. (Making this all continue to work is a burden on the implementation developers of course.) * not to argue which modern features are clean or not -- this is about SciPy.
- nrclark 7y agoIsn't Scipy on 1.4.1?
- kingbirdy 7y agoThe article appears to have been written in 2018, and just published recently. From the Discussion section: > SciPy has a strong developer community and a massive user base. GitHub traffic metrics report roughly 20,000 unique visitors to the source website between 14 May 2018 and 27 May 2018 (near the time of writing) SciPy 1.0 was released towards the end of 2017, so that timing makes sense for a 1.0 retrospective. Not sure why it took so long to get published.
- Areading314 7y agoPeer review takes a long time. There is often back and forth among reviewers, who are volunteers and not beholden to deadlines.
- Fomite 7y agoAlso it takes a long time to get from acceptance to "in print", especially for something that, with all due respect to the SciPy team (love you guys...), isn't time urgent.
- lrem 7y agoI have on my desk a 2017 best paper award, for a paper printed in 2016, submitted 2015, being an extended version of a 2013 paper for a problem we solved in 2012. Academic publishing is like that.
- MrEldritch 7y agoThis is a large part of why (at least in the deep learning field, which I'm more familiar with) scientific publishing takes place more or less exclusively through ArXiv, because the field has been moving fast enough that while peer-reviewed papers are also published, by the time they come out they're long obsolete. (Largely forgoing peer review as a scientific field is, however, not without problems. To say the least.)
- ddavis 7y agoCongratulations to all involved with this paper, an awesome accomplishment for the SciPy community-- pretty wild that it's almost 20 years old.
- timkam 7y agoGood that they got a paper in Nature. It's important such accomplishments get rewarded and acknowledged in the traditional academic way. I hope these people pile up citations. The engineers who build the foundations of modern scientific research get far less credit than they deserve.
- enriquto 7y agoIndeed! This publication is very good news, and I am happy that free software acquires such a prominent place in the academic citation space. On the other hand, notice that scipy is in large part an interface. The underlying solvers (e.g., numpy.linalg.solve) are often routines written in Fortran several decades ago, and already duly cited and academically recognized.
- mattip 7y agoThere is tremendous value that often goes unrecognised in taking best of brand routines and packaging them so real people can use them, consistently and reliably over decades.
- enriquto 7y agoSure, that's why people keep doing that: taking the same exact Fortran routine and giving the world access to it from whatever different languages and frameworks are fashionable.
- jononor 7y agoI'm looking forward to the second-order effects of open source scientific software being publishable in "high-impact" publications. It makes it much more attractive for scientists to use and develop open source, benefiting everyone.
- neonate 7y agohttps://web.archive.org/web/20200207035109/https://www.nature.com/articles/s41592-019-0686-2 https://web.archive.org/web/20200207035109/https://www.natur...
- tobmlt 7y agoSciPy, and especially numpy: the “free drugs”of the scientific computing world. In the best possible way.
- enriquto 7y ago> (...) In the best possible way. I do not see it that way. A whole generation of students is growing with the wrong impression that writing a "for" loop is inevitably inefficient. Also, they believe that it is OK for large arrays of numbers not to be a core language construct, requiring an external library like numpy. These two incorrect beliefs are incredibly damaging in my view. In a sane programming environment (e.g., with jit compilation), writing a matrix product using three loops should be just as efficient than calling a matrix product routine. The matrix product should also be rightly available without need to "import" anything.
- chpatrick 7y agoI don't think that's currently true in almost any mainstream language though. Even in C++, it's faster to use an optimized BLAS library working on a large set of data than your own for loop.
- enriquto 7y agoOf course using an optimized blas is somewhat faster, but it is not three orders of magnitude faster!
- kmbriedis 7y agoBut it can be 20x
- Sean1708 7y agoI actually did some benchmarking on this once upon a time[1] and it turns out that it's actually really easy to write something that is only ~5x slower[2] (until you get to matrices that don't fit in RAM): (4, 4, 4) (5, 5, 5) (32, 32, 32) (33, 33, 33) (256, 256, 256) (257, 257, 257) (512, 512, 512) (513, 513, 513) (1024, 1024, 1024) (1025, 1025, 1025) –––––––– ––––––––– ––––––––– –––––––––––– –––––––––––– ––––––––––––––– ––––––––––––––– ––––––––––––––– ––––––––––––––– –––––––––––––––––– –––––––––––––––––– :naive 0.0 0.0 1.3e-5 2.0e-5 0.0114 0.0133 0.0942 0.106 3.25 2.39 :tiled 0.0 0.0 2.7e-5 2.2e-5 0.0139 0.0121 0.154 0.101 1.25 0.888 :fastf77 0.0 0.0 8.0e-6 8.5e-6 0.00543 0.00563 0.0426 0.0445 0.437 0.448 :blas 4.5e-6 4.0e-6 1.9e-5 2.1e-5 0.000972 0.00109 0.00712 0.00744 0.0582 0.0607 (Units are seconds per multiplication.) Obviously OpenBLAS is so easy to package that it's not really worth avoiding it, but it was very eye-opening to see just how easy it is to get within an order of magnitude (easier, in fact, than getting into the 10x-20x range). [1]: https://gist.github.com/Sean1708/69c5694048e9a9ca7bd84fcbc9ea6f45 https://gist.github.com/Sean1708/69c5694048e9a9ca7bd84fcbc9e... [2]: 8-core 3.4GHz Haswell i7 with 32kB L1, 256kB L2, 8MB L3, and 8GB RAM.
- stelfer 7y agoI remember going to the first SciPy conference at Caltech in 2002. IIRC Most of the talk was about needing better package management. What happened since is just amazing.
- xianwen 7y agoMay I ask what the status of parallel computation is with SciPy?
- fock 7y agoOpenMP threaded blas-routines
- mud_dauber 7y agoI learned about SciPy via Numerical Python (Apress/GitHub: https://github.com/apress/numerical-python https://github.com/apress/numerical-python). That was a ton of fun, especially after I recovered from freshman calculus flashbacks.
- maxnoe 7y agoShameless plug: Here is a grand tour of scipy as a jupyternotebook me and a colleague used as intro to scipy for PhDs in astrophysics: https://github.com/maxnoe/scientific_python_notebooks https://github.com/maxnoe/scientific_python_notebooks You can try it without installing the software by clicking on the binder button.
- trentnelson 7y agoJust wanted to say I really like and appreciate this, great work!
- complex_pi 7y agoThere is also https://scipy-lectures.org/ https://scipy-lectures.org/ :-)
- valgor 7y agoGreat read! I'm ignorant of numerical computing, so I have to ask: is Fortran still used because it is that much faster than C or any other specialized language? I assume that is the case because I'm sure someone would have rewritten those routines by now otherwise.
- coliveira 7y agoFortran still is the language for numerical computing, most algorithms in this area have been implemented in Fortran. And this is very specialized code that has been test by experts in their areas. That's why a project like SciPy cannot simply replace it with something written in C or Python.