13 ms·
SciPy builds for Python 3.12 on Windows are a minor miracle
- pklausler 3y agoI'm very glad that f18 worked (so far as we know). I haven't tried building SciPy for any platform yet myself, much less Windows.
- gamache 3y agoOne nit to pick: as far as I am aware, "aarch64" and "arm64" are the same thing. Am I off?
- nerdponx 3y agoWhat I see in Python is that "aarch64" usually refers to Linux and "arm64" usually refers to MacOS ARM. I don't know enough about these things to understand why they have different names.
- mschuster91 3y agoThey are the same thing but there used to be two competing LLVM implementations on the backend side. [1] https://www.phoronix.com/news/MTY5ODk https://www.phoronix.com/news/MTY5ODk
- gamache 3y agoThanks, that is very helpful!
- stefan_ 3y agoObligatory: https://lkml.org/lkml/2012/7/15/133 https://lkml.org/lkml/2012/7/15/133
- pitaj 3y agoI love how he ranted so hard - without even mentioning the embedded or realtime variants.
- drtournier 3y agohttps://xkcd.com/2347/ https://xkcd.com/2347/
- josh-sematic 3y agoAlso https://xkcd.com/1987/ https://xkcd.com/1987/ Edit: not to imply that the work of the maintainers hasn’t been INCREDIBLE (it has). I just thought this XKCD was a funny take on how complex the python packaging ecosystem is.
- cozzyd 3y agothe build system churn in python is really hard to keep up with. I'm also curious about performance numbers on Windows (though, to first order, it doesn't matter... anything serious is probably running on a Linux machine).
- nerdponx 3y agoFortunately it should be slowing down now. The big transition was getting everyone on board with PEP 517, especially converting legacy Setuptools projects.
- lmm 3y agoI've been hearing something like this about Python builds every two or three years for decades. How long until the next "big transition" is "converting legacy PEP 517 projects"?
- nerdponx 3y agoHopefully never.
- lmm 3y agoHopefully, sure. But why should we believe that this time is different?
- oblvious-earth 3y agoThe main difference now is there is an actual standard rather than a de facto tool that becomes the standard. The idea is that different tools may rise and fall in popularity but they should all be following the standard so the compatability breaks should be minimal. Will it work? No idea, but it's the best attempt to make things work well for everyone yet. Python 3.12 might be the biggest churn moment, but there are probably a few more down the road, such as dropping legacy version specifiers.
- 3y ago
- tedunangst 3y ago> In particular, Meson was going to refuse to accept the MSVC+gfortran combination that was in use in conda-forge. This sounds like a bug? The point of a build tool is to run the commands you tell it, not tell you, sorry dave.
- rbancroft 3y agoThe article is well-written and detailed, however I was taken aback by the claim that meson is 'widely used for C & C++ projects'. I've come across bazel more often than meson. I guess because meson is written in python it seemed like a good choice for SciPy, and it worked out in the end so congratulations. But yeah, I think CMake is still the gold standard despite all its quirks, complexities and problems.
- bsder 3y agoMeson is a front end. Edit: Caution: This statement is incorrect--It can generate CMakefiles as a backend. It can also generate build files for MSVC. It can also generate standard Makefiles. Edit: Corrected downstream. CMake can make Ninja files which Meson also makes. I got this backwards but kept the edit so people won't get confused. CMake is NOT a gold standard. CMake is an agglomerative disaster. For example: try getting CMake to accept zig as your compiler. "Oh, your compiler command has a space in it? So sorry. I'm going to put everything after the space in all manner of weird places. Some correct--some broken--some totally random." If you're lucky CMake crashes with an inscrutable message. If you're unlucky, you wind up with a compiler command that fails in bizarre ways and no way to figure out why CMake is doing what it did. This is my experience with CMake every damn time--some absolutely inscrutable bug pops up until I figure out how to route around it. If I'm really unlucky, I have to file a bug report with CMake as I can't route around it. Sure, if some unfortunate soul has beaten CMake into submission and produced a functioning CMakefile, CMake works. If YOU are the poor slob having to create that CMakefile, you are in for worlds and worlds and worlds of pain.
- spookie 3y agoI agree, I stay away from CMake whenever possible. I've tried so much to understand it, and yet, it kicks me around like a pinata
- wenc 3y agoWhat an amazing read. Now I know why my pip installs are failing in 3.12 but we now have a brighter future ahead. Also while I love Python it’s helpful to understand why Python packaging is a (manageable) mess. It’s because of non standardization of build tools for C/C++/fortran and the immensity of the ecosystem, nothing to do with Python itself. It’s part irreducible complexity. It’s a miracle it works at all.
- IshKebab 3y agoIt's also to do with Python itself.
- wenc 3y agoHow?
- Kwpolska 3y agoThe Python packaging world is full of barely compatible tools and no clear vision. Even if you're consuming packages, or packaging pure Python code, it's often an incomprehensible mess.
- xapata 3y agoI haven't found it so. I've stuck with pip and adopted venv when it showed up, and haven't needed anything else. I use Docker for "pinned" builds.
- eviks 3y agovenv and docker are exactly the indicators of how Python is bad
- globular-toast 3y agoAre screws bad because you need a screwdriver?
- bee_rider 3y agoI’m under the impression that the best BLASes are mostly C (MKL, Blis, and OpenBlas). I wonder how far they could get with just C and Python. I wonder if they’d just go with libflame instead, if they started today. Of course there’s lots of other functionality in scipy; iterative stuff, sparse stuff, etc etc, so maybe Fortran is unavoidable (although, Fortran is a great language, I’m glad the tooling situation is at least starting to improve on Windows).
- pharrington 3y agonot gonna lie, seeing "Fortran is a great language, I’m glad the tooling situation is at least starting to improve on Windows" was not on my 2023 bingo card
- bee_rider 3y agoI assume that is because Fortran is such a great language that you just didn’t think it was possible that the tooling situation on Windows could be anything but perfect. :)
- cdavid 3y agoThe question of removing fortran from scipy has happened a few times, and never got anywhere for the reasons you gave. A lot of scipy itself contains fortran code that would take man years to rewrite. Several key parts using fortran have been removed, once it became possible. For example fft stuff.
- d3m0t3p 3y agoIn their table about compilers, isn't AArch64 and ARM64 the same ?
- Grimburger 3y ago> GCC..there are ways to make it work on Windows (through cygwin or MinGW) I assume it would be working on WSL also? Which seems like a simple solution to the mess that is developing on Windows, especially in Python.
- deleted 3y ago[deleted]
- mkoubaa 3y agoNo, Python on windows needs to work natively without wsl.
- Grimburger 3y agoI'm struggling to comprehend your comment here, python needs to work native on windows or is it something else, why is WSL such a barrier given it's generally available on a few billion devices worldwide?
- deleted 3y ago[deleted]
- FrustratedMonky 3y agoMaybe move away from Python altogether?
- deleted 3y ago[deleted]
- mkoubaa 3y agoGreat idea! Why don't you go suggest to the developers of sciPY??
- FrustratedMonky 3y agoThe developers of SciPY are really good, maybe their brains could be focused on developing tools for a different language that doesn't cause articles to be written about how crazy it is to integrate with it. At some point, move on, Python started as a toy, and grew into a behemoth.
- cycomanic 3y agoMaybe instead the solution should be moving away from Windows. Almost all the issues described in the article (and I would argue with most language packaging systems) is due Windows being such a pain to develop on. I'd argue it's going to be easier to convert all scientific users to Linux than rewrite all scientific code in a different language. (no I'm not seriously suggesting all Windows users abandon it, I'm just replying to someone who suggests to ditch the language which has the largest scientific codebase by likely several orders of magnitude)
- bluenose69 3y ago> At some point, move on For some in the scientific community, Julia is seen as a a nice alternative to python. It has some great libraries, and installing them is easy, perhaps because the system was well-designed from the start, and recently enough to build on experience with other systems. Julia is really handy for reproducible work, because you can specify which versions of which libraries you're using, and share that information with others easily so they can retrace your steps. That's a big factor in scientific work. The other thing (returning to the main topic) is that Julia does not have to rely on fortran or C/C++ for low-level work. Julia is very fast all by itself. Indeed, most of the Julia libraries are written in Julia itself, providing users with a tutorial for intensive work. In addition to these things, I have to admit that I love how Julia lets me program in unicode symbols, so my equations in code can look like my equations on paper. There are some downsides to Julia that I ought to point out. 1. The error messages can be very cryptic. 2. The documentation is poor (no worse than Python, I think, but a lot worse than R). 3. The development community is a bit ivory-tower-ish. The usual answer to "could you improve the documentation" is "please submit a PR". 4. There is a learning curve to get used to Julia. It's easier if you are already familiar with Fortran and Matlab. Then again, most scientists I know are familiar with both those languages (and Python, and R, and ...) so learning a new language is not a challenge. Scientists like to learn stuff -- that's the whole point of being in science, for most of us.
- mattbillenstein 3y agoVery naive question, but are the semantics of Fortran so different that it can't be translated to C first and then compiled using a C compiler? Perhaps maintained in C going forward? I can't imagine there are a lot of Fortran folks around maintaining these old libraries - they must need maintenance no?
- googl-free 3y agoyeah, fortran has native arrays
- lldb 3y agoWe have this already - “f2c” has been around for decades.
- ogrisel 3y agoIt's currently used to provide a hackish WASM compatible port of scipy for Pyodide and the Pyodide maintainers are eager to be able to drop the f2c hacks in favor of lfortran or any WASM-capable fotran compiler because f2c can cause very hard to debug low level crashes when running scipy or other downstream libraries tests (e.g. scikit-learn).
- mkoubaa 3y agoFortran is a higher level language than C. There are plenty of fortran developers. It's awful for application development, but that isn't it's niche
- not2b 3y agoIf you want the code to run slower, yes, you could do that. Because there are no pointers in Fortran, only arrays, and because arguments to functions aren't allowed to alias (let's ignore the horror of COMMON blocks for now), aggressive optimization and vectorization is easier. The standard Fortran math libraries just work, and they are fast. I should clarify that you can write C/C++ code that would have equivalent speed, especially with the C restrict keyword, but putting in an f2c step to translate the existing code will making things significantly worse in many cases.
- tsegratis 3y agoThe κατα in καταστροφή means 'down to' or 'according to' rather than sudden, and implies strongly a bad turn The inverse of κατα is often ανα though αναστροφή means literally "up turn" or invert So maybe ευστροφη a "good turn" (eustrophe) would be better coinage But arguing with JRR Tolkien on language coinage and being right would be a ... ευκαταστροφη i.e. good luck! But in general I love their noticing of the overflowing grace -- that's something that gives joy and happiness in the world
- coldtea 3y ago"Kata" in katastrofi actually means "against", so katastrofi is things "turning against".
- richwater 3y agoYet another reason the Python ecosystem and installation is unfixable.
- LispSporks22 3y agoI have thought that everyone was using WSL2 for this kind of thing and calling it a day. What are the reasons for even trying to build a native Windows version?
- RockRobotRock 3y agomsft and nvidia tackling CUDA drivers on WSL2 is a godsend. Especially when you can use Docker Desktop on it. Definitely making the best of a less than ideal situation.
- PLenz 3y agoWay more researchers use windows then you might think. Plus students.
- spookie 3y agoMaybe I'm biased, being in EU and all. But I've seen more Linux than Windows in those environments.
- aulin 3y agoSame, it's either linux or macos in compute intensive research for what I've seen in Europe. I've seen groups working with Windows based data analysis tools but those groups and the ones that need python rarely intersect. WSL2 is a godsend for people forced on Windows by blind IT policies, but fortunately IT doesn't have that kind of control in academia.
- pjmlp 3y agoI can assert that a large majority of people at CERN leave GNU/Linux for servers and use Windows/macOS on their desktops. It was like that 20 years ago, and it looks quite the same when I visit it for Alumni events.
- izacus 3y agoWhat you've "seen" has nothing to do with what's actually popular. Why does the concept of observer bias have to be explained on this site?
- aj7 3y agoDoes anyone else find the handling of arrays by Python so horrific that they can’t bring themselves to use it?
- kstrauser 3y agoIn what way?
- __MatrixMan__ 3y agoDo you mean lists, or is there some array type that I'm not thinking of?
- hurryer 3y agoThere is the array module, but it's cumbersome to use. https://docs.python.org/3/library/array.html https://docs.python.org/3/library/array.html But unlikely this is what parent meant.
- kevin_thibedeau 3y agoProbably meant ndarray.
- ketozhang 3y agoIt's not popular because you're mostly hearing from the science community who want more features in their array (vector/matrix/tensors). Why would you want to use C-like arrays in Python anyways?
- avidphantasm 3y agoGreat read. After spending a lot of time this year modernizing a CMake C++ project with Python bindings, which I successfully added to conda-forge as a new feedstock, I can say with great confidence that the first IT-related change I would make as God Emperor would be to extirpate Windows from all Universes for Eternity.
- panzi 3y agoAbout the compiler/architecture table: Whats the difference between arm64 and aarch64? The Wikipedia article to ARM64 redirects to AArch64.
- stingraycharles 3y agoI was thinking the same thing. Maybe they meant the armhfp architecture?
- pietroppeter 3y ago> > While Fortran has long been the butt of the joke in IT departments the world over, in a curious twist of fate, it has seen a dramatic resurgence over the last few years. While the reasons for this are not exactly obvious ... > It is obvious to me. :) https://x.com/OndrejCertik/status/1722364038212899274?s=20 https://x.com/OndrejCertik/status/1722364038212899274?s=20 resurrecting fortran: https://ondrejcertik.com/blog/2021/03/resurrecting-fortran/ https://ondrejcertik.com/blog/2021/03/resurrecting-fortran/
- bjornasm 3y agoThis was a great read, thx :) Of all the other languages than Python, I am the most drawn to Fortran. I used it for some university project and it was quite nice. Hope I'll find a good excuse to pick it up for some project.
- kragen 3y agomaybe the free software community should stop subsidizing microsoft's operating systems and let them port things like scipy to it themselves after all, if you want linux, you know where to find it; it's right there in wsl2 also microsoft could start shipping a fucking compiler with their sorry malware-ridden excuse for an operating system, like every single other operating system vendor has for sixty years
- xcv123 3y ago> microsoft could start shipping a fucking compiler Visual Studio is a free download. Most users are not developers. Waste of space to include it by default. https://visualstudio.microsoft.com/vs/community/ https://visualstudio.microsoft.com/vs/community/
- kragen 3y agothat's a ridiculous excuse unix v7 included a compiler and was three megabytes https://www.tuhs.org/Archive/Distributions/Research/Keith_Bostic_v7/ https://www.tuhs.org/Archive/Distributions/Research/Keith_Bo... https://opensimh.org/research-unix-7-pdp11-45-v2.0.pdf https://opensimh.org/research-unix-7-pdp11-45-v2.0.pdf the compiler was a tiny fraction of that gcc 9 is about 50 megabytes windows 11 is over 8 gigabytes; you need a 16 gig usb drive to install it there are probably individual audio files included in windows that are bigger than gcc also, tho, this is a lot like not including life jackets on a ship because most passengers don't get shipwrecked
- warangal 3y agoIndeed it was a great read about what goes behind the scenes for maintaining such packages and a problem generally every developer thinks about when such an extension has to be shipped along . I still wonder if targeting Python's LIMITED C API wouldn't help in this case. I use a tool (for Nim) which seems to target that Limited API and solves the problem of specific python version mismatching at-least for my specific code. I never had to upgrade a pure C/Nim extension due to python version ! Using zig-cc along with a fixed GLIBC (2.27) and generic CPU flag also made it possible to target linux ecosystem in case user is not a developer and just use the compiled extension shipped with the package.
- eviks 3y agoHeroes and miracles are indeed required in the swamps of bad designs
- vintagedave 3y agoAn amazing read. Most of the article notes that there was no Fortran compiler yet (though amazing how well the one they landed on worked -- a miracle alright!) but I was particularly struck by the ABI issues. The story of a lone developer hacking a MinGW build that used the right ABI struck home. I can provide some background on the ABI issues in Windows. The following is personal opinion not company opinion, and I am a product manager not an engineer so may have some technical details wrong. But: At Embarcadero we're moving our C++Builder toolchain forward to a new Clang[1], and using a new C and C++ RTL layer [also see 1]. Like SciPy, we're now using UCRT, and the key bits that cause difficulty are the C++ RTL and platform ABI. Boy howdy do we have some stories. Issues: * There is no standard Windows ABI beyond what msvc produces. This results in WinAPI (C-level, in other words!) APIs that could not in the past be reliably used from anything other than msvc because they could throw exceptions, rather than returning error codes, and Win64 SEH is not fully documented (clang-cl, which is open source, disappears into the closed source msvcrt to handle this.) * Or another issue: cdecl isn't standardised. This may surprise those of you who think -- as I used to think! -- that cdecl is simple and a known calling convention for any platform. We have issues where sret for return values[2] can be different between our toolchain and something built with MSVC. Since we need to change multiple languages, and one of our languages (Delphi) handles some returns (managed types) differently, changing this is more complex than it looks. We already did a lot of ABI compatibility work several years ago between multiple languages[3]. Back to the ABI: our new Clang is aiming at being fairly compatible with mingw-llvm on Windows. (And MSVC too, but we're starting with mingw-llvm as the basis.) That does not include a C++ ABI which the C++ committee has been resistant to, but it's a known good, working C++ toolchain, open source. If Windows ever did have a platform ABI, it would likely be based on MSVC, but I would suggest that we -- developers -- should resist that until or unless the entire toolchain including runtime internals that affect the ABI is open sourced, or at least documented so that other toolchains can match it. [1] https://blogs.embarcadero.com/win64-clang-toolchains-in-rad-studio-12/ https://blogs.embarcadero.com/win64-clang-toolchains-in-rad-... [2] sret is a hidden (?) or special return value used for structs, basically when a large (> register size) memory is required. I think. This is where I hope I don't embarass myself too much in the explanation. See eg https://stackoverflow.com/questions/66894013/when-calling-ffi-c-functions-from-llvm-code-when-is-an-sret-return-pointer-ne https://stackoverflow.com/questions/66894013/when-calling-ff... In other words, returning values is more complex than it seems, even for a plain simple C method returning a struct which should be _incredibly basic_ and becomes a complex ABI issue. [3] https://blogs.embarcadero.com/abi-changes-in-rad-studio-10-3/ https://blogs.embarcadero.com/abi-changes-in-rad-studio-10-3...
- bjackman 3y agoBack when Linux was a janky corner case run by loosely-coordinated hackers with sometimes-impractical ideological constraints, when brilliant people were expending heaps of effort to enable support for it, that was fantastic. Now that the janky corner case is a proprietary system run by cyber-landlords whose constraints are just... hostility, I feel less positive about the work being done to support it. On the one hand it's really great that these people care so deeply about making these tools usable for everyone. And I applaud them for doing it, absolutely not suggesting they should change course, just pondering. But... now instead of thinking "wow I'm so glad this work is happening" I do rather think "wow imagine what those wonderful people could achieve if they didn't have to work on this".
- SiempreViernes 3y agoWell, "they don't "have" to work on it, as was pointed out several times the SciPy devs are volunteers. Indeed most of the story is explaining why SciPy just had to hope someone else would make a open source Fortran compiler for windows, and it looks like it was mainly NVidia devs that provided salvation
- henrydark 3y agoI must go on compiling / You can't break that which isn't yours / I must go on packaging / I'm not my own, it's not my choice
- crabbone 3y agoIt's not even that. SciPy is paying the price of the idiotic (and highly biased) decisions of Python core-dev members who chose MSVC over MinGW for Python on Windows. (And their motivation is derived from Microsoft sponsorship. A bunch of people on core-dev list are straight-up Microsoft employees, who are paid by Microsoft to be on that list, and, on top of that, Microsoft pays for CI servers for CPython project). This whole problem could've been avoided if Python just didn't use proprietary tools in its toolchain.
- IcyWindows 3y ago
- Decabytes 3y agoSometimes I wonder if Rust’s biggest draw is not that is is safe, but that is removed a lot of the BS hoops you have to jump through to get a working program on your computer. I wonder if the Python alternatives will also get a similar boost for that reason. If maintaining the language and the ecosystem is that much of a bear, it would save a lot of human hours to do that. Even if we started back from scratch for a bit
- ngcc_hk 3y agoJust go to read: https://www.bitecode.dev/p/relieving-your-python-packaging-pain https://www.bitecode.dev/p/relieving-your-python-packaging-p... TL; DR The reason why we need virtual this and that in python is that some work in one and some work in the others. This is particularly problematic in macOS where sometimes the system fall back to the default (somewhat default) python compiler. I even have script to check version and lots of environment for running different program. It is a mess. After this magic, is there a new packaging environment, procedure, ... for the rest of us?
- crabbone 3y agoI just want to comment on this: > Meson was going to refuse to accept the MSVC+gfortran combination Back in the days, I went to Python's core-dev list and asked why. Why would any sane person ever use MSVC for a cross-platform language runtime. And guess what the answer was? Well... The answer was "Microsoft pays us, gives us servers to run CI on, and that's why we will use Microsoft's tools, goodby!" For reference, Ruby uses GCC for the same purpose as do plenty of other similar languages for this exact reason. To give you some context, I ran into this problem when writing bindings to kubectl. For those of you that don't know, in order to interface with Python from Go, one needs CGO, and on MS Windows it means MinGW. You could, in principle, build Python itself with GCC (a.k.a. MinGW) (and that's what MSYS2 a.k.a Cygwin a.k.a. Gitbash does), but this means no ABI compatibility with the garbage distributed from python.org. So, after I had a proof of concept bindings to kubectl working on Linux, I learned that there will be no way (well, no reasonably simple way) to get that working on Windows. So, the project died. (Btw, there still isn't a good Kubernetes client in Python). --- On the subject of packaging. I've decided to write my own Wheel packager. Just as a way to learn Ada. This made me read through the "spec" of this format while paying a lot more attention that I ever needed before. And what a dumpster fire this format is... It's insane that this atrocity is used by millions, and so much of critical infrastructure relies on this insanity to function. It's very sad that these things are only ever discussed by a very small, very biased, and not very smart group of people. But then their decisions affect so many w/o even the baseline knowledge of the decisions made by those few. I feel like Python users should be picking up pitchforks and torches and marching on PyPA (home-)offices and demand change. Alas, most those adversely affected by their work have no idea PyPA exists, forget the details of their work.
- mxz3000 3y agohindsight is 20/20. Don't criticise people for making certain decisions years ago when those don't match what you'd choose to do now. Often you'll find that they were very reasonable given the constraints at the time. Also the spec will have evolved over time with changes that would have been made under constraint of the existing system, which tends to produce things that are not as nice compared to something that was designed from the get-go to support the features. This is something that's seen very often in software engineering, and are probably partly a reason why long-lived codebases tend to be dumpster fires in general. Calling them 'very biased and not very smart' is not very constructive. That's not to say that the wheel format isn't a dumpster fire (I'll have to take your word on that), or hasn't morphed into one with time & revisions.
- hazrmard 3y agohttps://archive.is/SyeRt https://archive.is/SyeRt Archive link. Website doesn't load for me.
- carapace 3y agoHaving finally gotten over the whole 2->3 fiasco I'm still sitting out new Python development until the PyPA sorts out the new packaging & distribution story. They're working on it and making progress, but it's still pretty gnarly. Anyway, the complexity is too damn high!
- swingingFlyFish 3y agoThis is a really insightful article. Thank you for sharing it. it reminds me there is still so much for to be done in the FOSS community and how grateful we are to have it. Thank you!