7 ms·

What an amazing read. Now I know why my pip installs are failing in 3.12 but we now have a brighter future ahead. Also while I love Python it’s helpful to unde

• IshKebab 3y ago
  It's also to do with Python itself.
  • wenc 3y ago
    How?
    • Kwpolska 3y ago
      The Python packaging world is full of barely compatible tools and no clear vision. Even if you're consuming packages, or packaging pure Python code, it's often an incomprehensible mess.
      • xapata 3y ago
        I haven't found it so. I've stuck with pip and adopted venv when it showed up, and haven't needed anything else. I use Docker for "pinned" builds.
        • eviks 3y ago
          venv and docker are exactly the indicators of how Python is bad
          • globular-toast 3y ago
            Are screws bad because you need a screwdriver?
            • eviks 3y ago
              You're mistaking hammer for a screwdriver
            • IshKebab 3y ago
              That's a completely nonsensical analogy. Maybe you missed his point, but well designed programming language infrastructure does not need Docker or venv to work. The fact that you have to resort to the massive hack of Docker shows how bad the situation is. I do not have to use Docker or a venv for my Rust, Go or Deno builds.
              • xapata 3y ago
                I'm not a Rustacian, but so far I much prefer Python's packaging to Go's. Italicize all you like. I don't find the emphasis convincing. When I receive a Python program that I'd like to modify, I can. When I receive a Go program that I'd like to modify, I must beg for the source code. Do you "vendor" your database into your Go program? If not, you likely still need Docker, or something like it, for your program to work.
              • globular-toast 3y ago
                No, but you have to use Rust, Go or Deno.
                • IshKebab 3y ago
                  I get to use Rust, Go or Deno (for my own projects). I am forced to use Python for work unfortunately. It's not a terrible language for sure. It's just the packaging systems and tooling around it that are face-palmingly awful.
                  • globular-toast 3y ago
      • delfinom 3y ago
        Well, part of it is really Python's age and legacy. We are talking about going back to 1996. So much of python's development was ducktape through history in response to the changing world and whims of the contributors. I'm not saying it's an excuse but it's just how it got to where it was. Newer languages have alot of lessons learnt to build upon to be decent from day 0.
        • ecshafer 3y ago
          Java and ruby are similar ages as python and dependencies are much better stories there.
          • fbdab103 3y ago
            Ruby never had nearly the FFI/other language problem as Python so could almost entirely focus on Ruby-code delivery.
            • cesarb 3y ago
              The same for Java, since its ecosystem has an allergy to calling native (non-JVM) code, to the point of rewriting perfectly good libraries in Java. When they do call native code, it's often in horrible ways (like copying a native library from within the JAR file to a temporary directory, and loading it from there, the JAR file coming with pre-compiled native libraries for all possible operating system and architecture combinations). So the Java package managers mostly focus on Java (and other JVM languages) code building and delivery.
              • pjmlp 3y ago
                Maven and Gradle support building C and C++ libraries just fine.
        • IshKebab 3y ago
          While this may be true, they've also had literal decades to improve the situation and have barely got anywhere. In some ways they've gone backwards!
    • IshKebab 3y ago
      I'm assuming you're not a Python user otherwise you'd already know the many answers! This link might give you a taste: https://packaging.python.org/en/latest/key_projects/ https://packaging.python.org/en/latest/key_projects/
      • selcuka 3y ago
        I am a Python user, but never heard of most of the tools in that list. This is probably because everyone and their cousin attempts to write yet another package manager for Python. The built-in tools venv, pip, (together with requirements.txt and constraints.txt) meet 99% of real life dependency management needs.
        • starlevel003 3y ago
          The proliferation of requirements.txt is one of the key reasons why Python packaging sucks so much.
          • kelipso 3y ago
            Right what we need is a requirements.yaml (better yet, create an entirely new markup language for this particular project) and another new package manager for it. One day (one day!) I will start a project without python. One can hope.
            • starlevel003 3y ago
              > Right what we need is a requirements.yaml It already exists, it's called pyproject.toml. It already existed for years in the form of setup.py. Requirements.txt means that projects can't be automatically installed which contributes massively to the difficulty of getting packages to work.
              • mkesper 3y ago
                Pyproject.toml is a right step afaict but man is it complicated: "Please note that some of these configurations are deprecated, obsolete or at least discouraged, but they are made available to ensure portability." Core vs setuptools-specific etc. See https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html https://setuptools.pypa.io/en/latest/userguide/pyproject_con... and https://packaging.python.org/en/latest/specifications/declaring-project-metadata/ https://packaging.python.org/en/latest/specifications/declar...
    • oconnor663 3y ago
      This pair of articles is excellent, both for good practical advice about installing Python packages, and for its general attitude about how to teach difficult things to large groups of people: https://www.bitecode.dev/p/relieving-your-python-packaging-pain https://www.bitecode.dev/p/relieving-your-python-packaging-p... https://www.bitecode.dev/p/why-not-tell-people-to-simply-use https://www.bitecode.dev/p/why-not-tell-people-to-simply-use Every single decision point or edge case represents permanent failure for hudreds of people and intense frustration for thousands. Of course, none of this is really to do with Python the language. It's more about the wide userbase, large set of packages and use cases, and overlapping generations of legacy tools. But most of it isn't C/C++/Fortran's fault either.
• cdavid 3y ago
  Yes, that's a fundamental reason python packaging is a mess. Python success is largely due to the availability of key mixed language packages. No other mainstream language package manager has to deal with this. For example, cargo for rust, which is great, can assume to package mostly rust-only code. And while it is compiled, the language "owns" the compiler, which means building from sources as distribution strategy works. I don't know how/if cargo can deal with e.g. fortran out of the box, but I doubt cargo on windows would work well if top cargo packages required fortran code. The single biggest improvement for python ecosystem was the standardisation of a binary package format, wheel. It is only then that the whole scientific python ecosystem started to thrive on windows. But binary compatibility is a huge PITA, especially across languages and CPUs.
  • NewJazz 3y ago
    Doesn't R handle fortran and C++ packages?
    • azalemeth 3y ago
      It does -- and to a much lesser extent shims exist for other languages as well. R also has imperfect packaging but I think it's handled well albeit with a level of complexity that also goes up a lot unexpectedly at times. For a truly great experience, call python packages within R in a custom conda environment in order to get data out of pandas in a particularly unholy way...
      • disgruntledphd2 3y ago
        R has always shipped binary packages on Windows and Mac to avoid lots of the pain we see in Python. Also, all packages must build with the latest version of R, or they are removed from CRAN. This makes the dep problems a lot less severe than we see with Python.
  • winstonewert 3y ago
    Many rust crates actually do package code written in other languages. There are plenty of useful C/C++/Fortran libraries that nobody has rewritten into Rust but for which wrappers have been created that call into C. It works in Rust because the build.rs lets libraries do whatever they want during the build process, including invoking compilers for other languages. Various factors still make the Rust and Python story different (Python uses more mixed-language packages, the Rust demographic is more technically advanced, etc). But a big one is that in Rust, the FFI is defined in Rust. Recompiling just the Rust code gives you an updated FFI compatible with your version of Rust. In Python, the FFI is typically defined in C, so recompiling the python won't get you a compatible FFI. If Python did all FFI through something like ctypes, it would be much more smooth.
• smabie 3y ago
  Nothing to do with Python? These FFI bindings exist because Python is slow as dirt.
  • zzbn00 3y ago
    This is one of the reasons why Python got so popular. It is too slow to reimplement big pieces of software in it, so people just used bindings to existing code. And productivity rocketed!
  • zo1 3y ago
    And the bindings to python exist because those languages are a pain in the ass to work with. I don't want to work with Fortran, C++, Cobol, etc. And I sure as hell don't want to figure out how to integrate such wildly different languages into my existing and modern ecosystem.
    • oblio 3y ago
      You're probably replying to the original comment from the wrong angle. Ecosystems like Java, .NET, Golang, Rust, etc do away with this entire problem by virtue of... not calling into C 99.99% of the time, because they're <<fast enough>>.
      • BlueTemplar 3y ago
        My understanding is that, however, calls into Fortran happen because you want some subroutine to be «as fast as possible».
      • baq 3y ago
        There's no right or wrong angle here. There's the useful or not useful angle. Python was designed to call into C. It was always the solution to make Python fast: write the really slow parts in C and it might just turn out that will make the whole thing fast enough. Again: this is by design. The languages and VMs you list were designed to be fast enough without calling into C. If you need that, great, use them. People saying 'Python is slow' miss the point. It was never meant to be fast, it was always meant to be fast enough without qualifiers like 'no C'. If it isn't fast enough or otherwise not useful, don't use it, you've got plenty of alternatives.
• globalnode 3y ago
  yes it was very eye opening. i often see people comparing their favourite package manager to pythons and coming to the conclusion that python is terrible, but its not! one thing i dont quite understand, is why dont the python people just use a c/c++ math library instead of fortran?
  • patrick451 3y ago
    There often simply doesn't exist an equivalent library written in c/c++ (or any other language for that matter). The example I'm familiar with is SLICOT (Subroutine Library in Systems and Control Theory) [1], exposed in python through Slycot [2]. It as routines for pole placement, riccati solvers, various factorizations, and MIMO zero computations and a ton of other stuff. As far as I have been able to find, no c/c++/other-language comes close to supplying either the breadth or depth of this library. Further, many of the SLICOT subroutines were written by the original inventors of their respective algorithms, which I view as a big bonus. [1] http://slicot.org/ http://slicot.org/ [2] https://github.com/python-control/Slycot https://github.com/python-control/Slycot
    • avidphantasm 3y ago
      Right, lots of legacy code, plus lack of pointer aliasing in Fortran opens up more opportunities for optimization (or so I have read; this might have changed).
      • rightbyte 3y ago
        'restrict' has been an common extension in C since long ago and is now a proper keyword. I guess it is a matter of taste.
        • baq 3y ago
          also a matter of 'I'm not going to rewrite lapack in C because a platform which was never the intended target doesn't have a free compiler'.
    • 3y ago
• keithalewis 3y ago
  Indeed. The real problem is python seems to attract people with no training in software development. It is a mess on top of a mess.
  • dylan604 3y ago
    I thought that was what people said about php
    • josefx 3y ago
      A lot of the mockery PHP got was not because it attracted amateur developers, it was because the language itself was amateurishly implemented and because of the resulting mess when that leaked into how it behaved. Things like function names in the standard library optimized for a strlen based hash, a hand rolled parser that made it impossible to even guess in which contexts what features would work, proactive conversion of strings into numbers "0hello" == "0world", ... . There where entire communities dedicated not to mock the people working with PHP but the language itself.
    • oblio 3y ago
      And Basic, Visual Basic, HTML, Javascript, the list is probably endless :-)
  • tomlockwood 3y ago
    That's also a feature - by design it has to be friendly to new users, and not an arcane art only accessible to the Chosen One, as much as those Chosen Ones would like to be the only programmers.
• placebo 3y ago
  > It’s a miracle it works at all. I agree. In fact with what seems to be an exponential growth in complexity of software ecosystems, what's keeping it all from eventually getting to a "tower of Babel" catastrophe? Of course, this does not only apply to software, but it is a good example.
• sproketboy 3y ago
  [dead]