4 ms·

I don’t know how I feel about this. As a developer, I say good, I’m glad this impacts them. Many distributions cut apart and really break many aspects of Python

I don’t know how I feel about this. As a developer, I say good, I’m glad this impacts them. Many distributions cut apart and really break many aspects of Python such as breaking out pip and one off packages like virtual environment, often removing them from the standard library. On the other hand, python’s packaging system is such a mess, I’m not sure the alternative / replacement is actually better. I wish the ecosystem could do better here and had an easy way for people who want to help improve this to do so.

throwaway894345 5y ago
I've battled with Python's packaging systems and other tooling and performance problems for more than a decade. Every couple of years some new thing comes along promising to fix package management, but it always fails spectacularly on some important, mainstream package or use case. I suspect a lot of this complexity comes down to the significant degree in which the Python ecosystem depends on C: not only do we need to package Python programs but C programs as well, and some people think we should ship C source code and build on the target machine and others think we should ship pre-built C binaries and dynamically link against them on the target machine thus Python supports both mechanisms; however, both are perilous in the general case. And interestingly, Python depends so hard on C because CPython is very slow, and CPython is very slow because the C-extension API is virtually the whole of the CPython interpreter (many interesting optimizations to the interpreter would break compatibility in the ecosystem). So it certainly feels like all of Python's major problems come down to this decision to lean hard on C and specifically to make so much of the interpreter available for the C-extensions. The way out as far as I can tell is to define some narrower API surface (e.g., https://hpyproject.org https://hpyproject.org), get the ecosystem to consolidate around that, deprecate the old API surface, and then make the requisite breaking optimizations such that the ecosystem can feasibly do more in native Python. This requires leadership; however, and the hard truth is this has not historically seemed to be Python's strong suit--the Python maintainers seem unable to drive out big, necessary changes like this (which is certainly not to say that leadership is easy or that I could do better, particularly when Python is so established in many respects). Personally, I've come to use Go for 99% of my Python use cases and it's been great. There are dramatically fewer C bindings (<1% of the ecosystem by my crude estimation), so the build/packaging tooling and performance are orders of magnitude better than in Python. Static typing works well out of the box, real static binaries are not only feasible but trivial (as opposed to Python where you try to build a zip file with your dependencies and the result is hundreds of megabytes and it's still missing the runtime, std libs, and various .so files). Further still, builds, tests, and every kind of tooling are far faster than with Python, and far simpler to install and manage. Unless you're doing data science, I don't think you'll regret the transition. EDIT: On the subject of Python to Go, looks like this case study hit the front page shortly after posting this comment: https://gitlab.com/esr/reposurgeon/-/blob/master/GoNotes.adoc https://gitlab.com/esr/reposurgeon/-/blob/master/GoNotes.ado...
- cozzyd 5y ago
  Language-specific package managers make sense in some cases (generally only for developers, not users), but why should every language have its own siloed ecosystem? This just leads to people implementing things in other languages for no reason.
  - throwaway894345 5y ago
    I don't think anyone wishes that Go or Rust would have chosen to just use the package manager from JavaScript or Python or Java or .Net. There's still quite a lot of innovation in package management--Go's package management seems pretty ideal in my opinion, but a lot of languages expect tooling to cater to the narrow tail of use cases (e.g., arbitrary code execution at install time). In general, package managers are pretty tightly coupled to the ideals of a language community, which tend to vary a lot.
- dralley 5y ago
  >This requires leadership; however, and the hard truth is this has not historically seemed to be Python's strong suit--the Python maintainers seem unable to drive out big, necessary changes like this (which is certainly not to say that leadership is easy or that I could do better, particularly when Python is so established in many respects). I just want to point out that while I agree with this, the reason it also clear - lack of manpower. Somehow, despite Python being one of the top 3 most widely used programming languages, most of the critical infrastructure is maintained by very very few people, usually in their spare time.
  - mistrial9 5y ago
    yea agree -- and while my comment elsewhere is being downvoted to the pits, I do understand the Debian/Ubuntu python teams' sense of overwhelm; Debian/Ubuntu simply does not have enough hands to handle changing the OS dependencies on system Python, the moving library sets that are in fashion AND small changes to reset Python "not-three" to be just a tool, not a system component. So they chose to make good-enough package sets of Python binaries, which seem to please no one but work reliably. It's clear to me that there are immature people involved due to the name calling and basically vandalism of "not-three" Python on Debian, in addition to the incredibly smart and long-term committed people that make Debian uniquely great. I wrote email to Chris Lamb directly, but how could he work harder than he did. Herded cats would do better to listen sometimes. and while I was typing, Perl5 was updated TODAY
xorcist 5y ago
I see where you are coming from, but there are legitimate concerns with the language package systems such as pypi. For one thing, pypi is unvetted, and the problems that bring should be obvious by now. Another thing is that many move only the tip and older versions seldom gets fixed. If you introduce a dependency in a core package you take on the work of maintaining that version as long as the core package lives, which may be much longer than the upstream author wants to work on it. The language ecosystems usually don't bring any stability guarantees. Just because the upstream author thinks a new incompatible version is the bee's knees doesn't mean core packages can suddenly be rewritten in the middle of stable period. So there needs to be a system for maintaining these packages in place anyway. And if it's something distributions generally know how to do, it's packaging software, so it's pretty natural they use what tools they have available. The outcome isn't always great for the end user who wants unrestricted access to the language ecosystem, but that doesn't there aren't problems to solve here.

throwaway894345 5y ago

I've battled with Python's packaging systems and other tooling and performance problems for more than a decade. Every couple of years some new thing comes along promising to fix package management, but it always fails spectacularly on some important, mainstream package or use case. I suspect a lot of this complexity comes down to the significant degree in which the Python ecosystem depends on C: not only do we need to package Python programs but C programs as well, and some people think we should ship C source code and build on the target machine and others think we should ship pre-built C binaries and dynamically link against them on the target machine thus Python supports both mechanisms; however, both are perilous in the general case. And interestingly, Python depends so hard on C because CPython is very slow, and CPython is very slow because the C-extension API is virtually the whole of the CPython interpreter (many interesting optimizations to the interpreter would break compatibility in the ecosystem). So it certainly feels like all of Python's major problems come down to this decision to lean hard on C and specifically to make so much of the interpreter available for the C-extensions. The way out as far as I can tell is to define some narrower API surface (e.g., https://hpyproject.org https://hpyproject.org), get the ecosystem to consolidate around that, deprecate the old API surface, and then make the requisite breaking optimizations such that the ecosystem can feasibly do more in native Python. This requires leadership; however, and the hard truth is this has not historically seemed to be Python's strong suit--the Python maintainers seem unable to drive out big, necessary changes like this (which is certainly not to say that leadership is easy or that I could do better, particularly when Python is so established in many respects). Personally, I've come to use Go for 99% of my Python use cases and it's been great. There are dramatically fewer C bindings (<1% of the ecosystem by my crude estimation), so the build/packaging tooling and performance are orders of magnitude better than in Python. Static typing works well out of the box, real static binaries are not only feasible but trivial (as opposed to Python where you try to build a zip file with your dependencies and the result is hundreds of megabytes and it's still missing the runtime, std libs, and various .so files). Further still, builds, tests, and every kind of tooling are far faster than with Python, and far simpler to install and manage. Unless you're doing data science, I don't think you'll regret the transition. EDIT: On the subject of Python to Go, looks like this case study hit the front page shortly after posting this comment: https://gitlab.com/esr/reposurgeon/-/blob/master/GoNotes.adoc https://gitlab.com/esr/reposurgeon/-/blob/master/GoNotes.ado...

cozzyd 5y ago
Language-specific package managers make sense in some cases (generally only for developers, not users), but why should every language have its own siloed ecosystem? This just leads to people implementing things in other languages for no reason.
- throwaway894345 5y ago
  I don't think anyone wishes that Go or Rust would have chosen to just use the package manager from JavaScript or Python or Java or .Net. There's still quite a lot of innovation in package management--Go's package management seems pretty ideal in my opinion, but a lot of languages expect tooling to cater to the narrow tail of use cases (e.g., arbitrary code execution at install time). In general, package managers are pretty tightly coupled to the ideals of a language community, which tend to vary a lot.
dralley 5y ago
>This requires leadership; however, and the hard truth is this has not historically seemed to be Python's strong suit--the Python maintainers seem unable to drive out big, necessary changes like this (which is certainly not to say that leadership is easy or that I could do better, particularly when Python is so established in many respects). I just want to point out that while I agree with this, the reason it also clear - lack of manpower. Somehow, despite Python being one of the top 3 most widely used programming languages, most of the critical infrastructure is maintained by very very few people, usually in their spare time.
- mistrial9 5y ago
  yea agree -- and while my comment elsewhere is being downvoted to the pits, I do understand the Debian/Ubuntu python teams' sense of overwhelm; Debian/Ubuntu simply does not have enough hands to handle changing the OS dependencies on system Python, the moving library sets that are in fashion AND small changes to reset Python "not-three" to be just a tool, not a system component. So they chose to make good-enough package sets of Python binaries, which seem to please no one but work reliably. It's clear to me that there are immature people involved due to the name calling and basically vandalism of "not-three" Python on Debian, in addition to the incredibly smart and long-term committed people that make Debian uniquely great. I wrote email to Chris Lamb directly, but how could he work harder than he did. Herded cats would do better to listen sometimes. and while I was typing, Perl5 was updated TODAY

xorcist 5y ago

I see where you are coming from, but there are legitimate concerns with the language package systems such as pypi. For one thing, pypi is unvetted, and the problems that bring should be obvious by now. Another thing is that many move only the tip and older versions seldom gets fixed. If you introduce a dependency in a core package you take on the work of maintaining that version as long as the core package lives, which may be much longer than the upstream author wants to work on it. The language ecosystems usually don't bring any stability guarantees. Just because the upstream author thinks a new incompatible version is the bee's knees doesn't mean core packages can suddenly be rewritten in the middle of stable period. So there needs to be a system for maintaining these packages in place anyway. And if it's something distributions generally know how to do, it's packaging software, so it's pretty natural they use what tools they have available. The outcome isn't always great for the end user who wants unrestricted access to the language ecosystem, but that doesn't there aren't problems to solve here.