3 ms·

I agree. This really seems like high impact too. A first party, supported solution would solve this mess. Getting the Python dependency build story under contro

I agree. This really seems like high impact too. A first party, supported solution would solve this mess. Getting the Python dependency build story under control I would include in this category. In this department, I think Conda shows the way. They leverage Azure Pipelines to build much of the c dependencies ahead of time, and just send binaries down the pipe (at least, that's the high level overview as I understand it). I can't see a world where the Python Software Foundation can't get support for this, I think they have the contacts at the big companies to get it funded. The lead Python has on machine learning, data science, and scientific computing won't last forever (though, I think its safe to say no matter what, it ain't going anywhere.

brettcannon 7y ago
> I can't see a world where the Python Software Foundation can't get support for this, I think they have the contacts at the big companies to get it funded. You might be surprised at how hard it is to get money donated for this sort of thing. Large companies often have their own build infrastructure and thus build things for themselves from scratch, to the point of building their own build toolchains (e.g. look at Google and bazel, Facebook and buck). So the way they handle packaging means how the overall packaging ecosystem functions does not impact them like it does individuals or smaller companies. This is also why companies only started funding PyPI security work about two years ago; downloading source securely from PyPI is important to big companies as that's where they grab the source that they compile. But funding pip improvements only just happened in Decembe, starting with the new pip resolver work got money (https://pyfound.blogspot.com/2020/03/new-pip-resolver-to-roll-out-this-year.html https://pyfound.blogspot.com/2020/03/new-pip-resolver-to-rol...). And all that money came from Mozilla and the Chan Zuckerberg Initiative; two huge non-profits, not companies. Add on to the fact that the PSF is projected to have a $627,000 USD loss this year due to PyCon US 2020 being cancelled and it makes finding funding really hard (https://pyfound.blogspot.com/2020/03/psfs-projected-2020-financial-outcome.html https://pyfound.blogspot.com/2020/03/psfs-projected-2020-fin...). Python overall very much runs on volunteers, but the packaging system especially. The core team of Python itself might generously have a total of 3 devs/week being paid to work on CPython spread across all contributors (IOW it is nowhere near as impactful as 3 actual full-time devs who aren't context-switching constantly between "work work" and "Python work"). But for Python packaging in general? If you don't count conda I think packaging has less than even 3. But if anyone wants to donate to try and rectify this, please go to https://psfmember.org/civicrm/contribute/transact?reset=1&id=13 https://psfmember.org/civicrm/contribute/transact?reset=1&id... and donate to the packaging WG!
- throwaway894345 7y ago
  It would probably go a long way if they advertised a roadmap of things they’d like to work on given donations. Maybe they have this out there and I just haven’t seen it, but I imagine it would be easy enough for me to persuade my small employer to throw a few thousand USD their way each year until some sane packaging solution is developed.
  - brettcannon 7y ago
    There's a rough list at https://wiki.python.org/psf/Fundable%20Packaging%20Improvements https://wiki.python.org/psf/Fundable%20Packaging%20Improveme.... There's no official roadmap as getting a group of volunteers to agree to a list of priorities would be near impossible. :) If there's a specific pain point your employer would like to see fixed and is willing to donate money to help resolve that pain point then please go to https://donate.pypi.org/ https://donate.pypi.org/.
- no_wizard 7y ago
  First of all, as an aside, thank you for all the work you do on Python! I actually got started pretty early on in my professional career with Python (3.5+ too, which I am thankful for in hindsight). I was using VS Code and the Python Extension before MS even acquired its developer (is Don still there? I hope he is. Such a genuinely down to earth guy). Without the work you guys do I don't know if I would have taken the programming the same way, or at that the time I did in my life. I will say, I didn't realize the funding situation was _that_ dire, especially since on Talk Python I remember specifically Michael and some others talking about the massive influx of money into Python and its ecosystem in the last year and half or so. Its not billions of dollars but it sounded like the situation had a positive uptick substantially, so, in short, I was wrong! Has the PSF ever considered having an actual business? Like say, consulting and/or having developers work on features that businesses pay to have implemented? (A loose example of this would be Igalia[0], which makes surprisingly good amounts of money to implement features in web browsers on behalf of the major browser vendors) I've always thought this would be a pretty natural way to have a steady source of revenue and the profits could all feed back into the PSF. [0]https://www.igalia.com/ https://www.igalia.com/
BiteCode_dev 7y ago
There has been a long debate about it between core devs, pypa members, conda CEO and a few key elements of the community recently. The conclusion has been that: - distributing story is handled ok, but distributing program is not. - python has some unique problems due to its popularity, diversity of user base and long history - the py command should be available everywhere - communication on best practices is lacking, and people mixes project deployment, dependency management, lib packaging and program distribution - the way python deals with imports is the root of all evils, not the packaging system, which actually is a consequence of it The best course of action to take is still not clear though, as creating yet another layer on top of the mess is probably not the right thing to do.

brettcannon 7y ago

> I can't see a world where the Python Software Foundation can't get support for this, I think they have the contacts at the big companies to get it funded. You might be surprised at how hard it is to get money donated for this sort of thing. Large companies often have their own build infrastructure and thus build things for themselves from scratch, to the point of building their own build toolchains (e.g. look at Google and bazel, Facebook and buck). So the way they handle packaging means how the overall packaging ecosystem functions does not impact them like it does individuals or smaller companies. This is also why companies only started funding PyPI security work about two years ago; downloading source securely from PyPI is important to big companies as that's where they grab the source that they compile. But funding pip improvements only just happened in Decembe, starting with the new pip resolver work got money (https://pyfound.blogspot.com/2020/03/new-pip-resolver-to-roll-out-this-year.html https://pyfound.blogspot.com/2020/03/new-pip-resolver-to-rol...). And all that money came from Mozilla and the Chan Zuckerberg Initiative; two huge non-profits, not companies. Add on to the fact that the PSF is projected to have a $627,000 USD loss this year due to PyCon US 2020 being cancelled and it makes finding funding really hard (https://pyfound.blogspot.com/2020/03/psfs-projected-2020-financial-outcome.html https://pyfound.blogspot.com/2020/03/psfs-projected-2020-fin...). Python overall very much runs on volunteers, but the packaging system especially. The core team of Python itself might generously have a total of 3 devs/week being paid to work on CPython spread across all contributors (IOW it is nowhere near as impactful as 3 actual full-time devs who aren't context-switching constantly between "work work" and "Python work"). But for Python packaging in general? If you don't count conda I think packaging has less than even 3. But if anyone wants to donate to try and rectify this, please go to https://psfmember.org/civicrm/contribute/transact?reset=1&id=13 https://psfmember.org/civicrm/contribute/transact?reset=1&id... and donate to the packaging WG!

throwaway894345 7y ago
It would probably go a long way if they advertised a roadmap of things they’d like to work on given donations. Maybe they have this out there and I just haven’t seen it, but I imagine it would be easy enough for me to persuade my small employer to throw a few thousand USD their way each year until some sane packaging solution is developed.
- brettcannon 7y ago
  There's a rough list at https://wiki.python.org/psf/Fundable%20Packaging%20Improvements https://wiki.python.org/psf/Fundable%20Packaging%20Improveme.... There's no official roadmap as getting a group of volunteers to agree to a list of priorities would be near impossible. :) If there's a specific pain point your employer would like to see fixed and is willing to donate money to help resolve that pain point then please go to https://donate.pypi.org/ https://donate.pypi.org/.
no_wizard 7y ago
First of all, as an aside, thank you for all the work you do on Python! I actually got started pretty early on in my professional career with Python (3.5+ too, which I am thankful for in hindsight). I was using VS Code and the Python Extension before MS even acquired its developer (is Don still there? I hope he is. Such a genuinely down to earth guy). Without the work you guys do I don't know if I would have taken the programming the same way, or at that the time I did in my life. I will say, I didn't realize the funding situation was _that_ dire, especially since on Talk Python I remember specifically Michael and some others talking about the massive influx of money into Python and its ecosystem in the last year and half or so. Its not billions of dollars but it sounded like the situation had a positive uptick substantially, so, in short, I was wrong! Has the PSF ever considered having an actual business? Like say, consulting and/or having developers work on features that businesses pay to have implemented? (A loose example of this would be Igalia[0], which makes surprisingly good amounts of money to implement features in web browsers on behalf of the major browser vendors) I've always thought this would be a pretty natural way to have a steady source of revenue and the profits could all feed back into the PSF. [0]https://www.igalia.com/ https://www.igalia.com/

BiteCode_dev 7y ago

There has been a long debate about it between core devs, pypa members, conda CEO and a few key elements of the community recently. The conclusion has been that: - distributing story is handled ok, but distributing program is not. - python has some unique problems due to its popularity, diversity of user base and long history - the py command should be available everywhere - communication on best practices is lacking, and people mixes project deployment, dependency management, lib packaging and program distribution - the way python deals with imports is the root of all evils, not the packaging system, which actually is a consequence of it The best course of action to take is still not clear though, as creating yet another layer on top of the mess is probably not the right thing to do.