7 ms·

I'm going to admit that what I really want to see is a strong push to standardize and fully incorporate package management and distribution into python's core.

I'm going to admit that what I really want to see is a strong push to standardize and fully incorporate package management and distribution into python's core. Despite the work done on it, it's still a mess as far as I can see, and there is no single source of truth (that I know of) on how to do it. For that matter, pip can't even search for packages any more, and instead directs you to browse the pypi website in a browser. Whatever the technical reasons for that, its a user interface fail. Conda can do it!!!!! (as well as just about any package management system I've ever used)

asylteltine 3y ago
Python’s dependency management, or lack there of, its import system, and the lack of strong typing really make me hate it. It’s the first language I really felt adept with, but once I learned Go, I never looked back. Every time I have to use python it’s like coding with crayons.
- Aperocky 3y ago
  Premature optimization is root of all evil. Python isn't perfect, and mostly unsuitable for any system where performance is in consideration. That leaves everything else including personal utility scripts and packages I use each day to automate random stuff. And I hugely appreciate how fast and simple it is to develop in python, unlike certain languages that literally depends on IDEs due to the verbosity and unnecessary cognitive load.
  - switch007 3y ago
    > And I hugely appreciate how fast and simple it is to develop in python Indeed it’s crazy. A few pip installs and I had a multiprocessing pandas (dask) with a web gui, and a workflow system (also with a web gui), and a pipeline to convert csv to parquet in like 20 lines of code
- pnt12 3y ago
  I think type hints help a lot. A codebase with classes and type hints reads much better than one using ad-hoc dictionaries for every data structure. But on the other hand I don't really like Go, so maybe it's different languages for different tastes.
- wendyshu 3y ago
  What aspect of python's type system do you find insufficient?
  - asylteltine 3y ago
    [dead]
bandrami 3y ago
It's beyond "mess" well into "fiasco" and frankly I'm astounded people think there's a more important issue facing the language right now. Look, for an example of a high-prestige project, at Spleeter, which spends multiple pages of its wiki describing how to install it with Conda and then summarizes "Note: as of 2021 we no longer recommend using Conda to install Spleeter" and nothing else.
- noitpmeder 3y ago
  What are you smoking? The readme for spleeter clearly shows the two simple commands needed to install -- one being a conda install for system level dependencies and one being pip for the spleeter python package itself.
woodruffw 3y ago
> I'm going to admit that what I really want to see is a strong push to standardize and fully incorporate package management and distribution into python's core. Despite the work done on it, it's still a mess as far as I can see, and there is no single source of truth (that I know of) on how to do it. Package management is standardized in a series of PEPs[1]. Some of those PEPs are living documents that have versions maintained under the PyPA packaging specifications[2]. The Python Packaging User Guide[3] is, for most things, the canonical reference for how to do package distribution in Python. It's also maintained by the PyPA. (I happen to agree, even with all of this, that Python packaging is a bit of a mess. But it's a much better defined mess than it was even 5 years ago, and initiatives to bring packaging into the core need to address ~20 years of packaging debt.) [1]: https://peps.python.org/topic/packaging/ https://peps.python.org/topic/packaging/ [2]: https://packaging.python.org/en/latest/specifications/index.html https://packaging.python.org/en/latest/specifications/index.... [3]: https://packaging.python.org/en/latest/flow/ https://packaging.python.org/en/latest/flow/
- jonnycomputer 3y ago
  which is why I said, "Despite the work done on it"
  - woodruffw 3y ago
    Yes, that was meant more for the “source of truth” part.
    - jonnycomputer 3y ago
      No, I do appreciate you taking the time to do it (I was too lazy)
- jshen 3y ago
  Is there anything in there about managing dependencies within a python project? What is the canonical way to do that in python today?
globular-toast 3y ago
There it is. The obligatory comment on every Python thread on HN. It's most popular programming language in the world. Other people can figure it out, apparently.
- woodruffw 3y ago
  I think we can be more charitable than this: it's possible to be both immensely popular and to have a sub-par packaging experience that users put up with. That's where Python is.
  - globular-toast 3y ago
    The trouble is people compare it to greenfield languages of the past few years with nowhere near the scope, userbase or legacy of Python. Long time Python users like me don't have any of the problems that the non-Python users that always post these comments have. It would be nice to have improvements to packaging, sure, but it's always just completely non-constructive stuff like "it's not as easy as <brand new language with no legacy>".
    - jonnycomputer 3y ago
      Big assumptions here.
    - jshen 3y ago
      Java and Ruby both have much better dependency management experiences and both have been around for far longer than a few years.
      - deleted 3y ago
        [deleted]
      - antod 3y ago
        As someone who dealt with Java and Python 20yrs back, I don't think Java is a valid comparison. Java had a terrible or non existent OS integration story - it didn't even try to have OS native stuff. It was it's own separate island that worked best when you stayed on the island. On Linux, Python was included in the OS so you had the two worlds of distro packaging and application development/deployment dependencies already in conflict. Macs also shipped their own Python that you had to avoid messing up. And on Windows Python was also trying to support the standard download a setup.exe method for library distribution. Java only ever had the developer dependency usecase to think about. Before Maven most Java apps just manually vendored all their dependencies into their codebase, or you manually wrangled assembling stuff in place using application specific classpaths and additions to path env vars etc.
aranke 3y ago
I gave a talk about this at the Packaging Summit during Pycon which was well received, so the team is definitely aware of the problem. However, the sense I got was that it was going to be a lot of work to “fix Python packaging” which wasn't feasible with an all-volunteer group. At work, we're migrating away from pip as a distribution mechanism for this reason; I don't expect to see meaningful improvements to the developer experience anytime soon. This is especially true because pip today is roughly where npm was in 2015, so there's a lot of fundamental infrastructure work (including security) that still needs to happen. An example of this is that PyPI just got the ability to namespace packages.
- LordKeren 3y ago
  > we're migrating away from pip as a distribution mechanism for this reason Could you elaborate on what you’re using as a replacement?
  - pnt12 3y ago
    Not the parent but pipenv is decent, poetry is even better: - clear separation of dev and production dependencies - lock file with the current version of all dependencies for reproducible builds (this is slightly difference than the dependency specification) - no accidental global installs because you forgot to activate a virtual environment - (not sure if supported by pip) allows installing libraries directly from a git repo, which is very useful if you have internal libraries - easier updates
  - aranke 3y ago
    We distribute a CLI tool (dbt), so we’re migrating to distributing using the following mechanisms: 1. curl script that installs dbt for end users 2. zipped snapshots for dbt Cloud, the SaaS hosted version of dbt Eventually we want to create Docker images from 2, but we’re not there yet.
- di 3y ago
  > An example of this is that PyPI just got the ability to namespace packages. You're thinking of organizations, which are not namespaces: https://blog.pypi.org/posts/2023-04-23-introducing-pypi-organizations/ https://blog.pypi.org/posts/2023-04-23-introducing-pypi-orga...

asylteltine 3y ago

Python’s dependency management, or lack there of, its import system, and the lack of strong typing really make me hate it. It’s the first language I really felt adept with, but once I learned Go, I never looked back. Every time I have to use python it’s like coding with crayons.

Aperocky 3y ago
Premature optimization is root of all evil. Python isn't perfect, and mostly unsuitable for any system where performance is in consideration. That leaves everything else including personal utility scripts and packages I use each day to automate random stuff. And I hugely appreciate how fast and simple it is to develop in python, unlike certain languages that literally depends on IDEs due to the verbosity and unnecessary cognitive load.
- switch007 3y ago
  > And I hugely appreciate how fast and simple it is to develop in python Indeed it’s crazy. A few pip installs and I had a multiprocessing pandas (dask) with a web gui, and a workflow system (also with a web gui), and a pipeline to convert csv to parquet in like 20 lines of code
pnt12 3y ago
I think type hints help a lot. A codebase with classes and type hints reads much better than one using ad-hoc dictionaries for every data structure. But on the other hand I don't really like Go, so maybe it's different languages for different tastes.
wendyshu 3y ago
What aspect of python's type system do you find insufficient?
- asylteltine 3y ago
  [dead]

bandrami 3y ago

It's beyond "mess" well into "fiasco" and frankly I'm astounded people think there's a more important issue facing the language right now. Look, for an example of a high-prestige project, at Spleeter, which spends multiple pages of its wiki describing how to install it with Conda and then summarizes "Note: as of 2021 we no longer recommend using Conda to install Spleeter" and nothing else.

noitpmeder 3y ago
What are you smoking? The readme for spleeter clearly shows the two simple commands needed to install -- one being a conda install for system level dependencies and one being pip for the spleeter python package itself.

woodruffw 3y ago

> I'm going to admit that what I really want to see is a strong push to standardize and fully incorporate package management and distribution into python's core. Despite the work done on it, it's still a mess as far as I can see, and there is no single source of truth (that I know of) on how to do it. Package management is standardized in a series of PEPs[1]. Some of those PEPs are living documents that have versions maintained under the PyPA packaging specifications[2]. The Python Packaging User Guide[3] is, for most things, the canonical reference for how to do package distribution in Python. It's also maintained by the PyPA. (I happen to agree, even with all of this, that Python packaging is a bit of a mess. But it's a much better defined mess than it was even 5 years ago, and initiatives to bring packaging into the core need to address ~20 years of packaging debt.) [1]: https://peps.python.org/topic/packaging/ https://peps.python.org/topic/packaging/ [2]: https://packaging.python.org/en/latest/specifications/index.html https://packaging.python.org/en/latest/specifications/index.... [3]: https://packaging.python.org/en/latest/flow/ https://packaging.python.org/en/latest/flow/

jonnycomputer 3y ago
which is why I said, "Despite the work done on it"
- woodruffw 3y ago
  Yes, that was meant more for the “source of truth” part.
  - jonnycomputer 3y ago
    No, I do appreciate you taking the time to do it (I was too lazy)
jshen 3y ago
Is there anything in there about managing dependencies within a python project? What is the canonical way to do that in python today?

globular-toast 3y ago

There it is. The obligatory comment on every Python thread on HN. It's most popular programming language in the world. Other people can figure it out, apparently.

woodruffw 3y ago
I think we can be more charitable than this: it's possible to be both immensely popular and to have a sub-par packaging experience that users put up with. That's where Python is.
- globular-toast 3y ago
  The trouble is people compare it to greenfield languages of the past few years with nowhere near the scope, userbase or legacy of Python. Long time Python users like me don't have any of the problems that the non-Python users that always post these comments have. It would be nice to have improvements to packaging, sure, but it's always just completely non-constructive stuff like "it's not as easy as <brand new language with no legacy>".
  - jonnycomputer 3y ago
    Big assumptions here.
  - jshen 3y ago
    Java and Ruby both have much better dependency management experiences and both have been around for far longer than a few years.
    - deleted 3y ago
      [deleted]
    - antod 3y ago
      As someone who dealt with Java and Python 20yrs back, I don't think Java is a valid comparison. Java had a terrible or non existent OS integration story - it didn't even try to have OS native stuff. It was it's own separate island that worked best when you stayed on the island. On Linux, Python was included in the OS so you had the two worlds of distro packaging and application development/deployment dependencies already in conflict. Macs also shipped their own Python that you had to avoid messing up. And on Windows Python was also trying to support the standard download a setup.exe method for library distribution. Java only ever had the developer dependency usecase to think about. Before Maven most Java apps just manually vendored all their dependencies into their codebase, or you manually wrangled assembling stuff in place using application specific classpaths and additions to path env vars etc.

aranke 3y ago

I gave a talk about this at the Packaging Summit during Pycon which was well received, so the team is definitely aware of the problem. However, the sense I got was that it was going to be a lot of work to “fix Python packaging” which wasn't feasible with an all-volunteer group. At work, we're migrating away from pip as a distribution mechanism for this reason; I don't expect to see meaningful improvements to the developer experience anytime soon. This is especially true because pip today is roughly where npm was in 2015, so there's a lot of fundamental infrastructure work (including security) that still needs to happen. An example of this is that PyPI just got the ability to namespace packages.

LordKeren 3y ago
> we're migrating away from pip as a distribution mechanism for this reason Could you elaborate on what you’re using as a replacement?
- pnt12 3y ago
  Not the parent but pipenv is decent, poetry is even better: - clear separation of dev and production dependencies - lock file with the current version of all dependencies for reproducible builds (this is slightly difference than the dependency specification) - no accidental global installs because you forgot to activate a virtual environment - (not sure if supported by pip) allows installing libraries directly from a git repo, which is very useful if you have internal libraries - easier updates
- aranke 3y ago
  We distribute a CLI tool (dbt), so we’re migrating to distributing using the following mechanisms: 1. curl script that installs dbt for end users 2. zipped snapshots for dbt Cloud, the SaaS hosted version of dbt Eventually we want to create Docker images from 2, but we’re not there yet.
di 3y ago
> An example of this is that PyPI just got the ability to namespace packages. You're thinking of organizations, which are not namespaces: https://blog.pypi.org/posts/2023-04-23-introducing-pypi-organizations/ https://blog.pypi.org/posts/2023-04-23-introducing-pypi-orga...