4 ms·

If your project has any third-party dependencies, and so (nowadays) you're going to set up requirements.txt and virtualenv and whatever anyway, I can see that y

If your project has any third-party dependencies, and so (nowadays) you're going to set up requirements.txt and virtualenv and whatever anyway, I can see that you're going to think things like "this XML parser in the standard library is just getting in the way; I can get a better one from PyPi". But I think a lot of the value of a large standard library is that it makes it possible to write more programs without needing that first third-party dependency. This is particularly good if you're using Python as a piece of glue inside something that isn't principally a Python project. It's easy to imagine a Python script doing a little bit of code generation in the build system of some larger project that wants to parse an XML file.

nullwasamistake 7y ago
What they need is an Apache Commons or Guava of Python. They're both defacto part of the standard java library.
- dehrmann 7y ago
  I try to avoid Guava because they have a habit of making incompatible breaking changes, and because so many libraries depend on it, it's likely to cause version conflicts. The way Apache Commons puts the major version in the package is much better in that regard.
  - nullwasamistake 7y ago
    I have not experienced this running guava 16-23 in various apps. Maybe incompatible but they're good about security patches for old versions. I have never seen a version conflict between guava releases
    - BeeOnRope 7y ago
      It's very easy to get a Guava version conflict because (a) Guava frequently adds new stuff, and (b) Guava semi-frequently deprecates and removes stuff a couple versions later. So all you need is one dep that needs Guava version X with method M that is removed in version X+2 (say) and another dep that needs something new introduced in version X+2, and you have a Guava version conflict. That's, Guava releases are not backwards compatible due to removal of classes and methods. You can sometimes fix this with a technology like shade or OSGi or whatever to allow private copies but it does not always work.
    - aphexairlines 7y ago
      Transitive dependencies on Guava 19, 20, 21 can lead to runtime crashes if your dependencies differ in what guava versions they expected when they were compiled: https://www.google.com/search?q=guava+nosuchmethoderror https://www.google.com/search?q=guava+nosuchmethoderror
guitarbill 7y ago
the problem is it can fall apart quickly. the XML parsing in the standard library is limited and slow, so most people consume lxml instead [0]. so it depends on the case. counterpoint: e.g. pathlib being in included is great. it was at least inspired by 3rd party libraries, but the features are relatively stable and the scope defined, and relatively few dependencies, and so moving it into the standard library is a win IMO. not only for import ease, but for consistency. [0] https://pypi.org/project/lxml/ https://pypi.org/project/lxml/
- maxerickson 7y ago
  ElementTree is in the stdlib. It isn't slow and has incremental parsing and so on. It's also a nice API for dealing with XML.
  - UncleEntity 7y ago
    > ElementTree is in the stdlib. It isn't slow and has incremental parsing and so on. I had enough trouble using it efficiently that I went and wrapped Boost property tree[0] and can happily churn out all sorts of data queries (including calling into python for the sorting function from the C++ lib) in almost no time. I was taking daily(ish) updates of an rss feed and appending it to a master rss file but sorting was pretty slow using list comprehensions so now I convert it automagically to json and append it as is. No more list comprehensions either, just hand it a lambda and it outputs a sorted C++ iterator. Though I probably should've just thrown the data into a database and learned SQL like a normal person... [0] https://github.com/eponymous/python3-property_tree https://github.com/eponymous/python3-property_tree
rogerbinns 7y ago
I think the biggest problem is going from zero third-party dependencies to one and more. Adding that very first one is a huge pain since there are many ways of doing it with many different trade offs. It is also time consuming and tedious. The various tools like you mention are best at adding even more dependencies, but are hurdles for the very first one.
- tachyonbeam 7y ago
  Not true. The more external dependencies you add, the more likely it is that one of them will break. I try to have as few external dependencies as possible, and to pick dependencies that are robust and reliably maintained. There is so much Python code on GitHub that is just broken out of the box. When people try your software and it fails to install because your nth dependency is broken or won't build on their system, you're lucky if they open an issue. Most potential users will just end up looking for an alternative and not even report the problem.
  - Groxx 7y ago
    They're typically broken out of the box because they don't pin their dependencies. pip-tools[1] or pipenv[2], and tox[3] if it's a lib, should be considered bare minimum necessities - if a project isn't using them, consider abandoning it ASAP, since apparently they don't know what they're doing and haven't paid attention to the ecosystem for years. [1] https://github.com/jazzband/pip-tools https://github.com/jazzband/pip-tools [2] https://docs.pipenv.org/en/latest/ https://docs.pipenv.org/en/latest/ [3] https://tox.readthedocs.io/en/latest/ https://tox.readthedocs.io/en/latest/
    - tachyonbeam 7y ago
      It's trickier than just pinning dependencies because some libraries also need to build C code, etc. Once you bring in external build tools, you have that many more potential points of failure. It's great. Also, what happens if your dependencies don't pin their dependencies? Possibly, uploading a package to pipy should require freezing dependencies or do it automatically.
wirrbel 7y ago
Last week I have written a Python script purely relying on the stdlib. Basically a coworkers shell script had to be adjusted to account for newer datafiles I was processing and I am not that experienced with shell script magic. I typed down 15 lines of Python code only relying on the std lib and was happy, it ran on the server's 2.x Python without an issue. This is the primary selling point of a larger std. library. But, I do wonder whether it needs to grow. Python is in a stage where adoption of new std library features is inherently slow, not only in third-party libraries like twisted, but also in applications, even if they use newer Python versions. What kills Python here is that it is most commonly bundled with the linux distribution or the OS (true also on mac). This reduces cycle times drastically. Compare this to newr language platforms that people like to install in newer versions on older platforms quite regularly. Some recent additions would be fine additions at an early stage of language development, but surely not for Python.
- petre 7y ago
  > What kills Python here is that it is most commonly bundled with the linux distribution or the OS (true also on mac) It doesn't kill it, quite the contrary, rather makes it ubiquitous. If you want another version you just install virtualenv. It's the same with Perl. We use the Perl version shipped with the distribution (openSuSE) and deploy to that. It's older but it's stable and it works. On our dev environment (mac) we have the same version with all of the modules installed in plenv. We also chose a framework with as little deoendencies as possible (Mojolicious). It looks like it was a great choice.
porker 7y ago
> and so (nowadays) you're going to set up requirements.txt and virtualenv and whatever anyway If only that were the norm amongst long-tail python users. Heck I don't do it; I have the anaconda distribution installed on Windows and when I need to do a bit of data analysis hope I have the correct version of packages installed. Making this core to the python workflow (bundling virtualenv? updating all docs to say "Set up a virtualenv first"?) is the first required change, before thinking about unbundling stdlib
- schlenk 7y ago
  I develop in python since 10+ years. And setting up a virtualenv only happens when developing patches for 3rd party packages. (due to the other commercial environment that doesn't work with venvs). And it is a totally miserable experience on Windows, every single time.
- pas 7y ago
  The stdlib would still ship with Python I presume, it would be simply updatable. But that doesn't mean it wouldn't work without a "requirements.txt" (use Pipenv which is light-years ahead).

nullwasamistake 7y ago

What they need is an Apache Commons or Guava of Python. They're both defacto part of the standard java library.

dehrmann 7y ago
I try to avoid Guava because they have a habit of making incompatible breaking changes, and because so many libraries depend on it, it's likely to cause version conflicts. The way Apache Commons puts the major version in the package is much better in that regard.
- nullwasamistake 7y ago
  I have not experienced this running guava 16-23 in various apps. Maybe incompatible but they're good about security patches for old versions. I have never seen a version conflict between guava releases
  - BeeOnRope 7y ago
    It's very easy to get a Guava version conflict because (a) Guava frequently adds new stuff, and (b) Guava semi-frequently deprecates and removes stuff a couple versions later. So all you need is one dep that needs Guava version X with method M that is removed in version X+2 (say) and another dep that needs something new introduced in version X+2, and you have a Guava version conflict. That's, Guava releases are not backwards compatible due to removal of classes and methods. You can sometimes fix this with a technology like shade or OSGi or whatever to allow private copies but it does not always work.
  - aphexairlines 7y ago
    Transitive dependencies on Guava 19, 20, 21 can lead to runtime crashes if your dependencies differ in what guava versions they expected when they were compiled: https://www.google.com/search?q=guava+nosuchmethoderror https://www.google.com/search?q=guava+nosuchmethoderror

guitarbill 7y ago

the problem is it can fall apart quickly. the XML parsing in the standard library is limited and slow, so most people consume lxml instead [0]. so it depends on the case. counterpoint: e.g. pathlib being in included is great. it was at least inspired by 3rd party libraries, but the features are relatively stable and the scope defined, and relatively few dependencies, and so moving it into the standard library is a win IMO. not only for import ease, but for consistency. [0] https://pypi.org/project/lxml/ https://pypi.org/project/lxml/

maxerickson 7y ago
ElementTree is in the stdlib. It isn't slow and has incremental parsing and so on. It's also a nice API for dealing with XML.
- UncleEntity 7y ago
  > ElementTree is in the stdlib. It isn't slow and has incremental parsing and so on. I had enough trouble using it efficiently that I went and wrapped Boost property tree[0] and can happily churn out all sorts of data queries (including calling into python for the sorting function from the C++ lib) in almost no time. I was taking daily(ish) updates of an rss feed and appending it to a master rss file but sorting was pretty slow using list comprehensions so now I convert it automagically to json and append it as is. No more list comprehensions either, just hand it a lambda and it outputs a sorted C++ iterator. Though I probably should've just thrown the data into a database and learned SQL like a normal person... [0] https://github.com/eponymous/python3-property_tree https://github.com/eponymous/python3-property_tree

rogerbinns 7y ago

I think the biggest problem is going from zero third-party dependencies to one and more. Adding that very first one is a huge pain since there are many ways of doing it with many different trade offs. It is also time consuming and tedious. The various tools like you mention are best at adding even more dependencies, but are hurdles for the very first one.

tachyonbeam 7y ago
Not true. The more external dependencies you add, the more likely it is that one of them will break. I try to have as few external dependencies as possible, and to pick dependencies that are robust and reliably maintained. There is so much Python code on GitHub that is just broken out of the box. When people try your software and it fails to install because your nth dependency is broken or won't build on their system, you're lucky if they open an issue. Most potential users will just end up looking for an alternative and not even report the problem.
- Groxx 7y ago
  They're typically broken out of the box because they don't pin their dependencies. pip-tools[1] or pipenv[2], and tox[3] if it's a lib, should be considered bare minimum necessities - if a project isn't using them, consider abandoning it ASAP, since apparently they don't know what they're doing and haven't paid attention to the ecosystem for years. [1] https://github.com/jazzband/pip-tools https://github.com/jazzband/pip-tools [2] https://docs.pipenv.org/en/latest/ https://docs.pipenv.org/en/latest/ [3] https://tox.readthedocs.io/en/latest/ https://tox.readthedocs.io/en/latest/
  - tachyonbeam 7y ago
    It's trickier than just pinning dependencies because some libraries also need to build C code, etc. Once you bring in external build tools, you have that many more potential points of failure. It's great. Also, what happens if your dependencies don't pin their dependencies? Possibly, uploading a package to pipy should require freezing dependencies or do it automatically.

wirrbel 7y ago

Last week I have written a Python script purely relying on the stdlib. Basically a coworkers shell script had to be adjusted to account for newer datafiles I was processing and I am not that experienced with shell script magic. I typed down 15 lines of Python code only relying on the std lib and was happy, it ran on the server's 2.x Python without an issue. This is the primary selling point of a larger std. library. But, I do wonder whether it needs to grow. Python is in a stage where adoption of new std library features is inherently slow, not only in third-party libraries like twisted, but also in applications, even if they use newer Python versions. What kills Python here is that it is most commonly bundled with the linux distribution or the OS (true also on mac). This reduces cycle times drastically. Compare this to newr language platforms that people like to install in newer versions on older platforms quite regularly. Some recent additions would be fine additions at an early stage of language development, but surely not for Python.

petre 7y ago
> What kills Python here is that it is most commonly bundled with the linux distribution or the OS (true also on mac) It doesn't kill it, quite the contrary, rather makes it ubiquitous. If you want another version you just install virtualenv. It's the same with Perl. We use the Perl version shipped with the distribution (openSuSE) and deploy to that. It's older but it's stable and it works. On our dev environment (mac) we have the same version with all of the modules installed in plenv. We also chose a framework with as little deoendencies as possible (Mojolicious). It looks like it was a great choice.

porker 7y ago

> and so (nowadays) you're going to set up requirements.txt and virtualenv and whatever anyway If only that were the norm amongst long-tail python users. Heck I don't do it; I have the anaconda distribution installed on Windows and when I need to do a bit of data analysis hope I have the correct version of packages installed. Making this core to the python workflow (bundling virtualenv? updating all docs to say "Set up a virtualenv first"?) is the first required change, before thinking about unbundling stdlib

schlenk 7y ago
I develop in python since 10+ years. And setting up a virtualenv only happens when developing patches for 3rd party packages. (due to the other commercial environment that doesn't work with venvs). And it is a totally miserable experience on Windows, every single time.
pas 7y ago
The stdlib would still ship with Python I presume, it would be simply updatable. But that doesn't mean it wouldn't work without a "requirements.txt" (use Pipenv which is light-years ahead).