5 ms·

Argh, Python is moving too fast. 3.8 has barely settled in with the walrus operator. For people doing docker based stuff with the latest and greatest always it

by aprdm 6y ago

Argh, Python is moving too fast. 3.8 has barely settled in with the walrus operator. For people doing docker based stuff with the latest and greatest always it might be OK, however, for enterprise environments where OS versions do not move as fast and you have runtime dependencies everywhere it isn't good. We have to build C++ libs against python versions, have thousands of users having their workstation in a sane version of python globally where all dependencies (1k+ in-house/external packages) work well and etc. etc. (think CAD kind of software / Qt UIs) We were still in the process of migrating towards python3.7 and it was a huge effort for barely any real gain. Hopefully new libraries do not use 3.8/3.9 only features otherwise we're in for a bad time..

cozzyd 6y ago
The Python packaging "solution" is to have a whole Python world for everything you want to run (aka insanity).
- postpawl 6y ago
  Heard of virtualenv?
  - cozzyd 6y ago
    Yes... that's exactly what I was referring to. The problem is this: https://github.com/pypa/packaging-problems/issues/328 https://github.com/pypa/packaging-problems/issues/328
    - rbanffy 6y ago
      I've been thinking of offering a patch to `fdupes` to turn duplicates into CoW copies. ZFS and BtrFS have them covered, but I'm not sure about APFS and I bet NTFS doesn't. In any case, it's really hard to get a workstation with less than a terabyte of storage in it. I have 2.9 gigs under mine and that's for 53 environments. I'm not sure that's a huge issue.
      - cozzyd 6y ago
        Sure in some use cases it doesn't matter. In others (e.g. on an HPC cluster where you have a quota in your home directory measured in GB) it does. CoW won't help on GPFS or Lustre, probably. Arguably Python has no place in HPC but a lot of people use it. And on my laptop technically I have the space but I am loathe to give it up to duplicate various packages many times.
        aprdm 6y ago
        What would you suggest instead for HPC?
        cozzyd 6y ago
        Well, generally you want something with better performance characteristics, which increasingly is not the case (see e.g. https://www.nature.com/articles/s41550-020-1208-y%C2%A0 https://www.nature.com/articles/s41550-020-1208-y%C2%A0). Though, even if your stuff is implemented in Fortran or C or C++ or on a GPU, Python is still a natural scripting language that is often the primary interface to some other piece of code, thanks to swig or boost::python or cppyy. But Python's fast rate of change and lack of backwards compatibility makes it problematic for this use. Our HPC cluster recently forced everyone to upgrade to Python 3 and that caused all sorts of problems for everyone. I understand there isn't really a choice with the lack of Python 2.7 support, but this has not been a great experience for anybody.
        aprdm 6y ago
        Yeah totally. Our use cases seem very similar. We still haven't transitioned to python3 at our HPC. Well.. we do have it available but only around ~10% of the workloads make use of python3 and their libraries. We have created the two different paths in our NFS and if you're in python3 then you can't access python2 and vice-versa. We will be migrating everything to python3 eventually but as you noted it is a big effort with a lot of possibility for chaos.
        rbanffy 6y ago
- abiogenesis 6y ago
  Yes, and snap/flatpak's idea of software installation is a whole container per software. You should choose one: Isolated environments or dependency hell. Each has its own pros and cons.
abiogenesis 6y ago
Python 3.8 will still be supported until 2024, that is, 5 years from its release. I guess that's fair enough.
techdragon 6y ago
Genuinely asking because you’ve described the sort of CI/CD problem that there are existing solutions to, things like the Open SUSE build service, red hat have their own tools, commercial services exist from several vendors designed to tackle these “operating system permutations” build problems... is one release a year with security fixes going back four versions... really too fast? All the gory details about the release timeline are in PEP 602 - https://www.python.org/dev/peps/pep-0602/ https://www.python.org/dev/peps/pep-0602/
- aprdm 6y ago
  I believe the biggest issue is having non compatible additions to the language... for example, any code written in 3.8 with walrus operator will not work in python3.7. That is IMO not ideal. If it was security fixes / new features / improvements with no new syntax then I would be OK. I would say a lot of companies still have a very big legacy code base in 2.7... new services are created in docker using the latest python3 but desktop applications aren't as easy to upgrade constantly. Some companies also use NFS to store python libraries (we release 100s of times per day) and make some wrangling of PYTHONPATH to allow applications to find dependencies. Having to support multiple (incompatible) versions of python at a workstation isn't a good problem to have.
  - Flockster 6y ago
    Backwards compatibility I understand, but which languages give forward compatibility? I mean how would that with new features even work?
    - aprdm 6y ago
      Hm not exactly what I tried to convey. I guess I would love if python stopped trying to add new syntax so often. Walrus operator, fstring, etc. should happen maybe once every two years ideally? Did we really need the walrus operator? The cost of adding it is making any library using it in python 3.8+ incompatible with runtimes of python3.8<.
orf 6y ago
Are you migrating to 3.7 from 2.7, or 3.6?
- aprdm 6y ago
  I would say 90% of our code base is still in 2.7 and 10% in 3.6. That’s for the desktop applications, we have a bunch of services in 3.8 already.
rbanffy 6y ago
> Hopefully new libraries do not use 3.8/3.9 only features Ideally, for production environments, you'll want to pin version numbers for everything you can, so new packages don't intrude. I also like to keep testing environments with minimal dependencies and no pinning (I built `pip-chill` for that). Wouldn't that work for you?
- coldtea 6y ago
  That helps not to break an existing codebase. But not if there's a nice new library you really have a need for that uses new 3.8 features, and you can't use it because you're stuck with 3.6 for example...
  - rbanffy 6y ago
    We can always offer a PR that allows the library to run on older versions. I am limited to 3.6 on many environments and would be extremely happy to know others on 3.6 have better library support. Having said that, It's not something I feel. I know some things can get faster and some syntax can be improved, but, so far, I'm pretty OK with those envs running on 3.6. As for me, locally I use 3.8. If I do any 3.8-ism, CI will catch it.
- aprdm 6y ago
  That works but doesn’t solve the problem mentioned by the other comment. We also have to support a matrix of Python runtimes and dependencies in all of our machines. We can’t pre-bake the environments. It’s a very unique problem for HPC I guess. We don’t use pip, have a look in the Rez package manager if you haven’t yet for an idea of the problem. For our webservices it is much less of a problem as we can isolate the runtime from software installed in machines or third party software dependencies that we integrate with (think a blender plugin, blender comes with its own versions of dependencies)
  - rbanffy 6y ago
    > We don’t use pip What's the problem?
HelloNurse 6y ago
The problem is the "huge effort", not the "barely any real gain": you are not automating enough. Building and packaging "1k+ in-house/external packages" should be longer than building only one, but not more difficult with proper scripting and build systems.
- aprdm 6y ago
  Yeah of course :) we have terabytes of Python code written over the past 20 years that support our business. Good luck justifying to the business to stop producing code that generates revenue to “modernize” all that stable code.
barumi 6y ago
> We were still in the process of migrating towards python3.7 and it was a huge effort for barely any real gain. Besides the pursuit for the latest and greatest, is there really anything forcing you to upgrade? I mean, Python 3.7 will be around for at least 2 more years. https://www.python.org/dev/peps/pep-0537/ https://www.python.org/dev/peps/pep-0537/

by aprdm 6y ago

cozzyd 6y ago

The Python packaging "solution" is to have a whole Python world for everything you want to run (aka insanity).

postpawl 6y ago
Heard of virtualenv?
- cozzyd 6y ago
  Yes... that's exactly what I was referring to. The problem is this: https://github.com/pypa/packaging-problems/issues/328 https://github.com/pypa/packaging-problems/issues/328
  - rbanffy 6y ago
    I've been thinking of offering a patch to `fdupes` to turn duplicates into CoW copies. ZFS and BtrFS have them covered, but I'm not sure about APFS and I bet NTFS doesn't. In any case, it's really hard to get a workstation with less than a terabyte of storage in it. I have 2.9 gigs under mine and that's for 53 environments. I'm not sure that's a huge issue.
    - cozzyd 6y ago
      Sure in some use cases it doesn't matter. In others (e.g. on an HPC cluster where you have a quota in your home directory measured in GB) it does. CoW won't help on GPFS or Lustre, probably. Arguably Python has no place in HPC but a lot of people use it. And on my laptop technically I have the space but I am loathe to give it up to duplicate various packages many times.
      - aprdm 6y ago
        What would you suggest instead for HPC?
        cozzyd 6y ago
        Well, generally you want something with better performance characteristics, which increasingly is not the case (see e.g. https://www.nature.com/articles/s41550-020-1208-y%C2%A0 https://www.nature.com/articles/s41550-020-1208-y%C2%A0). Though, even if your stuff is implemented in Fortran or C or C++ or on a GPU, Python is still a natural scripting language that is often the primary interface to some other piece of code, thanks to swig or boost::python or cppyy. But Python's fast rate of change and lack of backwards compatibility makes it problematic for this use. Our HPC cluster recently forced everyone to upgrade to Python 3 and that caused all sorts of problems for everyone. I understand there isn't really a choice with the lack of Python 2.7 support, but this has not been a great experience for anybody.
        aprdm 6y ago
        Yeah totally. Our use cases seem very similar. We still haven't transitioned to python3 at our HPC. Well.. we do have it available but only around ~10% of the workloads make use of python3 and their libraries. We have created the two different paths in our NFS and if you're in python3 then you can't access python2 and vice-versa. We will be migrating everything to python3 eventually but as you noted it is a big effort with a lot of possibility for chaos.
      - rbanffy 6y ago
abiogenesis 6y ago
Yes, and snap/flatpak's idea of software installation is a whole container per software. You should choose one: Isolated environments or dependency hell. Each has its own pros and cons.

abiogenesis 6y ago

Python 3.8 will still be supported until 2024, that is, 5 years from its release. I guess that's fair enough.

techdragon 6y ago

Genuinely asking because you’ve described the sort of CI/CD problem that there are existing solutions to, things like the Open SUSE build service, red hat have their own tools, commercial services exist from several vendors designed to tackle these “operating system permutations” build problems... is one release a year with security fixes going back four versions... really too fast? All the gory details about the release timeline are in PEP 602 - https://www.python.org/dev/peps/pep-0602/ https://www.python.org/dev/peps/pep-0602/

aprdm 6y ago
I believe the biggest issue is having non compatible additions to the language... for example, any code written in 3.8 with walrus operator will not work in python3.7. That is IMO not ideal. If it was security fixes / new features / improvements with no new syntax then I would be OK. I would say a lot of companies still have a very big legacy code base in 2.7... new services are created in docker using the latest python3 but desktop applications aren't as easy to upgrade constantly. Some companies also use NFS to store python libraries (we release 100s of times per day) and make some wrangling of PYTHONPATH to allow applications to find dependencies. Having to support multiple (incompatible) versions of python at a workstation isn't a good problem to have.
- Flockster 6y ago
  Backwards compatibility I understand, but which languages give forward compatibility? I mean how would that with new features even work?
  - aprdm 6y ago
    Hm not exactly what I tried to convey. I guess I would love if python stopped trying to add new syntax so often. Walrus operator, fstring, etc. should happen maybe once every two years ideally? Did we really need the walrus operator? The cost of adding it is making any library using it in python 3.8+ incompatible with runtimes of python3.8<.

orf 6y ago

Are you migrating to 3.7 from 2.7, or 3.6?

aprdm 6y ago
I would say 90% of our code base is still in 2.7 and 10% in 3.6. That’s for the desktop applications, we have a bunch of services in 3.8 already.

rbanffy 6y ago

> Hopefully new libraries do not use 3.8/3.9 only features Ideally, for production environments, you'll want to pin version numbers for everything you can, so new packages don't intrude. I also like to keep testing environments with minimal dependencies and no pinning (I built `pip-chill` for that). Wouldn't that work for you?

coldtea 6y ago
That helps not to break an existing codebase. But not if there's a nice new library you really have a need for that uses new 3.8 features, and you can't use it because you're stuck with 3.6 for example...
- rbanffy 6y ago
  We can always offer a PR that allows the library to run on older versions. I am limited to 3.6 on many environments and would be extremely happy to know others on 3.6 have better library support. Having said that, It's not something I feel. I know some things can get faster and some syntax can be improved, but, so far, I'm pretty OK with those envs running on 3.6. As for me, locally I use 3.8. If I do any 3.8-ism, CI will catch it.
aprdm 6y ago
That works but doesn’t solve the problem mentioned by the other comment. We also have to support a matrix of Python runtimes and dependencies in all of our machines. We can’t pre-bake the environments. It’s a very unique problem for HPC I guess. We don’t use pip, have a look in the Rez package manager if you haven’t yet for an idea of the problem. For our webservices it is much less of a problem as we can isolate the runtime from software installed in machines or third party software dependencies that we integrate with (think a blender plugin, blender comes with its own versions of dependencies)
- rbanffy 6y ago
  > We don’t use pip What's the problem?

HelloNurse 6y ago

The problem is the "huge effort", not the "barely any real gain": you are not automating enough. Building and packaging "1k+ in-house/external packages" should be longer than building only one, but not more difficult with proper scripting and build systems.

aprdm 6y ago
Yeah of course :) we have terabytes of Python code written over the past 20 years that support our business. Good luck justifying to the business to stop producing code that generates revenue to “modernize” all that stable code.

barumi 6y ago

> We were still in the process of migrating towards python3.7 and it was a huge effort for barely any real gain. Besides the pursuit for the latest and greatest, is there really anything forcing you to upgrade? I mean, Python 3.7 will be around for at least 2 more years. https://www.python.org/dev/peps/pep-0537/ https://www.python.org/dev/peps/pep-0537/