5 ms·

>Python 3.11 support on Anaconda Platform >Due to lack of Python 3.11 support for packages that PyTorch depends on, including NumPy, SciPy, SymPy, Pillow and o

>Python 3.11 support on Anaconda Platform >Due to lack of Python 3.11 support for packages that PyTorch depends on, including NumPy, SciPy, SymPy, Pillow and others on the Anaconda platform. We will not be releasing Conda binaries compiled with Python 3.11 for PyTorch Release 2.0. The Pip packages with Python 3.11 support will be released, hence if you intend to use PyTorch 2.0 with Python 3.11 please use our Pip packages. It really sucks that anaconda always lags behind. I know the reasoning*, and I know it makes sense for what a lot of teams use it for... but on our side we are now looking more and more into dropping it since we are more of an R&D team. We already use containers for most of our pipelines, so just using pip might be viable. *Though I guess Anaconda chewed more than it can handle w.r.t managing an entire Python universe, and keeping up to date. Conda-forge is already almost a requirement but using the official package (with pip, in this case) has its own benefits for very complex packages like pytorch.

brucethemoose2 4y ago
The Arch Linux PyTorch 2.0 packages are great if you are looking for "cutting edge," as they are compiled against CUDA 12.1 now, instead of 11.8 like the official nightly releases. You can also get AVX2 patched Python and optimized C Python packages through CachyOS or ALHP. But even Arch is still stuck on Python 3.10
DreamFlasher 4y ago
Afaik NumPy, SciPy, SymPy and Pillow are not managed/owned by Anaconda? At least here: https://numpy.org/about/ https://numpy.org/about/ Anaconda isn't mentioned.
- DreamFlasher 4y ago
  Ah, yeah they do have a Python 3.11 release, just not on anaconda. Okay, yeah, for a couple of years now there isn't a good reason anymore to use anaconda anyways.
  - mardifoufs 4y ago
    Yes that's the issue! Most of the software is already ready, usable and just works... unless you use anaconda. Now that I think about it, is there some technical reason for that? I always thought it was mostly about stability, but I can't imagine python 3.11 being so unstable as to warrant waiting a whole year before even porting.
  - PaulHoule 4y ago
    For one of my projects, conda ‘just works’ to get working with the GPU but following the instructions for pip doesn’t work. On the other hand there’s another package I am interested in using where I need to build out of GitHub and it’s a very different story. I see myself as interested in commercial exploitation of transformers right now and I am delighted with the results. The first time I tried clustering all the Ukraine articles lumped together, all the sports were lumped, it runs 5x faster than my LDA-based clustering system and I think does a better job. With results like this I am happy to trade ‘cutting edge’ for convenience. I have thought about a ‘path less followed’ in Python which is a truly sound package manager like maven for Python (as opposed to Poetry which I’m not sure is sound but it sure is slow) and I can say I like the way conda works I just would rather do it with wheels. One beef I have with conda is that the bzip2 files are slow to decompress and even over a DSL line I would trade a little more downloading for faster installs.
  - Vox_Leone 4y ago
    >>for a couple of years now there isn't a good reason anymore to use anaconda anyways. I think of Anaconda envs as great back ends for Jupyter notebooks. There is still a place for it.
- hmaarrfk 4y ago
  All the projects you listed are community projects that have grown over the years to become the backbone of scientific computing. conda-forge, has evolved as one of the major conda community projects that helps release the latest releases from the projects you listed. You can help improve the state of pytorch packaging on conda-forge too! We've even released Pytorch 1.13 + Python 3.11 on linux and OSX! Give it a shot and let us know what you think! edit: Link to the conda-forge pytorch development repository https://github.com/conda-forge/pytorch-cpu-feedstock https://github.com/conda-forge/pytorch-cpu-feedstock
varispeed 4y ago
Why people use conda? I couldn't figure out what it brings to the table. I use pyenv on Linux and Mac and it basically works and with virtualenv plugin I can quickly switch environments to different Python versions.
- DethNinja 4y ago
  I find it far better for DevOps. It is really easy to get the team up and running via conda environment files.
- aldanor 4y ago
  - Managing non-python / binary dependencies alongside python ones - Managing a mixture of python / binary dependencies in local registries - conda-forge managing builds of some really flaky binary python packages that are sometimes a nightmare to build locally
  - nl 4y ago
    These are all legitimate reasons, but my personal experience (and perhaps preference?) is to use Docker for anything that is more complex than pip can handle. > conda-forge managing builds of some really flaky binary python packages that are sometimes a nightmare to build locally Yeah this is fair. Fortunately it's becoming rarer.
    - aldanor 4y ago
      Well. To each their own, use cases differ so wildly it's hard to compare them. The key audience for conda is ML/DS space, where most if not all packages come from either C/C++/Rust/Fortran and have to be compiled, while also requiring a consistent set of external C libraries like libblas, etc. As I said, some of those packages are a completely nightmare to build locally. Conda simplifies this by a lot in that you can just 'conda create -n myenv some=1.0 crazy=2.0 deps=2.0' and in a few seconds (if you use mamba and not conda) you have a working Python environment so off you go; no dockers, no local builds etc.
vimy 4y ago
I use anaconda environments and install everything with pip in them. Am I doing it wrong?
- mardifoufs 4y ago
  That's basically where we are at for tons of our pipelines, but it kind of defeats the purpose since a dockerfile with a proper base image is basically equivalent at that point.
  - disgruntledphd2 4y ago
    Conda is also really annoying to get working in a Dockerfile, because (if I remember correctly) of differences between login and non-login shells.
    - nerdponx 4y ago
      In general you shouldn't need to "activate" a Conda environment in the shell. Things generally "just work" if you use absolute paths. Something like this: COPY conda.yaml /env/conda.yaml RUN conda env create -f /env/conda.yaml -p /env/conda COPY foo.py /env/foo.py CMD ["/env/conda/bin/python", "/env/foo.py"] What is a little funny is installing a consistent version of Conda inside a container, because the official Miniconda installers are rolling-release only. However you might be able to downgrade to your desired version of Conda after installation.
- blandcoffee 4y ago
  What's the point of this vs. creating your own venv (python -m venv .venv) and having a much smaller library of intentional tools?
  - tagh 4y ago
    Each environment can have its own version of python which downloads with everything else (venv would require separate system installs for this).
dheera 4y ago
> It really sucks that anaconda always lags behind. I usually just go for virtualenv (if python library versions are the only issue) or go for docker (if it's more than that). Both let you just use the latest and greatest without any friction. conda sits in a weird middle ground that I hate.
disgruntledphd2 4y ago
> *Though I guess Anaconda chewed more than it can handle w.r.t managing an entire Python universe, and keeping up to date. Conda-forge is already almost a requirement but using the official package (with pip, in this case) has its own benefits for very complex packages like pytorch. Yeah, I absolutely adore conda, but they really need support.
urthor 4y ago
Anaconda is really a solution for a personal dev box yes. For containers a version controlled requirements.txt or Dockerfile is the way. Realistically, Anaconda is a venerable solution from the truly god awful old days. The ecosystem is vastly improved.
- deleted 4y ago
  [deleted]

brucethemoose2 4y ago

The Arch Linux PyTorch 2.0 packages are great if you are looking for "cutting edge," as they are compiled against CUDA 12.1 now, instead of 11.8 like the official nightly releases. You can also get AVX2 patched Python and optimized C Python packages through CachyOS or ALHP. But even Arch is still stuck on Python 3.10

DreamFlasher 4y ago

Afaik NumPy, SciPy, SymPy and Pillow are not managed/owned by Anaconda? At least here: https://numpy.org/about/ https://numpy.org/about/ Anaconda isn't mentioned.

DreamFlasher 4y ago
Ah, yeah they do have a Python 3.11 release, just not on anaconda. Okay, yeah, for a couple of years now there isn't a good reason anymore to use anaconda anyways.
- mardifoufs 4y ago
  Yes that's the issue! Most of the software is already ready, usable and just works... unless you use anaconda. Now that I think about it, is there some technical reason for that? I always thought it was mostly about stability, but I can't imagine python 3.11 being so unstable as to warrant waiting a whole year before even porting.
- PaulHoule 4y ago
  For one of my projects, conda ‘just works’ to get working with the GPU but following the instructions for pip doesn’t work. On the other hand there’s another package I am interested in using where I need to build out of GitHub and it’s a very different story. I see myself as interested in commercial exploitation of transformers right now and I am delighted with the results. The first time I tried clustering all the Ukraine articles lumped together, all the sports were lumped, it runs 5x faster than my LDA-based clustering system and I think does a better job. With results like this I am happy to trade ‘cutting edge’ for convenience. I have thought about a ‘path less followed’ in Python which is a truly sound package manager like maven for Python (as opposed to Poetry which I’m not sure is sound but it sure is slow) and I can say I like the way conda works I just would rather do it with wheels. One beef I have with conda is that the bzip2 files are slow to decompress and even over a DSL line I would trade a little more downloading for faster installs.
- Vox_Leone 4y ago
  >>for a couple of years now there isn't a good reason anymore to use anaconda anyways. I think of Anaconda envs as great back ends for Jupyter notebooks. There is still a place for it.
hmaarrfk 4y ago
All the projects you listed are community projects that have grown over the years to become the backbone of scientific computing. conda-forge, has evolved as one of the major conda community projects that helps release the latest releases from the projects you listed. You can help improve the state of pytorch packaging on conda-forge too! We've even released Pytorch 1.13 + Python 3.11 on linux and OSX! Give it a shot and let us know what you think! edit: Link to the conda-forge pytorch development repository https://github.com/conda-forge/pytorch-cpu-feedstock https://github.com/conda-forge/pytorch-cpu-feedstock

varispeed 4y ago

Why people use conda? I couldn't figure out what it brings to the table. I use pyenv on Linux and Mac and it basically works and with virtualenv plugin I can quickly switch environments to different Python versions.

DethNinja 4y ago
I find it far better for DevOps. It is really easy to get the team up and running via conda environment files.
aldanor 4y ago
- Managing non-python / binary dependencies alongside python ones - Managing a mixture of python / binary dependencies in local registries - conda-forge managing builds of some really flaky binary python packages that are sometimes a nightmare to build locally
- nl 4y ago
  These are all legitimate reasons, but my personal experience (and perhaps preference?) is to use Docker for anything that is more complex than pip can handle. > conda-forge managing builds of some really flaky binary python packages that are sometimes a nightmare to build locally Yeah this is fair. Fortunately it's becoming rarer.
  - aldanor 4y ago
    Well. To each their own, use cases differ so wildly it's hard to compare them. The key audience for conda is ML/DS space, where most if not all packages come from either C/C++/Rust/Fortran and have to be compiled, while also requiring a consistent set of external C libraries like libblas, etc. As I said, some of those packages are a completely nightmare to build locally. Conda simplifies this by a lot in that you can just 'conda create -n myenv some=1.0 crazy=2.0 deps=2.0' and in a few seconds (if you use mamba and not conda) you have a working Python environment so off you go; no dockers, no local builds etc.

vimy 4y ago

I use anaconda environments and install everything with pip in them. Am I doing it wrong?

mardifoufs 4y ago
That's basically where we are at for tons of our pipelines, but it kind of defeats the purpose since a dockerfile with a proper base image is basically equivalent at that point.
- disgruntledphd2 4y ago
  Conda is also really annoying to get working in a Dockerfile, because (if I remember correctly) of differences between login and non-login shells.
  - nerdponx 4y ago
    In general you shouldn't need to "activate" a Conda environment in the shell. Things generally "just work" if you use absolute paths. Something like this: COPY conda.yaml /env/conda.yaml RUN conda env create -f /env/conda.yaml -p /env/conda COPY foo.py /env/foo.py CMD ["/env/conda/bin/python", "/env/foo.py"] What is a little funny is installing a consistent version of Conda inside a container, because the official Miniconda installers are rolling-release only. However you might be able to downgrade to your desired version of Conda after installation.
blandcoffee 4y ago
What's the point of this vs. creating your own venv (python -m venv .venv) and having a much smaller library of intentional tools?
- tagh 4y ago
  Each environment can have its own version of python which downloads with everything else (venv would require separate system installs for this).

dheera 4y ago

> It really sucks that anaconda always lags behind. I usually just go for virtualenv (if python library versions are the only issue) or go for docker (if it's more than that). Both let you just use the latest and greatest without any friction. conda sits in a weird middle ground that I hate.

disgruntledphd2 4y ago

> *Though I guess Anaconda chewed more than it can handle w.r.t managing an entire Python universe, and keeping up to date. Conda-forge is already almost a requirement but using the official package (with pip, in this case) has its own benefits for very complex packages like pytorch. Yeah, I absolutely adore conda, but they really need support.

urthor 4y ago

Anaconda is really a solution for a personal dev box yes. For containers a version controlled requirements.txt or Dockerfile is the way. Realistically, Anaconda is a venerable solution from the truly god awful old days. The ecosystem is vastly improved.

deleted 4y ago
[deleted]