8 ms·

I’m using Anaconda because it was recommended in a step by step tutorial for playing with deep learning. What would be involved in removing it from my system a

by natch 9y ago

I’m using Anaconda because it was recommended in a step by step tutorial for playing with deep learning. What would be involved in removing it from my system and moving instead to this set of tools? Not necessarily looking for s step by step answer, just for general suggestions. My guess is: find out which python the deep learning tools are using, remove Anaconda, and reinstall the python version needed, using the tools from this post. I’ll need to read up on the tools too. Any pitfalls with this approach?

sreevisakh 9y ago
Just in case you wish to retain anaconda, you could simply remove anaconda path from your PATH variable in .bashrc file (Assuming you are on Linux). If you need anaconda again, you could add the path to the variable, or run `source ~/anaconda3/bin/activate`
guitarbill 9y ago
I think you could probably use both, side-by-side (but not at the same time/for the same project). Anaconda might be nicer if the packages you need are C-based and would need compiling on your platform. Depends on your use-case. For experimentation/playing around, stick with what works for you. But if a project required me to install Anaconda, I wouldn’t take it seriously. And pipenv is pretty easy to use, too. So if you plan on distributing it or open-sourcing it, understanding how most other people manage dependencies ouside of conda is going to be useful.
- stared 9y ago
  Why not seriously? Many data science environments are pretty much based on Anaconda installations. (See e.g. these dockers: https://github.com/jupyter/docker-stacks/tree/master/scipy-notebook https://github.com/jupyter/docker-stacks/tree/master/scipy-n...) I mean, all packages I know DONT require Anaconda. But if you need the whole environment, sometimes Anaconda is the easiest tool too install all dependencies.
  - guitarbill 9y ago
    > Anaconda is the easiest tool too install all dependencies. For hobbyist stuff that’s fine, and I applaud lowering the entry barrier. My issue would be if I needed Anaconda to deploy the project/dependencies into “prod” in some professional capacity, instead of standard Python build tools. Having said that, I’m not terribly familiar with Anaconda, and it seems to leverage virtualenv under the hood, possibly with pre-compiled packages (like wheels?).
    - stared 9y ago
      "Hobbyist"? It's widely used in research in data science & machine learning. If you want a setup which works on various systems, and uses Python numeric packages, usually it is the most failsafe way to use with various OS. (Unless you want to put everything in docker.) Unless by "professional" you mean "building", well - then you have a point.
      - Fomite 9y ago
        Yeah. Unless "Hobbyist" now also means my research university's 104 node, 2548 core HPC cluster...
    - pwang 9y ago
      Anaconda does not leverage virtualenv under the hood. Anaconda does not use wheels as part of that. However, you can use pip and virtualenvs within Anaconda, but you'll probably get some inconsistent results. The reason that Anaconda exists is to make it easier for people to have consistency between their dev environment, the build environment, and the prod environment. If you're seriously going to build all of the scientific Python stack from scratch, correctly and optimally, for your prod environment (and then also do the same for every dev environment you need to support), then you have way too much time and probably haven't actually tried doing it over any real period of time. The problem isn't that it's impossible. The problem is that there are dozens of different ways to almost succeed, and you don't run into the problems until way too late.
      - 9y ago
beagle3 9y ago
What do you think you'll get from any if these tools that you do not get from {ana,mini}conda? I must be missing something, but all these use cases (and more) seem to have been covered by miniconda ages ago. conda works with perfectly well with pip, and creates virtual environments that are at least (in my experience) as good as virtualenv does.
james_the_cat 9y ago
Anaconda does most of the stuff mentioned, and also makes it much easier to install packages based on C/C++ libraries (which most deep learning things are). So you're better off staying with anaconda. It's widely used in commercial data science projects so the idea that noone "takes it seriously" as someone else suggests is a bit silly. I assume they're thinking about a different context to data science projects. That said anaconda does have a whole variety of extremely annoying quirks, like packages not being backwards compatible with old versions of conda, or conda going crazy and reinstalling itself, or the way the conda-forge repo has far more packages than the official conda repo. It's very far from perfect. But for data science I think it's basically the standard package manager in Python land.
- Rotareti 9y ago
  > ... and also makes it much easier to install packages based on C/C++ libraries I hear this often, though I cannot remember ever running into a pip package where this was an issue. Out of curiosity, could someone point me to a pip package and its conda equivalent where this is the case?
  - jhayward 9y ago
    Install scipy on Windows and include the Intel linear matrix libraries.
    - akx 9y ago
      The numpy+mkl wheel is available on Gohlke's site.
  - _coveredInBees 9y ago
    Try pip installing scipy or Numpy and you'll see the value of conda.
    - zeptomu 9y ago
      Although I mostly use the system's numpy package (usually shipped with the distribution), I just tried installing it via pip and had no problem (obviously just one data point).
closed 9y ago
A few years ago Anaconda helped overcome install issues with some packages, but I almost never run into those problems anymore. One thing to be aware of is that Anaconda modifies various Jupyter configs / installs some of its own kernels. So it can be a hassle to get back to system python + plain jupyter. https://github.com/jupyter/notebook/issues/1630 https://github.com/jupyter/notebook/issues/1630
adamson 9y ago
None of these should conflict with Anaconda (or at least Continuum conda, I'm not that familiar with what else is out there) unless you put the conda root bin directory on your path.
natch 9y ago
Thanks for the replies everybody. One thing that's still confusing to Python tourists (my word for myself since I am usually programming in a different language, but come to Python occasionally to do something) is that everyone talks about pip, when actually it seems pip3 is required to install when using Python 3. Is this no longer the case? Or do people just say "pip" when they mean "pip3"? Or are people actually still living in 2.x land? I'm talking about pip the command line command/binary executable file, not pip the concept / tool name. Similar to the distinction between the capitalized "Python" (name of the language) versus "python" (command entered on the command line/name of the binary on the system).
- mixmastamyk 9y ago
  Any of those could be the case. Guess it's understood that you will put a 2 or 3 on the end of the pip when you want to choose one or the other. And it goes further: /usr/local/bin/pip* /usr/local/bin/pip2* /usr/local/bin/pip2.7* /usr/local/bin/pip3* /usr/local/bin/pip3.5* /usr/local/bin/pip3.6*

by natch 9y ago

sreevisakh 9y ago

Just in case you wish to retain anaconda, you could simply remove anaconda path from your PATH variable in .bashrc file (Assuming you are on Linux). If you need anaconda again, you could add the path to the variable, or run `source ~/anaconda3/bin/activate`

guitarbill 9y ago

I think you could probably use both, side-by-side (but not at the same time/for the same project). Anaconda might be nicer if the packages you need are C-based and would need compiling on your platform. Depends on your use-case. For experimentation/playing around, stick with what works for you. But if a project required me to install Anaconda, I wouldn’t take it seriously. And pipenv is pretty easy to use, too. So if you plan on distributing it or open-sourcing it, understanding how most other people manage dependencies ouside of conda is going to be useful.

stared 9y ago
Why not seriously? Many data science environments are pretty much based on Anaconda installations. (See e.g. these dockers: https://github.com/jupyter/docker-stacks/tree/master/scipy-notebook https://github.com/jupyter/docker-stacks/tree/master/scipy-n...) I mean, all packages I know DONT require Anaconda. But if you need the whole environment, sometimes Anaconda is the easiest tool too install all dependencies.
- guitarbill 9y ago
  > Anaconda is the easiest tool too install all dependencies. For hobbyist stuff that’s fine, and I applaud lowering the entry barrier. My issue would be if I needed Anaconda to deploy the project/dependencies into “prod” in some professional capacity, instead of standard Python build tools. Having said that, I’m not terribly familiar with Anaconda, and it seems to leverage virtualenv under the hood, possibly with pre-compiled packages (like wheels?).
  - stared 9y ago
    "Hobbyist"? It's widely used in research in data science & machine learning. If you want a setup which works on various systems, and uses Python numeric packages, usually it is the most failsafe way to use with various OS. (Unless you want to put everything in docker.) Unless by "professional" you mean "building", well - then you have a point.
    - Fomite 9y ago
      Yeah. Unless "Hobbyist" now also means my research university's 104 node, 2548 core HPC cluster...
  - pwang 9y ago
    Anaconda does not leverage virtualenv under the hood. Anaconda does not use wheels as part of that. However, you can use pip and virtualenvs within Anaconda, but you'll probably get some inconsistent results. The reason that Anaconda exists is to make it easier for people to have consistency between their dev environment, the build environment, and the prod environment. If you're seriously going to build all of the scientific Python stack from scratch, correctly and optimally, for your prod environment (and then also do the same for every dev environment you need to support), then you have way too much time and probably haven't actually tried doing it over any real period of time. The problem isn't that it's impossible. The problem is that there are dozens of different ways to almost succeed, and you don't run into the problems until way too late.
    - 9y ago

beagle3 9y ago

What do you think you'll get from any if these tools that you do not get from {ana,mini}conda? I must be missing something, but all these use cases (and more) seem to have been covered by miniconda ages ago. conda works with perfectly well with pip, and creates virtual environments that are at least (in my experience) as good as virtualenv does.

james_the_cat 9y ago

Anaconda does most of the stuff mentioned, and also makes it much easier to install packages based on C/C++ libraries (which most deep learning things are). So you're better off staying with anaconda. It's widely used in commercial data science projects so the idea that noone "takes it seriously" as someone else suggests is a bit silly. I assume they're thinking about a different context to data science projects. That said anaconda does have a whole variety of extremely annoying quirks, like packages not being backwards compatible with old versions of conda, or conda going crazy and reinstalling itself, or the way the conda-forge repo has far more packages than the official conda repo. It's very far from perfect. But for data science I think it's basically the standard package manager in Python land.

Rotareti 9y ago
> ... and also makes it much easier to install packages based on C/C++ libraries I hear this often, though I cannot remember ever running into a pip package where this was an issue. Out of curiosity, could someone point me to a pip package and its conda equivalent where this is the case?
- jhayward 9y ago
  Install scipy on Windows and include the Intel linear matrix libraries.
  - akx 9y ago
    The numpy+mkl wheel is available on Gohlke's site.
- _coveredInBees 9y ago
  Try pip installing scipy or Numpy and you'll see the value of conda.
  - zeptomu 9y ago
    Although I mostly use the system's numpy package (usually shipped with the distribution), I just tried installing it via pip and had no problem (obviously just one data point).

closed 9y ago

A few years ago Anaconda helped overcome install issues with some packages, but I almost never run into those problems anymore. One thing to be aware of is that Anaconda modifies various Jupyter configs / installs some of its own kernels. So it can be a hassle to get back to system python + plain jupyter. https://github.com/jupyter/notebook/issues/1630 https://github.com/jupyter/notebook/issues/1630

adamson 9y ago

None of these should conflict with Anaconda (or at least Continuum conda, I'm not that familiar with what else is out there) unless you put the conda root bin directory on your path.

natch 9y ago

Thanks for the replies everybody. One thing that's still confusing to Python tourists (my word for myself since I am usually programming in a different language, but come to Python occasionally to do something) is that everyone talks about pip, when actually it seems pip3 is required to install when using Python 3. Is this no longer the case? Or do people just say "pip" when they mean "pip3"? Or are people actually still living in 2.x land? I'm talking about pip the command line command/binary executable file, not pip the concept / tool name. Similar to the distinction between the capitalized "Python" (name of the language) versus "python" (command entered on the command line/name of the binary on the system).

mixmastamyk 9y ago
Any of those could be the case. Guess it's understood that you will put a 2 or 3 on the end of the pip when you want to choose one or the other. And it goes further: /usr/local/bin/pip* /usr/local/bin/pip2* /usr/local/bin/pip2.7* /usr/local/bin/pip3* /usr/local/bin/pip3.5* /usr/local/bin/pip3.6*