5 ms·

I'm surprised the article doesn't mention Anaconda, which is Python with all the things he lists pre-installed for you. I've been a fan for some time now: https

I'm surprised the article doesn't mention Anaconda, which is Python with all the things he lists pre-installed for you. I've been a fan for some time now: https://www.continuum.io/why-anaconda https://www.continuum.io/why-anaconda

alceufc 11y ago
I also think that Anaconda is great. However, I hope that in the future we could install numpy, matplotlib, jupyter, etc. just using pip.
- stared 11y ago
  I like Anaconda (and I recommend it as the easiest installation for data sci), but on OS X it is easy to install Python and relevant numerical packages with Homebrew and pip.
- zo1 11y ago
  I did that earlier today: pip install jupyter pip install numpy pip install scipy pip install scikit-learn pip install matplotlib The only problem I had was with OpenCV, which requires manual make installation if you want the contrib package. The other problem was when trying to install scikit-learn, it requires manual pip installation of scipy.
  - yeukhon 11y ago
    The reason you can't is because of the C libraries levaaged must be installed prior to install numpy and scipy. For example, you can't get through PyYaml unless python-dev is installed on Ubuntu. I am not sure if wheel would fix it but I don't think so.
    - takeda 11y ago
      Wheel is a binary distribution, so files would already be compiled and therefore python-dev would not be needed anymore.
      - yeukhon 11y ago
        Is wheel not default? Or play it elementary :-) how would I avoid this issue in the future?
        takeda 11y ago
        I did not write anything worthy to send to PIP so don't know exactly but looks like it is up to the developers[1] Anyway I just noticed that PyPI only supports binary packages for Windows and Mac OS X. Although, you could still generate wheels of packages that you use by using something like this: pip wheel -r requirements.txt You can then install them with pip install <file> or (unfortunately I forgot the option, perhaps it was -i) you can use an option to point to a directory containing wheels and pip install to install the main package. It should use all dependencies in that directory as well. [1] http://pythonwheels.com/ http://pythonwheels.com/
        techdragon 11y ago
        the issue with Wheels on Linux, and this is very much a Linux problem, is that the 'pre built' nature of libraries in wheels doesnt play nice with the raw chaos of Linux's package management + distro + kernel ecosystem. I raised the question of FreeBSD and Solaris based wheels at a PyCon when I was in a face to face discussion with someone more knowledgeable, and the answer was 'in theory that should work like Windows & OSX, no one has done the hard work yet.' So yeah Linux is not the most friendly environment for Python Wheels.
  - timClicks 11y ago
    I would always be worried with installing numpy/scipy via pip that I wouldn't be linking to BLAS/LAPACK correctly.
    - rhodysurf 11y ago
      What do you mean? I have always installed via pip with zero errors on all three platforms.
      - moyix 11y ago
        The worry is not that it will throw an error, it's that it will silently link against lower-performance math libraries and make your code inexplicably slow.
- drauh 11y ago
  Currently, pip install of numpy (and scipy) does not pick up MKL, which you would want if you do lots of linear algebra and FFTs.
- p4wnc6 11y ago
  conda supports pip installation within the context of an environment (much more gracefully, in fact, than does virtualenv). So you're not giving anything up by using conda in this regard. I don't see the desire to have pip as the baseline. For me, the conda packaging is much more informative and placing everything you need for multiplatform support into an /info directory with a meta.yaml is a lot more effective than going through the steps of PyPI. conda also makes uploading and hosting on anaconda.org extremely easy. Normally there is the whole "gee, I don't want to learn another package manager" -- but conda / anaconda.org is extremely worth it. It really is a major engineering step forward from the existing package deployment strategies in Python. I even configure my travis.yml CI scripts to download Miniconda, create a conda environment from a requiremenets.txt, and then build and test my code via conda on the contiguous integration VM itself. The only worry is how strongly tied conda and anaconda.org are to the future of Continuum. Given how much Continuum speaks of open-source work, one would hope that these projects essentially live independently (or that forks of them would) but you never know. I do admit that is a major downside.
deleted 11y ago
[deleted]
smortaz 11y ago
Another data point on Anaconda from microsoft's Azure ML team - we've been using Anaconda for the past few years and have been very happy with it. One key feature we like is that we can deploy a nearly identical suite of pkgs on Windows, Linux and MacOS w/o having to worry about mismatched versions, installation/update issues, etc. Case in point - the Azure ML Jupyter notebooks run on a farm of Linux/docker machines: https://blogs.technet.microsoft.com/machinelearning/2016/02/10/python-notebooks-in-the-cortana-analytics-gallery/ https://blogs.technet.microsoft.com/machinelearning/2016/02/... Once a model is built, it can be deployed for production/scale to the Azure ML backend which runs entirely on a farm of Windows machines. Both environments have matching Anaconda distros running underneath which makes running on two different OS's practically a non-issue. In fact 95% of our users probably have no idea their notebooks & production site run two OS's (in large part thanks to Python, Linux, Docker, Anaconda, ... and lots of other great open source software).
ktamura 11y ago
I totally echo this. In my opinion, if Python was the Linux Kernel, Anaconda would be Ubuntu: i.e., it ensures that you are actually up and running within minutes and not struggling in the missing-header-files or dependencies-management hell. Of course, there's time and place for eschewing Anaconda, especially if you are trying to ensure your installation and dependencies are minimal. That said, for most data scientists who are workaday *NIX users at best, Anaconda is such a productivity boost. EDIT: Actually, my analogy needs further qualifications. You _can_ get into dependencies/missing-header-files hell in Ubuntu, but it usually doesn't happen in the first few hour/days. Both Ubuntu and Anaconda have great "first 5 minutes to 5 days" experience, and that goes a long way in ensuring the adoption of underlying technology.
a3n 11y ago
I have anaconda installed at work/windows and home/linux. One minor annoyance is that it doesn't play well (at all?) with virtualenv. They worked so hard to get a huge suite of off the shelf third party libraries integrated, it's curious that they would come up with a replacement for virtualenv. I'm halfway hoping that I don't know what I'm talking about.
- spdustin 11y ago
  http://conda.pydata.org/docs/using/envs.html http://conda.pydata.org/docs/using/envs.html Glad to help :)
  - a3n 11y ago
    Right, I know about that. What I'm pointing out is that anaconda doesn't support http://docs.python-guide.org/en/latest/dev/virtualenvs/ http://docs.python-guide.org/en/latest/dev/virtualenvs/ It's nice that they have an env tool, but why didn't they just use the existing virtualenv?
    - spdustin 11y ago
      Because using theirs means not having to recompile numpy (for example) with each environment. Why doesn't anaconda's env work for you?
      - a3n 11y ago
        Not saying it doesn't work, just wondering why the duplication of functionality. Which you answered, thanks.
        spdustin 11y ago
        conda env also installs other dependencies for code. Really, the whole point of it all is that conda replaces pip and virtualenv. Conda manages more than just Python packages, but other dependencies as well. For example, here's list of packages in a new environment I created with $ conda create -n hn python $ source activate hn $ conda list -c openssl-1.0.2g-0 pip-8.1.1-py35_0 python-3.5.1-0 readline-6.2-2 setuptools-20.3-py35_0 sqlite-3.9.2-0 tk-8.5.18-0 wheel-0.29.0-py35_0 xz-5.0.5-1 zlib-1.2.8-0 setuptools and wheel are the only Python packages installed. The others are non-Python packages that conda is managing for my environment. For all the talk of vendoring in package manager tools these days, I really, really like the way conda manages this stuff.

alceufc 11y ago

I also think that Anaconda is great. However, I hope that in the future we could install numpy, matplotlib, jupyter, etc. just using pip.

stared 11y ago
I like Anaconda (and I recommend it as the easiest installation for data sci), but on OS X it is easy to install Python and relevant numerical packages with Homebrew and pip.
zo1 11y ago
I did that earlier today: pip install jupyter pip install numpy pip install scipy pip install scikit-learn pip install matplotlib The only problem I had was with OpenCV, which requires manual make installation if you want the contrib package. The other problem was when trying to install scikit-learn, it requires manual pip installation of scipy.
- yeukhon 11y ago
  The reason you can't is because of the C libraries levaaged must be installed prior to install numpy and scipy. For example, you can't get through PyYaml unless python-dev is installed on Ubuntu. I am not sure if wheel would fix it but I don't think so.
  - takeda 11y ago
    Wheel is a binary distribution, so files would already be compiled and therefore python-dev would not be needed anymore.
    - yeukhon 11y ago
      Is wheel not default? Or play it elementary :-) how would I avoid this issue in the future?
      - takeda 11y ago
        I did not write anything worthy to send to PIP so don't know exactly but looks like it is up to the developers[1] Anyway I just noticed that PyPI only supports binary packages for Windows and Mac OS X. Although, you could still generate wheels of packages that you use by using something like this: pip wheel -r requirements.txt You can then install them with pip install <file> or (unfortunately I forgot the option, perhaps it was -i) you can use an option to point to a directory containing wheels and pip install to install the main package. It should use all dependencies in that directory as well. [1] http://pythonwheels.com/ http://pythonwheels.com/
        techdragon 11y ago
        the issue with Wheels on Linux, and this is very much a Linux problem, is that the 'pre built' nature of libraries in wheels doesnt play nice with the raw chaos of Linux's package management + distro + kernel ecosystem. I raised the question of FreeBSD and Solaris based wheels at a PyCon when I was in a face to face discussion with someone more knowledgeable, and the answer was 'in theory that should work like Windows & OSX, no one has done the hard work yet.' So yeah Linux is not the most friendly environment for Python Wheels.
- timClicks 11y ago
  I would always be worried with installing numpy/scipy via pip that I wouldn't be linking to BLAS/LAPACK correctly.
  - rhodysurf 11y ago
    What do you mean? I have always installed via pip with zero errors on all three platforms.
    - moyix 11y ago
      The worry is not that it will throw an error, it's that it will silently link against lower-performance math libraries and make your code inexplicably slow.
drauh 11y ago
Currently, pip install of numpy (and scipy) does not pick up MKL, which you would want if you do lots of linear algebra and FFTs.
p4wnc6 11y ago
conda supports pip installation within the context of an environment (much more gracefully, in fact, than does virtualenv). So you're not giving anything up by using conda in this regard. I don't see the desire to have pip as the baseline. For me, the conda packaging is much more informative and placing everything you need for multiplatform support into an /info directory with a meta.yaml is a lot more effective than going through the steps of PyPI. conda also makes uploading and hosting on anaconda.org extremely easy. Normally there is the whole "gee, I don't want to learn another package manager" -- but conda / anaconda.org is extremely worth it. It really is a major engineering step forward from the existing package deployment strategies in Python. I even configure my travis.yml CI scripts to download Miniconda, create a conda environment from a requiremenets.txt, and then build and test my code via conda on the contiguous integration VM itself. The only worry is how strongly tied conda and anaconda.org are to the future of Continuum. Given how much Continuum speaks of open-source work, one would hope that these projects essentially live independently (or that forks of them would) but you never know. I do admit that is a major downside.

deleted 11y ago

[deleted]

smortaz 11y ago

Another data point on Anaconda from microsoft's Azure ML team - we've been using Anaconda for the past few years and have been very happy with it. One key feature we like is that we can deploy a nearly identical suite of pkgs on Windows, Linux and MacOS w/o having to worry about mismatched versions, installation/update issues, etc. Case in point - the Azure ML Jupyter notebooks run on a farm of Linux/docker machines: https://blogs.technet.microsoft.com/machinelearning/2016/02/10/python-notebooks-in-the-cortana-analytics-gallery/ https://blogs.technet.microsoft.com/machinelearning/2016/02/... Once a model is built, it can be deployed for production/scale to the Azure ML backend which runs entirely on a farm of Windows machines. Both environments have matching Anaconda distros running underneath which makes running on two different OS's practically a non-issue. In fact 95% of our users probably have no idea their notebooks & production site run two OS's (in large part thanks to Python, Linux, Docker, Anaconda, ... and lots of other great open source software).

ktamura 11y ago

I totally echo this. In my opinion, if Python was the Linux Kernel, Anaconda would be Ubuntu: i.e., it ensures that you are actually up and running within minutes and not struggling in the missing-header-files or dependencies-management hell. Of course, there's time and place for eschewing Anaconda, especially if you are trying to ensure your installation and dependencies are minimal. That said, for most data scientists who are workaday *NIX users at best, Anaconda is such a productivity boost. EDIT: Actually, my analogy needs further qualifications. You _can_ get into dependencies/missing-header-files hell in Ubuntu, but it usually doesn't happen in the first few hour/days. Both Ubuntu and Anaconda have great "first 5 minutes to 5 days" experience, and that goes a long way in ensuring the adoption of underlying technology.

a3n 11y ago

I have anaconda installed at work/windows and home/linux. One minor annoyance is that it doesn't play well (at all?) with virtualenv. They worked so hard to get a huge suite of off the shelf third party libraries integrated, it's curious that they would come up with a replacement for virtualenv. I'm halfway hoping that I don't know what I'm talking about.

spdustin 11y ago
http://conda.pydata.org/docs/using/envs.html http://conda.pydata.org/docs/using/envs.html Glad to help :)
- a3n 11y ago
  Right, I know about that. What I'm pointing out is that anaconda doesn't support http://docs.python-guide.org/en/latest/dev/virtualenvs/ http://docs.python-guide.org/en/latest/dev/virtualenvs/ It's nice that they have an env tool, but why didn't they just use the existing virtualenv?
  - spdustin 11y ago
    Because using theirs means not having to recompile numpy (for example) with each environment. Why doesn't anaconda's env work for you?
    - a3n 11y ago
      Not saying it doesn't work, just wondering why the duplication of functionality. Which you answered, thanks.
      - spdustin 11y ago
        conda env also installs other dependencies for code. Really, the whole point of it all is that conda replaces pip and virtualenv. Conda manages more than just Python packages, but other dependencies as well. For example, here's list of packages in a new environment I created with $ conda create -n hn python $ source activate hn $ conda list -c openssl-1.0.2g-0 pip-8.1.1-py35_0 python-3.5.1-0 readline-6.2-2 setuptools-20.3-py35_0 sqlite-3.9.2-0 tk-8.5.18-0 wheel-0.29.0-py35_0 xz-5.0.5-1 zlib-1.2.8-0 setuptools and wheel are the only Python packages installed. The others are non-Python packages that conda is managing for my environment. For all the talk of vendoring in package manager tools these days, I really, really like the way conda manages this stuff.