5 ms·

You could alternately use conda, as much of the scientific computing community does: http://conda.pydata.org/docs/ http://conda.pydata.org/docs/

ForHackernews 11y ago
No, no no. No 'alternately'. The python community needs to pick ONE standard and, uh...standardize on it.
- dvanduzer 11y ago
  This isn't the issue. Wheels and Conda are almost completely orthogonal. Conda is a higher level abstraction than pip. Wheels is operating at the lowest level of abstraction. Multiple package management tools are relatively fine. Having multiple competing format standards (egg vs wheel vs lots of other options) would be a disaster. (For reference, there's .deb versus .rpm, but also dpkg versus apt-get versus yum versus up2date versus...)
  - shoo 11y ago
    From a continuum.io blog post about Conda [1]: > Having been involved in the python world for so long, we are all aware of pip, easy_install, and virtualenv, but these tools did not meet all of our specific requirements. The main problem is that they are focused around Python, neglecting non-Python library dependencies, such as HDF5, MKL, LLVM, etc., which do not have a setup.py in their source code and also do not install files into Python’s site-packages directory. I haven't used Conda, but I totally get their main point here. Python packages sometimes depend upon non-Python packages. Any Python-specific packaging solution that cannot express that e.g. a Python package depends upon a native library does not really solve this problem. I don't regard packaging, dependency management, and isolation as programming-language-dependent problems. [1] http://www.continuum.io/blog/conda http://www.continuum.io/blog/conda
    - dvanduzer 11y ago
      I haven't evaluated Conda either, but the wheel vs egg distinction is categorically different from managing build chains for huge tools like Hadoop.
    - ngoldbaum 11y ago
      Replace "sometimes" with "nearly always" in the scientific world.
      - IanOzsvald 11y ago
        +1 that conda solves "most of our packaging problems" in the science/data science world (I've lived within it for 1.5 years, it really simplifies things). I use conda on Linux, Mac and Windows for r&d and deployment for tools including numpy, sklearn, matplotlib along with flask, pymongo etc.
    - e12e 11y ago
      Conda certainly sounds ambitious -- but I'm doubtful how useful it really is. As I see it, it mainly fills two needs: the lack of a package manager on OS X and the lack of a package manager on Windows. Windows is getting a package manager now, and OS X arguably have the App Store. Everywhere else (read: Linux/BSD) -- most of what's packaged by Conda, is already packaged by the vendor. In short: I don't think it's the best way forward (but it may very well be the best way for many usecases/users right now and in the near future).
      - shoo 11y ago
        I agree. There's probably also some element of "it isn't the best general solution, but it's super convenient for some particular group of people, e.g. python users".
      - cossatot 11y ago
        A big part of Conda's usefulness is not system-wide package management a la apt or yum or whatever, but being able to have reproducible, isolated environments for scientific computing or other times when reproducibility is very important, and/or when you need multiple versions of packages for compatibility with other packages. I believe it uses virtualenv for this, but I may be wrong. Regardless, having this feature (as well as no interference with the system Python) is one of the killer features for me, and others I'm sure.
- analog31 11y ago
  As I understand it, it's still the same standard. The distributions that I've used, unpack themselves to look just like a mainstream installation, including pip. So, if you have to install a missing package, you still do so in the same way. The distributions just give you a head start.
- ubernostrum 11y ago
  There are at least two major classes of people using Python. On the one hand, there are people who write, distribute and deploy Unix-y-type software -- system utilities, web applications, service daemons, etc. These users write for users who are like themselves, or who are sysadmins of Unix-y systems. Their concern is with automatable, repeatable, command-line-friendly packaging, distribution and deployment. They benefit from wheels because it saves them a potentially-time-consuming build step during installation, and will generally prefer wheels or tarballs as a distribution format. On the other hand, there are people who use Python as a scientific computing and number-crunching stack. Many of them do not write and distribute their own applications; they distribute the results of their code. Many of them run on desktop-type machines using Windows. What they primarily want is a single .exe they can download and run which will provide Python plus all the numeric/scientific libraries in one install, plus (increasingly) IPython Notebook as a way to share their work. There is no single standard which works well for both of these groups.
  - e12e 11y ago
    > There is no single standard which works well for both of these groups. Sure there is. A single exe that bootstraps python and leverage wheels to install what the scientific user needs. In fact, with so many new tools comming to windows lately (as in the past decade) -- that exe could probably be a powershell-script. And an equivalent shell-script could probably do the same for more unix-y platforms. Or one might make a script that builds an installer that bundles up things downloaded via the wheels framework.
- pekk 11y ago
  I'm sorry if things aren't working well for you. The Python community already did pick a standard, which is PyPI, pip and wheels. All this is covered in PEPs. Unfortunately, the creators of Conda present their own tool to compete with the standard with a complete fork of all of Python's packaging, right down to virtualenv, which really only works for the packages that they wanted to support. That isn't the fault of the broader Python community. There is nothing anyone else can do to prevent that.
ngoldbaum 11y ago
Why not both (at least on OSX and Windows)? Build packages where the users are. Even in the scientific space, many of your users build packages using pip.
analog31 11y ago
The packaged distributions are definitely a blessing, when introducing beginners to Python. Even when package installation goes swimmingly, it still takes effort. And it's handy to keep just one big installer on a flash drive for putting Python on non-networked computers. Since I use Python for lab work, I often have it running on several computers at once.
wclax04 11y ago
Unless you want things compiled with SSL, then you're kind of hosed at the moment.
- infinite8s 11y ago
  Which packages?
  - wclax04 11y ago
    psycopg2
lqdc13 11y ago
You can't use conda in many (most?) cases. Specifically, there aren’t conda packages for every Python package. And if there are, they are behind their pip alternatives in many cases. Sometimes you then need to fall back to pip. However, it is impossible to install some packages into conda environments via pip. So basically the whole solution falls apart because the python libs you install via pip can't find C libraries installed via conda. Edit: One example is: https://github.com/ContinuumIO/anaconda-issues/issues/177 https://github.com/ContinuumIO/anaconda-issues/issues/177 Their solution was to include openssl into conda. Clearly, this solution doesn't generalize to other libs. Just things that use openssl.
acdha 11y ago
Can you expand on why what makes Conda better?
hogu 11y ago
I want to address several of the comments. I used to work for continuum, but I no longer do. I didn't work on the conda team, but as a part of the other teams at continuum, and at my new gig I'm a heavy conda user 1. linux package management is sufficient - this isn't the case. linux package management couples your library versions with your distro verison. Meaning if I want to do something like try out my code with the latest pandas, or run 2 jobs with different versions of numpy I'm out of luck 2. conda went ahead and forked everything without regard for the community - this is completely untrue. We sat down with guido for the very first pydata at google offices in 2012 to discuss many issues, packaging being one of them. Guido acknowledged that the python packaging ecosystem isn't sufficient for the scientific community, and we should go out and solve the problem ourselves - so we did. Honestly on this point, you should just read travis' words about the issue http://technicaldiscovery.blogspot.sg/2013/12/why-i-promote-conda.html http://technicaldiscovery.blogspot.sg/2013/12/why-i-promote-... Scientific users need to be able to package non-python libs along with python libs (think zmq, or libhdf5, or R) we need a pkg mgmt solution that sits OUTSIDE of python for this reason. You can think of conda as a little in between place between virtualenv and docker

ForHackernews 11y ago

No, no no. No 'alternately'. The python community needs to pick ONE standard and, uh...standardize on it.

dvanduzer 11y ago
This isn't the issue. Wheels and Conda are almost completely orthogonal. Conda is a higher level abstraction than pip. Wheels is operating at the lowest level of abstraction. Multiple package management tools are relatively fine. Having multiple competing format standards (egg vs wheel vs lots of other options) would be a disaster. (For reference, there's .deb versus .rpm, but also dpkg versus apt-get versus yum versus up2date versus...)
- shoo 11y ago
  From a continuum.io blog post about Conda [1]: > Having been involved in the python world for so long, we are all aware of pip, easy_install, and virtualenv, but these tools did not meet all of our specific requirements. The main problem is that they are focused around Python, neglecting non-Python library dependencies, such as HDF5, MKL, LLVM, etc., which do not have a setup.py in their source code and also do not install files into Python’s site-packages directory. I haven't used Conda, but I totally get their main point here. Python packages sometimes depend upon non-Python packages. Any Python-specific packaging solution that cannot express that e.g. a Python package depends upon a native library does not really solve this problem. I don't regard packaging, dependency management, and isolation as programming-language-dependent problems. [1] http://www.continuum.io/blog/conda http://www.continuum.io/blog/conda
  - dvanduzer 11y ago
    I haven't evaluated Conda either, but the wheel vs egg distinction is categorically different from managing build chains for huge tools like Hadoop.
  - ngoldbaum 11y ago
    Replace "sometimes" with "nearly always" in the scientific world.
    - IanOzsvald 11y ago
      +1 that conda solves "most of our packaging problems" in the science/data science world (I've lived within it for 1.5 years, it really simplifies things). I use conda on Linux, Mac and Windows for r&d and deployment for tools including numpy, sklearn, matplotlib along with flask, pymongo etc.
  - e12e 11y ago
    Conda certainly sounds ambitious -- but I'm doubtful how useful it really is. As I see it, it mainly fills two needs: the lack of a package manager on OS X and the lack of a package manager on Windows. Windows is getting a package manager now, and OS X arguably have the App Store. Everywhere else (read: Linux/BSD) -- most of what's packaged by Conda, is already packaged by the vendor. In short: I don't think it's the best way forward (but it may very well be the best way for many usecases/users right now and in the near future).
    - shoo 11y ago
      I agree. There's probably also some element of "it isn't the best general solution, but it's super convenient for some particular group of people, e.g. python users".
    - cossatot 11y ago
      A big part of Conda's usefulness is not system-wide package management a la apt or yum or whatever, but being able to have reproducible, isolated environments for scientific computing or other times when reproducibility is very important, and/or when you need multiple versions of packages for compatibility with other packages. I believe it uses virtualenv for this, but I may be wrong. Regardless, having this feature (as well as no interference with the system Python) is one of the killer features for me, and others I'm sure.
analog31 11y ago
As I understand it, it's still the same standard. The distributions that I've used, unpack themselves to look just like a mainstream installation, including pip. So, if you have to install a missing package, you still do so in the same way. The distributions just give you a head start.
ubernostrum 11y ago
There are at least two major classes of people using Python. On the one hand, there are people who write, distribute and deploy Unix-y-type software -- system utilities, web applications, service daemons, etc. These users write for users who are like themselves, or who are sysadmins of Unix-y systems. Their concern is with automatable, repeatable, command-line-friendly packaging, distribution and deployment. They benefit from wheels because it saves them a potentially-time-consuming build step during installation, and will generally prefer wheels or tarballs as a distribution format. On the other hand, there are people who use Python as a scientific computing and number-crunching stack. Many of them do not write and distribute their own applications; they distribute the results of their code. Many of them run on desktop-type machines using Windows. What they primarily want is a single .exe they can download and run which will provide Python plus all the numeric/scientific libraries in one install, plus (increasingly) IPython Notebook as a way to share their work. There is no single standard which works well for both of these groups.
- e12e 11y ago
  > There is no single standard which works well for both of these groups. Sure there is. A single exe that bootstraps python and leverage wheels to install what the scientific user needs. In fact, with so many new tools comming to windows lately (as in the past decade) -- that exe could probably be a powershell-script. And an equivalent shell-script could probably do the same for more unix-y platforms. Or one might make a script that builds an installer that bundles up things downloaded via the wheels framework.
pekk 11y ago
I'm sorry if things aren't working well for you. The Python community already did pick a standard, which is PyPI, pip and wheels. All this is covered in PEPs. Unfortunately, the creators of Conda present their own tool to compete with the standard with a complete fork of all of Python's packaging, right down to virtualenv, which really only works for the packages that they wanted to support. That isn't the fault of the broader Python community. There is nothing anyone else can do to prevent that.

ngoldbaum 11y ago

Why not both (at least on OSX and Windows)? Build packages where the users are. Even in the scientific space, many of your users build packages using pip.

analog31 11y ago

The packaged distributions are definitely a blessing, when introducing beginners to Python. Even when package installation goes swimmingly, it still takes effort. And it's handy to keep just one big installer on a flash drive for putting Python on non-networked computers. Since I use Python for lab work, I often have it running on several computers at once.

wclax04 11y ago

Unless you want things compiled with SSL, then you're kind of hosed at the moment.

infinite8s 11y ago
Which packages?
- wclax04 11y ago
  psycopg2

lqdc13 11y ago

You can't use conda in many (most?) cases. Specifically, there aren’t conda packages for every Python package. And if there are, they are behind their pip alternatives in many cases. Sometimes you then need to fall back to pip. However, it is impossible to install some packages into conda environments via pip. So basically the whole solution falls apart because the python libs you install via pip can't find C libraries installed via conda. Edit: One example is: https://github.com/ContinuumIO/anaconda-issues/issues/177 https://github.com/ContinuumIO/anaconda-issues/issues/177 Their solution was to include openssl into conda. Clearly, this solution doesn't generalize to other libs. Just things that use openssl.

acdha 11y ago

Can you expand on why what makes Conda better?

hogu 11y ago

I want to address several of the comments. I used to work for continuum, but I no longer do. I didn't work on the conda team, but as a part of the other teams at continuum, and at my new gig I'm a heavy conda user 1. linux package management is sufficient - this isn't the case. linux package management couples your library versions with your distro verison. Meaning if I want to do something like try out my code with the latest pandas, or run 2 jobs with different versions of numpy I'm out of luck 2. conda went ahead and forked everything without regard for the community - this is completely untrue. We sat down with guido for the very first pydata at google offices in 2012 to discuss many issues, packaging being one of them. Guido acknowledged that the python packaging ecosystem isn't sufficient for the scientific community, and we should go out and solve the problem ourselves - so we did. Honestly on this point, you should just read travis' words about the issue http://technicaldiscovery.blogspot.sg/2013/12/why-i-promote-conda.html http://technicaldiscovery.blogspot.sg/2013/12/why-i-promote-... Scientific users need to be able to package non-python libs along with python libs (think zmq, or libhdf5, or R) we need a pkg mgmt solution that sits OUTSIDE of python for this reason. You can think of conda as a little in between place between virtualenv and docker