6 ms·

One solution is to use Ana / Miniconda. If you are not that tech savvy, if you don't mind downloading several hundred megabytes you can get away with this easil

by ericol 6y ago

One solution is to use Ana / Miniconda. If you are not that tech savvy, if you don't mind downloading several hundred megabytes you can get away with this easily. In my case, and contrary to what one of the other commentators said, I'm using docker. It took a while to get it ironed out, but I use a "base" folder where I have already downloaded all the packages I use by default (Top of my mind are pandas, numpy and youtube-dl ;) ) I have all the related configuration in an env file that tells pip to save the packages in the base folder so they don't disappear when I shut down the container - that I always run with --rm so they get removed when dead - and for creating a new env I just have to copy that folder. As I use a base folder for all of this I don't need to remember the names of the envs, as just need to list folders. Only downside to this - because I'm lazy and it still hasn't bothered me enough to fix it - is that new files & folders are owned by root. I use just a base image with python, and for different versions that work with this - supposedly - I just have to download a different image.

theelous3 6y ago
As someone who's been hanging out in freenode python channels for years helping noobs, I recommend strongly against any kind of conda setup for new programmers. It often has strange issues that noobs have difficulty even understanding how to ask for help with, nevermind actually solve. Learning how to use a simple barebones venv is extremely easy, saves a ton of time both in the short and long run, and generalises better. pip install virtualenv cd your/project/location which python virtualenv -p result_of_which_python env source env/bin/activate pip install anything_you_like and do whatever you want from there. Those commands get 100% of the basics out of the way for noobs, and cover like 90% of the stuff you use to do more serious stuff. > but I use a "base" folder where I have already downloaded all the packages I use by default [...etc...] Your setup sounds outrageously complex, and I don't understand why you would do any of that.
- aldanor 6y ago
  I would strongly recommend to use conda because: - It would work the same way on Windows, Linux and macOS - Most importantly (!), it is not a Python package manager. That is, if a package needs BLAS or MKL or HDF5 or whatever else, you won't be crossing fingers and hoping your system-installed version would work for all your venvs; instead, those binary libraries are properly managed per environment. Pro tip: use mamba instead of conda to get a free 4x speed boost.
  - theelous3 6y ago
    Ah yes, all of those noobs who need weird numpy extensions and highly domain oriented and specific odd data storage formats. That's the generalisation part I mentioned. The distinction you're making is non-trival for a wee noobie. Better they get the ground work in and expand to where they need to go.
    - aldanor 6y ago
      One of the things most noobs will install first is pandas. Pandas needs numpy. Numpy needs lapack, blas etc, even if they're not aware it. Yes, there's wheels, but that's limited to python-only stuff. Then they're reading on some blog about protobuf, or hdf5, or arrow or whatever, and want to use it - but using either from Python needs installed C libraries. On linux/macos you'll need to dig into your package managers and then hope things are compatible, on Windows it's a complete pain. In conda, they can just 'conda install h5py' or whatever, and get proper hdf5 installed without having to figure out the nitty gritty details.
      - theelous3 6y ago
        > most noobs Well, already I can tell you're disconnected from what "most noobs" are actually like. Presumably you think "most noobs" are data scientists, which just isn't the case in my experience.
        aldanor 6y ago
        > Presumably you think "most noobs" are data scientists, which just isn't the case in my experience It is the case in my personal experience though. I don't think I'm disconnected from what 'noobs' are as I've taught a lot of them through my line of work. Assume you're a noob and are exploring Python package universe, fire a new private tab and google 'most popular python packages'; in my case the first 3 links are: - https://www.activestate.com/blog/top-10-must-have-python-packages/ https://www.activestate.com/blog/top-10-must-have-python-pac... - https://www.ubuntupit.com/best-python-libraries-and- https://www.ubuntupit.com/best-python-libraries-and- packages-for-beginners/ - https://www.edureka.co/blog/python-libraries/ https://www.edureka.co/blog/python-libraries/ All of which list numpy, pandas and the rest e.g. tensorflow and friends.
- I_complete_me 6y ago
  I would add the following: 1. state that you may need to install pip for your OS 2. show how to deactivate the virtualenv
  - theelous3 6y ago
    Sure, but I'm not teaching here. Even then, unbuntu will tell you how to install pip (actually it probably comes installed), and closing the terminal is sufficient to close the env for most.
    - jlbnjmn 6y ago
      Noobs aren't using Ubuntu though.
      - theelous3 6y ago
        They very often are, and if they're not, it only takes a couple of mins to install the win10 ubu subsystem.
- ericol 6y ago
  > Your setup sounds outrageously complex, and I don't understand why you would do any of that But isn't that similar to what virtualenv does? The differences are that with virtualenv, folders are created elsewhere. Besides that, friction so far hasn't enough for me to automate this even more (as in, creating a script that would do the folder creation) but I guess I'll do eventually. Also, this way I have a set of python packages already installed. And it was fun learning how to command pip / python to do things as I wanted them. That I guess answers this: > I don't understand why you would do any of that P.S.: virtualenv -p $(which python) env Saves you one line there ;)
  - ptx 6y ago
    > Also, this way I have a set of python packages already installed. Isn't this what virtualenv was created to get away from? If several of your projects rely on one of the "base" packages, upgrading it (which you might want for one project) could break the other projects. Also, having implicit dependencies on packages might lead to problems with deployment (or for other developers) because they don't know that this package is required (and you might not have realized either, since it's always available on your machine). (I might have misunderstood your solution, though. I've never used Docker.)
    - ericol 6y ago
      Hi, nop, the reasoning behind having a base set of packages installed is that I more often than not use those. But as I start docker off a particular folder, I could decide not to use that base folder, and then having no package pre installed, or a complete different set. This other comment [1] clarifies my whole set up, and sheds a lot more light on how it works [1] https://news.ycombinator.com/item?id=24028953 https://news.ycombinator.com/item?id=24028953
  - slavik81 6y ago
    You can shorten that to: virtualenv env
  - theelous3 6y ago
    The reasoning here is lost on me. We want independent envs for independent codebases. Why do you have some static base to build off? What's with the root stuff? Why is any of this taking place outside of the env that actually needs it? You're creating huge amounts of complexity and dependency problems to save yourself a couple of literal ms of download / path lookup per env. The friction here sounds like a tire fire. I think you are just not aware at the moment of how much less convoluted this can be. You've become used to this complexity.
- ptx 6y ago
  I agree, but it can be simplified. Python 3 nowadays comes with all the needed tools built-in, so there is no need to install virtualenv: cd your/project/location python3 -m venv env source env/bin/activate pip install anything_you_like Also, on Windows, use the Python launcher[1] (that comes with the standard install of Python for Windows) to avoid having to mess with PATH settings as you would have had to historically. Instead of calling "python" or "python3" and hoping for the right version, just call "py" and tell it which version you want: py -3.6-32 -m venv env That gets you a venv with 32-bit Python 3.6, for example. Use "py -0" to list available versions. [1] https://docs.python.org/3/using/windows.html#python-launcher-for-windows https://docs.python.org/3/using/windows.html#python-launcher...
  - theelous3 6y ago
    Yep, this is good. Just trying to be general here on hn, or I'll get someone saying the other case ;)
- bllguo 6y ago
  the only time i've appreciated conda is on windows systems. recently I've been playing with pystan, which - after hours of trial and error - im not even sure is possible to install correctly without conda. but otherwise, i agree entirely
  - theelous3 6y ago
    Ah yeah, but when you're already sitting in a boiling pot you may as well add some seasoning.

by ericol 6y ago

theelous3 6y ago

As someone who's been hanging out in freenode python channels for years helping noobs, I recommend strongly against any kind of conda setup for new programmers. It often has strange issues that noobs have difficulty even understanding how to ask for help with, nevermind actually solve. Learning how to use a simple barebones venv is extremely easy, saves a ton of time both in the short and long run, and generalises better. pip install virtualenv cd your/project/location which python virtualenv -p result_of_which_python env source env/bin/activate pip install anything_you_like and do whatever you want from there. Those commands get 100% of the basics out of the way for noobs, and cover like 90% of the stuff you use to do more serious stuff. > but I use a "base" folder where I have already downloaded all the packages I use by default [...etc...] Your setup sounds outrageously complex, and I don't understand why you would do any of that.

aldanor 6y ago
I would strongly recommend to use conda because: - It would work the same way on Windows, Linux and macOS - Most importantly (!), it is not a Python package manager. That is, if a package needs BLAS or MKL or HDF5 or whatever else, you won't be crossing fingers and hoping your system-installed version would work for all your venvs; instead, those binary libraries are properly managed per environment. Pro tip: use mamba instead of conda to get a free 4x speed boost.
- theelous3 6y ago
  Ah yes, all of those noobs who need weird numpy extensions and highly domain oriented and specific odd data storage formats. That's the generalisation part I mentioned. The distinction you're making is non-trival for a wee noobie. Better they get the ground work in and expand to where they need to go.
  - aldanor 6y ago
    One of the things most noobs will install first is pandas. Pandas needs numpy. Numpy needs lapack, blas etc, even if they're not aware it. Yes, there's wheels, but that's limited to python-only stuff. Then they're reading on some blog about protobuf, or hdf5, or arrow or whatever, and want to use it - but using either from Python needs installed C libraries. On linux/macos you'll need to dig into your package managers and then hope things are compatible, on Windows it's a complete pain. In conda, they can just 'conda install h5py' or whatever, and get proper hdf5 installed without having to figure out the nitty gritty details.
    - theelous3 6y ago
      > most noobs Well, already I can tell you're disconnected from what "most noobs" are actually like. Presumably you think "most noobs" are data scientists, which just isn't the case in my experience.
      - aldanor 6y ago
        > Presumably you think "most noobs" are data scientists, which just isn't the case in my experience It is the case in my personal experience though. I don't think I'm disconnected from what 'noobs' are as I've taught a lot of them through my line of work. Assume you're a noob and are exploring Python package universe, fire a new private tab and google 'most popular python packages'; in my case the first 3 links are: - https://www.activestate.com/blog/top-10-must-have-python-packages/ https://www.activestate.com/blog/top-10-must-have-python-pac... - https://www.ubuntupit.com/best-python-libraries-and- https://www.ubuntupit.com/best-python-libraries-and- packages-for-beginners/ - https://www.edureka.co/blog/python-libraries/ https://www.edureka.co/blog/python-libraries/ All of which list numpy, pandas and the rest e.g. tensorflow and friends.
I_complete_me 6y ago
I would add the following: 1. state that you may need to install pip for your OS 2. show how to deactivate the virtualenv
- theelous3 6y ago
  Sure, but I'm not teaching here. Even then, unbuntu will tell you how to install pip (actually it probably comes installed), and closing the terminal is sufficient to close the env for most.
  - jlbnjmn 6y ago
    Noobs aren't using Ubuntu though.
    - theelous3 6y ago
      They very often are, and if they're not, it only takes a couple of mins to install the win10 ubu subsystem.
ericol 6y ago
> Your setup sounds outrageously complex, and I don't understand why you would do any of that But isn't that similar to what virtualenv does? The differences are that with virtualenv, folders are created elsewhere. Besides that, friction so far hasn't enough for me to automate this even more (as in, creating a script that would do the folder creation) but I guess I'll do eventually. Also, this way I have a set of python packages already installed. And it was fun learning how to command pip / python to do things as I wanted them. That I guess answers this: > I don't understand why you would do any of that P.S.: virtualenv -p $(which python) env Saves you one line there ;)
- ptx 6y ago
  > Also, this way I have a set of python packages already installed. Isn't this what virtualenv was created to get away from? If several of your projects rely on one of the "base" packages, upgrading it (which you might want for one project) could break the other projects. Also, having implicit dependencies on packages might lead to problems with deployment (or for other developers) because they don't know that this package is required (and you might not have realized either, since it's always available on your machine). (I might have misunderstood your solution, though. I've never used Docker.)
  - ericol 6y ago
    Hi, nop, the reasoning behind having a base set of packages installed is that I more often than not use those. But as I start docker off a particular folder, I could decide not to use that base folder, and then having no package pre installed, or a complete different set. This other comment [1] clarifies my whole set up, and sheds a lot more light on how it works [1] https://news.ycombinator.com/item?id=24028953 https://news.ycombinator.com/item?id=24028953
- slavik81 6y ago
  You can shorten that to: virtualenv env
- theelous3 6y ago
  The reasoning here is lost on me. We want independent envs for independent codebases. Why do you have some static base to build off? What's with the root stuff? Why is any of this taking place outside of the env that actually needs it? You're creating huge amounts of complexity and dependency problems to save yourself a couple of literal ms of download / path lookup per env. The friction here sounds like a tire fire. I think you are just not aware at the moment of how much less convoluted this can be. You've become used to this complexity.
ptx 6y ago
I agree, but it can be simplified. Python 3 nowadays comes with all the needed tools built-in, so there is no need to install virtualenv: cd your/project/location python3 -m venv env source env/bin/activate pip install anything_you_like Also, on Windows, use the Python launcher[1] (that comes with the standard install of Python for Windows) to avoid having to mess with PATH settings as you would have had to historically. Instead of calling "python" or "python3" and hoping for the right version, just call "py" and tell it which version you want: py -3.6-32 -m venv env That gets you a venv with 32-bit Python 3.6, for example. Use "py -0" to list available versions. [1] https://docs.python.org/3/using/windows.html#python-launcher-for-windows https://docs.python.org/3/using/windows.html#python-launcher...
- theelous3 6y ago
  Yep, this is good. Just trying to be general here on hn, or I'll get someone saying the other case ;)
bllguo 6y ago
the only time i've appreciated conda is on windows systems. recently I've been playing with pystan, which - after hours of trial and error - im not even sure is possible to install correctly without conda. but otherwise, i agree entirely
- theelous3 6y ago
  Ah yeah, but when you're already sitting in a boiling pot you may as well add some seasoning.