3 ms·

> Your setup sounds outrageously complex, and I don't understand why you would do any of that But isn't that similar to what virtualenv does? The differences a

by ericol 6y ago

> Your setup sounds outrageously complex, and I don't understand why you would do any of that But isn't that similar to what virtualenv does? The differences are that with virtualenv, folders are created elsewhere. Besides that, friction so far hasn't enough for me to automate this even more (as in, creating a script that would do the folder creation) but I guess I'll do eventually. Also, this way I have a set of python packages already installed. And it was fun learning how to command pip / python to do things as I wanted them. That I guess answers this: > I don't understand why you would do any of that P.S.: virtualenv -p $(which python) env Saves you one line there ;)

ptx 6y ago
> Also, this way I have a set of python packages already installed. Isn't this what virtualenv was created to get away from? If several of your projects rely on one of the "base" packages, upgrading it (which you might want for one project) could break the other projects. Also, having implicit dependencies on packages might lead to problems with deployment (or for other developers) because they don't know that this package is required (and you might not have realized either, since it's always available on your machine). (I might have misunderstood your solution, though. I've never used Docker.)
- ericol 6y ago
  Hi, nop, the reasoning behind having a base set of packages installed is that I more often than not use those. But as I start docker off a particular folder, I could decide not to use that base folder, and then having no package pre installed, or a complete different set. This other comment [1] clarifies my whole set up, and sheds a lot more light on how it works [1] https://news.ycombinator.com/item?id=24028953 https://news.ycombinator.com/item?id=24028953
slavik81 6y ago
You can shorten that to: virtualenv env
theelous3 6y ago
The reasoning here is lost on me. We want independent envs for independent codebases. Why do you have some static base to build off? What's with the root stuff? Why is any of this taking place outside of the env that actually needs it? You're creating huge amounts of complexity and dependency problems to save yourself a couple of literal ms of download / path lookup per env. The friction here sounds like a tire fire. I think you are just not aware at the moment of how much less convoluted this can be. You've become used to this complexity.
- ericol 6y ago
  > We want independent envs for independent codebases OK. I understand where are you coming from, and I feel your pain, being a developer myself and have been way too many times that I want to admit on the "But, why?!" position. You sound really distressed by this, and at the same time honestly curious (And baffled can probably shoved in there) so 'll explain things a little more. I think all this starts with the fact that I explained myself rather poorly. I _do_ have different envs for different codebases. My set up is like this: Everything python related is in a folder where I store my "dockerized" projects: ~/projects/docker/python/ In that folder I have a pip.env file that I use when setting up the container. It tells pip where to pick things up from: UN_PIP_TARGET=/work/packages PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/work/packages/bin UN_PYTHONPATH=/usr/local/lib/python3.7/site-packages:/work/packages PYTHONPATH=/work/packages PYTHONUSERBASE=/work/packages PIP_USER=yes I take it it might be a little messy (and may be thee are some redundant confs there) but it works. Once I got it working, I didn't mind about removing what was not necessary. then I have a "base" project folder, that I use as a starting point for _most_ of my projects: base ├── packages ├── notebooks This folder has installed a few "default" packages (pandas, numpy, matplotlib, youtube-dl). So, as you can see the packages go all in a custom folder. This is because when I start the container, the project folder would be shared inside the container, and python / pip will pick the packages from there. If I install new packages, it will be persisted in the "env" folder system of the guest OS, so next time I "start" than environment all them packages are there. Whenever I want to start a new project, the steps I need to do is: python.sh env_name python.sh is a bash scripts that checks if env_name is there (as a folder). if not, it copies the base env and all it's contents (That, just so that it's extra clear, is a barebones folder _except_ for the already stated packages installed in it). In case it receives a second parameter, it uses that as starting folder (So, I could duplicate an existing project, or use an even emptier base folder). Once the folder is there, it just starts a container in interactive mode with the (environment) base folder always shared as /work In case I need some files to work there, I just copy them. Again, the _only_ issue so far - that annoys me -, is that if I create files when inside the container they are owned by root. Eventually I'll grow tired of this and will fix it, but not there yet. for me starting a certain env is just one line of code. Obviously you could do the same with your method. Said that I didn't have the script, for me creating a new env goes like this: cp -r ./base new_env docker run -it -rm --name new_env_container -v $(pwd)/new_env:/work -w="/work" python:3.7 /bin/bash Removing an environment is just: rm -r new_env What I like of this set up is that all the files related to a certain environment are in a certain place. Also, moving envs from one machine to another is rather trivial (as long as I didn't install anything not related to python in that container). This way, the container is removed every time you exit it, but it has a name so you can log into another console would you need it. the only issue with this is that it's a barebones OS, so if you need some program for it you have to install it (And that is lost of you don't keep the container: in my todo list there's a change for python.sh where it would be possible to persist the containers, and run them from the one already existing if found. The downside to this is that every container you persist is several hundred mb... But space is cheap, right?). BTW, I do have a container I persist for use with youtube-dl as it depends heavily on ffmpeg. From my point of view, that works the same a venv. Your mileage might vary. Caveat: python:3.7 is not the default name of the python image; I renamed it because the default was too long. I used to do it this way until I got tired of writing the same long command.

by ericol 6y ago

ptx 6y ago

> Also, this way I have a set of python packages already installed. Isn't this what virtualenv was created to get away from? If several of your projects rely on one of the "base" packages, upgrading it (which you might want for one project) could break the other projects. Also, having implicit dependencies on packages might lead to problems with deployment (or for other developers) because they don't know that this package is required (and you might not have realized either, since it's always available on your machine). (I might have misunderstood your solution, though. I've never used Docker.)

ericol 6y ago
Hi, nop, the reasoning behind having a base set of packages installed is that I more often than not use those. But as I start docker off a particular folder, I could decide not to use that base folder, and then having no package pre installed, or a complete different set. This other comment [1] clarifies my whole set up, and sheds a lot more light on how it works [1] https://news.ycombinator.com/item?id=24028953 https://news.ycombinator.com/item?id=24028953

slavik81 6y ago

You can shorten that to: virtualenv env

theelous3 6y ago

The reasoning here is lost on me. We want independent envs for independent codebases. Why do you have some static base to build off? What's with the root stuff? Why is any of this taking place outside of the env that actually needs it? You're creating huge amounts of complexity and dependency problems to save yourself a couple of literal ms of download / path lookup per env. The friction here sounds like a tire fire. I think you are just not aware at the moment of how much less convoluted this can be. You've become used to this complexity.

ericol 6y ago
> We want independent envs for independent codebases OK. I understand where are you coming from, and I feel your pain, being a developer myself and have been way too many times that I want to admit on the "But, why?!" position. You sound really distressed by this, and at the same time honestly curious (And baffled can probably shoved in there) so 'll explain things a little more. I think all this starts with the fact that I explained myself rather poorly. I _do_ have different envs for different codebases. My set up is like this: Everything python related is in a folder where I store my "dockerized" projects: ~/projects/docker/python/ In that folder I have a pip.env file that I use when setting up the container. It tells pip where to pick things up from: UN_PIP_TARGET=/work/packages PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/work/packages/bin UN_PYTHONPATH=/usr/local/lib/python3.7/site-packages:/work/packages PYTHONPATH=/work/packages PYTHONUSERBASE=/work/packages PIP_USER=yes I take it it might be a little messy (and may be thee are some redundant confs there) but it works. Once I got it working, I didn't mind about removing what was not necessary. then I have a "base" project folder, that I use as a starting point for _most_ of my projects: base ├── packages ├── notebooks This folder has installed a few "default" packages (pandas, numpy, matplotlib, youtube-dl). So, as you can see the packages go all in a custom folder. This is because when I start the container, the project folder would be shared inside the container, and python / pip will pick the packages from there. If I install new packages, it will be persisted in the "env" folder system of the guest OS, so next time I "start" than environment all them packages are there. Whenever I want to start a new project, the steps I need to do is: python.sh env_name python.sh is a bash scripts that checks if env_name is there (as a folder). if not, it copies the base env and all it's contents (That, just so that it's extra clear, is a barebones folder _except_ for the already stated packages installed in it). In case it receives a second parameter, it uses that as starting folder (So, I could duplicate an existing project, or use an even emptier base folder). Once the folder is there, it just starts a container in interactive mode with the (environment) base folder always shared as /work In case I need some files to work there, I just copy them. Again, the _only_ issue so far - that annoys me -, is that if I create files when inside the container they are owned by root. Eventually I'll grow tired of this and will fix it, but not there yet. for me starting a certain env is just one line of code. Obviously you could do the same with your method. Said that I didn't have the script, for me creating a new env goes like this: cp -r ./base new_env docker run -it -rm --name new_env_container -v $(pwd)/new_env:/work -w="/work" python:3.7 /bin/bash Removing an environment is just: rm -r new_env What I like of this set up is that all the files related to a certain environment are in a certain place. Also, moving envs from one machine to another is rather trivial (as long as I didn't install anything not related to python in that container). This way, the container is removed every time you exit it, but it has a name so you can log into another console would you need it. the only issue with this is that it's a barebones OS, so if you need some program for it you have to install it (And that is lost of you don't keep the container: in my todo list there's a change for python.sh where it would be possible to persist the containers, and run them from the one already existing if found. The downside to this is that every container you persist is several hundred mb... But space is cheap, right?). BTW, I do have a container I persist for use with youtube-dl as it depends heavily on ffmpeg. From my point of view, that works the same a venv. Your mileage might vary. Caveat: python:3.7 is not the default name of the python image; I renamed it because the default was too long. I used to do it this way until I got tired of writing the same long command.