4 ms·

My two cents is that conda and conda-forge is the “right” way forward, and attention focused on pipenv is not suboptimal because of pipenv (though you can make

by mlthoughts2018 8y ago

My two cents is that conda and conda-forge is the “right” way forward, and attention focused on pipenv is not suboptimal because of pipenv (though you can make engineering arguments against it— after all, it is still very young and has not yet matured the way other packaging tools have). Rather it’s bad because it’s yet another set of spec files and environment tools to learn, potentially further fracturing the community and taking attention away from other approaches. To be fair, it’s not really a problem with pipenv, but rather with any “official” community endorsement of pipenv. Saying it’s a third party, optional tool and you’re free to have at it if it’s good for your project is fine. But the minute there’s any degree of officialness to it, it just creates yet another fissure in the anarchy of Python packaging. My reaction is more malaise. Guess I’ll have to learn about this new packaging gadget too, to be current and signal how with-it I am. But in any case I have a choice, I’ll still just use conda and Docker for application & environment management, supplemented by pip and virtualenv when appropriate.

kalefranz 8y ago
Current conda dev lead here. Just want to reference an issue we'll be tackling ideally on the timeline of the next 12 months. https://github.com/conda/conda/issues/7248 https://github.com/conda/conda/issues/7248 Conda is not without its problems in the workflow that pipenv addresses. We know we have a deficiency here, and addressing that deficiency is one of our enumerated highest priorities.
- mlthoughts2018 8y ago
  Thanks for your work! I agree these are valuable design features to add and to get right, and also agree that pipenv offers tools for this workflow. From my perspective, I’ve tended to prefer Docker for this. This allows me to version control environment components like environment variables in the form of the Dockerfile. I usually have a Makefile for my project which then uses multistage builds or other techniques to automatically detect when environment.yml (or package changes, like to setup.py) take place, requiring a rebuild of the container, so that downstream things like running tests, dropping into an interactive debugger in a full environment (with the right python environment, environment variables, etc.), all just become simple Make directives. It doesn’t cover everything. Notably, for situations when there are many target environments, like testing a library I intend to distribute cross-platform, across Python 2/3, etc., I might still use a tool like appveyor rather than suffering the multiplicity of a bunch of Dockerfiles and Make commands to target multiple environments. But for most applications I develop with an intention of targeting a small set of production or dev environments, it turns out to be better IMO to separate the Python environment & package definition entirely from the larger runtime environment (ENV vars, etc.), and use Docker for that part, with Make as the bridge. Combined with making a library choose sensible defaults for ENV vars, copiously documenting them, and providing examples, this has worked well enough that I do not need or want to mix the external environment configuration with the app’s config files or package spec, in a “separation of concerns” sense. Your mileage may vary, but this pattern has really worked well for me, and makes me skeptical of solutions that approach combining environment configuration with packaging as a first-class feature, rather than as a special case.

kalefranz 8y ago

Current conda dev lead here. Just want to reference an issue we'll be tackling ideally on the timeline of the next 12 months. https://github.com/conda/conda/issues/7248 https://github.com/conda/conda/issues/7248 Conda is not without its problems in the workflow that pipenv addresses. We know we have a deficiency here, and addressing that deficiency is one of our enumerated highest priorities.

mlthoughts2018 8y ago
Thanks for your work! I agree these are valuable design features to add and to get right, and also agree that pipenv offers tools for this workflow. From my perspective, I’ve tended to prefer Docker for this. This allows me to version control environment components like environment variables in the form of the Dockerfile. I usually have a Makefile for my project which then uses multistage builds or other techniques to automatically detect when environment.yml (or package changes, like to setup.py) take place, requiring a rebuild of the container, so that downstream things like running tests, dropping into an interactive debugger in a full environment (with the right python environment, environment variables, etc.), all just become simple Make directives. It doesn’t cover everything. Notably, for situations when there are many target environments, like testing a library I intend to distribute cross-platform, across Python 2/3, etc., I might still use a tool like appveyor rather than suffering the multiplicity of a bunch of Dockerfiles and Make commands to target multiple environments. But for most applications I develop with an intention of targeting a small set of production or dev environments, it turns out to be better IMO to separate the Python environment & package definition entirely from the larger runtime environment (ENV vars, etc.), and use Docker for that part, with Make as the bridge. Combined with making a library choose sensible defaults for ENV vars, copiously documenting them, and providing examples, this has worked well enough that I do not need or want to mix the external environment configuration with the app’s config files or package spec, in a “separation of concerns” sense. Your mileage may vary, but this pattern has really worked well for me, and makes me skeptical of solutions that approach combining environment configuration with packaging as a first-class feature, rather than as a special case.