3 ms·

It was an attempt to directly address the FUD about running python code during dependency resolution. You're doing this to run more python code from the project

It was an attempt to directly address the FUD about running python code during dependency resolution. You're doing this to run more python code from the project's author so this seemed odd and an irrational fear. Maybe my tone came across condescending. That wasn't the intent. It was meant in jest in hopes the reader would second guess the fear. KISS is great, but this project is here because it's not already working for everyone. There is benefit to a flexible dependency system that keeps the concepts of "acceptable versions" separate from "actual versions last used" while also adopting an extendible DSL ready for edge cases that haven't been thought of.

witten 10y ago
To be clear, I totally get the benefits of formal support for an "actual versions last used". That is a topic near and dear to my heart, having been burnt by floating versions more than once. It's just that I don't want to swallow a too-flexible DSL in order to get that benefit, because I've also been burnt multiple times by using a DSL when a declarative data format will suffice. And you don't need both. You can pip freeze a frozen_requirements.txt. You can ship around a whole virtual environment. And I can conceive of more requirements.txt 2.0 solutions that don't require over-wrought DSLs.
- examancer 10y ago
  You guys were really burned by this setup.py stuff, huh? I empathize. I also find it a little odd since a destructive project author can still find ways to mess things up without any code executing during setup, right? Since this is for project level dependencies it doesn't seem like the potential for abuse is too high since you'll likely have a single Pipfile per project and the project will control it. I don't know that anything will prevent destructive developers from being destructive. I would hate to throw out all DSLs because of a few bad actors. For what it's worth in Ruby even library dependencies are written using a ruby DSL that runs unprotected during install, as well as some post installation hook facilities. There have been minor abuses like annoyingly long post-install messages and other crap that has occurred, but through simple community pressure the offending libraries are eventually pushed back in line and it's not an issue Ruby developers regularly encounter. Communities are different but Python seems like a community that appreciates best practices and is maybe better at enforcing them than Ruby is, so I'm not sure preventing one minor way a terrible developer can bite you is worth hamstringing your build system. Yes, an over-wrought DSL is not a prerequisite. You can do deterministic builds and even grouped dependencies without it. But complex platform specific dependencies, git based dependencies, and dependency edge cases we haven't yet envisioned are more easily captured in an extendible native DSL than a more static data format. The DSL also has the advantage of being "just Python" so it will be very easy to remember for Python developers. Maybe the benefits aren't worth the trade-offs but don't be so quick to count potential or perceived trade-offs as actual ones until you try out the new build system and see how it impacts your workflow. I know you got burned but this isn't setup.py and potential for abuse isn't the same as abuse... yet.
  - HelloNurse 10y ago
    It's a matter of opaqueness, not of abuse: the install system will be unable to reason about what an arbitrary script does, severely limiting functionality. For example, pip will be unable to ensure dependencies remain constant (after installing something else, after DST begins or ends, between a dry run and an actual install, after unrelated software alters environment variables, etc.). This is much worse than a 95% correct dependencies manifest that can be easily hacked to 100% correct.
    - lukeschlather 10y ago
      pip will be able to do everything you suggest by using the lockfile. The whole point is that if you want things to be reproducible you don't even need to look in the Pipfile. requirements.txt already supports conditionals, so the Pipfile.lock is strictly simpler to reason about in that sense.
- kingosticks 10y ago
  Can you give some concrete examples of the issues with using a setup.py/DSL? I've only been using python intermittently for a few years and I'm probably not exposed to these things. Am I setting myself up for a fall by using setup.py?
  - shakna 10y ago
    Not as such, but I've encountered setups that shell out to Inkscape, ImageMagick and other tools, and don't gracefully handle failure, sometimes to the point where the only error is "Failed to install egg".
  - witten 10y ago
    Here's a setup.py that includes an interactive prompt, and thus breaks all ability to do automated installs! https://github.com/pypa/pip/issues/2732 https://github.com/pypa/pip/issues/2732 Here's a setup.py that cannot even be imported unless another package (numpy) is already installed: https://github.com/scipy/scipy/blob/master/setup.py https://github.com/scipy/scipy/blob/master/setup.py I've seen setup.pys that read random files from the filesystem. I've seen setup.pys that shell out to random system commands that may or may not be installed. It's just a horrible format for what it's trying to accomplish.. because it's not a format! It's a programming language.

witten 10y ago

To be clear, I totally get the benefits of formal support for an "actual versions last used". That is a topic near and dear to my heart, having been burnt by floating versions more than once. It's just that I don't want to swallow a too-flexible DSL in order to get that benefit, because I've also been burnt multiple times by using a DSL when a declarative data format will suffice. And you don't need both. You can pip freeze a frozen_requirements.txt. You can ship around a whole virtual environment. And I can conceive of more requirements.txt 2.0 solutions that don't require over-wrought DSLs.

examancer 10y ago
You guys were really burned by this setup.py stuff, huh? I empathize. I also find it a little odd since a destructive project author can still find ways to mess things up without any code executing during setup, right? Since this is for project level dependencies it doesn't seem like the potential for abuse is too high since you'll likely have a single Pipfile per project and the project will control it. I don't know that anything will prevent destructive developers from being destructive. I would hate to throw out all DSLs because of a few bad actors. For what it's worth in Ruby even library dependencies are written using a ruby DSL that runs unprotected during install, as well as some post installation hook facilities. There have been minor abuses like annoyingly long post-install messages and other crap that has occurred, but through simple community pressure the offending libraries are eventually pushed back in line and it's not an issue Ruby developers regularly encounter. Communities are different but Python seems like a community that appreciates best practices and is maybe better at enforcing them than Ruby is, so I'm not sure preventing one minor way a terrible developer can bite you is worth hamstringing your build system. Yes, an over-wrought DSL is not a prerequisite. You can do deterministic builds and even grouped dependencies without it. But complex platform specific dependencies, git based dependencies, and dependency edge cases we haven't yet envisioned are more easily captured in an extendible native DSL than a more static data format. The DSL also has the advantage of being "just Python" so it will be very easy to remember for Python developers. Maybe the benefits aren't worth the trade-offs but don't be so quick to count potential or perceived trade-offs as actual ones until you try out the new build system and see how it impacts your workflow. I know you got burned but this isn't setup.py and potential for abuse isn't the same as abuse... yet.
- HelloNurse 10y ago
  It's a matter of opaqueness, not of abuse: the install system will be unable to reason about what an arbitrary script does, severely limiting functionality. For example, pip will be unable to ensure dependencies remain constant (after installing something else, after DST begins or ends, between a dry run and an actual install, after unrelated software alters environment variables, etc.). This is much worse than a 95% correct dependencies manifest that can be easily hacked to 100% correct.
  - lukeschlather 10y ago
    pip will be able to do everything you suggest by using the lockfile. The whole point is that if you want things to be reproducible you don't even need to look in the Pipfile. requirements.txt already supports conditionals, so the Pipfile.lock is strictly simpler to reason about in that sense.
kingosticks 10y ago
Can you give some concrete examples of the issues with using a setup.py/DSL? I've only been using python intermittently for a few years and I'm probably not exposed to these things. Am I setting myself up for a fall by using setup.py?
- shakna 10y ago
  Not as such, but I've encountered setups that shell out to Inkscape, ImageMagick and other tools, and don't gracefully handle failure, sometimes to the point where the only error is "Failed to install egg".
- witten 10y ago
  Here's a setup.py that includes an interactive prompt, and thus breaks all ability to do automated installs! https://github.com/pypa/pip/issues/2732 https://github.com/pypa/pip/issues/2732 Here's a setup.py that cannot even be imported unless another package (numpy) is already installed: https://github.com/scipy/scipy/blob/master/setup.py https://github.com/scipy/scipy/blob/master/setup.py I've seen setup.pys that read random files from the filesystem. I've seen setup.pys that shell out to random system commands that may or may not be installed. It's just a horrible format for what it's trying to accomplish.. because it's not a format! It's a programming language.