3 ms·

Iteratively. You don't need to solve all those problems at once. Version pinning can be done in setup.py using the same syntax you would see in a requirements.

Iteratively. You don't need to solve all those problems at once. Version pinning can be done in setup.py using the same syntax you would see in a requirements.txt file. You should be very conservative when pinning versions in a library, though. You can lean on your ci tool (eg. Github actions) to handle testing, hash checking, credential management, etc. But I recommend all of this start as a bunch of locally runnable scripts. I typically bump version directly in a version file and move on with my life. This stuff usually builds up iteratively and at least for me has never been the starting point. Starting point should be a library worth sharing. It is not the end of the world of you release the first few versions manually.

neolog 6y ago
> You don't need to solve all those problems at once. Why spend time tweaking the setup when you can just get it right in the first place with less work?
- zomglings 6y ago
  Do you recommend poetry? I've been meaning to try it but haven't had the time to migrate an existing project to poetry.
  - neolog 6y ago
    Yeah it's pretty good. There are plenty of problems and holes of course, but for most pure-Python packages it does a nice job most of the time. If you're an expert, you can do it with setup.py or setup.cfg, but I don't think it's normally worth the trouble.
  - atoav 6y ago
    I used poetry for multiple projects already and I quite like it because it makes version management really straightforward and you can just go like poetry new foo And it will setup all the basic stuff for you right away. Having all the project dependecies and Metadata in pyproject.toml makes sense and reduces cognitive overload. Having poetry manage your venvs autmagically is a good extra. There is still room for improvement, e.g. I had some trouble with their install script at times (python vs python3)
ehsankia 6y ago
To be fair, while this is a single article, if you only look at step 1, 2 and 3, you get a fully published package with only one tool used (flit) and not much extra. It's the succeeding sections (A, B, C, D, E) that get more advanced, but they're all optional. You should definitely do A, but the rest I'd say it's a lot more opinionated and definitely not needed.
- zomglings 6y ago
  Fair point. I wouldn't recommend this article to someone just starting out with python, though. It's often good to understand generation n-1's way of doing things but not too be married to it.
staticassertion 6y ago
TBH as someone trying to use Python professionally it is extremely frustrating that basic things with regards to package management are something you have to iterate towards, as opposed to just being obvious and default.
- zomglings 6y ago
  I sympathize. It is unfortunate that the python community never settled around a tool like leiningen for clojure or cargo for rust or npm for node. What we saw with npm was the entire community iterating towards a feature set and everyone reaping the benefits automatically with npm updates. package-lock.json is a good example of this.
  - staticassertion 6y ago
    I completely agree. Elsewhere we see sanity by default, in Python there's still open questions and hopeful but early stage projects.
  - marcinzm 6y ago
    The issue is that every time the community settles on a tool a new tool is made to fix the issues with the old tool rather than just refactoring the old tool.
  - brundolf 6y ago
    Worth noting is that cargo and npm weren't "settled around"; they were developed and presented, from the beginning, alongside the relevant compiler and runtime. There was never a question; the batteries were included. Leiningen is the weird one where people did actually settle fairly well around an unofficial solution in the absence of an official one. I think the norm with languages that forego official tooling is closer to what we've seen in Python.
    - xapata 6y ago
      The Python community has considered an "official" packaging tool in the past, but in those conversations found that the community had too many preferences to find a good compromise. That's the trouble with having a highly diverse set of uses and integrations, and lots of legacy. If you're curious, the email threads about Conda and defining wheels are interesting.
dagenix 6y ago
> Version pinning can be done in setup.py using the same syntax you would see in a requirements.txt file The problem with this approach is that it doesn't handle transitive dependencies well. Say you depend on version 1.4.6 of a particular library. And then that library depends on version >= 2 of some other library. When you install your package, you know that you'll get version 1.4.6 of the first library but have no idea what version you'll get of the second library. You can of course pin all the transitive dependencies - except that clutters up the setup.py and is a massive pain to keep up to date as you bump individual dependency versions.
- knome 6y ago
  Seems like a solid argument for a switch to use go's minimal version selection the version selected during a build is the one with the most minimal version that satisfies all other constraints. this means if you have libA that needs dep>=1.1 and libB that needs dep>=1.3, you get dep=1.3 even if dep1.9 is out. your build never changes because of a new version release, as long as they release with proper semantic versioning. if you later include libC that needs dep>=1.8, you'll get that version. but because you changed your immediate dependencies, not due to a surprise down the dependency line. https://research.swtch.com/vgo-principles https://research.swtch.com/vgo-principles
  - dagenix 6y ago
    Its at least consistent - which, IMO, is better than just getting a random version. I do, however, think it's a bit unfortunate that it prevents picking up bug / security fixes in transitive dependencies. Imagine that you depend on library A of a particular version which itself depends on library B. With minimal version selection, as long you don't bump you dependency on library A (or some other library that depends on library B), you'll continue to get that same version of library B. But, then library B releases a critical security fix. With minimal version selection, there isn't a great way to pick up that fix. You can _hope_ that library A releases a new version that requires the fix - but that may or may not happen and could take a while. Or, you could add an explicit dependency on the new version of library B - which is unfortunate, since, your main package doesn't depend on library B directly. Lock files solve this problem. You can depend on whatever version of library A that you need and lock the transitive dependencies. And once library B releases its fix, you can update your lock file without having up bump the version of library A. Tools like poetry providing the features to automate this workflow.
BerislavLopac 6y ago
> You should be very conservative when pinning versions in a library, though No; you should be very conservative when pinning versions in an application, not in a library. Check this article for the explanation: https://caremad.io/posts/2013/07/setup-vs-requirement/ https://caremad.io/posts/2013/07/setup-vs-requirement/
- toyg 6y ago
  Anybody familiar with the history of requests knows this is bad advice.
  - BerislavLopac 6y ago
    requests is definitely not a good example here. :/

neolog 6y ago

> You don't need to solve all those problems at once. Why spend time tweaking the setup when you can just get it right in the first place with less work?

zomglings 6y ago
Do you recommend poetry? I've been meaning to try it but haven't had the time to migrate an existing project to poetry.
- neolog 6y ago
  Yeah it's pretty good. There are plenty of problems and holes of course, but for most pure-Python packages it does a nice job most of the time. If you're an expert, you can do it with setup.py or setup.cfg, but I don't think it's normally worth the trouble.
- atoav 6y ago
  I used poetry for multiple projects already and I quite like it because it makes version management really straightforward and you can just go like poetry new foo And it will setup all the basic stuff for you right away. Having all the project dependecies and Metadata in pyproject.toml makes sense and reduces cognitive overload. Having poetry manage your venvs autmagically is a good extra. There is still room for improvement, e.g. I had some trouble with their install script at times (python vs python3)

ehsankia 6y ago

To be fair, while this is a single article, if you only look at step 1, 2 and 3, you get a fully published package with only one tool used (flit) and not much extra. It's the succeeding sections (A, B, C, D, E) that get more advanced, but they're all optional. You should definitely do A, but the rest I'd say it's a lot more opinionated and definitely not needed.

zomglings 6y ago
Fair point. I wouldn't recommend this article to someone just starting out with python, though. It's often good to understand generation n-1's way of doing things but not too be married to it.

staticassertion 6y ago

TBH as someone trying to use Python professionally it is extremely frustrating that basic things with regards to package management are something you have to iterate towards, as opposed to just being obvious and default.

zomglings 6y ago
I sympathize. It is unfortunate that the python community never settled around a tool like leiningen for clojure or cargo for rust or npm for node. What we saw with npm was the entire community iterating towards a feature set and everyone reaping the benefits automatically with npm updates. package-lock.json is a good example of this.
- staticassertion 6y ago
  I completely agree. Elsewhere we see sanity by default, in Python there's still open questions and hopeful but early stage projects.
- marcinzm 6y ago
  The issue is that every time the community settles on a tool a new tool is made to fix the issues with the old tool rather than just refactoring the old tool.
- brundolf 6y ago
  Worth noting is that cargo and npm weren't "settled around"; they were developed and presented, from the beginning, alongside the relevant compiler and runtime. There was never a question; the batteries were included. Leiningen is the weird one where people did actually settle fairly well around an unofficial solution in the absence of an official one. I think the norm with languages that forego official tooling is closer to what we've seen in Python.
  - xapata 6y ago
    The Python community has considered an "official" packaging tool in the past, but in those conversations found that the community had too many preferences to find a good compromise. That's the trouble with having a highly diverse set of uses and integrations, and lots of legacy. If you're curious, the email threads about Conda and defining wheels are interesting.

dagenix 6y ago

> Version pinning can be done in setup.py using the same syntax you would see in a requirements.txt file The problem with this approach is that it doesn't handle transitive dependencies well. Say you depend on version 1.4.6 of a particular library. And then that library depends on version >= 2 of some other library. When you install your package, you know that you'll get version 1.4.6 of the first library but have no idea what version you'll get of the second library. You can of course pin all the transitive dependencies - except that clutters up the setup.py and is a massive pain to keep up to date as you bump individual dependency versions.

knome 6y ago
Seems like a solid argument for a switch to use go's minimal version selection the version selected during a build is the one with the most minimal version that satisfies all other constraints. this means if you have libA that needs dep>=1.1 and libB that needs dep>=1.3, you get dep=1.3 even if dep1.9 is out. your build never changes because of a new version release, as long as they release with proper semantic versioning. if you later include libC that needs dep>=1.8, you'll get that version. but because you changed your immediate dependencies, not due to a surprise down the dependency line. https://research.swtch.com/vgo-principles https://research.swtch.com/vgo-principles
- dagenix 6y ago
  Its at least consistent - which, IMO, is better than just getting a random version. I do, however, think it's a bit unfortunate that it prevents picking up bug / security fixes in transitive dependencies. Imagine that you depend on library A of a particular version which itself depends on library B. With minimal version selection, as long you don't bump you dependency on library A (or some other library that depends on library B), you'll continue to get that same version of library B. But, then library B releases a critical security fix. With minimal version selection, there isn't a great way to pick up that fix. You can _hope_ that library A releases a new version that requires the fix - but that may or may not happen and could take a while. Or, you could add an explicit dependency on the new version of library B - which is unfortunate, since, your main package doesn't depend on library B directly. Lock files solve this problem. You can depend on whatever version of library A that you need and lock the transitive dependencies. And once library B releases its fix, you can update your lock file without having up bump the version of library A. Tools like poetry providing the features to automate this workflow.

BerislavLopac 6y ago

> You should be very conservative when pinning versions in a library, though No; you should be very conservative when pinning versions in an application, not in a library. Check this article for the explanation: https://caremad.io/posts/2013/07/setup-vs-requirement/ https://caremad.io/posts/2013/07/setup-vs-requirement/

toyg 6y ago
Anybody familiar with the history of requests knows this is bad advice.
- BerislavLopac 6y ago
  requests is definitely not a good example here. :/