13 ms·
>sharing your code is even pain with colleagues even if they are using the same operating system, mainly because the Python requirement file doesn't pin depende
by ActorNightly 4y ago
>sharing your code is even pain with colleagues even if they are using the same operating system, mainly because the Python requirement file doesn't pin dependencies,
wat? Pretty sure you can use == in requirements.txt
Also, its very possible, and quite easy to just include the code for the library in your package, which effectively "locks in" the version. We did this all the time when building AWS lambda deployments.
- AlphaSite 4y agoI recommend using poetry and just moving on with your life, it’s far easier than trying to use pip (which is totally doable, but not clean).
- LtWorf 4y agoBut poetry requires pip… you can't use it without using pip
- gen220 4y agoPoetry requires pip in the way that `go mod` requires `go get`, i.e. Poetry allows one to operate at a higher level of abstraction, where it's harder to make mistakes and generally easier to manage your dependency tree.
- LtWorf 4y agoAbstractions are amazing until they break and you have to fix them.
- gen220 4y agoSure, this is true for every abstraction: some are more air-tight or leaky than others, but IME Poetry and go mod are fairly solid. I haven’t had to use pip in a long time. Just like I haven’t had to use `go get` in a while. I mean it’s not the most rock solid abstraction, but introducing sane package management in an OSS environment with a decade plus of history is very hard. Against that background, they are doing pretty well, IMO.
- biorach 4y agoThey clearly mean using pip on its own, presumably by `pip freeze > requirements.txt`
- ryanianian 4y agoKind of. It's intended to be a system package/tool. In that regard you can use, wait for it, yet another python tool - pipx. So instead of installing poetry to a venv or whatever using pip, you can use `pipx run poetry`. Now you have to install pipx...
- alexanderh 4y agoThis is what I came to the comment section to say... You absolutely can pin dependencies.... da fudge? Sounds like this guy needs to finish learning Python before he learns something else. From what you suggested, to containerizing things with something like Docker, there are ways to make Python more easily distributable.
- anacoluthe 4y agoWhat if the depedencies you pinned have non-pinned depedencies? packageA==1.0.0 depends itself on packageB Therefore, you can find yourself with a different set of deps. Had a bug like this once.
- im3w1l 4y agoPip freeze will pin explicit as well as transitive dependencies
- fnord123 4y agoIt's a hassle to do this correctly and upgrade the dependencies. Use poetry.
- robertlagrant 4y agopip freeze > requirements.txt
- fnord123 4y agoThat only generates a lock file. When you want to upgrade some of your dependencies and recalculate the correct versions, it doesn't help.
- robertlagrant 4y agoHow's that an issue? Here's an example of what happens: https://gist.github.com/robertlagrant/23489d8970ef6b4996030785f1e3ee58 https://gist.github.com/robertlagrant/23489d8970ef6b49960307...
- iasay 4y agoThat’s fine until it wont compile on someone else’s machine. Had that many times before with python. SciPy is a bastard on macs for example.
- geraneum 4y agoOn the other hand tools on par with SciPy are not common in other languages.
- jossclimb 4y agoYep, I have no idea how they do not even understand the basics of python dependency management. pip freeze > requirements.txt will do it all for you. No wonder they found rust to hard.
- vietthan 4y agorecommend pipreqs
- fnord123 4y ago`pip freeze > requirements.txt` will generate a lock file. You want a requirements listing as well as a lock file (like rust, poetry, golang, npm, ruby).
- deleted 4y ago[deleted]
- shrimpx 4y agoYour dismissal of the author is unwarranted. You can do you want you suggest, but it's an operational pain in the ass. You need to maintain two files, the actual requirements.txt and the `pip freeze` one that locks the environment. And you better never `pip install` anything by hand or you'll capture random packages in your frozen file, or else always take care to create your frozen file in a fresh virtualenv. And if you don't want to install your dev packages into the production environment, then you need to maintain two requirements.txt and two of those frozen files. The author mentions Poetry which does solve these issues with a nice interface.
- pletnes 4y agoSure can pin versions, it’s easy too, if you just use «pip freeze».
- anyfactor 4y agoBetter option is to use pipreqs. Virtual Environment gets on my way. Looking at you Jupyter notebook.
- pid-1 4y agoNow you lost track of which are your top level dependencies and which are dependencies of dependencies.
- _Wintermute 4y agoThen use pip compile.
- pid-1 4y agoSure. Or poetry, or pipenv, or whatever third party packaging system you enjoy. The whole point of GP is that python lacks a native sane packaging system, which I agree.
- pletnes 4y agoPython has several. It’s overall a good thing. It’s the reason you can package both one-file scripts and huge C++ / Fortran packages, which is the reason it has become a huge language in HPC / ML / AI / astronomy / data science / … I agree that it makes life more confusing for newbies, though.
- ska 4y agoThat only solves part of the problem. something like poetry's approach is the right one here; you need a list of core dependencies (not derived ones), you need a solver for when anything changes to find a new set of viable versions, and you need a lock file of some sort to uniquely & reproducible construct an environment.
- dagw 4y agoAlso, its very possible, and quite easy to just include the code for the library in your package This only works if installing on exactly the same os and architecture. It can also make the installer for your quick little command line tool hundreds of megabytes. That being said packing up the python interpreter and all dependencies is the approach I ended up using when shipping non-trivial python applications in the past.
- jeremydw 4y agoThis comment section itself clearly shows how crazy dependency and environment management is in Python. In this thread alone, we've received instructions to... - poetry - "Just pin the dependencies and use Docker" - pip freeze - Vendoring in dependency code - pipreqs - virtualenv This is simply a mess and it's handled much better in other languages. I manage a small agency team and there are some weeks where I feel like we need a full-time devops person to just help resolve environment issues with Python projects around the team.
- throwawaymaths 4y agoNow try installing tensorflow. Treat yourself to ice cream if you get it to install without having to reinstall Linux and without borking the active project you're on.
- raffraffraff 4y agoAnd unless things have gotten a lot better in the 2 years since I last did `pip install numpy` on ARM, prepare for a very long wait because you'll be building it from source.
- derac 4y agoThe major issues you'll see involve library version mismatches. It's a very good idea to use venv with these tools since there are often mismatches between projects.
- syassami 4y agoYou can go even further now with PEP508 https://peps.python.org/pep-0508/ https://peps.python.org/pep-0508/
- paulsef11 4y agoThere's lots of cases where you wouldn't want to pin your requirements.txt, the main one being if you're authoring a package. You need to leave the versions unpinned, preferably just bound to a major version, allowing some variability for the users of your package in case there's a shared dependency. I have a feeling that's what the author is describing here, because Poetry solves this dilemma by introducing a poetry.lock file which pins the dev versions of all the dependencies, but publishes a package with unpinned deps.
- giantrobot 4y agoThe issue is transitive dependencies. A dependency you pin isn't guaranteed to pin its own dependencies. A bug somewhere in a grandchild dependency can manifest for you even if you have a version pinned but the dependency did not. It's not automatically a problem but it certainly can become one.
- shrimpx 4y ago> Pretty sure you can use == in requirements.txt You can, but one of those packages that you depend on will have a loose version spec for one of its dependencies, making your `pip install -r requirements.txt` non-deterministic. Poetry and Pipenv solve this, though, by pinning all dependencies in a lock file.
- ActorNightly 4y agoPip freeze will grab all of the versions that are installed.
- linsomniac 4y agoI'm a huge, long time, python fan. I don't tend to have a lot of problems with dependencies, but it clearly is something that certain situations have problems with. As I understand the post, the author is saying "It sure is nice to be able to just hand off a go executable and be done with it." And I think we can agree that the Python runtime situation is far from this. I largely control my work environment, so this isn't a huge issue for me. But I'm right at this very moment converting a "python" to "python3" in a wrapper script because the underlying code got converted to py3 and can't "import configparser". (Actually, I'm removing "python" entirely and adding a shebamg). I've been looking at writing a small tool in Python and then porting it to Go (I don't know go, so seems like a reasonable way to approach it), because the main target of the app is my developers, who probably don't have the right Python environment set up, and I just want them to be able to use it, not give them a chore.
- Too 4y agoThis is wrong and OMG all other 15 answers to this comment even more wrong. Pinning a dependency in requirements.txt does not pin its transitive dependencies. This does NOT mean you should pip freeze everything into requirements.txt, because then how do you distinguish top level dependencies from transitive? The correct answer is to use lock file. No third party tools needed (1). In Python they named it constraints and requires an extra flag to use. pip freeze > constraints.txt. Then next time pip install -r requirements.txt -c constraints.txt. Oh, and always always always use a venv for each project. Globally installed packages is a recipe for disaster. (1) Poetry and Pipenv are still nice additions, with nicer project declarations in pyproject.toml and to save you from remembering pip flags or forgetting activation of venvs. But they are not strictly necessary and it’s honestly just 2 commands extra without them.