13 ms·

Python's New Package Landscape

325 points by teajunky 8y ago

samwillis 8y ago
Not really packaging but related, my favourite new tool is Pyenv (https://github.com/pyenv/pyenv https://github.com/pyenv/pyenv) it made getting a new laptop setup with various versions of Python so much quicker. I haven’t used Pipenv yet but it works with pyenv to create virtual envs with a specified puthon version as well as all the correct packages.
- rhizome31 8y ago
  Polyglots might also be interested in asdf (https://github.com/asdf-vm/asdf https://github.com/asdf-vm/asdf). It's like Pyenv but supports various languages via a plugin system (eg. https://github.com/danhper/asdf-python https://github.com/danhper/asdf-python).
  - Bogdanp 8y ago
    For those wondering (like I did): it doesn't seem like this has any relation to Common Lisp's ASDF.
Bogdanp 8y ago
Having used pure pip + virtualenv{,wrapper}, pip-tools + virtualenv, poetry and Pipenv for medium to large applications, I'm going to be sticking to pip-tools for the time being for apps. Poetry is fine, but pip-tools is faster and there's less to learn. Pipenv is unbearably slow for large applications and often buggy. For libraries, I've been using Poetry for molten[1] and pure setuptools for dramatiq[2] and, at least for my needs, pure setuptools seems to be the way to go. [1]: https://github.com/Bogdanp/molten https://github.com/Bogdanp/molten [2]: https://github.com/Bogdanp/dramatiq https://github.com/Bogdanp/dramatiq
- epage 8y ago
  The problem I ran into with pip-tools (and I assume pipenv has this) is that the lock files are platform and python-version specific (evaluates all of the conditionals for the given platform) when I do a lot of stuff cross-platform.
- deleted 8y ago
  [deleted]
whitehouse3 8y ago
While pipenv has garnered a lot of attention and praise for ease of use, it falls over whenever I integrate it with any serious work. Pipenv lock can take 20-30 minutes on a small flask app (~18 dependencies). And it often mixes up virtualenvs, enabling the wrong one with seemingly no remedy. I see the problems on Windows, MacOS and Ubuntu. 2018 is not the year of pipenv, for me. I'm sticking with regular virtualenvs and the manual-hell of requirements.txt. I hope it gets better eventually.
- prepend 8y ago
  Am I weird that I use anaconda instead of virtualenvs? I guess it’s overkill if you aren’t using the other conda features.
  - azag0 8y ago
    It is an overkill for pure-python packages or packages with simple C extensions. Conda was developed specifically to handle non-python dependencies, which would be difficult to build in setup.py. Also, a conda package is not a replacement for a distutils/setuptools package. When building a conda package, one still calls setup.py. So every python conda package has to be a distutils/setuptools package anyway.
    - 131012 8y ago
      Thanks for caveat. Nonetheless, anaconda makes my life so much easier when working with python libraries. If anybody got any other reasons to be careful of it, I'm interested!
      - flatfilefan 8y ago
        If you need to work with cutting-edge python tools (e.g.from GitHub) it’s often easier to use virtualenv to control the versions you need installed.
        dagw 8y ago
dlitvakb 8y ago
I have been using purely setuptools for all of our open source Python libraries at Contentful, but have found that lately I've been getting deprecation warnings from PyPI not to use `setup.py upload` anymore. What should the alternative be now? Edit: I'm reading about twine right now, but I cannot begin to comprehend why it's not bundled directly if this is what they are intending for us to use to upload packages.
- Bogdanp 8y ago
  Twine seems to be the recommended[1] way. It's pretty straightforward to use, thankfully. [1]: https://packaging.python.org/tutorials/packaging-projects/#uploading-the-distribution-archives https://packaging.python.org/tutorials/packaging-projects/#u...
- deleted 8y ago
  [deleted]
- toyg 8y ago
  Anything PyPI-related has recently gone into the (terrible) habit of recommending very recent (and often half-baked) tools that live entirely outside of stdlib. It seems pretty silly to me, considering Python core developers made significant efforts to bundle and support pip and virtualenv (venv) in the stdlib precisely to avoid having a lot of de-facto essential libraries outside the core distribution. If the problem is that stdlib cannot move as fast as PyPI-related development requires, maybe that should be fixed, rather than trying to bypass all quality checks and then relying on obscure shared knowledge to navigate the ecosystem. Maybe there should be a system where specific network-sensitive stdlib modules could be updated faster than the rest.
  - di 8y ago
    You're mostly right, the problem is also that users don't upgrade their Python distribution very often, so they miss out on new features. > Maybe there should be a system where specific network-sensitive stdlib modules could be updated faster than the rest. This is essentially what `setuptools` does, by putting a package on PyPI that monkeypatches/plugs in to the stdlib.
liveoneggs 8y ago
this incredible complexity is what docker is really good at simplifying
- emptysea 8y ago
  Maybe for deployment, but this problem has been solved well by both Yarn and Cargo.
  - liveoneggs 8y ago
    pretty sure this is about python
- deleted 8y ago
  [deleted]
- yxhuvud 8y ago
  Uh, docker doesn't help at all with handling dependency upgrades. Good package-manager/version-manager combinations does just that.
scrollaway 8y ago
Published May 11th, 2018. But it's interesting it's popping up again. It's a good explanation of the landscape as of 2018, though Pipenv has since gone in a weird direction. There's a lot of recommendations for it, but I sometimes get the feeling people don't understand what they're recommending, such as replacing some things that work (setup.cfg) by things that don't do the same thing (Pipfile). Man, the Python packaging ecosystem is one of those things which really bring me down regarding the state of Python, because there is such an extremely high barrier for breaking backwards compatibility and nothing really works. The JS ecosystem is far better in this regard. Pipenv was most promising because it followed in Yarn's footsteps, but it didn't go all the way in replacing pip (which it really should have). So now there's still a bunch of stuff handled by pip, which pipenv does not / cannot know about, and this isn't really fixable. The end result is that instead of telling people about pip + virtualenv, we now have pip, virtualenv and pipenv to talk about. And people who don't understand the full stack, and the exact role of each tool, can't really understand how to properly do the tasks we choose to recommend delegating to each one of them. There's three separate-but-related use cases: - "Installing a library" (npm install; pip install). - "Publishing a library" (setup.py. Or Twine if you're using a tool. Both use setuptools.). - "Deploying a Project", local dev or production (pipenv. Well, if it's configured with a pipfile, otherwise virtualenv, and who knows where your dependencies are, maybe requirements.txt. Pipenv does create a virtualenv anyway, so you can use that. Anyway you should be in docker, probably. Make sure you have pip installed systemwide. Yes I know it comes with python, but some distributions remove it from Python. Stop asking why, it's simple. What do you mean this uses Python 3.6 but there's only Python 3.5 available on Debian? Wait, no, don't install pyenv, that's not a good idea! COME BACK!) The JS ecosystem manages to have two tools, both of which can do all of this. I don't know how we keep messing up when we have good prior work to look at.
- deleted 8y ago
  [deleted]
- acdha 8y ago
  > - "Deploying a Project", local dev or production (pipenv. Well, if it's configured with a pipfile, otherwise virtualenv, and who knows where your dependencies are, maybe requirements.txt. Pipenv does create a virtualenv anyway, so you can use that. Anyway you should be in docker, probably. Make sure you have pip installed systemwide. Yes I know it comes with python, but some distributions remove it from Python. Stop asking why, it's simple. What do you mean this uses Python 3.6 but there's only Python 3.5 available on Debian? Wait, no, don't install pyenv, that's not a good idea! COME BACK!) This makes the situation sound a lot more complex than it actually is by conflating separate layers: the system distribution issue is exactly the same for both Python and JS (if Debian ships an old v8 you either need to install a new one, perhaps using Docker to make that easy and isolated). Similarly, the question of whether you install the app using pip or pipenv is a different layer from whether you're using Docker or not, just as Docker is unrelated to the question of whether you use npm or yarn. For a new project in 2018, you can simply say “Use pipenv. Deploy in Docker, using pipenv.” and it works as well as the JS world. People sometimes choose to make their projects too complicated or to manage things at the wrong level but that's a social problem which is hard to solve with tooling.
ericcholis 8y ago
I gave conda a shot and found it to be better than pip + virtualenv, but still not amazing.
- fifnir 8y ago
  I did the same and found pip_venv to be much superior
antpls 8y ago
No mention of containers? I didn't write Python code since a while now, but it would have been nice to have a comparison with container technologies, which weren't available at the creation time of pypi and pip. Containers solve both the problems of the article : isolation and repeatability, for any language. Are virtual env tools still needed in the container era?
- Bogdanp 8y ago
  At least on non-Linux systems, containers are far more heavyweight than something like virtualenv.
- bovermyer 8y ago
  Containers would seem to solve the isolation part of the problem, but dependency management is not something containers can deal with effectively.
  - antpls 8y ago
    Big monolithic Python projects will face dependency issues for sure. However, softwares structured into simpler, smaller components, using the right languages for the right tasks, will probably have simpler dependencies for each modules. That's what Go and tools like Bazel allows for : static builds, which forces to modularize the project into smaller independent components. In case of static builds, the protocol between components is the C ABI, or an RPC protocol, but it could be a mesh of microservices too. What is currently happening with the explosions of tools with Python is the result (take it with a grain of salt, only my opinion) of people only working with Python and not exploring enough outside of it
    - aprdm 8y ago
      Can you give practical examples of what you are mentioning? Like, how to achieve ""static builds"" with python? Got me interested!
      - xapata 8y ago
hultner 8y ago
I've migrated to pipenv in most of my projects, it's simple and great for application development but I still write everything to work with pure pip as well so the Pipfile basically lists my application as a dependency and I mainly use it for the lock files. For library development I target pure pip/setuptools but still use pipenv during development phase. There have been a few cases where pipenv had problems and I had to either remove my virtualenv and reinitialize it or even remove my pip-file/lockfile, but since I still have my setup.py it's not a big deal for me. As for uploading etc I use twine but I wrap everything in a makefile to make handling easier. A problem I noticed recently was a case where one of my developers used a tool which was implicitly installed in the testing environment since it was a subdependency of a testing tool but it was not installed into the production image. This resulted in "faulty" code passing the CI/CD and got automatically deployed to the live development environment where it broke (so it never reached staging). Caused a little bit of a headache before I found the cause.
azag0 8y ago
For pure-python library projects, I found Poetry the best option these days (haven't tried Hatch). But it is still heavily under development, so it's not necessarily a black-box solution. The biggest pain point of Pipenv for me is that it cannot as yet selectively update a single dependency without updating the whole environment.
datavirtue 8y ago
Who curates the packages to prevent security issues?
- blattimwind 8y ago
  That's such a cute idea.
- cpburns2009 8y ago
  This is a glaring issue with Python. Does any other language package repository implement any security? (I honestly don't know the state with other languages.)
phren0logy 8y ago
I'm not a programmer by trade, but I dabble, and these issues make it much less fun. In my limited experience, Clojure's Leinengen is a far more pleasant way to solve these problems. I'm sure there are many other examples in other languages, but in the few I've used, nothing comes close. Each project has versioned dependencies, and they stay in their own little playground. A REPL started from within the project finds everything. Switch directories to a different project, and that all works as expected, too. It's a dream. [https://leiningen.org/ https://leiningen.org/]
- deleted 8y ago
  [deleted]
CMCDragonkai 8y ago
I've tried a lot of solutions, but nix-shell hands down is the best I've used. I wrote a little gist detailing how to develop in python using a Nix: https://gist.github.com/CMCDragonkai/b2337658ff40294d251cc79d12b34224 https://gist.github.com/CMCDragonkai/b2337658ff40294d251cc79...
epage 8y ago
The biggest problem I have with python packaging tools is how do I start using them. I'd rather not install all of them in my global site-packages. Do I need to create a virtualenv just to get a tool to manage my virtualenv's? I have seen poetry is working on their bootstrapping story. I could not get their current solution to work on Ubuntu. Maybe what they are developing towards will work. https://github.com/sdispater/poetry/issues/342 https://github.com/sdispater/poetry/issues/342
- blattimwind 8y ago
  pip install --user
  - gtycomb 8y ago
    I stumbled on this (--user flag) only the other day and it simplified things immensely. Too much information, various ways of doing things explained in bits and pieces over the decades makes it look confusing. Often there is a very simple way, even in Python packaging and deploying. In my situation the easiest way began along these lines -- 1. Install python3 for the local user from the source distribution (make sure you have compilers etc that the configure check lists out) 2. After compiling the sources and finishing with 'make install', make Python available in your local search path 3. And use pip with this magical --user flag as needed. No virtual env, conda, etc etc. 4. Leave HOMEPATH etc alone as this conflicts with the setup of the admin's system wide installs (when you su) Things can go smoothly with pip alone.
  - epage 8y ago
    Doesn't that just install into a global-to-the-user path? Isn't one of the things we're trying to avoid is conflicts between these tools. For example, pip is now more freely breaking compatibility. I now need to ensure if my different packaging tools are compatible with my version of pip all installed in my user location.
    - mixmastamyk 8y ago
      Yes, and that's fine for most folks. The one exception I make is developing a huge app at work, use a virtualenv for that to keep it separate.
mattdeboard 8y ago
Interestingly this URL gets blocked by my work's security thing. Never saw that before. edit: I requested an exemption but corp IT staff came back and said there's definitely been malware identified on that site. So... be careful with your clicks. edit2: Well who knows where the malware alert is coming from, might be an ad or something.
andybak 8y ago
In case this scares any new users, I've used nothing more than pip and virtualenv for several years with no issues of note.
- linuxftw 8y ago
  Same. I feel people invent problems in order to not use these tools.
- wepower_ico 8y ago
  Totally agree. And why not just invest time into pip than invent again.
- philosopherlawr 8y ago
  There are lots of errors when it comes to reproducing the build on other machines. Pip install -r requirements.txt does not guarantee that you will install the same version of packages on a new machine, and in fact, you will typically not.
  - jabwork 8y ago
    I've never had problems with this when requirements.txt contains package versions Have you, or are you not using explicit versions supplied by eg pip freeze?
- peterkelly 8y ago
  Seconded. I read this article and thought "nope, not touching any of those tools". I'd rather spend my time building products than spending a week researching the landscape of 10 different package management approaches.
- SSilver2k2 8y ago
  Same. pip + virtualenv just works.
sambe 8y ago
Whenever talk in Python-world goes towards packaging, I feel like I have been transported to Javascript-world: it's never clear to me what concrete problems are being solved by the new tools/libraries. This article seems well-written and well-intentioned. Despite reading it, I don't know why I would not have loose dependencies in setup.py and concrete, pinned dependencies in requirements.txt. It's never felt hard to manage or to sync up - the hard part is wading through all the different tools and recommendations.
- lmm 8y ago
  > loose dependencies in setup.py How does that work? How would someone else coming to work on your project use them? > concrete, pinned dependencies in requirements.txt How do you maintain that requirements.txt? And while that might work for applications, what do you do for libraries?
  - sambe 8y ago
    I assume that someone working on the project would do: pip install -e . in a virtual environment. I thought this was quite well-established. Is there a problem with it that I'm not aware of? pip freeze > requirements.txt for requirements.txt generation. For libraries just omit this? I'm not sure I understand the question. The article also mentions that several of the new tools aren't appropriate for libraries anyway.
    - lmm 8y ago
      > I assume that someone working on the project would do: pip install -e . in a virtual environment. I thought this was quite well-established. Is there a problem with it that I'm not aware of? So ignoring your requirements.txt, and potentially working with different versions of dependencies from the ones you were working with and encountering different bugs? (Also managing your virtual environments "by hand" is tedious and error-prone when you're working on multiple projects). > pip freeze > requirements.txt for requirements.txt generation. The problem with this is that it's not reproducible - if two people try to run it they might get different results, and it's not at all obvious who should "win" when the time comes to merge. If you mess up the merge and re-run then maybe you get a different result again, and have to do all your testing etc. over again. > For libraries just omit this? Maybe, but then you'll face a lot of bug reports from people who end up running your library against different versions of upstream libraries from the ones that you tested against.
patagonia 8y ago
Imagine I’m just getting started with Python, and I see this article. I think to myself, “Awesome, a primer!” Then I start reading (these comments)... mayyybe I should try Julia... or anything else, at least while I’m still getting started.
- vxNsr 8y ago
  Kinda exactly what went through my mind, I've dabbled in python, but only to use other ppl's projects, which sometimes required setting up pip, etc. So I was looking forward to reading about how it'd been simplified :(
- zitterbewegung 8y ago
  This isn’t a primer but more like a survey of packaging options. Using pip and virtualenv is usually fine or using pipenv .
  - patagonia 8y ago
    “Usually” is great until it’s not. I don’t want to get invested in a languages only to discover issues down the line. So, if something as basic as packaging is potentially problematic and there are other options available, I might look elsewhere before rolling the dice that this problem is not too problematic. Perhaps “Primer” was the wrong word, but I believe the sentiment is valid. Reading the comments, there is simply no consensus. If code readablility is important since code will be read more than written, packing is important because in many many scenarios that count code will be distributed more than it will be read. It is simply frustrating that the typical response to comments like my original comment is some form of “it’s not as bad as you think”. Look, we have a problem here. A problem many other languages deem important enough to solve upfront. It’s been a problem for a long long time.
    - aprdm 8y ago
      pip and virtualenv has been solving the problem and is the standard for ever. other people tried different approaches, like, pipenv and is just that, a separate project trying to solve the same problem. I don't know why / who said that pipenv is the official recommended way, if it is it should not be and I hope it is not.
breatheoften 8y ago
I’m looking forward to a future where I no longer have to use languages that require the use of different mechanisms to reference functionality from library code than one uses to take advantage of your own source ... all the incidental complexity around custom compilation processes are in reality, just enormously non-productive relics of the past. In the future — you have a set of entry points to your program, these are crawled by the language aware tool chain to identify and assemble all the requirements for the program (including 3rd party functionality). There’s no need for separate tools to manage packages, caches, and virtual environments — let’s just put all this logic into the compiler(s) — where necessary let the application describe the necessary state of the external world and empower language toolchains to ensure that it’s so ... let’s live in the future already ...
agumonkey 8y ago
(incf confusion)
cik 8y ago
We just went through this cycle - ultimately we build packages (debs) and dockers, for deployment within VMs. Our build process - depending on the component pushes the deb to repos, or uses the deb in the docker. After trying to replace pip with Pipenv, we had to stop. The dependency resolution time for 20 declared dependencies (that in turn pull down > 100 components) takes well over 5 minutes. With poetry - it takes less than 33 seconds on a clean system. The times are consistent for both Ubuntu 16.04 and Mac OS X. Our only goal is to get to the point we're now in - tracking dependencies, and separate dev requirements (like ipython and pdbpp) from our other requirements. Poetry made it fast, simple, and made me an addict. Over two days, I moved our entire codebase and every single (active) personal project I had to poetry. I don't regret it :)
- abalaji 8y ago
  I can second this... poetry is a seriously good project and matches PEP standards for the pyproject.toml, you will definitely see more projects jumping onto that standard soon.
  - cik 8y ago
    That was the part that made me try it, literally while waiting for Pipenv to resolve.
  - ihumanable 8y ago
    Sébastien Eustace is a really awesome developer, I've used Pendulum and Orator as well as Poetry for projects, the documentation is beautiful and complete, the APIs are well thought out, and if you find an issue contributing back to the project is simple and straightforward (I'm happy to have a few PRs accepted into Orator). There's a certain joy working with tools when it's clear that the person making those tools actually cares about the developer and making it work well.
- loop0 8y ago
  I did have the same problem, and apart from taking too long for dependency resolution it felt like breaking up at each update, sometimes from pipenv update, sometimes from pip update making pipenv breaking. I migrated to poetry and never been happier.
xapata 8y ago
No mention of Anaconda?! How strange. I recommend using `conda` instead of virtualenv, and instead of pip where possible. A Python project does not only depend on Python modules, but non-Python modules as well. Beyond Python, conda helps manage your other dependencies, like your database. I use Miniconda instead of Anaconda, to avoid the initial mega-download.
Gaelan 8y ago
Huh, OpenDNS blocks this due to "a security threat that was discovered by the Cisco Umbrella security researchers."
deleted 8y ago
[deleted]
qwerty456127 8y ago
Sadly there are things you can't always install reliably from the Python repository. E.g. you may have to install things like scipy and keras from the OS or a 3-rd party (like brew or conda) repository as pip install would fail in the build process.
reilly3000 8y ago
Why is it that Python is geared towards archiving packages at the site level by default while npm, composer, et all tend towards including packages in the project's folder? Is it convention from a time when disk space was less plentiful?
- gary_bernhardt 8y ago
  "Why" is hard, but Python's packaging system was created when global installation was just how packaging worked. There may have been exceptions, but the first local package installation tool I knew of was workingenv (2006). It was the predecessor of virtualenv, which I think led directly to pipenv.
EamonnMR 8y ago
We tried Pipenv last year, ran into a number of bugs, this one being the most irritating: https://github.com/pypa/pipenv/issues/786 https://github.com/pypa/pipenv/issues/786
TheOtherHobbes 8y ago
I was trying to set up pipenv on a Mac earlier in the week. Being able to select P2 or P3 environments is great. Unfortunately it decided all my packages were in /var/mail. No patience to debug it, so I gave up on it.
binalpatel 8y ago
I find conda far and above the best tool to manage python packages and dependencies. Being able to concisely contain all Python and binary dependencies together is invaluable. I recently wrote about it as blog post, using conda within containers has solved almost every pain point we had with python packaging and how to get things into production reliably.
- orf 8y ago
  Counterpoint: it's terrible. A lack of a lockfile is a killer, plus it not really fitting well with the general ecosystem (it's not really a python dependency manager). It's an all or nothing tool, which sucks to be honest Now most projects have wheels pip is pretty damn good. The conda CLI is also just terrible. It's good for ad-hoc research, but for big deployments? No thanks, I've had enough pain using it.
erikb 8y ago
If you only think in Python then packaging really might be a painpoint. But honestly if we look at other languages it's not so bad actually, very specifically calling out Golang here because it claimed exactly this topic as an initial design goal and to this day has basically failed at delivering it. There are even languages like C++ where the community as a (w)hole has given up on that topic and instead opts for completely building every tool by building the underlying libraries up first manually. Considering all this, who can actually beat Python at this point? Java maybe? Is Ruby still competing? How is NodeJS doing? Currently with what I see around me (mostly Go and C++) I don't feel too bad about setuptools+pip+virtualenv anymore.
- johnlinvc 8y ago
  Ruby's bundler is pretty neat. It's one of the first systems that introduced lock files.
luord 8y ago
I rarely even use requirements.txt and never use it in my personal projects. I just pin the project's direct dependencies in the setup.py file and install the folder directly. I know it might cause bugs with different developers (or the CI) using different versions of the upstream dependencies but I guess I trust the developers who create each library I'm using. The moment I directly import something from what used to be an upstream dependency, I pin it too. So far this approach hasn't given me trouble, but I'll still take a look at poetry based on what I read in the comments here.
jimbo1qaz 8y ago
Pipenv is unusable for me, since launching your app only works when your current working directory is the Pipfile directory. If you want to launch an app via a shell script from another directory, you have to first cd to the Pipfile dir, pipenv shell (maybe you can pass in a second shell script as an argument). The article mentions Pipsi is designed to make command-line apps globally accessible, and I'll try it out. Additionally, adding git/src/package/module.py may be fine when you're using an IDE, but when browsing in a file manager, you must navigate 3 directories deep to even see any source files, which seems to be trending towards the inconvenience amd pain of Java projects.

325 points by teajunky 8y ago

Not really packaging but related, my favourite new tool is Pyenv (https://github.com/pyenv/pyenv https://github.com/pyenv/pyenv) it made getting a new laptop setup with various versions of Python so much quicker. I haven’t used Pipenv yet but it works with pyenv to create virtual envs with a specified puthon version as well as all the correct packages.

rhizome31 8y ago
Polyglots might also be interested in asdf (https://github.com/asdf-vm/asdf https://github.com/asdf-vm/asdf). It's like Pyenv but supports various languages via a plugin system (eg. https://github.com/danhper/asdf-python https://github.com/danhper/asdf-python).
- Bogdanp 8y ago
  For those wondering (like I did): it doesn't seem like this has any relation to Common Lisp's ASDF.

Bogdanp 8y ago

Having used pure pip + virtualenv{,wrapper}, pip-tools + virtualenv, poetry and Pipenv for medium to large applications, I'm going to be sticking to pip-tools for the time being for apps. Poetry is fine, but pip-tools is faster and there's less to learn. Pipenv is unbearably slow for large applications and often buggy. For libraries, I've been using Poetry for molten[1] and pure setuptools for dramatiq[2] and, at least for my needs, pure setuptools seems to be the way to go. [1]: https://github.com/Bogdanp/molten https://github.com/Bogdanp/molten [2]: https://github.com/Bogdanp/dramatiq https://github.com/Bogdanp/dramatiq

epage 8y ago
The problem I ran into with pip-tools (and I assume pipenv has this) is that the lock files are platform and python-version specific (evaluates all of the conditionals for the given platform) when I do a lot of stuff cross-platform.
deleted 8y ago
[deleted]

whitehouse3 8y ago

While pipenv has garnered a lot of attention and praise for ease of use, it falls over whenever I integrate it with any serious work. Pipenv lock can take 20-30 minutes on a small flask app (~18 dependencies). And it often mixes up virtualenvs, enabling the wrong one with seemingly no remedy. I see the problems on Windows, MacOS and Ubuntu. 2018 is not the year of pipenv, for me. I'm sticking with regular virtualenvs and the manual-hell of requirements.txt. I hope it gets better eventually.

prepend 8y ago
Am I weird that I use anaconda instead of virtualenvs? I guess it’s overkill if you aren’t using the other conda features.
- azag0 8y ago
  It is an overkill for pure-python packages or packages with simple C extensions. Conda was developed specifically to handle non-python dependencies, which would be difficult to build in setup.py. Also, a conda package is not a replacement for a distutils/setuptools package. When building a conda package, one still calls setup.py. So every python conda package has to be a distutils/setuptools package anyway.
  - 131012 8y ago
    Thanks for caveat. Nonetheless, anaconda makes my life so much easier when working with python libraries. If anybody got any other reasons to be careful of it, I'm interested!
    - flatfilefan 8y ago
      If you need to work with cutting-edge python tools (e.g.from GitHub) it’s often easier to use virtualenv to control the versions you need installed.
      - dagw 8y ago

dlitvakb 8y ago

I have been using purely setuptools for all of our open source Python libraries at Contentful, but have found that lately I've been getting deprecation warnings from PyPI not to use `setup.py upload` anymore. What should the alternative be now? Edit: I'm reading about twine right now, but I cannot begin to comprehend why it's not bundled directly if this is what they are intending for us to use to upload packages.

Bogdanp 8y ago
Twine seems to be the recommended[1] way. It's pretty straightforward to use, thankfully. [1]: https://packaging.python.org/tutorials/packaging-projects/#uploading-the-distribution-archives https://packaging.python.org/tutorials/packaging-projects/#u...
deleted 8y ago
[deleted]
toyg 8y ago
Anything PyPI-related has recently gone into the (terrible) habit of recommending very recent (and often half-baked) tools that live entirely outside of stdlib. It seems pretty silly to me, considering Python core developers made significant efforts to bundle and support pip and virtualenv (venv) in the stdlib precisely to avoid having a lot of de-facto essential libraries outside the core distribution. If the problem is that stdlib cannot move as fast as PyPI-related development requires, maybe that should be fixed, rather than trying to bypass all quality checks and then relying on obscure shared knowledge to navigate the ecosystem. Maybe there should be a system where specific network-sensitive stdlib modules could be updated faster than the rest.
- di 8y ago
  You're mostly right, the problem is also that users don't upgrade their Python distribution very often, so they miss out on new features. > Maybe there should be a system where specific network-sensitive stdlib modules could be updated faster than the rest. This is essentially what `setuptools` does, by putting a package on PyPI that monkeypatches/plugs in to the stdlib.

liveoneggs 8y ago

this incredible complexity is what docker is really good at simplifying

emptysea 8y ago
Maybe for deployment, but this problem has been solved well by both Yarn and Cargo.
- liveoneggs 8y ago
  pretty sure this is about python
deleted 8y ago
[deleted]
yxhuvud 8y ago
Uh, docker doesn't help at all with handling dependency upgrades. Good package-manager/version-manager combinations does just that.

scrollaway 8y ago

Published May 11th, 2018. But it's interesting it's popping up again. It's a good explanation of the landscape as of 2018, though Pipenv has since gone in a weird direction. There's a lot of recommendations for it, but I sometimes get the feeling people don't understand what they're recommending, such as replacing some things that work (setup.cfg) by things that don't do the same thing (Pipfile). Man, the Python packaging ecosystem is one of those things which really bring me down regarding the state of Python, because there is such an extremely high barrier for breaking backwards compatibility and nothing really works. The JS ecosystem is far better in this regard. Pipenv was most promising because it followed in Yarn's footsteps, but it didn't go all the way in replacing pip (which it really should have). So now there's still a bunch of stuff handled by pip, which pipenv does not / cannot know about, and this isn't really fixable. The end result is that instead of telling people about pip + virtualenv, we now have pip, virtualenv and pipenv to talk about. And people who don't understand the full stack, and the exact role of each tool, can't really understand how to properly do the tasks we choose to recommend delegating to each one of them. There's three separate-but-related use cases: - "Installing a library" (npm install; pip install). - "Publishing a library" (setup.py. Or Twine if you're using a tool. Both use setuptools.). - "Deploying a Project", local dev or production (pipenv. Well, if it's configured with a pipfile, otherwise virtualenv, and who knows where your dependencies are, maybe requirements.txt. Pipenv does create a virtualenv anyway, so you can use that. Anyway you should be in docker, probably. Make sure you have pip installed systemwide. Yes I know it comes with python, but some distributions remove it from Python. Stop asking why, it's simple. What do you mean this uses Python 3.6 but there's only Python 3.5 available on Debian? Wait, no, don't install pyenv, that's not a good idea! COME BACK!) The JS ecosystem manages to have two tools, both of which can do all of this. I don't know how we keep messing up when we have good prior work to look at.

deleted 8y ago
[deleted]
acdha 8y ago
> - "Deploying a Project", local dev or production (pipenv. Well, if it's configured with a pipfile, otherwise virtualenv, and who knows where your dependencies are, maybe requirements.txt. Pipenv does create a virtualenv anyway, so you can use that. Anyway you should be in docker, probably. Make sure you have pip installed systemwide. Yes I know it comes with python, but some distributions remove it from Python. Stop asking why, it's simple. What do you mean this uses Python 3.6 but there's only Python 3.5 available on Debian? Wait, no, don't install pyenv, that's not a good idea! COME BACK!) This makes the situation sound a lot more complex than it actually is by conflating separate layers: the system distribution issue is exactly the same for both Python and JS (if Debian ships an old v8 you either need to install a new one, perhaps using Docker to make that easy and isolated). Similarly, the question of whether you install the app using pip or pipenv is a different layer from whether you're using Docker or not, just as Docker is unrelated to the question of whether you use npm or yarn. For a new project in 2018, you can simply say “Use pipenv. Deploy in Docker, using pipenv.” and it works as well as the JS world. People sometimes choose to make their projects too complicated or to manage things at the wrong level but that's a social problem which is hard to solve with tooling.

ericcholis 8y ago

I gave conda a shot and found it to be better than pip + virtualenv, but still not amazing.

fifnir 8y ago
I did the same and found pip_venv to be much superior

antpls 8y ago

No mention of containers? I didn't write Python code since a while now, but it would have been nice to have a comparison with container technologies, which weren't available at the creation time of pypi and pip. Containers solve both the problems of the article : isolation and repeatability, for any language. Are virtual env tools still needed in the container era?

Bogdanp 8y ago
At least on non-Linux systems, containers are far more heavyweight than something like virtualenv.
bovermyer 8y ago
Containers would seem to solve the isolation part of the problem, but dependency management is not something containers can deal with effectively.
- antpls 8y ago
  Big monolithic Python projects will face dependency issues for sure. However, softwares structured into simpler, smaller components, using the right languages for the right tasks, will probably have simpler dependencies for each modules. That's what Go and tools like Bazel allows for : static builds, which forces to modularize the project into smaller independent components. In case of static builds, the protocol between components is the C ABI, or an RPC protocol, but it could be a mesh of microservices too. What is currently happening with the explosions of tools with Python is the result (take it with a grain of salt, only my opinion) of people only working with Python and not exploring enough outside of it
  - aprdm 8y ago
    Can you give practical examples of what you are mentioning? Like, how to achieve ""static builds"" with python? Got me interested!
    - xapata 8y ago

hultner 8y ago

I've migrated to pipenv in most of my projects, it's simple and great for application development but I still write everything to work with pure pip as well so the Pipfile basically lists my application as a dependency and I mainly use it for the lock files. For library development I target pure pip/setuptools but still use pipenv during development phase. There have been a few cases where pipenv had problems and I had to either remove my virtualenv and reinitialize it or even remove my pip-file/lockfile, but since I still have my setup.py it's not a big deal for me. As for uploading etc I use twine but I wrap everything in a makefile to make handling easier. A problem I noticed recently was a case where one of my developers used a tool which was implicitly installed in the testing environment since it was a subdependency of a testing tool but it was not installed into the production image. This resulted in "faulty" code passing the CI/CD and got automatically deployed to the live development environment where it broke (so it never reached staging). Caused a little bit of a headache before I found the cause.

azag0 8y ago

For pure-python library projects, I found Poetry the best option these days (haven't tried Hatch). But it is still heavily under development, so it's not necessarily a black-box solution. The biggest pain point of Pipenv for me is that it cannot as yet selectively update a single dependency without updating the whole environment.

datavirtue 8y ago

Who curates the packages to prevent security issues?

blattimwind 8y ago
That's such a cute idea.
cpburns2009 8y ago
This is a glaring issue with Python. Does any other language package repository implement any security? (I honestly don't know the state with other languages.)

phren0logy 8y ago

I'm not a programmer by trade, but I dabble, and these issues make it much less fun. In my limited experience, Clojure's Leinengen is a far more pleasant way to solve these problems. I'm sure there are many other examples in other languages, but in the few I've used, nothing comes close. Each project has versioned dependencies, and they stay in their own little playground. A REPL started from within the project finds everything. Switch directories to a different project, and that all works as expected, too. It's a dream. [https://leiningen.org/ https://leiningen.org/]

deleted 8y ago
[deleted]

CMCDragonkai 8y ago

I've tried a lot of solutions, but nix-shell hands down is the best I've used. I wrote a little gist detailing how to develop in python using a Nix: https://gist.github.com/CMCDragonkai/b2337658ff40294d251cc79d12b34224 https://gist.github.com/CMCDragonkai/b2337658ff40294d251cc79...

epage 8y ago

The biggest problem I have with python packaging tools is how do I start using them. I'd rather not install all of them in my global site-packages. Do I need to create a virtualenv just to get a tool to manage my virtualenv's? I have seen poetry is working on their bootstrapping story. I could not get their current solution to work on Ubuntu. Maybe what they are developing towards will work. https://github.com/sdispater/poetry/issues/342 https://github.com/sdispater/poetry/issues/342

blattimwind 8y ago
pip install --user
- gtycomb 8y ago
  I stumbled on this (--user flag) only the other day and it simplified things immensely. Too much information, various ways of doing things explained in bits and pieces over the decades makes it look confusing. Often there is a very simple way, even in Python packaging and deploying. In my situation the easiest way began along these lines -- 1. Install python3 for the local user from the source distribution (make sure you have compilers etc that the configure check lists out) 2. After compiling the sources and finishing with 'make install', make Python available in your local search path 3. And use pip with this magical --user flag as needed. No virtual env, conda, etc etc. 4. Leave HOMEPATH etc alone as this conflicts with the setup of the admin's system wide installs (when you su) Things can go smoothly with pip alone.
- epage 8y ago
  Doesn't that just install into a global-to-the-user path? Isn't one of the things we're trying to avoid is conflicts between these tools. For example, pip is now more freely breaking compatibility. I now need to ensure if my different packaging tools are compatible with my version of pip all installed in my user location.
  - mixmastamyk 8y ago
    Yes, and that's fine for most folks. The one exception I make is developing a huge app at work, use a virtualenv for that to keep it separate.

mattdeboard 8y ago

Interestingly this URL gets blocked by my work's security thing. Never saw that before. edit: I requested an exemption but corp IT staff came back and said there's definitely been malware identified on that site. So... be careful with your clicks. edit2: Well who knows where the malware alert is coming from, might be an ad or something.

andybak 8y ago

In case this scares any new users, I've used nothing more than pip and virtualenv for several years with no issues of note.

linuxftw 8y ago
Same. I feel people invent problems in order to not use these tools.
wepower_ico 8y ago
Totally agree. And why not just invest time into pip than invent again.
philosopherlawr 8y ago
There are lots of errors when it comes to reproducing the build on other machines. Pip install -r requirements.txt does not guarantee that you will install the same version of packages on a new machine, and in fact, you will typically not.
- jabwork 8y ago
  I've never had problems with this when requirements.txt contains package versions Have you, or are you not using explicit versions supplied by eg pip freeze?
peterkelly 8y ago
Seconded. I read this article and thought "nope, not touching any of those tools". I'd rather spend my time building products than spending a week researching the landscape of 10 different package management approaches.
SSilver2k2 8y ago
Same. pip + virtualenv just works.

sambe 8y ago

Whenever talk in Python-world goes towards packaging, I feel like I have been transported to Javascript-world: it's never clear to me what concrete problems are being solved by the new tools/libraries. This article seems well-written and well-intentioned. Despite reading it, I don't know why I would not have loose dependencies in setup.py and concrete, pinned dependencies in requirements.txt. It's never felt hard to manage or to sync up - the hard part is wading through all the different tools and recommendations.

lmm 8y ago
> loose dependencies in setup.py How does that work? How would someone else coming to work on your project use them? > concrete, pinned dependencies in requirements.txt How do you maintain that requirements.txt? And while that might work for applications, what do you do for libraries?
- sambe 8y ago
  I assume that someone working on the project would do: pip install -e . in a virtual environment. I thought this was quite well-established. Is there a problem with it that I'm not aware of? pip freeze > requirements.txt for requirements.txt generation. For libraries just omit this? I'm not sure I understand the question. The article also mentions that several of the new tools aren't appropriate for libraries anyway.
  - lmm 8y ago
    > I assume that someone working on the project would do: pip install -e . in a virtual environment. I thought this was quite well-established. Is there a problem with it that I'm not aware of? So ignoring your requirements.txt, and potentially working with different versions of dependencies from the ones you were working with and encountering different bugs? (Also managing your virtual environments "by hand" is tedious and error-prone when you're working on multiple projects). > pip freeze > requirements.txt for requirements.txt generation. The problem with this is that it's not reproducible - if two people try to run it they might get different results, and it's not at all obvious who should "win" when the time comes to merge. If you mess up the merge and re-run then maybe you get a different result again, and have to do all your testing etc. over again. > For libraries just omit this? Maybe, but then you'll face a lot of bug reports from people who end up running your library against different versions of upstream libraries from the ones that you tested against.

patagonia 8y ago

Imagine I’m just getting started with Python, and I see this article. I think to myself, “Awesome, a primer!” Then I start reading (these comments)... mayyybe I should try Julia... or anything else, at least while I’m still getting started.

vxNsr 8y ago
Kinda exactly what went through my mind, I've dabbled in python, but only to use other ppl's projects, which sometimes required setting up pip, etc. So I was looking forward to reading about how it'd been simplified :(
zitterbewegung 8y ago
This isn’t a primer but more like a survey of packaging options. Using pip and virtualenv is usually fine or using pipenv .
- patagonia 8y ago
  “Usually” is great until it’s not. I don’t want to get invested in a languages only to discover issues down the line. So, if something as basic as packaging is potentially problematic and there are other options available, I might look elsewhere before rolling the dice that this problem is not too problematic. Perhaps “Primer” was the wrong word, but I believe the sentiment is valid. Reading the comments, there is simply no consensus. If code readablility is important since code will be read more than written, packing is important because in many many scenarios that count code will be distributed more than it will be read. It is simply frustrating that the typical response to comments like my original comment is some form of “it’s not as bad as you think”. Look, we have a problem here. A problem many other languages deem important enough to solve upfront. It’s been a problem for a long long time.
  - aprdm 8y ago
    pip and virtualenv has been solving the problem and is the standard for ever. other people tried different approaches, like, pipenv and is just that, a separate project trying to solve the same problem. I don't know why / who said that pipenv is the official recommended way, if it is it should not be and I hope it is not.

breatheoften 8y ago

I’m looking forward to a future where I no longer have to use languages that require the use of different mechanisms to reference functionality from library code than one uses to take advantage of your own source ... all the incidental complexity around custom compilation processes are in reality, just enormously non-productive relics of the past. In the future — you have a set of entry points to your program, these are crawled by the language aware tool chain to identify and assemble all the requirements for the program (including 3rd party functionality). There’s no need for separate tools to manage packages, caches, and virtual environments — let’s just put all this logic into the compiler(s) — where necessary let the application describe the necessary state of the external world and empower language toolchains to ensure that it’s so ... let’s live in the future already ...

agumonkey 8y ago

(incf confusion)

cik 8y ago

We just went through this cycle - ultimately we build packages (debs) and dockers, for deployment within VMs. Our build process - depending on the component pushes the deb to repos, or uses the deb in the docker. After trying to replace pip with Pipenv, we had to stop. The dependency resolution time for 20 declared dependencies (that in turn pull down > 100 components) takes well over 5 minutes. With poetry - it takes less than 33 seconds on a clean system. The times are consistent for both Ubuntu 16.04 and Mac OS X. Our only goal is to get to the point we're now in - tracking dependencies, and separate dev requirements (like ipython and pdbpp) from our other requirements. Poetry made it fast, simple, and made me an addict. Over two days, I moved our entire codebase and every single (active) personal project I had to poetry. I don't regret it :)

abalaji 8y ago
I can second this... poetry is a seriously good project and matches PEP standards for the pyproject.toml, you will definitely see more projects jumping onto that standard soon.
- cik 8y ago
  That was the part that made me try it, literally while waiting for Pipenv to resolve.
- ihumanable 8y ago
  Sébastien Eustace is a really awesome developer, I've used Pendulum and Orator as well as Poetry for projects, the documentation is beautiful and complete, the APIs are well thought out, and if you find an issue contributing back to the project is simple and straightforward (I'm happy to have a few PRs accepted into Orator). There's a certain joy working with tools when it's clear that the person making those tools actually cares about the developer and making it work well.
loop0 8y ago
I did have the same problem, and apart from taking too long for dependency resolution it felt like breaking up at each update, sometimes from pipenv update, sometimes from pip update making pipenv breaking. I migrated to poetry and never been happier.

xapata 8y ago

No mention of Anaconda?! How strange. I recommend using `conda` instead of virtualenv, and instead of pip where possible. A Python project does not only depend on Python modules, but non-Python modules as well. Beyond Python, conda helps manage your other dependencies, like your database. I use Miniconda instead of Anaconda, to avoid the initial mega-download.

Gaelan 8y ago

Huh, OpenDNS blocks this due to "a security threat that was discovered by the Cisco Umbrella security researchers."

deleted 8y ago

[deleted]

qwerty456127 8y ago

Sadly there are things you can't always install reliably from the Python repository. E.g. you may have to install things like scipy and keras from the OS or a 3-rd party (like brew or conda) repository as pip install would fail in the build process.

reilly3000 8y ago

Why is it that Python is geared towards archiving packages at the site level by default while npm, composer, et all tend towards including packages in the project's folder? Is it convention from a time when disk space was less plentiful?

gary_bernhardt 8y ago
"Why" is hard, but Python's packaging system was created when global installation was just how packaging worked. There may have been exceptions, but the first local package installation tool I knew of was workingenv (2006). It was the predecessor of virtualenv, which I think led directly to pipenv.

EamonnMR 8y ago

We tried Pipenv last year, ran into a number of bugs, this one being the most irritating: https://github.com/pypa/pipenv/issues/786 https://github.com/pypa/pipenv/issues/786

TheOtherHobbes 8y ago

I was trying to set up pipenv on a Mac earlier in the week. Being able to select P2 or P3 environments is great. Unfortunately it decided all my packages were in /var/mail. No patience to debug it, so I gave up on it.

binalpatel 8y ago

I find conda far and above the best tool to manage python packages and dependencies. Being able to concisely contain all Python and binary dependencies together is invaluable. I recently wrote about it as blog post, using conda within containers has solved almost every pain point we had with python packaging and how to get things into production reliably.

orf 8y ago
Counterpoint: it's terrible. A lack of a lockfile is a killer, plus it not really fitting well with the general ecosystem (it's not really a python dependency manager). It's an all or nothing tool, which sucks to be honest Now most projects have wheels pip is pretty damn good. The conda CLI is also just terrible. It's good for ad-hoc research, but for big deployments? No thanks, I've had enough pain using it.

erikb 8y ago

If you only think in Python then packaging really might be a painpoint. But honestly if we look at other languages it's not so bad actually, very specifically calling out Golang here because it claimed exactly this topic as an initial design goal and to this day has basically failed at delivering it. There are even languages like C++ where the community as a (w)hole has given up on that topic and instead opts for completely building every tool by building the underlying libraries up first manually. Considering all this, who can actually beat Python at this point? Java maybe? Is Ruby still competing? How is NodeJS doing? Currently with what I see around me (mostly Go and C++) I don't feel too bad about setuptools+pip+virtualenv anymore.

johnlinvc 8y ago
Ruby's bundler is pretty neat. It's one of the first systems that introduced lock files.

luord 8y ago

I rarely even use requirements.txt and never use it in my personal projects. I just pin the project's direct dependencies in the setup.py file and install the folder directly. I know it might cause bugs with different developers (or the CI) using different versions of the upstream dependencies but I guess I trust the developers who create each library I'm using. The moment I directly import something from what used to be an upstream dependency, I pin it too. So far this approach hasn't given me trouble, but I'll still take a look at poetry based on what I read in the comments here.

jimbo1qaz 8y ago

Pipenv is unusable for me, since launching your app only works when your current working directory is the Pipfile directory. If you want to launch an app via a shell script from another directory, you have to first cd to the Pipfile dir, pipenv shell (maybe you can pass in a second shell script as an argument). The article mentions Pipsi is designed to make command-line apps globally accessible, and I'll try it out. Additionally, adding git/src/package/module.py may be fine when you're using an IDE, but when browsing in a file manager, you must navigate 3 directories deep to even see any source files, which seems to be trending towards the inconvenience amd pain of Java projects.