16 ms·
Python Wheels Crosses 90%
- groodt 6y ago325 of top 360 Python packages are now distributed as Wheels.
- navait 6y agoI don’t know a lot about Python tooling, but in general my experiences with pip have been pleasant, so I appreciate all the work done by the maintainers to make it pleasant.
- ciarannolan 6y agoCould someone give a beginner's tl;dr of wheels vs. eggs? I use Python extensively but the "Advantages of wheels" section on this site is way over my head. edit: Thanks everyone :)
- rantwasp 6y agohttp://packaging.python.org/discussions/wheel-vs-egg/ http://packaging.python.org/discussions/wheel-vs-egg/ use wheels
- thundergolfer 6y agoThis page does a good comparing them: https://packaging.python.org/discussions/wheel-vs-egg/ https://packaging.python.org/discussions/wheel-vs-egg/ Is anything in that unclear? In practice, this bit I think it's pretty important: > Wheel has a richer file naming convention. A single wheel archive can indicate its compatibility with a number of Python language versions and implementations, ABIs, and system architectures.
- polm23 6y agoThis is a pretty short overview. https://packaging.python.org/discussions/wheel-vs-egg/ https://packaging.python.org/discussions/wheel-vs-egg/ Having packaged wheels but not eggs, my understanding is somewhat limited, but eggs being unversioned and having weaker filename conventions sounds like it would be a pain. I mainly use binary wheels for projects that include compiled C extensions, so I need a different wheel for each supported platform. There's a straightforward filename convention for that, and all the tools support it, so I don't have to even think about that when generating or uploading wheels.
- groodt 6y agoWheels are newer and have an official PEP (PEP 427) Eggs contain compiled Python bytecode which requires publishing egg versions for every version of Python. While Wheels are distribution formats that can support more than a single Python version.
- u801e 6y agoOne thing I found when converting our python application packaging from RPM to wheels is that wheels don't properly handle the data_files parameter in the setup call. That is, it places files under the python library directory instead of in the absolute path as specified. This means that sample configuration files and init scripts end up in the wrong place on the file system. In order to get around this, we had to upload the source distribution to our devpi instance and run pip install with the --no-binary option which would then place those files in the correct directories. The other issue is that there's no equivalent of the %(config) RPM spec directive to prevent the config file from being overwritten if it already exists on the file system. So, for libraries, wheels are a good cross-platform packaging solution, but not so much for applications that require configuration files and init scripts.
- groodt 6y agoAgree. I don't think the Wheel specification was intended to replace application distribution. It is aimed at the distribution of individual Python libraries.
- enriquto 6y agoWhat's the difference between "applications" and "libraries"? Why can't they be distributed in the same way?
- oefrha 6y agoLibraries are self-contained and can unpack into any prefix. Applications tend to have config files, service description files, documentation, etc. scattered all over the system at specific locations in order to comply with host system conventions and inter-operate with supervisors and other applications. Wheel is an unpack-into-any-prefix (and stays in that prefix) binary distribution format.
- pansa2 6y agoPython libraries are distributed to other Python developers. Applications are distributed to end-users, who may not even know what Python is, let alone details of any Python installation on their machine.
- downerending 6y agoSo far it's just a bit of smoke on the horizon, but I'm noticing some packages abandoning 'pip' installs entirely in favor of 'conda'. It's a bit early to tell if this trend will take off, but it does seem plausible.
- dijksterhuis 6y agoI really hope conda stays on the user end. Like, I understand it's great for managing package dependencies/setting up environments etc for a single project. But I've found it's an absolute nightmare for building docker images/generally doing build stuff when it's involved. Not to mention conda installs seem to take waaaay longer than pip. apt + pip works far more sensibly and reliably in my experience.
- blt 6y agoconda seems to spend a lot of time in a SAT solver or something to deal with the known NP-complete problem of dependency resolution [1]. Maybe pip doesn't do the same thing for requirements.txt setups? [1] https://research.swtch.com/version-sat https://research.swtch.com/version-sat
- downerending 6y agoIt can be pretty bad, yes. A primary issue seems to be that conda package authors don't always do their deps right. Pip, on the other hand, often "wins" by not even bothering to notice the versions of low-level libraries, etc. If that doesn't work, you get to keep both pieces.
- wdobbels 6y agoIndeed, pip has no dependency resolver as of now, although it is being actively worked on [1], and they did secure some funding [2]. [1] https://github.com/pypa/pip/issues/988 https://github.com/pypa/pip/issues/988 [2] https://pyfound.blogspot.com/2019/11/seeking-developers-for-paid-contract.html https://pyfound.blogspot.com/2019/11/seeking-developers-for-...
- 6y ago
- fortran77 6y agoBut snakes don't have wheels. Why not call it "skins?"
- xapata 6y agoYou get wheels of cheese from the Cheese Shop. https://www.youtube.com/watch?v=Hz1JWzyvv8A https://www.youtube.com/watch?v=Hz1JWzyvv8A
- haberman 6y agoToday I spent at least an hour fighting with Python packaging. The more I think about it, the more I feel that self-contained static binaries are the way to go. Trying to load source files from all over the filesystem at runtime is hell. Or at least it's hell to debug when it goes wrong. I would love to see a move towards "static" binaries that package everything together into a single, self-contained unit.
- rtpg 6y agoPyOxidizer might be of interest to you.
- haberman 6y agoYes, that looks like exactly what I had in mind. Thanks for the reference.
- shoo 6y agoAround a decade ago I worked for a place that packaged and shipped python applications to windows desktops using a similar bundling tool, py2exe. It worked pretty fine. Pretty much just unzips into a directory: there's your self contained python app hiding inside an exe and some associated data files/dlls in the directory. We still used wheel archives etc for managing some python dependencies during our dev and build process, but it was never a complete solution: some python packages depend upon non python shared libraries so you need to install them using an operating system level package manager. Then when shipping the software to users you cannot assume the user has a working version of python, or even if they do you don't want to increase your support burden by having your application running inside some arbitrary python environment.
- pansa2 6y agoThanks, I hadn’t heard of PyOxidizer before. Looks like a significant improvement over tools like PyInstaller and Shiv, which produce self-extracting executables that are slow and/or leave files hanging around after running.
- 6y ago
- jessaustin 6y agoThis seems like a really effective way to set up a page for shaming "laggards". It would be interesting to track over time how many github issues are just links to this page.
- rlayton2 6y agoAs noted on the page, but this idea (at least in the python world) started with such a "Wall of shame" for libraries that hadn't updated to Python 3 support. I'm not sure how much progress was directly attributable to that site, but it was fairly widely referenced.
- jessaustin 6y agoHaha python is so authoritarian.
- thehiddenbrain 6y agoFor anyone working on python wheels requiring to compile c/cpp/fortran, this may be useful. See https://scikit-build.org https://scikit-build.org (Disclaimer: I am one of the maintainer)
- dralley 6y agoThanks for your work! I've used skbuild to package a couple of C library dependencies we have that would otherwise only be available to us as RPMs.
- awinter-py 6y agohave always wondered why pypi doesn't generate whl files for pure-python sdists and why companies like travis / github aren't more active in language-level packaging work github gives away so much free docker time -- faster installation would save them money directly
- X-Istence 6y agoBecause there is no way to know if something is pure-python or not. Even more so with pep517 and being able to use different build systems.
- awinter-py 6y agowait then how does bdist_wheel know to make platform-dependent vs platform-independent whl files?
- icegreentea2 6y agoYour hunch is right, setuptools is able to determine if it's pure python or not (https://packaging.python.org/guides/distributing-packages-using-setuptools/#pure-python-wheels https://packaging.python.org/guides/distributing-packages-us...) Maybe it's not reliable enough? In any case, I can see why PyPI wouldn't want to increase the scope of their work. Sometimes is super valuable to just be good at one thing (distribute what others give you).
- wirrbel 6y agoI think the pypi folks just don't have the resources / capacity to implement this. With appropriate funding this could have been done (development efforts wouldn't be that big, but probably operational costs for the build farms). In the past I ran (devpi builder)[https://github.com/blue-yonder/devpi-builder/ https://github.com/blue-yonder/devpi-builder/] with (devpi)[https://pypi.org/project/devpi/ https://pypi.org/project/devpi/] to build wheels for all my dependencies and keep them stored in a place I have control over.
- 6y ago
- usr1106 6y agoThat doesn't seem to be a particular fast adoption. I remember seeing the first wheels when working at a job I quit early 2010. So it must be over 10 years. Edit: A web search points to 2012, so maybe it's "only" 8 years? Edit 2: Pip came in 2008, so something changed somewhat before 2010 as I remembered. But what did it install if not wheels?
- deleted 6y ago[deleted]
- groodt 6y agoIf you were using PyPI you would have encountered source archives or Wheels. If you were not using PyPI you could have encountered source archives or Eggs.
- usr1106 6y ago> If you were using PyPI you would have encountered source archives or Wheels. Before 2010? That's what I believed to remember. But it looks like wheels did not exist before 2012, so that can't be true.
- groodt 6y agoIf you were using PyPI before Wheels were available, you would have encountered source distributions (sdist).
- jacobush 6y agoEggs?
- foldr 6y agoAh, so I've been confusing PyPI with PyPy. Someone made a great naming decision there.
- young_unixer 6y agoI confuse PyPI with PyPA, the maintainers of pip, which allows you to interact with PyPI.
- deleted 6y ago[deleted]
- wodenokoto 6y agoA too little known fact: PyPI is pronounced Py-pee-ai, whereas PyPy is pronounced Py-py. If everybody knew that, I think the confusion would mostly disappear.
- JTon 6y ago> A too little known fact: PyPI is pronounced Py-pee-ai So I was reading this comment and thinking to myself, how does one even pronounce "ai"? Language is not my strong suit. After searching the web, I found out that "ai" is a diphthong and it sounds like "eye" or the letter "i". Leaving that info here for others who may be confused. Sound: https://www.youtube.com/watch?v=uyKgPH0kmrU https://www.youtube.com/watch?v=uyKgPH0kmrU
- wodenokoto 6y agoJust goes to show how difficult conveying sounds and pronunciation can be in text! I’ll keep this in mind to next time I want to write out the letter I and use eye instead.
- wil421 6y agoGlad I wasnt the only one who thought how does “ai” sound.
- 6y ago
- dmix 6y agoThat’s a great way to track tech adoption within a community.
- fouc 6y agoIs Python Wheels the equivalent of Ruby Gems? Or is it better?
- groodt 6y agoGems
- quietbritishjim 6y agoHonest questions from non-user of Ruby: * Do Ruby gems contain all binary depedencies statically linked into them? * Is there a list of exceptions that are guaranteed to be present on target platform (see manylinux [1]). The fact that you answered "Gems" rather than "something better" means the answer to both must be "yes" but I wanted to check. [1] https://github.com/pypa/manylinux https://github.com/pypa/manylinux
- groodt 6y agoApologies, I think the question was edited/clarified after my comment. Gems are more similar to Python sdist’s. Native code is compiled on the installing host. There is no specification like manylinux1,2010,2014 like there are for Python Wheels. In this sense, Python Wheels can be considered more sophisticated than Ruby Gems.
- arusahni 6y agoAny idea if musl support is on the horizon?
- groodt 6y agoI think still a while off https://mail.python.org/archives/list/distutils-sig@python.org/thread/H3323AXRRLJAYOY2XZKS74IOUQMJUOYD/ https://mail.python.org/archives/list/distutils-sig@python.o...
- dingdingdang 6y agoSince pip is used to install Wheels it would probably be best to have a new separate 3rd party meta tool to install package managers themselves to avoid the confusion. Preferably this should have it's own additional PEP and integrate PyPI and PyPy along with other packages that could make life simpler for the (hopefully now happier) end user.
- schwag09 6y agoAt one point in time I created a Python package to highlight this benefit of wheels: "Avoids arbitrary code execution for installation. (Avoids setup.py)" - https://github.com/mschwager/0wned https://github.com/mschwager/0wned Of course Python imports can have side-effects, so you can achieve the same results with 'import malicious_package', but the installation avenue was surprising to me at the time so I created a simple demo. Also consider that 'import malicious_package' is typically not run as root whereas 'pip install' is often run with 'sudo'.
- ghshephard 6y agoI thought the rule was never run `sudo` with `pip install` or you'll screw up the permissions on your system.
- sigjuice 6y agoFor other reasons, it is just as bad when using python from Homebrew on macOS where sudo isn't necessary. `pip install` will install modules in `/usr/local` where they will get mixed with Homebrew-provided python packages. I was hoping there would be a way to make `pip install --user` the default, but I couldn't figure it out the last time I checked.
- djlewald 6y agoIf you're using a *nix system, you could create an alias. Something like, `alias "pip install" "pip install --user"`
- ghshephard 6y agoThis is exactly why you want to do all (as in 100%) of your python work in a virtual environment, so the packages are completely isolated in your ~/.virtualenvs/[ENVNAME]/lib/pythonx.x/site-packages. Never, ever, do a pip install in your non virtualenv environment.
- daveisfera 6y agoAny news on supporting Alpine Linux with wheels?