5 ms·

One thing I found when converting our python application packaging from RPM to wheels is that wheels don't properly handle the data_files parameter in the setup

by u801e 6y ago

One thing I found when converting our python application packaging from RPM to wheels is that wheels don't properly handle the data_files parameter in the setup call. That is, it places files under the python library directory instead of in the absolute path as specified. This means that sample configuration files and init scripts end up in the wrong place on the file system. In order to get around this, we had to upload the source distribution to our devpi instance and run pip install with the --no-binary option which would then place those files in the correct directories. The other issue is that there's no equivalent of the %(config) RPM spec directive to prevent the config file from being overwritten if it already exists on the file system. So, for libraries, wheels are a good cross-platform packaging solution, but not so much for applications that require configuration files and init scripts.

groodt 6y ago
Agree. I don't think the Wheel specification was intended to replace application distribution. It is aimed at the distribution of individual Python libraries.
- enriquto 6y ago
  What's the difference between "applications" and "libraries"? Why can't they be distributed in the same way?
  - oefrha 6y ago
    Libraries are self-contained and can unpack into any prefix. Applications tend to have config files, service description files, documentation, etc. scattered all over the system at specific locations in order to comply with host system conventions and inter-operate with supervisors and other applications. Wheel is an unpack-into-any-prefix (and stays in that prefix) binary distribution format.
  - pansa2 6y ago
    Python libraries are distributed to other Python developers. Applications are distributed to end-users, who may not even know what Python is, let alone details of any Python installation on their machine.
  - BiteCode_dev 6y ago
    Applications are for the end user. The end user don't need the source code, don't want to compile anything, don't want to bother about installing Python or even knowing it's written in Python, wants it work with his system of choice (app store, package manager, simple exe...), wants OS integration (entry in menus, icons, etc.) and want it fast. Libs are for the dev: you want the source code, you want to manage versions, you want it to integrate with your dev env, you want a well delimited scope, a way to install it using your usual setup (pip/poetry/etc) and to isolate it from project to project.
    - enriquto 6y ago
      As a "dev" who develops mostly for other "devs" this distinction is not so clear to me.
      - BiteCode_dev 6y ago
        Do you provide your lib as a dmg for mac users or through the windows app store ? Do you provide a entry in the start menu for it ? An icon on the desktop ?
        enriquto 6y ago
        Lol no I just send them an email with the url of a private git repo containing my code/experiments, and let them sort it out XD The pleasures of academia.
jeanvalmarc 6y ago
Is it desirable for a distributed Python package to write files to a hardcoded absolute path? Wheels seem to respect the `package_data` setup key, which is definitely inside the library path and seems to work pretty well for library data. I'd be a bit horrified if a package I installed from Pypi attempted to write stuff in `/etc/` or whatever. I guess I'm just asking, aren't these two separate problems (config management and Python packaging)?
- u801e 6y ago
  > Is it desirable for a distributed Python package to write files to a hardcoded absolute path? The established behavior of python setuptools when creating an RPM spec file via the bdist_rpm command was to create a spec file that would invoke the build command in the %build section and the install command in the %install section. The install command would place those files in the absolute paths specified in the data_files named parameter. It has worked this way even back when distutils.core, the predecessor of setuptools, was used. As for whether it's desirable, if you're missing the config file, init script, and other things necessary for an application to work, then you essentially have an incomplete installation and will have to copy those files from other sources and update them yourself via config management before you're able to run it. > I guess I'm just asking, aren't these two separate problems (config management and Python packaging)? In my view, config management should update a config file that's already there and also restart the service as needed (either when the package is updated, the config file is updated, or both). It shouldn't be responsible for placing the config file and init script itself. For example, if I install an application like redis or rabbitmq-server, the package manager will place the included sample config file in the correct directory (e.g., /etc). I can further modify that file as necessary before starting the service. It will also include the scripts necessary to start the service as a daemon. Config management could be used to update the config file for environment specific concerns, but it doesn't need to essentially duplicate what the package manager has already done (create the directories, set permissions, place the file, etc). Python applications, regardless of how they're packaged, shouldn't really be different in this regard.
  - orf 6y ago
kstrauser 6y ago
Python has a couple of mechanisms for fetching files from inside the wheel files (or eggs or...) without you having to know or care about exactly where they’re located. See https://stackoverflow.com/questions/6028000/how-to-read-a-static-file-from-inside-a-python-package https://stackoverflow.com/questions/6028000/how-to-read-a-st... for examples.
sandGorgon 6y ago
You need to reopen this https://github.com/pypa/wheel/issues/92 https://github.com/pypa/wheel/issues/92
pjc50 6y ago
> it places files under the python library directory instead of in the absolute path as specified How is this supposed to work in virtualenv environments? What happens if you have two of them?
raziel2p 6y ago
Wheels don't, and shouldn't, replace RPM or deb packages, and I don't think anyone advertised them as such. They replace eggs or installing from source .py files, you should still build RPMs or debs on top of that if you want to do any system-level stuff like set up systemd services, config files in /etc and so on.
- u801e 6y ago
  > Wheels don't, and shouldn't, replace RPM or deb packages Part of the motivation for using wheels in our case was to allow for pulling in later versions of certain python packages as part of the dependency resolution process when installing or updating the package. Using RPM while attempting to get the updated dependencies would mean having to repackage every single dependency (direct and indirect) from pypi as RPMs and upload them to our yum repo so that yum would be able to handle dependency resolution. > I don't think anyone advertised them as such. They replace eggs or installing from source .py files, The setup method from setuptools does have the necessary features for handling system-level files and the bdist_rpm command leveraged that when building RPMs. It just seems that if this was a bug rather than a feature, then it should have been noted when transitioning from egg based distributions to wheels. Nothing in the documentation for eggs or wheels states that they don't support deploying system files in absolute file paths. The fact that doing something like: pip install git+ssh://git@github.com/org/project.git@1.2.3 and pip install project==1.2.3 have different results for where files listed in the data_files named parameter of the setup end up on the file system suggest that the lack of absolute path support when installing wheels is a bug rather than an intentional change.
  - raziel2p 6y ago
    That makes sense. What I did in the past was include wheels of the project itself and all dependencies (pip download) in the RPM/deb, then create the virtualenv and install dependencies as a post-install script. You could upgrade dependencies by calling pip install -U from the virtualenv, however any time you installed the RPM/deb it would "reset" to the packaged version (though there should be ways to prevent that, e.g. tell pip to never downgrade packages).

by u801e 6y ago

groodt 6y ago

Agree. I don't think the Wheel specification was intended to replace application distribution. It is aimed at the distribution of individual Python libraries.

enriquto 6y ago
What's the difference between "applications" and "libraries"? Why can't they be distributed in the same way?
- oefrha 6y ago
  Libraries are self-contained and can unpack into any prefix. Applications tend to have config files, service description files, documentation, etc. scattered all over the system at specific locations in order to comply with host system conventions and inter-operate with supervisors and other applications. Wheel is an unpack-into-any-prefix (and stays in that prefix) binary distribution format.
- pansa2 6y ago
  Python libraries are distributed to other Python developers. Applications are distributed to end-users, who may not even know what Python is, let alone details of any Python installation on their machine.
- BiteCode_dev 6y ago
  Applications are for the end user. The end user don't need the source code, don't want to compile anything, don't want to bother about installing Python or even knowing it's written in Python, wants it work with his system of choice (app store, package manager, simple exe...), wants OS integration (entry in menus, icons, etc.) and want it fast. Libs are for the dev: you want the source code, you want to manage versions, you want it to integrate with your dev env, you want a well delimited scope, a way to install it using your usual setup (pip/poetry/etc) and to isolate it from project to project.
  - enriquto 6y ago
    As a "dev" who develops mostly for other "devs" this distinction is not so clear to me.
    - BiteCode_dev 6y ago
      Do you provide your lib as a dmg for mac users or through the windows app store ? Do you provide a entry in the start menu for it ? An icon on the desktop ?
      - enriquto 6y ago
        Lol no I just send them an email with the url of a private git repo containing my code/experiments, and let them sort it out XD The pleasures of academia.

jeanvalmarc 6y ago

Is it desirable for a distributed Python package to write files to a hardcoded absolute path? Wheels seem to respect the `package_data` setup key, which is definitely inside the library path and seems to work pretty well for library data. I'd be a bit horrified if a package I installed from Pypi attempted to write stuff in `/etc/` or whatever. I guess I'm just asking, aren't these two separate problems (config management and Python packaging)?

u801e 6y ago
> Is it desirable for a distributed Python package to write files to a hardcoded absolute path? The established behavior of python setuptools when creating an RPM spec file via the bdist_rpm command was to create a spec file that would invoke the build command in the %build section and the install command in the %install section. The install command would place those files in the absolute paths specified in the data_files named parameter. It has worked this way even back when distutils.core, the predecessor of setuptools, was used. As for whether it's desirable, if you're missing the config file, init script, and other things necessary for an application to work, then you essentially have an incomplete installation and will have to copy those files from other sources and update them yourself via config management before you're able to run it. > I guess I'm just asking, aren't these two separate problems (config management and Python packaging)? In my view, config management should update a config file that's already there and also restart the service as needed (either when the package is updated, the config file is updated, or both). It shouldn't be responsible for placing the config file and init script itself. For example, if I install an application like redis or rabbitmq-server, the package manager will place the included sample config file in the correct directory (e.g., /etc). I can further modify that file as necessary before starting the service. It will also include the scripts necessary to start the service as a daemon. Config management could be used to update the config file for environment specific concerns, but it doesn't need to essentially duplicate what the package manager has already done (create the directories, set permissions, place the file, etc). Python applications, regardless of how they're packaged, shouldn't really be different in this regard.
- orf 6y ago

kstrauser 6y ago

Python has a couple of mechanisms for fetching files from inside the wheel files (or eggs or...) without you having to know or care about exactly where they’re located. See https://stackoverflow.com/questions/6028000/how-to-read-a-static-file-from-inside-a-python-package https://stackoverflow.com/questions/6028000/how-to-read-a-st... for examples.

sandGorgon 6y ago

You need to reopen this https://github.com/pypa/wheel/issues/92 https://github.com/pypa/wheel/issues/92

pjc50 6y ago

> it places files under the python library directory instead of in the absolute path as specified How is this supposed to work in virtualenv environments? What happens if you have two of them?

raziel2p 6y ago

Wheels don't, and shouldn't, replace RPM or deb packages, and I don't think anyone advertised them as such. They replace eggs or installing from source .py files, you should still build RPMs or debs on top of that if you want to do any system-level stuff like set up systemd services, config files in /etc and so on.

u801e 6y ago
> Wheels don't, and shouldn't, replace RPM or deb packages Part of the motivation for using wheels in our case was to allow for pulling in later versions of certain python packages as part of the dependency resolution process when installing or updating the package. Using RPM while attempting to get the updated dependencies would mean having to repackage every single dependency (direct and indirect) from pypi as RPMs and upload them to our yum repo so that yum would be able to handle dependency resolution. > I don't think anyone advertised them as such. They replace eggs or installing from source .py files, The setup method from setuptools does have the necessary features for handling system-level files and the bdist_rpm command leveraged that when building RPMs. It just seems that if this was a bug rather than a feature, then it should have been noted when transitioning from egg based distributions to wheels. Nothing in the documentation for eggs or wheels states that they don't support deploying system files in absolute file paths. The fact that doing something like: pip install git+ssh://git@github.com/org/project.git@1.2.3 and pip install project==1.2.3 have different results for where files listed in the data_files named parameter of the setup end up on the file system suggest that the lack of absolute path support when installing wheels is a bug rather than an intentional change.
- raziel2p 6y ago
  That makes sense. What I did in the past was include wheels of the project itself and all dependencies (pip download) in the RPM/deb, then create the virtualenv and install dependencies as a post-install script. You could upgrade dependencies by calling pip install -U from the virtualenv, however any time you installed the RPM/deb it would "reset" to the packaged version (though there should be ways to prevent that, e.g. tell pip to never downgrade packages).