Python's packaging is a lot better with Poetry. Use Poetry and build the wheel with poetry build. The lockfile lets you create an exact virtual environment.

Python's packaging is a lot better with Poetry. Use Poetry and build the wheel with poetry build. The lockfile lets you create an exact virtual environment. Like you mentioned, wheel files are not nearly as customizable as JARs. You can easily pick & choose the dependencies to include in JARs (fat JARs) and change namespaces of dependencies via shading to prevent collisions. I've generally found it easier to add JARs to Spark clusters compared to attaching wheel files to PySpark clusters.