6 ms·
We've been using pex for a long while to package PySpark jobs with dependencies into a single file. Saves a ton of time vs. the old way of building/deploying do
by khaledh 2y ago
We've been using pex for a long while to package PySpark jobs with dependencies into a single file. Saves a ton of time vs. the old way of building/deploying docker images.
https://spark.apache.org/docs/latest/api/python/user_guide/python_packaging.html#using-pex https://spark.apache.org/docs/latest/api/python/user_guide/p...
- ttyprintk 2y agoWe shiv all the dependencies into a single file, bake it into a Docker layer, and refer to it in each scripts shebang.
- fire_lake 2y agoI would prefer Docker images since you they are easier to “cross compile”. However EMR requires the same Docker image for the entire cluster - why not per job!?!?
- khaledh 2y agoThat's fair. Pex does support specifying a target platform though, which can package linux binary dependencies on a Mac for example. That being said, if a native dependency doesn't provide binaries (i.e. source only) we're out of luck. We solve this issue by precompiling and publishing those dependencies on a private package index. Thankfully there are only a handful of dependencies that need this.
- fire_lake 2y agoUnfortunately it’s the most important dependencies that tend to be native - Spark, TensorFlow, Torch, etc…