3 ms·

I would prefer Docker images since you they are easier to “cross compile”. However EMR requires the same Docker image for the entire cluster - why not per job!

• khaledh 2y ago
  That's fair. Pex does support specifying a target platform though, which can package linux binary dependencies on a Mac for example. That being said, if a native dependency doesn't provide binaries (i.e. source only) we're out of luck. We solve this issue by precompiling and publishing those dependencies on a private package index. Thankfully there are only a handful of dependencies that need this.
  • fire_lake 2y ago
    Unfortunately it’s the most important dependencies that tend to be native - Spark, TensorFlow, Torch, etc…