3 ms·
I saw that dynamic allocation is enabled by default. I thought dynamic allocation does not work well on k8s if the executors need to be kept around for serving
by sg47 6y ago
I saw that dynamic allocation is enabled by default. I thought dynamic allocation does not work well on k8s if the executors need to be kept around for serving shuffle files. How does it work in your case?
- jstephan 6y agoThanks, great question ! Dynamic allocation is only enabled on our Spark 3.0 image (from the 3.0-preview branch, since the official 3.0 isn't released yet). It works by tracking which executors are storing active shuffle files. These executors will not be removed when downscaling. More info here: https://issues.apache.org/jira/browse/SPARK-27963 https://issues.apache.org/jira/browse/SPARK-27963 It's not perfect, but there are more improvements for dynamic allocation being worked on (remote shuffle service for Kubernetes).