3 ms·
(Disclaimer: work on Dataproc) Though Dataproc doesn't natively have an autoscaler, Spotify wrote one as part of Spydra: https://github.com/spotify/spydra https
by tawkspace 9y ago
(Disclaimer: work on Dataproc) Though Dataproc doesn't natively have an autoscaler, Spotify wrote one as part of Spydra: https://github.com/spotify/spydra https://github.com/spotify/spydra
- Xeago 9y agoThe autoscaler here doesn't solve the downscaling issues mentioned. We've seen it work fine for scaling up, quickly. For now we're not scaling down, as the job will finish quicker than individual tasks will time out when unlucky.