2 ms·
in case you are using PySpark - a good framework to move to is Dask. https://docs.dask.org/en/latest/ https://docs.dask.org/en/latest/ it is also natively int
by sandGorgon 7y ago
in case you are using PySpark - a good framework to move to is Dask. https://docs.dask.org/en/latest/ https://docs.dask.org/en/latest/
it is also natively integrated with K8s - https://github.com/dask/dask-kubernetes https://github.com/dask/dask-kubernetes - and Yarn https://yarn.dask.org/en/latest/ https://yarn.dask.org/en/latest/