if you're doing spark or hadoop today and are a python shop...you should definitely look at Dask https://dask.org/ https://dask.org/ works as good as spark. ve

if you're doing spark or hadoop today and are a python shop...you should definitely look at Dask https://dask.org/ https://dask.org/ works as good as spark. very lightweight. works through docker. Ground up integrated with kubernetes (runs on EKS/GKE,etc). and no serialization betweek java/python, fatjar stuff, etc