You're definitely correct about Spark being a nightmare to configure. Luckily we've got data platform teams who take care of that. Analysts like I still do conf

You're definitely correct about Spark being a nightmare to configure. Luckily we've got data platform teams who take care of that. Analysts like I still do configure parameters depending on the scenario. You might be correct about data in the 1-15 GB being more common. I'm an analyst so most of the time, I've to aggregate data before I do any analysis. But data scientists in my team typically work on smaller datasets and use Sagemaker instances.