I've worked with both Hadoop/Spark and Greenplum. Hadoop for pre-processing and Greenplum for analytics. There was way too much time spent on the platform and o

I've worked with both Hadoop/Spark and Greenplum. Hadoop for pre-processing and Greenplum for analytics. There was way too much time spent on the platform and optimizations. The use of both didn't improve our workflow either, it was substantially more complex and looking back would think that full rack of Oracle Exadata would have been better. This was a bit over 10 years ago and if I could do it all today I would stick to what we are doing now with dbt and Snowflake. All the promises of the power of these platforms was lost in the additional effort in managing the complexities. Developer productivity for the work that I've been doing is way more important than hw/sw costs. I can't think of anytime that this trade worked better where I had more focus on the sw/hw costs rather than developer time.