I find Spark almost as easy to use on S3 as it is on HDFS. I mean you add your keys to the config (or use an instance that already has an IAM profile), and inst

I find Spark almost as easy to use on S3 as it is on HDFS. I mean you add your keys to the config (or use an instance that already has an IAM profile), and instead of typing hdfs://namenode:8020/ you type s3a://bucket/. Done. I haven't tried running Spark entirely without HDFS, so I don't know how well it works to have application log history, etc. go to S3. But certainly for some use cases HDFS doesn't bring anything to the table. If you need stronger consistency guarantees or the higher throughput is a big deal (e.g. very large MapReduce jobs or sequences of them), then there are still teams for which heavier use of HDFS makes total sense.