I recommend reading the GFS paper and consider that there were/are use cases for horizontally scalable, fault-tolerant object storage, with the nuance of unders

I recommend reading the GFS paper and consider that there were/are use cases for horizontally scalable, fault-tolerant object storage, with the nuance of understanding that some of your storage/data nodes may also be compute nodes and there can be a benefit or preference for assigning applications to run where underlying data is stored. In the HDFS case with Hadoop's ecosystem, consider Hive, BigTable, Drill, and even Spark when running on YARN. In the peak days of Hadoop, many organizations were primarily on-prem, and S3 or S3 compatible object stores were mostly reserved for people using AWS.