Big companies work around this problem by having their own metadata store which is a (non-consistent) replica of S3/GCS' metadata. Netflix built s3mpr to stop

Big companies work around this problem by having their own metadata store which is a (non-consistent) replica of S3/GCS' metadata. Netflix built s3mpr to stop problems with Hadoop jobs inserting files, other jobs then starting but not finding the file(s) when they execute a listing for it: http://techblog.netflix.com/2014/01/s3mper-consistency-in-cloud.html http://techblog.netflix.com/2014/01/s3mper-consistency-in-cl... Spotify are doing the same for GCS. AWS have even done it for EMR - http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-fs.html http://docs.aws.amazon.com/ElasticMapReduce/latest/Developer...