> I haven't found a fundamental reason why one would not base their batch processing on S3 (or your object store of choice) Naming Consistency for things like

> I haven't found a fundamental reason why one would not base their batch processing on S3 (or your object store of choice) Naming Consistency for things like list after create. I had to debug through data loss issues with an object store[1] since they treat paths as a key lookup instead of being hierarchical - because rename("/tmp/x", "/tmp/y") has race conditions, unlike what's usually expected of a filesystem. I recommend reading on about S3mper (fi) from Netflix. [1] - https://cloud.google.com/storage/docs/gsutil/commands/mv?hl=en#non-atomic-operation https://cloud.google.com/storage/docs/gsutil/commands/mv?hl=...