At Hopsworks we built HopsFS-S3 to improve things like listing (becomes a partition pruned scan of a in-memory DB), atomic renames and added a block/object cach

At Hopsworks we built HopsFS-S3 to improve things like listing (becomes a partition pruned scan of a in-memory DB), atomic renames and added a block/object caching layer using NVMe drives. You can read the research paper here if you are curious: https://www.hopsworks.ai/research-papers/hopsfs-s3-extending-object-stores-with-posix-like-semantics https://www.hopsworks.ai/research-papers/hopsfs-s3-extending...