5 ms·

> 100X the performance of S3 for file move/rename operations I don’t see how it can be useful. Moving or renaming files in S3 seems more like maintenance than

> 100X the performance of S3 for file move/rename operations I don’t see how it can be useful. Moving or renaming files in S3 seems more like maintenance than something you want to do on a regular basis.

colinwilson 6y ago
Systems like Hadoop and Spark will run multiple versions of the same task writing out to a FS at once but to a temporary directory and when the first finishes the output data is just moved to the final place. It's not uncommon for a job to "complete" writing data to S3 and just sit there and hang as the FS move commands run copying the data and deleting the old version. It is just assumed a rename/move is a no-op in some systems.
stingraycharles 6y ago
We do this in our ETL jobs on several hundreds of thousands of files a day. Not a reason to switch to a different system, but there are definitely non-maintenance use cases for this.
- haodemon 6y ago
  If someone is affected by the "copy-delete on move" problem on S3: there is a way to mitigate this https://hadoop.apache.org/docs/r3.1.1/hadoop-aws/tools/hadoop-aws/committers.html https://hadoop.apache.org/docs/r3.1.1/hadoop-aws/tools/hadoo...
- otterley 6y ago
  Any particular reason you don't tag the objects instead? That's a significantly lighter-weight operation, since S3 doesn't have native renaming capability.
  - jamesblonde 6y ago
    In HopsFS, you can also tag objects with arbitrary metadata (the xattr API), and the JSON objects you tag with are automatically (eventually consistent) replicated to Elastic search. So, you can search the FS namespace with free-text search on elastic. However, the lag is up to 100ms from the MySQL Cluster metadata. See the epipe paper in the blog post for details.
  - bfrydl 6y ago
    You can't list objects by tag.
    - dragonwriter 6y ago
      IIRC, you can, but it's bit elliptical: because IAM access policies can key off tags, you can adopt an IAM role that can only see the desired tags, and list objects.
    - otterley 6y ago
      Ah hah - but see https://medium.com/zendesk-engineering/resurrecting-amazon-simpledb-9404034ec506 https://medium.com/zendesk-engineering/resurrecting-amazon-s...
randallsquared 6y ago
Also, I presume that "move" means "update the index, held in another object or objects" (no slight! If I understand correctly, that's what most on-disk filesystems do as well). That's the only way I can imagine getting performance that much better.

colinwilson 6y ago

Systems like Hadoop and Spark will run multiple versions of the same task writing out to a FS at once but to a temporary directory and when the first finishes the output data is just moved to the final place. It's not uncommon for a job to "complete" writing data to S3 and just sit there and hang as the FS move commands run copying the data and deleting the old version. It is just assumed a rename/move is a no-op in some systems.

stingraycharles 6y ago

We do this in our ETL jobs on several hundreds of thousands of files a day. Not a reason to switch to a different system, but there are definitely non-maintenance use cases for this.

haodemon 6y ago
If someone is affected by the "copy-delete on move" problem on S3: there is a way to mitigate this https://hadoop.apache.org/docs/r3.1.1/hadoop-aws/tools/hadoop-aws/committers.html https://hadoop.apache.org/docs/r3.1.1/hadoop-aws/tools/hadoo...
otterley 6y ago
Any particular reason you don't tag the objects instead? That's a significantly lighter-weight operation, since S3 doesn't have native renaming capability.
- jamesblonde 6y ago
  In HopsFS, you can also tag objects with arbitrary metadata (the xattr API), and the JSON objects you tag with are automatically (eventually consistent) replicated to Elastic search. So, you can search the FS namespace with free-text search on elastic. However, the lag is up to 100ms from the MySQL Cluster metadata. See the epipe paper in the blog post for details.
- bfrydl 6y ago
  You can't list objects by tag.
  - dragonwriter 6y ago
    IIRC, you can, but it's bit elliptical: because IAM access policies can key off tags, you can adopt an IAM role that can only see the desired tags, and list objects.
  - otterley 6y ago
    Ah hah - but see https://medium.com/zendesk-engineering/resurrecting-amazon-simpledb-9404034ec506 https://medium.com/zendesk-engineering/resurrecting-amazon-s...

randallsquared 6y ago

Also, I presume that "move" means "update the index, held in another object or objects" (no slight! If I understand correctly, that's what most on-disk filesystems do as well). That's the only way I can imagine getting performance that much better.