4 ms·
We do this in our ETL jobs on several hundreds of thousands of files a day. Not a reason to switch to a different system, but there are definitely non-maintenan
by stingraycharles 6y ago
We do this in our ETL jobs on several hundreds of thousands of files a day. Not a reason to switch to a different system, but there are definitely non-maintenance use cases for this.
- haodemon 6y agoIf someone is affected by the "copy-delete on move" problem on S3: there is a way to mitigate this https://hadoop.apache.org/docs/r3.1.1/hadoop-aws/tools/hadoop-aws/committers.html https://hadoop.apache.org/docs/r3.1.1/hadoop-aws/tools/hadoo...
- otterley 6y agoAny particular reason you don't tag the objects instead? That's a significantly lighter-weight operation, since S3 doesn't have native renaming capability.
- jamesblonde 6y agoIn HopsFS, you can also tag objects with arbitrary metadata (the xattr API), and the JSON objects you tag with are automatically (eventually consistent) replicated to Elastic search. So, you can search the FS namespace with free-text search on elastic. However, the lag is up to 100ms from the MySQL Cluster metadata. See the epipe paper in the blog post for details.
- bfrydl 6y agoYou can't list objects by tag.
- dragonwriter 6y agoIIRC, you can, but it's bit elliptical: because IAM access policies can key off tags, you can adopt an IAM role that can only see the desired tags, and list objects.
- otterley 6y agoAh hah - but see https://medium.com/zendesk-engineering/resurrecting-amazon-simpledb-9404034ec506 https://medium.com/zendesk-engineering/resurrecting-amazon-s...