3 ms·
I used the same approach based on Rclone for a long time. I wondered what makes Regatta Storage different than Rclone. Here is the answer: "When performing muta
by garganzol 2y ago
I used the same approach based on Rclone for a long time. I wondered what makes Regatta Storage different than Rclone. Here is the answer: "When performing mutating operations on the file system (including writes, renames, and directory changes), Regatta first stages this data on its high-speed caching layer to provide strong consistency to other file clients." [0].
Rclone, on the contrary, has no layer that would guarantee consistency among parallel clients.
[0] https://docs.regattastorage.com/details/architecture#overview https://docs.regattastorage.com/details/architecture#overvie...
- huntaub 2y agoThis is exactly right, and something that we think is particularly important for applications that care about data consistency. Often times, we see that customers want to be able to quickly hand off tasks from one instance to another which can be incredibly complex if you don't have guarantees that your new operations will be seen by the second instance!
- wanderingmind 2y agoMight be useful to show the differences with Rclone, s3fs as a table to make it obvious
- huntaub 2y agoI agree, I plan to put up a table soon.
- freedomben 2y agoThanks, this was my thought as well. I use and love rclone and it wasn't immediately clear what this offered above that
- benatkin 2y agoThe headline seems misleading, then. rclone can work with AWS' different offerings, some of which at least partially address this: https://aws.amazon.com/blogs/aws/new-amazon-s3-express-one-zone-high-performance-storage-class/ https://aws.amazon.com/blogs/aws/new-amazon-s3-express-one-z...
- huntaub 2y agoI'm not totally sure what you mean. I don't think that S3 Express One Zone offers any additional atomic semantics in the file system world.
- benatkin 2y agoFor the misleading part, I probably should have said confusing because I don't think you intended that, I mean that instead of introducing your caching layer you make it about S3, where the Object Storage provider seems totally interchangeable. Though it seems to work for a lot of your audience, from what I can tell from other comments here. As for Express One Zone providing consistency, it would make more groups of operations consistent, provided that the clients could access the endpoints with low latency. It wouldn't be a guarantee but it would be practical for some applications. It depends on what the problem is - for instance, do you want someone to never see noticeably stale data? I can definitely see that happening with Express One Zone if it's as described.
- huntaub 2y agoYes, I think this is something that I’m actually struggling with. What’s the most exciting part for users? Is it the fact that we’re building a super fast file system or is it that we have this synchronization to S3? Ultimately, there just isn’t space for it all — but I appreciate the feedback.
- benatkin 2y agoI think they both go together. It might take about 10 minutes to give a good high level explanation of it, including how the S3 syncing works - that the S3 lags slightly behind the caching layer for reads, and that you can still write to S3. 2-way sync. I imagine that S3 would be treated sort of like another client if updates came from S3 and the clients at the same time. It would probably be not so great to write to S3 if you aren't writing to somewhere that's being actively edited, but if you want to write to a dormant area of S3 directly, that's fine.
- dheera 2y agoI suppose rclone doesn't provide byte range file locking? Running sqlite over rclone would be a disaster.