3 ms·

Lots of people here talking about S3, and yet: > The system currently supports Azure Blob Storage and Google Cloud Storage. I'd interpret that as either a har

Lots of people here talking about S3, and yet: > The system currently supports Azure Blob Storage and Google Cloud Storage. I'd interpret that as either a hard "fuck you" to AWS, or a sign that the S3 API is somehow more difficult to use for this purpose.

deleted 3y ago
[deleted]
FooBarWidget 3y ago
The S3 API doesn't have the concurrency control primitives necessary to guarantee consistency in the face of concurrent writes. I wrote a distributed lock on Google Cloud Storage. https://www.joyfulbikeshedding.com/blog/2021-05-19-robust-distributed-locking-algorithm-based-on-google-cloud-storage.html https://www.joyfulbikeshedding.com/blog/2021-05-19-robust-di... During my research it was quickly evident that GCS has more concurrency control primitives than S3. Heck S3 didn't even guarantee strong read-after-write until recently.
- fs111 3y ago
  > until recently Almost 3 years https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-read-after-write-consistency/ https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-rea...
- janejeon 3y ago
  Ah, I'm guessing that's the same reason why many "cloud storage as a backend" stuff (e.g. Terraform/Vault storage backends, Delta.io backends) require DynamodDB + S3
- electroly 3y ago
  While I'm sure you're right, this VFS implementation doesn't use any of that stuff. They just handwave concurrent writes away by saying the application has to ensure a single writer on its own. An S3 implementation appears to be planned; it's mentioned in the comments in the bcvmodule.c file.
stevefan1999 3y ago
I highly suspect it would be the latter because if you just want to "fuck" S3 then you could just list Minio, Swift and Ceph RADOSGW support instead.
mbreese 3y ago
I thought it was a sign of who paid for the feature. I have no insight as to whether or not this is true, but leaving out S3 made me assume either Google or Microsoft (or a third party heavily tied to one of these clouds) paid for this to be added.

deleted 3y ago

[deleted]

FooBarWidget 3y ago

The S3 API doesn't have the concurrency control primitives necessary to guarantee consistency in the face of concurrent writes. I wrote a distributed lock on Google Cloud Storage. https://www.joyfulbikeshedding.com/blog/2021-05-19-robust-distributed-locking-algorithm-based-on-google-cloud-storage.html https://www.joyfulbikeshedding.com/blog/2021-05-19-robust-di... During my research it was quickly evident that GCS has more concurrency control primitives than S3. Heck S3 didn't even guarantee strong read-after-write until recently.

fs111 3y ago
> until recently Almost 3 years https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-read-after-write-consistency/ https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-rea...
janejeon 3y ago
Ah, I'm guessing that's the same reason why many "cloud storage as a backend" stuff (e.g. Terraform/Vault storage backends, Delta.io backends) require DynamodDB + S3
electroly 3y ago
While I'm sure you're right, this VFS implementation doesn't use any of that stuff. They just handwave concurrent writes away by saying the application has to ensure a single writer on its own. An S3 implementation appears to be planned; it's mentioned in the comments in the bcvmodule.c file.

stevefan1999 3y ago

I highly suspect it would be the latter because if you just want to "fuck" S3 then you could just list Minio, Swift and Ceph RADOSGW support instead.

mbreese 3y ago

I thought it was a sign of who paid for the feature. I have no insight as to whether or not this is true, but leaving out S3 made me assume either Google or Microsoft (or a third party heavily tied to one of these clouds) paid for this to be added.