3 ms·

? Its like GET <namespace>/object, PUT <namespace>/object. To me its the most obvious mapping of HTTP to immutable object key value storage you could imagine.

? Its like GET <namespace>/object, PUT <namespace>/object. To me its the most obvious mapping of HTTP to immutable object key value storage you could imagine. It is bad that the control plane responses can be malformed XML (e.g keys are not escaped right if you put XML control characters in object paths) but that can be forgiven as an oversight. Its not perfect but I don't think its a strange API at all.

candiddevmike 10mo ago
Everything uses poorly documented, sometimes inconsistent HTTP headers that read like afterthoughts/tech debt. An S3 standard implementation has to have amazon branding all over it (x-amz) which is gross.
- christina97 10mo ago
  I mean… it’s straight up an Amazon product, not like it’s an IETF standard or something.
- drob518 10mo ago
  I suspect they learned a lot over the years and the API shows the scars. In their defense, they did go first.
perbu 10mo ago
Last I checked the user guide to the API was 3500 pages. 3500 pages to describe upload and download, basically. That is pretty strange in my book.
- nine_k 10mo ago
  Even download and upload get tricky if you consider stuff like serving buckets like static sites, or stuff like siged upload URLs. Now with the trivial part off the table, let's consder storage classes, security and ACLs, lifecycle management, events, etc.
PunchyHamster 10mo ago
It gets complex with ACLs for permissions, lifecycle controls, header controls and a bunch of other features that are needed on S3 scale but not at smaller provider scale. And many S3-compatible alternatives (probably most but the big ones like Ceph) don't implement all of the features. For example for lifecycles backblaze have completely different JSON syntax
jerf 10mo ago
That may be what S3 is like, but what the S3 API is is this: https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/service/s3 https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/service/s3 My browser prints that out to 413 pages with a naive print preview. You can squeeze it to 350 pretty reasonably with a bit of scaling before it starts getting to awfully small type on the page. Yes, there's a simple API with simple capabilities struggling to get out there, but pointing that out is merely the first step on the thousand-mile journey of determining what, exactly, that is. "Everybody uses 10% of Microsoft Word, the problem is, they all use a different 10%", basically. If you sat down with even 5 relevant stakeholders and tried to define that "simple API" you'd be shocked what you discover and how badly Hyrum's Law will bite you even at that scale.
- eproxus 10mo ago
  That page crashes Safari for me on iOS.
- zokier 10mo ago
  > That may be what S3 is like, but what the S3 API is is this: https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/service/s3 https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/service/s3 > My browser prints that out to 413 pages with a naive print preview. You can squeeze it to 350 pretty reasonably with a bit of scaling before it starts getting to awfully small type on the page. idk why you link to Go SDK docs when you can link to the actual API reference documentation: https://docs.aws.amazon.com/AmazonS3/latest/API/API_Operations_Amazon_Simple_Storage_Service.html https://docs.aws.amazon.com/AmazonS3/latest/API/API_Operatio... and its PDF version: https://docs.aws.amazon.com/pdfs/AmazonS3/latest/API/s3-api.pdf https://docs.aws.amazon.com/pdfs/AmazonS3/latest/API/s3-api.... (just 3874 pages)
  - tlarkworthy 10mo ago
    It's better to link to a leading S3 compatible API docs page. You get a better measure of the essential complexity https://developers.cloudflare.com/r2/api/s3/api/ https://developers.cloudflare.com/r2/api/s3/api/ It's not that much, most of weirder S3 APIs are optional, orthogonal APIs, which is good design.
paulddraper 10mo ago
!!! I’ve seen a lot of bad takes and this is one of them. Listing keys is weird (is it V1 or V2)? The authentication relies on an obtuse and idiosyncratic signature algorithm. And S3 in practice responds with malformed XML, as you point out. Protocol-wise, I have trouble liking it over WebDAV. And that's depressing.
deleted 10mo ago
[deleted]
KaiserPro 10mo ago
HTTP isn't really a great back plane for object storage.

candiddevmike 10mo ago

Everything uses poorly documented, sometimes inconsistent HTTP headers that read like afterthoughts/tech debt. An S3 standard implementation has to have amazon branding all over it (x-amz) which is gross.

christina97 10mo ago
I mean… it’s straight up an Amazon product, not like it’s an IETF standard or something.
drob518 10mo ago
I suspect they learned a lot over the years and the API shows the scars. In their defense, they did go first.

perbu 10mo ago

Last I checked the user guide to the API was 3500 pages. 3500 pages to describe upload and download, basically. That is pretty strange in my book.

nine_k 10mo ago
Even download and upload get tricky if you consider stuff like serving buckets like static sites, or stuff like siged upload URLs. Now with the trivial part off the table, let's consder storage classes, security and ACLs, lifecycle management, events, etc.

PunchyHamster 10mo ago

It gets complex with ACLs for permissions, lifecycle controls, header controls and a bunch of other features that are needed on S3 scale but not at smaller provider scale. And many S3-compatible alternatives (probably most but the big ones like Ceph) don't implement all of the features. For example for lifecycles backblaze have completely different JSON syntax

jerf 10mo ago

That may be what S3 is like, but what the S3 API is is this: https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/service/s3 https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/service/s3 My browser prints that out to 413 pages with a naive print preview. You can squeeze it to 350 pretty reasonably with a bit of scaling before it starts getting to awfully small type on the page. Yes, there's a simple API with simple capabilities struggling to get out there, but pointing that out is merely the first step on the thousand-mile journey of determining what, exactly, that is. "Everybody uses 10% of Microsoft Word, the problem is, they all use a different 10%", basically. If you sat down with even 5 relevant stakeholders and tried to define that "simple API" you'd be shocked what you discover and how badly Hyrum's Law will bite you even at that scale.

eproxus 10mo ago
That page crashes Safari for me on iOS.
zokier 10mo ago
> That may be what S3 is like, but what the S3 API is is this: https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/service/s3 https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/service/s3 > My browser prints that out to 413 pages with a naive print preview. You can squeeze it to 350 pretty reasonably with a bit of scaling before it starts getting to awfully small type on the page. idk why you link to Go SDK docs when you can link to the actual API reference documentation: https://docs.aws.amazon.com/AmazonS3/latest/API/API_Operations_Amazon_Simple_Storage_Service.html https://docs.aws.amazon.com/AmazonS3/latest/API/API_Operatio... and its PDF version: https://docs.aws.amazon.com/pdfs/AmazonS3/latest/API/s3-api.pdf https://docs.aws.amazon.com/pdfs/AmazonS3/latest/API/s3-api.... (just 3874 pages)
- tlarkworthy 10mo ago
  It's better to link to a leading S3 compatible API docs page. You get a better measure of the essential complexity https://developers.cloudflare.com/r2/api/s3/api/ https://developers.cloudflare.com/r2/api/s3/api/ It's not that much, most of weirder S3 APIs are optional, orthogonal APIs, which is good design.

paulddraper 10mo ago

!!! I’ve seen a lot of bad takes and this is one of them. Listing keys is weird (is it V1 or V2)? The authentication relies on an obtuse and idiosyncratic signature algorithm. And S3 in practice responds with malformed XML, as you point out. Protocol-wise, I have trouble liking it over WebDAV. And that's depressing.

deleted 10mo ago

[deleted]

KaiserPro 10mo ago

HTTP isn't really a great back plane for object storage.