12 ms·

S3 is showing its age

260 points by pulisse 2y ago

fred_is_fred 2y ago
The title should have been "S3 is missing features others have". There are no references that I saw to "Product age" causing his concerns.
- vrosas 2y ago
  You could argue that the reticence to add features to your service due to the the sheer complexity, time commitment, or flat out unwillingness to not upset the apple cart is a sign of age.
  - bigstrat2003 2y ago
    I would argue it doesn't have much to do with age, but rather with market dominance. If you are the top dog you don't need to fight for users by implementing various features.
  - andyferris 2y ago
    I personally doubt age has much to do with it. Consider Windows - how old is it now? MS are adding crazy new features on top now more than ever.
KaiserPro 2y ago
S3 was/is optimised for a specific bunch of use cases. Because EC2's hypervisor was (when it was launched) lacking features (no hot swap, not shared block storage, no host movements, no online backup/clone, no live recovery, no Highly available host failover ) S3 had to step in to pick up some of the slack that proper block or file storage would have taken. For better or for worse, people adopted the ephemeral style of computing, and used s3 as the state store. S3 got away with it because it was the only practical object store in town. The biggest drawback that is still has (and will likely always have) is that you can't write parts to a file. its either replace or nothing. But that's by design. So I suspect it'll stay like that.
- cduzz 2y ago
  At scale, running on commodity servers, why would want those features even today?
  - kbolino 2y ago
    Partial updates could be useful for certain kinds of (mostly binary) files, but block storage is going to handle that much better in general than object storage. The concurrency and consistency guarantees are quite different between object and block storage. Making partial updates atomic would be quite difficult in general in S3, though simple preconditions like compare-and-swap (which is sorely needed anyway) might be sufficient to make it possible for certain use cases.
    - sillysaurusx 2y ago
      I think gcloud has CAS by comparing etags. Surprised to hear S3 can’t.
      - nathants 2y ago
        the common approach is cas dynamodb pointers to uuid named objects in a named s3 prefix. would it be better to merge ddb and s3? maybe, maybe not. gcp/azure/etc are there to provide fancy cloud solutions. no need for aws to serve that market as well.
        kbolino 2y ago
        One of the important premises of object storage is that if your PutObject or multipart upload succeed, the entire object is atomically replaced. It is eventually consistent, so you may not immediately retrieve the just-uploaded object with GetObject, but you should see the new version eventually, and never see part of one version mixed with part of another. This should natively support compare-and-swap: "hey if the existing etag is what I expect, apply my change, otherwise ignore my change and tell me so". This has nothing to do with DynamoDB and is not reimplementing its feature set. It is just a natural extension of how the service already works (from an API consumer perspective, not necessarily an implementation perspective).
laurencerowe 2y ago
The lack of If-Match/If-None-Match preconditions in S3 is definitely frustrating.
- loevborg 2y ago
  This or some other way of CAS would enable so many useful applications, like a simple mutex based on an S3 object.
  - DaiPlusPlus 2y ago
    Content-Addressable-Storage? You can do that yourself: just use the hash of an S3 object as its name, and mark the object as immutable.
    - foota 2y ago
      Compare and swap
      - nathants 2y ago
        in an immutable context, cas updates the content hash a named pointer points to.
- wiml 2y ago
  Until recently, S3 had an eventual-consistency model (a given file/object would be internally consistent, but different readers would see creations/deletions/etc in different orders). It's favoring availability over consistency. Using file operations for mutexes makes sense in Unix because of the filesystem semantics there but it makes less sense in a distributed object store.
  - jameshart 2y ago
    S3 has had read-after-write strong consistency since 2020, so... yeah, I guess that's still 'recently', given that S3 existed for 14 years prior to that with eventual consistency.
watermelon0 2y ago
> S3 doesn’t have dual-region or multi-region buckets This is true, but S3 does support replication (including deletion markers), and even 2-way replication, between two regions. Definitely not the same thing as a dual-region bucket, but it can satisfy many use cases where dual-region bucket would be used otherwise. https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/replic...
- beastman82 2y ago
  think it costs 1 or 2 cents/GB depending on region
  - szundi 2y ago
    Multi region pricing probably would be higher anyway
    - klodolph 2y ago
      One two-region bucket would be cheaper than two one-region buckets. I’ve done analyses of similar systems and calculated the costs necessary. You pay for durability and availability in the form of disk overhead, CPU, and network. Each encoding scheme has some expected cost. If your overhead for one-region is $X per gigabyte, then generally speaking, the overhead for two-region is going to be less than $2X—each region is more durable + available because of the copy stored in the other region.
- deanCommie 2y ago
  It can do more than 2. Also you can then create a single multi-region endpoint for those so I don't know why this person says it doesn't exist: https://docs.aws.amazon.com/AmazonS3/latest/userguide/MultiRegionAccessPoints.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/MultiR...
miniman1337 2y ago
S3 Object Lambda solves a lot of these "problems" with this design
- sea-gold 2y ago
  https://aws.amazon.com/s3/features/object-lambda/ https://aws.amazon.com/s3/features/object-lambda/
chucke1992 2y ago
How is S3 right now in comparison to the Azure Storage Account?
- _pqg4 2y ago
  Azure Storage Accounts have major problems with key partitioning that you have to be aware of at the application level that S3 has no problem with. Additionally, Azure Storage accounts have bandwidth throttle limits that can force you to shard across multiple accounts which is pretty painful. Azure SDKs outside of C# and Java are also not well supported in my experience. EDIT: This was based on my experience from ~2 years ago. The Blob storage team has reached out to me via email and let me know that both the key partitioning and throttling issues have been fixed since then.
  - mcaravey 2y ago
    I would consider the documented performance targets [0] for a standard Azure Blob account to be very good. We're talking 60 Gbps in/120 Gbps out, with 20,000 requests per second as the default request rate. From what I can tell, the S3 request rate is about 9,000 requests per second [1] split between reads and writes for a single partition. From my perspective it really just depends on what you're trying to build but I don't see the performance of Azure Storage as being an issue in any way for a typical application. Partitioning will also depend heavily on what kind of application you're building, but the documentation does point out that load balancing will kick in once it starts to see a lot of traffic on a partition [2]. Since you have to use partitioning for S3 in order to get better performance, I don't really see how that's a point against Azure. As for SDKs [3] I have no idea how good support is, but they all have commits within the last day. [0] https://learn.microsoft.com/en-us/azure/storage/common/scalability-targets-standard-account#scale-targets-for-standard-storage-accounts https://learn.microsoft.com/en-us/azure/storage/common/scala... [1] https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimi... [2] https://learn.microsoft.com/en-us/azure/storage/blobs/storage-performance-checklist#partitioning https://learn.microsoft.com/en-us/azure/storage/blobs/storag... [3] https://azure.microsoft.com/en-us/downloads/ https://azure.microsoft.com/en-us/downloads/
kylehotchkiss 2y ago
I want somewhere reasonably priced to keep my files where I can confidently know they'll be there a decade from now. For that purpose, I think S3's age is a killer feature. I won't be surprised when we see the HN post "google cloud storage has been sent to the graveyard"
- szundi 2y ago
  Google can do that. How could they kill even the IoT core, my fleet of devices just died. (No resources to migrate)
  - kylehotchkiss 2y ago
    Their killing of domains, selling that to Squarespace, then Squarespace selling to PE was the most incredible degradation of service I've seen in a while!
    - dpkirchner 2y ago
      It was such a bizarre choice, I don't understand. Why wouldn't they want new businesses to come to them first? Naming a company often involves getting a domain and then paying for hosting, email, and/or other services. This all makes me think gsuite will be next on the chopping block.
      - vineyardmike 2y ago
        > This all makes me think gsuite will be next on the chopping block. GSuite is the most profitable part of GCP, and it's totally propping that division up. It'll be the last to go - it's a lot easier to recognize bad management. Domains probably was expensive to maintain or understaffed for some reason, and some exec knew a guy a Squarespace.
      - vel0city 2y ago
        I don't see how domains going away would suggest Google suite is near the chopping block. I doubt many major customers of GSuite even bothered with Domains. It was such a a small and bare bones business, pretty unrelated to the rest of GSuite.
AtlasBarfed 2y ago
The biggest problem is the cost. Actual storage cost is "meh" but... My god the absolute highway robbery of the bandwidth costs.
- mjhay 2y ago
  Amazon's profit margins on data transfer are out of this world, like probably > 80%. Data transfer costs are hard and confusing to predict (especially with different regions, etc), so it makes sense they'd gouge on it more than storage. https://www.cnbc.com/2021/09/05/how-amazon-web-services-makes-money-estimated-margins-by-service.html https://www.cnbc.com/2021/09/05/how-amazon-web-services-make...
- cageface 2y ago
  Cloudflare R2 has an S3 compatible API and no egress costs.
  - beeeeerp 2y ago
    I like Cloudflare, but this is also what makes me skeptical. There's no such thing as a free lunch, and I've had the rug pulled out from underneath me many times before. I wish they at least charged a sustainable amount for egress, because I feel like it's coming eventually.
    - cageface 2y ago
      Yeah you do have to wonder if this is destined for enshittification at some point. It does seem a bit to good to be true.
    - kondro 2y ago
      Supposedly it’s not applicable to R2, but their Enterprise egress pricing is around the $0.05/GB rate. They won’t contact you though until you start to get close to the $5,000/month usage level, which itself can be a problem as your fee goes from $0/month to $5,000/month. R2 still has per-request fees which, given PUTs are still $4.50/million and GETs $0.36/million (only about 10% off S3’s), they still have significant margin there to cover egress for objects of reasonable size.
akira2501 2y ago
> By embracing DynamoDB as your metadata layer, systems stand to gain a lot. I just implemented a "posix-like" filesystem on top of it. Which means large object offload to s3 is not a problem or even an "ugly abstraction." In fact it looks quite natural once you get down to the layer that does this. You also get something like regular file locks which extend to s3 objects, and if you're using cognito, you can simplify your permissions management and run it all through the file system as well.
someguy4242 2y ago
Can anybody clarify how the author proposed two phase commit/write with DDB and S3?
- nvartolomei 2y ago
  Example: https://delta.io/blog/2022-05-18-multi-cluster-writes-to-delta-lake-storage-in-s3/ https://delta.io/blog/2022-05-18-multi-cluster-writes-to-del...
- paulgb 2y ago
  Assuming you want to write to dest_path with put-if-absent semantics, here's a sketch: - Write to a unique temporary path (call this temp_path) - Commit a record to DynamoDB with put-if-absent semantics on the key (dest_path) and the value (temp_path, "incomplete") - Three possible outcomes: 1. Your write succeeded; proceed to issue a CopyObject call from temp_path to dest_path and if successful mark as complete in DynamoDB. 2. The row already existed in DynamoDB as "complete" -> do nothing 3. The row already existed in DynamoDB as "incomplete" -> another write has been committed, but is not complete; attempt to repair it by issuing a CopyObject for the path in DynamoDB. On success, mark as complete in DynamoDB. ("repair" step) Reads could also hit DynamoDB before S3 in order to perform the "repair" step if applicable.
pryz 2y ago
> By embracing DynamoDB as your metadata layer, systems stand to gain a lot. Yes yes yes. However, DynamoDB can be expensive very quickly :]
- kennu 2y ago
  Interestingly, DynamoDB is cheaper than S3 though, compared by number of requests. DynamoDB costs $1.25 per million write request units and $0.25 per million read request units. While S3 is $5 per million PUT requests and $0.4 per million GET requests.
  - Skinney 2y ago
    Keep in mind that a write capacity unit in DDB is capped to 1kb. Writing a 10kb file to DDB? That's 10 write capacity units.
    - kennu 2y ago
      That's a good point, only very small data is cheaper in DynamoDB than S3. Also adding global secondary indexes tends to add cost, since writes are charged for each of them.
deleted 2y ago
[deleted]
Lucasoato 2y ago
One feature that I definitely miss is the lack of a strongly consistent putIfAbsent API. A lot of big data table formats like Delta.io would benefit so much from it, right now you need to work around it by connecting to DynamoDB :/
menacingly 2y ago
Good. This is a part of the stack I want to be boring. I don't need innovative features. It's the actual federal hate crime that are the egress costs I could do without.
- klodolph 2y ago
  CAS and multi-region are pretty boring. I’d go as far as to say they’re not even innovative.
QuadrupleA 2y ago
Good. The first S in S3 is simple, despite nobody in software appreciating simplicity anymore. Adding features makes documentation more complicated, makes the tech harder to learn, makes libraries bigger, likely harms performance a bit, increases bug surface area, etc. When it gets too out of hand, people will paper it over with a new, simpler abstraction layer, and the process starts again, only with a layer of garbage spaghetti underneath. Show your age and be proud, Simple Storage Service.
- charlie0 2y ago
  This. There's always a small group of people pushing feature requests (ahem scope creep) into services that were never designed for those things. Unfortunately, those people win a lot of the time. Ie, see all the simple JS frameworks that were initially meant to solve relatively simple problems, only for them to become bloated and be replaced by something else that promised simplicity.
- klodolph 2y ago
  Multi-region is pretty essential IMO. You can’t get the same cost effectiveness by building your own multi-region S3, stapling multiple buckets together. The basic premise of S3 is that you get a keystore interface, and S3 handles error correction and distributing the chunks among multiple physical machines on the cheap. Multi-region is the same product, at a larger scale. The complicated part is that somebody has to pay for the network bandwidth (whereas in a single region, it’s so cheap it’s unmetered). The CAS thing is also pretty essential. Everybody wants to build some simple storage system on top of S3 and without CAS it’s pretty damn hard to have any kind of consistency guarantees, even for very simple systems… you end up having to build something outside S3 to manage consistency. An easy to understand use case is backups. Suppose you are using S3 as a backend for a backup system which deduplicates backups. You want to make backups from multiple locations, you want to deduplicate because there’s a lot of duplicate data floating around (maybe you have terabytes of video files getting copied, ML model weights, something else big that you copy around), and you want to expire old backups. You can almost build this on top of plain S3, and the only reason you can’t is because it’s unsafe to expire old data in such a system if any backup is writing (because the other backup may add a new reference to data, racing against the expiration / garbage collection process). A simple CAS gives you a lot of tools to solve this. The alternative to CAS is doing something kinda silly, like running a DynamoDB table as a layer of indirection. Neither of these things add much complexity to S3. (I think append is less useful and potentially a lot more complicated, both in terms of its API implications and in terms of the underlying complexity. If I want “append”, then I can use multipart uploads, or just upload multiple objects and reassemble them on the client side.)
estebarb 2y ago
Append would allow to build a lot of other systems. I mean, the only functional difference between S3 and GFS is append operation. Google build BigTable, Megastore and who knows what more over GFS. You can't do the same with S3 (without having to implement the append somewhere else yourself).
- klodolph 2y ago
  GFS? Google hasn’t used GFS for, like, fifteen years. You can totally build stuff like Megastore or Bigtable (or Spanner) on top of S3. You use a log-structured merge tree. That’s how these systems work in the first place. In the log-structured merge tree, you have a set of files containing your data but you don’t modify them. Instead, you write new files containing the changes (the log). Eventually you compact them by writing a complete copy and deleting the old versions. This works just fine on S3, and there are even some key-value stores built on top of S3 that work this way. Colossus is cheaper for short-lived data.
  - dpkirchner 2y ago
    I haven't used S3: does compacting work within their system, like with an API call, or do you have to download all the chunks, upload the concatenated result, and then delete the chunks?
    - klodolph 2y ago
      Compacting is a database operation. It would happen in your database, not at the underlying storage layer. Your database may use multiple underlying storage layers anyway.
      - dpkirchner 2y ago
        Ah, I guess I misunderstood the part about writing the complete copy. The data isn't really written, it's just abstracted access to multiple chunks.
  - estebarb 2y ago
mythz 2y ago
S3 does simple, reliable object storage well. If anything its draconian pricing model for charging exuberant pricing for bandwidth is showing its age and why we've moved to Cloudflare R2 for their zero egress fees. Being able to reuse the s3 command-line and existing S3 libraries has made the migration painless, so I'm thankful they've created a defacto standard that other S3 compatible object storage providers can implement.
niuzeta 2y ago
This is perfectly fine. S3 is simple and stable, and that's their selling point. There's no competition to the history and the proven stability rather than chasing shiny features.
jonstewart 2y ago
When I have a problem figuring something out with S3, I slide into my shared slack channel with my AWS account rep and solutions engineers and ask them for help, and they help. Is this a feature other cloud storage providers have? (But this rarely happens with S3, because it's so simple.)
at_a_remove 2y ago
I suppose that means I have to learn it now. I'm about half-joking: I've deliberately sought to limit myself to tech further on in the hype cycle.
robertclaus 2y ago
It feels like the S3 Team has managed to avoid chasing the marginal user with features. A great example of restraint. It does beg the question of whether the system could be improved in other areas like optimizations to reduce price.
esteer 2y ago
That is a very cool name for a blog.
nathants 2y ago
s3 is not perfect, but close enough. the ideal design is to use s3 for immutable objects, then build a similar system on ec2 nvme for mutable/ephemeral data. the one i use by default is s4[1]. 1. https://github.com/nathants/s4 https://github.com/nathants/s4
personomas 2y ago
S3 _does_ support CAS on copyObject [0]. Cloudflare supports CAS on copyObject and on putObject [1]. It doesn't support CAS on deleteObject. I don't know about the others though (ABS, GCS, Tigris, MinIO). https://developers.cloudflare.com/r2/api/s3/extensions/#conditional-operations-in-putobject https://developers.cloudflare.com/r2/api/s3/extensions/#cond... [0] https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html#API_CopyObject_RequestSyntax https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObje... [1] https://developers.cloudflare.com/r2/api/s3/extensions/#conditional-operations-in-putobject https://developers.cloudflare.com/r2/api/s3/extensions/#cond...
- yencabulator 2y ago
  Does it? The x-amz-copy-source-if-match etc headers seem to be talking about the source, not the target. I see nothing there to express "fail if target object exists already".
  - personomas 2y ago
    Sure, but the source one is also completely relevant and important.
andyjohnson0 2y ago
The mere fact that abstractions like S3 even exist still boggles my mind. Infinitely scalable, indefinitely persistent, inexpensive, super-high reliability, software-addressable storage accessible from (almost) anywhere on the planet. I'm sure tfa's critique is valid, but also... we have miraculous tools.
brcmthrowaway 2y ago
How is s3 implemented? Is there a data center with many solid state disks? Are tape drives used?
- ted_dunning 2y ago
  Presumably, you could have searched and found this https://medium.com/@ivaramme/systems-design-notes-aws-s3-6ef82a101e15 https://medium.com/@ivaramme/systems-design-notes-aws-s3-6ef...
dabinat 2y ago
I really wish S3 had the ability to rename / move files without having to copy the data all over again. It seems like something that should not be necessary, given that that information is just metadata. Even if there is some technical reason why the data needs copying, S3 could at least pretend that the file is in the new place until it’s actually there.
- passion__desire 2y ago
  There is copyobject or something which is exactly equivalent to renaming.

260 points by pulisse 2y ago

The title should have been "S3 is missing features others have". There are no references that I saw to "Product age" causing his concerns.

vrosas 2y ago
You could argue that the reticence to add features to your service due to the the sheer complexity, time commitment, or flat out unwillingness to not upset the apple cart is a sign of age.
- bigstrat2003 2y ago
  I would argue it doesn't have much to do with age, but rather with market dominance. If you are the top dog you don't need to fight for users by implementing various features.
- andyferris 2y ago
  I personally doubt age has much to do with it. Consider Windows - how old is it now? MS are adding crazy new features on top now more than ever.

KaiserPro 2y ago

S3 was/is optimised for a specific bunch of use cases. Because EC2's hypervisor was (when it was launched) lacking features (no hot swap, not shared block storage, no host movements, no online backup/clone, no live recovery, no Highly available host failover ) S3 had to step in to pick up some of the slack that proper block or file storage would have taken. For better or for worse, people adopted the ephemeral style of computing, and used s3 as the state store. S3 got away with it because it was the only practical object store in town. The biggest drawback that is still has (and will likely always have) is that you can't write parts to a file. its either replace or nothing. But that's by design. So I suspect it'll stay like that.

cduzz 2y ago
At scale, running on commodity servers, why would want those features even today?
- kbolino 2y ago
  Partial updates could be useful for certain kinds of (mostly binary) files, but block storage is going to handle that much better in general than object storage. The concurrency and consistency guarantees are quite different between object and block storage. Making partial updates atomic would be quite difficult in general in S3, though simple preconditions like compare-and-swap (which is sorely needed anyway) might be sufficient to make it possible for certain use cases.
  - sillysaurusx 2y ago
    I think gcloud has CAS by comparing etags. Surprised to hear S3 can’t.
    - nathants 2y ago
      the common approach is cas dynamodb pointers to uuid named objects in a named s3 prefix. would it be better to merge ddb and s3? maybe, maybe not. gcp/azure/etc are there to provide fancy cloud solutions. no need for aws to serve that market as well.
      - kbolino 2y ago
        One of the important premises of object storage is that if your PutObject or multipart upload succeed, the entire object is atomically replaced. It is eventually consistent, so you may not immediately retrieve the just-uploaded object with GetObject, but you should see the new version eventually, and never see part of one version mixed with part of another. This should natively support compare-and-swap: "hey if the existing etag is what I expect, apply my change, otherwise ignore my change and tell me so". This has nothing to do with DynamoDB and is not reimplementing its feature set. It is just a natural extension of how the service already works (from an API consumer perspective, not necessarily an implementation perspective).

laurencerowe 2y ago

The lack of If-Match/If-None-Match preconditions in S3 is definitely frustrating.

loevborg 2y ago
This or some other way of CAS would enable so many useful applications, like a simple mutex based on an S3 object.
- DaiPlusPlus 2y ago
  Content-Addressable-Storage? You can do that yourself: just use the hash of an S3 object as its name, and mark the object as immutable.
  - foota 2y ago
    Compare and swap
    - nathants 2y ago
      in an immutable context, cas updates the content hash a named pointer points to.
wiml 2y ago
Until recently, S3 had an eventual-consistency model (a given file/object would be internally consistent, but different readers would see creations/deletions/etc in different orders). It's favoring availability over consistency. Using file operations for mutexes makes sense in Unix because of the filesystem semantics there but it makes less sense in a distributed object store.
- jameshart 2y ago
  S3 has had read-after-write strong consistency since 2020, so... yeah, I guess that's still 'recently', given that S3 existed for 14 years prior to that with eventual consistency.

watermelon0 2y ago

> S3 doesn’t have dual-region or multi-region buckets This is true, but S3 does support replication (including deletion markers), and even 2-way replication, between two regions. Definitely not the same thing as a dual-region bucket, but it can satisfy many use cases where dual-region bucket would be used otherwise. https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/replic...

beastman82 2y ago
think it costs 1 or 2 cents/GB depending on region
- szundi 2y ago
  Multi region pricing probably would be higher anyway
  - klodolph 2y ago
    One two-region bucket would be cheaper than two one-region buckets. I’ve done analyses of similar systems and calculated the costs necessary. You pay for durability and availability in the form of disk overhead, CPU, and network. Each encoding scheme has some expected cost. If your overhead for one-region is $X per gigabyte, then generally speaking, the overhead for two-region is going to be less than $2X—each region is more durable + available because of the copy stored in the other region.
deanCommie 2y ago
It can do more than 2. Also you can then create a single multi-region endpoint for those so I don't know why this person says it doesn't exist: https://docs.aws.amazon.com/AmazonS3/latest/userguide/MultiRegionAccessPoints.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/MultiR...

miniman1337 2y ago

S3 Object Lambda solves a lot of these "problems" with this design

sea-gold 2y ago
https://aws.amazon.com/s3/features/object-lambda/ https://aws.amazon.com/s3/features/object-lambda/

chucke1992 2y ago

How is S3 right now in comparison to the Azure Storage Account?

_pqg4 2y ago
Azure Storage Accounts have major problems with key partitioning that you have to be aware of at the application level that S3 has no problem with. Additionally, Azure Storage accounts have bandwidth throttle limits that can force you to shard across multiple accounts which is pretty painful. Azure SDKs outside of C# and Java are also not well supported in my experience. EDIT: This was based on my experience from ~2 years ago. The Blob storage team has reached out to me via email and let me know that both the key partitioning and throttling issues have been fixed since then.
- mcaravey 2y ago
  I would consider the documented performance targets [0] for a standard Azure Blob account to be very good. We're talking 60 Gbps in/120 Gbps out, with 20,000 requests per second as the default request rate. From what I can tell, the S3 request rate is about 9,000 requests per second [1] split between reads and writes for a single partition. From my perspective it really just depends on what you're trying to build but I don't see the performance of Azure Storage as being an issue in any way for a typical application. Partitioning will also depend heavily on what kind of application you're building, but the documentation does point out that load balancing will kick in once it starts to see a lot of traffic on a partition [2]. Since you have to use partitioning for S3 in order to get better performance, I don't really see how that's a point against Azure. As for SDKs [3] I have no idea how good support is, but they all have commits within the last day. [0] https://learn.microsoft.com/en-us/azure/storage/common/scalability-targets-standard-account#scale-targets-for-standard-storage-accounts https://learn.microsoft.com/en-us/azure/storage/common/scala... [1] https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimi... [2] https://learn.microsoft.com/en-us/azure/storage/blobs/storage-performance-checklist#partitioning https://learn.microsoft.com/en-us/azure/storage/blobs/storag... [3] https://azure.microsoft.com/en-us/downloads/ https://azure.microsoft.com/en-us/downloads/

kylehotchkiss 2y ago

I want somewhere reasonably priced to keep my files where I can confidently know they'll be there a decade from now. For that purpose, I think S3's age is a killer feature. I won't be surprised when we see the HN post "google cloud storage has been sent to the graveyard"

szundi 2y ago
Google can do that. How could they kill even the IoT core, my fleet of devices just died. (No resources to migrate)
- kylehotchkiss 2y ago
  Their killing of domains, selling that to Squarespace, then Squarespace selling to PE was the most incredible degradation of service I've seen in a while!
  - dpkirchner 2y ago
    It was such a bizarre choice, I don't understand. Why wouldn't they want new businesses to come to them first? Naming a company often involves getting a domain and then paying for hosting, email, and/or other services. This all makes me think gsuite will be next on the chopping block.
    - vineyardmike 2y ago
      > This all makes me think gsuite will be next on the chopping block. GSuite is the most profitable part of GCP, and it's totally propping that division up. It'll be the last to go - it's a lot easier to recognize bad management. Domains probably was expensive to maintain or understaffed for some reason, and some exec knew a guy a Squarespace.
    - vel0city 2y ago
      I don't see how domains going away would suggest Google suite is near the chopping block. I doubt many major customers of GSuite even bothered with Domains. It was such a a small and bare bones business, pretty unrelated to the rest of GSuite.

AtlasBarfed 2y ago

The biggest problem is the cost. Actual storage cost is "meh" but... My god the absolute highway robbery of the bandwidth costs.

mjhay 2y ago
Amazon's profit margins on data transfer are out of this world, like probably > 80%. Data transfer costs are hard and confusing to predict (especially with different regions, etc), so it makes sense they'd gouge on it more than storage. https://www.cnbc.com/2021/09/05/how-amazon-web-services-makes-money-estimated-margins-by-service.html https://www.cnbc.com/2021/09/05/how-amazon-web-services-make...
cageface 2y ago
Cloudflare R2 has an S3 compatible API and no egress costs.
- beeeeerp 2y ago
  I like Cloudflare, but this is also what makes me skeptical. There's no such thing as a free lunch, and I've had the rug pulled out from underneath me many times before. I wish they at least charged a sustainable amount for egress, because I feel like it's coming eventually.
  - cageface 2y ago
    Yeah you do have to wonder if this is destined for enshittification at some point. It does seem a bit to good to be true.
  - kondro 2y ago
    Supposedly it’s not applicable to R2, but their Enterprise egress pricing is around the $0.05/GB rate. They won’t contact you though until you start to get close to the $5,000/month usage level, which itself can be a problem as your fee goes from $0/month to $5,000/month. R2 still has per-request fees which, given PUTs are still $4.50/million and GETs $0.36/million (only about 10% off S3’s), they still have significant margin there to cover egress for objects of reasonable size.

akira2501 2y ago

> By embracing DynamoDB as your metadata layer, systems stand to gain a lot. I just implemented a "posix-like" filesystem on top of it. Which means large object offload to s3 is not a problem or even an "ugly abstraction." In fact it looks quite natural once you get down to the layer that does this. You also get something like regular file locks which extend to s3 objects, and if you're using cognito, you can simplify your permissions management and run it all through the file system as well.

someguy4242 2y ago

Can anybody clarify how the author proposed two phase commit/write with DDB and S3?

nvartolomei 2y ago
Example: https://delta.io/blog/2022-05-18-multi-cluster-writes-to-delta-lake-storage-in-s3/ https://delta.io/blog/2022-05-18-multi-cluster-writes-to-del...
paulgb 2y ago
Assuming you want to write to dest_path with put-if-absent semantics, here's a sketch: - Write to a unique temporary path (call this temp_path) - Commit a record to DynamoDB with put-if-absent semantics on the key (dest_path) and the value (temp_path, "incomplete") - Three possible outcomes: 1. Your write succeeded; proceed to issue a CopyObject call from temp_path to dest_path and if successful mark as complete in DynamoDB. 2. The row already existed in DynamoDB as "complete" -> do nothing 3. The row already existed in DynamoDB as "incomplete" -> another write has been committed, but is not complete; attempt to repair it by issuing a CopyObject for the path in DynamoDB. On success, mark as complete in DynamoDB. ("repair" step) Reads could also hit DynamoDB before S3 in order to perform the "repair" step if applicable.

pryz 2y ago

> By embracing DynamoDB as your metadata layer, systems stand to gain a lot. Yes yes yes. However, DynamoDB can be expensive very quickly :]

kennu 2y ago
Interestingly, DynamoDB is cheaper than S3 though, compared by number of requests. DynamoDB costs $1.25 per million write request units and $0.25 per million read request units. While S3 is $5 per million PUT requests and $0.4 per million GET requests.
- Skinney 2y ago
  Keep in mind that a write capacity unit in DDB is capped to 1kb. Writing a 10kb file to DDB? That's 10 write capacity units.
  - kennu 2y ago
    That's a good point, only very small data is cheaper in DynamoDB than S3. Also adding global secondary indexes tends to add cost, since writes are charged for each of them.

deleted 2y ago

[deleted]

Lucasoato 2y ago

One feature that I definitely miss is the lack of a strongly consistent putIfAbsent API. A lot of big data table formats like Delta.io would benefit so much from it, right now you need to work around it by connecting to DynamoDB :/

menacingly 2y ago

Good. This is a part of the stack I want to be boring. I don't need innovative features. It's the actual federal hate crime that are the egress costs I could do without.

klodolph 2y ago
CAS and multi-region are pretty boring. I’d go as far as to say they’re not even innovative.

QuadrupleA 2y ago

Good. The first S in S3 is simple, despite nobody in software appreciating simplicity anymore. Adding features makes documentation more complicated, makes the tech harder to learn, makes libraries bigger, likely harms performance a bit, increases bug surface area, etc. When it gets too out of hand, people will paper it over with a new, simpler abstraction layer, and the process starts again, only with a layer of garbage spaghetti underneath. Show your age and be proud, Simple Storage Service.

charlie0 2y ago
This. There's always a small group of people pushing feature requests (ahem scope creep) into services that were never designed for those things. Unfortunately, those people win a lot of the time. Ie, see all the simple JS frameworks that were initially meant to solve relatively simple problems, only for them to become bloated and be replaced by something else that promised simplicity.
klodolph 2y ago
Multi-region is pretty essential IMO. You can’t get the same cost effectiveness by building your own multi-region S3, stapling multiple buckets together. The basic premise of S3 is that you get a keystore interface, and S3 handles error correction and distributing the chunks among multiple physical machines on the cheap. Multi-region is the same product, at a larger scale. The complicated part is that somebody has to pay for the network bandwidth (whereas in a single region, it’s so cheap it’s unmetered). The CAS thing is also pretty essential. Everybody wants to build some simple storage system on top of S3 and without CAS it’s pretty damn hard to have any kind of consistency guarantees, even for very simple systems… you end up having to build something outside S3 to manage consistency. An easy to understand use case is backups. Suppose you are using S3 as a backend for a backup system which deduplicates backups. You want to make backups from multiple locations, you want to deduplicate because there’s a lot of duplicate data floating around (maybe you have terabytes of video files getting copied, ML model weights, something else big that you copy around), and you want to expire old backups. You can almost build this on top of plain S3, and the only reason you can’t is because it’s unsafe to expire old data in such a system if any backup is writing (because the other backup may add a new reference to data, racing against the expiration / garbage collection process). A simple CAS gives you a lot of tools to solve this. The alternative to CAS is doing something kinda silly, like running a DynamoDB table as a layer of indirection. Neither of these things add much complexity to S3. (I think append is less useful and potentially a lot more complicated, both in terms of its API implications and in terms of the underlying complexity. If I want “append”, then I can use multipart uploads, or just upload multiple objects and reassemble them on the client side.)

estebarb 2y ago

Append would allow to build a lot of other systems. I mean, the only functional difference between S3 and GFS is append operation. Google build BigTable, Megastore and who knows what more over GFS. You can't do the same with S3 (without having to implement the append somewhere else yourself).

klodolph 2y ago
GFS? Google hasn’t used GFS for, like, fifteen years. You can totally build stuff like Megastore or Bigtable (or Spanner) on top of S3. You use a log-structured merge tree. That’s how these systems work in the first place. In the log-structured merge tree, you have a set of files containing your data but you don’t modify them. Instead, you write new files containing the changes (the log). Eventually you compact them by writing a complete copy and deleting the old versions. This works just fine on S3, and there are even some key-value stores built on top of S3 that work this way. Colossus is cheaper for short-lived data.
- dpkirchner 2y ago
  I haven't used S3: does compacting work within their system, like with an API call, or do you have to download all the chunks, upload the concatenated result, and then delete the chunks?
  - klodolph 2y ago
    Compacting is a database operation. It would happen in your database, not at the underlying storage layer. Your database may use multiple underlying storage layers anyway.
    - dpkirchner 2y ago
      Ah, I guess I misunderstood the part about writing the complete copy. The data isn't really written, it's just abstracted access to multiple chunks.
- estebarb 2y ago

mythz 2y ago

S3 does simple, reliable object storage well. If anything its draconian pricing model for charging exuberant pricing for bandwidth is showing its age and why we've moved to Cloudflare R2 for their zero egress fees. Being able to reuse the s3 command-line and existing S3 libraries has made the migration painless, so I'm thankful they've created a defacto standard that other S3 compatible object storage providers can implement.

niuzeta 2y ago

This is perfectly fine. S3 is simple and stable, and that's their selling point. There's no competition to the history and the proven stability rather than chasing shiny features.

jonstewart 2y ago

When I have a problem figuring something out with S3, I slide into my shared slack channel with my AWS account rep and solutions engineers and ask them for help, and they help. Is this a feature other cloud storage providers have? (But this rarely happens with S3, because it's so simple.)

at_a_remove 2y ago

I suppose that means I have to learn it now. I'm about half-joking: I've deliberately sought to limit myself to tech further on in the hype cycle.

robertclaus 2y ago

It feels like the S3 Team has managed to avoid chasing the marginal user with features. A great example of restraint. It does beg the question of whether the system could be improved in other areas like optimizations to reduce price.

esteer 2y ago

That is a very cool name for a blog.

nathants 2y ago

s3 is not perfect, but close enough. the ideal design is to use s3 for immutable objects, then build a similar system on ec2 nvme for mutable/ephemeral data. the one i use by default is s4[1]. 1. https://github.com/nathants/s4 https://github.com/nathants/s4

personomas 2y ago

S3 _does_ support CAS on copyObject [0]. Cloudflare supports CAS on copyObject and on putObject [1]. It doesn't support CAS on deleteObject. I don't know about the others though (ABS, GCS, Tigris, MinIO). https://developers.cloudflare.com/r2/api/s3/extensions/#conditional-operations-in-putobject https://developers.cloudflare.com/r2/api/s3/extensions/#cond... [0] https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html#API_CopyObject_RequestSyntax https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObje... [1] https://developers.cloudflare.com/r2/api/s3/extensions/#conditional-operations-in-putobject https://developers.cloudflare.com/r2/api/s3/extensions/#cond...

yencabulator 2y ago
Does it? The x-amz-copy-source-if-match etc headers seem to be talking about the source, not the target. I see nothing there to express "fail if target object exists already".
- personomas 2y ago
  Sure, but the source one is also completely relevant and important.

andyjohnson0 2y ago

The mere fact that abstractions like S3 even exist still boggles my mind. Infinitely scalable, indefinitely persistent, inexpensive, super-high reliability, software-addressable storage accessible from (almost) anywhere on the planet. I'm sure tfa's critique is valid, but also... we have miraculous tools.

brcmthrowaway 2y ago

How is s3 implemented? Is there a data center with many solid state disks? Are tape drives used?

ted_dunning 2y ago
Presumably, you could have searched and found this https://medium.com/@ivaramme/systems-design-notes-aws-s3-6ef82a101e15 https://medium.com/@ivaramme/systems-design-notes-aws-s3-6ef...

dabinat 2y ago

I really wish S3 had the ability to rename / move files without having to copy the data all over again. It seems like something that should not be necessary, given that that information is just metadata. Even if there is some technical reason why the data needs copying, S3 could at least pretend that the file is in the new place until it’s actually there.

passion__desire 2y ago
There is copyobject or something which is exactly equivalent to renaming.