6 ms·

S3 was/is optimised for a specific bunch of use cases. Because EC2's hypervisor was (when it was launched) lacking features (no hot swap, not shared block stor

S3 was/is optimised for a specific bunch of use cases. Because EC2's hypervisor was (when it was launched) lacking features (no hot swap, not shared block storage, no host movements, no online backup/clone, no live recovery, no Highly available host failover ) S3 had to step in to pick up some of the slack that proper block or file storage would have taken. For better or for worse, people adopted the ephemeral style of computing, and used s3 as the state store. S3 got away with it because it was the only practical object store in town. The biggest drawback that is still has (and will likely always have) is that you can't write parts to a file. its either replace or nothing. But that's by design. So I suspect it'll stay like that.

cduzz 2y ago
At scale, running on commodity servers, why would want those features even today?
- kbolino 2y ago
  Partial updates could be useful for certain kinds of (mostly binary) files, but block storage is going to handle that much better in general than object storage. The concurrency and consistency guarantees are quite different between object and block storage. Making partial updates atomic would be quite difficult in general in S3, though simple preconditions like compare-and-swap (which is sorely needed anyway) might be sufficient to make it possible for certain use cases.
  - sillysaurusx 2y ago
    I think gcloud has CAS by comparing etags. Surprised to hear S3 can’t.
    - nathants 2y ago
      the common approach is cas dynamodb pointers to uuid named objects in a named s3 prefix. would it be better to merge ddb and s3? maybe, maybe not. gcp/azure/etc are there to provide fancy cloud solutions. no need for aws to serve that market as well.
      - kbolino 2y ago
        One of the important premises of object storage is that if your PutObject or multipart upload succeed, the entire object is atomically replaced. It is eventually consistent, so you may not immediately retrieve the just-uploaded object with GetObject, but you should see the new version eventually, and never see part of one version mixed with part of another. This should natively support compare-and-swap: "hey if the existing etag is what I expect, apply my change, otherwise ignore my change and tell me so". This has nothing to do with DynamoDB and is not reimplementing its feature set. It is just a natural extension of how the service already works (from an API consumer perspective, not necessarily an implementation perspective).
  - skrtskrt 2y ago
    The paradigm can be flipped now for these distributed storage systems. The blocks of a filesystem can now be objects, replicated or erasure-coded - like Ceph running filesystems on top of its low-level object storage protocol, which is done on raw disks, not a filesystem. This can't be done for something like Minio just running on your filesystem, but if you're building the storage system from the ground up it can. We will see more and more of these products appearing. Vast Data is another interesting one. Global deduplication and compression for your data, with an S3, block, or NFS interface. Storing differential backups for thousands or millions of VMs? You'll only store the data from base Ubuntu image once.
  - cduzz 2y ago
    I was referring more to the list of hypervisor features not implemented: >> Because EC2's hypervisor was (when it was launched) lacking features (no hot swap, not shared block storage, no host movements, no online backup/clone, no live recovery, no Highly available host failover ) S3 had to step in to pick up some of the slack that proper block or file storage would have taken. I don't want any of that nonsense in my compute layer or an application (at scale) that relies on shared block storage or host movements or live recovery. I'm sure S3 append would be super handy.
- KaiserPro 2y ago
  At scale? transparent HA means that I can fail over services to other regions without having to get the programmers to think about it. Most of the busy work at scale is managing state or, more correctly recovering state from broken machines. If I can make something else do that reliably, rather than engineer it myself, thats a win. So much of the work of standing up a cluster (be it k8s or something else) is getting to the point where you can arbitrarily kill a datastore and it self heal. If you're talking about s3 partial updates, its about cost/and or performance. If you dealing with megabyte chunks, and you want to flip a few bytes over hundreds of thousands, thats going to eat into transfer costs. Sure you could chunk up the files even smaller, but then you hit into access latency (s3 aint that fast. )
  - cduzz 2y ago
    I was referring to the notion that "failings" in the hypervisor layer like "hot swap, shared block storage, host movements, online backup/clone, live recovery, Highly available host failover" are a problem. At scale, I don't want my application to rely on any of that magic. Reliability is always your problem not something to be punted to another layer of the stack that lets you pretend stuff doesn't go wrong.
    - KaiserPro 2y ago
      > Reliability is always your problem yup, which is why relying on devs to engineer it is a pain in the arse. Having online migration is such a useful tool to avoid accidental overloads when doing maintenance, its also a grat tool to have when testing config changes. Currently I work at a place that has its own container spec and scheduler. This makes sense because we have literal millions of machines to manage. but thats an edge case. For something like a global newspaper (when I used to work) it would be a massive overkill, we spent far too long making K8s act like a mainframe, when we could have bought one 20 times over, and still have change for a good party every week. or, just used hosted databases and liberal caches.
      - cduzz 2y ago
        Oh sure -- for piddly enterprise nonsense, having some VM yeeting magic to HA a thing that's not HA is .... yeah, I guess. Ideally in combination with tested backups for when the HA magic corrupts instead of protects, but such is life. But that's not "at scale" that's just some great plains accounting app that's been dragged from one pickle jar to another.
vlovich123 2y ago
> The biggest drawback that is still has (and will likely always have) is that you can't write parts to a file. its either replace or nothing. You can emulate it though via UploadPartCopy in many cases, no?
- KaiserPro 2y ago
  I mean kinda, but you're still doing a copy on write, its just (from my understanding so I could easily be wrong) you are telling S3 copy the parts of file you don't upload. If you are spooling through a file and changing a few chars at a time, you'll be generating a boat load of writes (and or reading from the wrong file too. )
rowanseymour 2y ago
Sigh append would be such game changer for us. Pulling a big S3 object over the wire to append to it and send it back is far from ideal.
- afavour 2y ago
  Try Azure’s equivalent. I’ve already forgotten what it’s called but it has append operations.
  - magicalhippo 2y ago
    So simple it's hard: append blobs[1]. [1]: https://learn.microsoft.com/en-us/rest/api/storageservices/understanding-block-blobs--append-blobs--and-page-blobs#about-append-blobs https://learn.microsoft.com/en-us/rest/api/storageservices/u...
  - derefr 2y ago
    The design of Azure's blob-store abstraction is great progress toward the ideal form of an object-store: a serverless managed store for all types of data buffers, where handles can be efficiently transformed/exchanged between between different "states of matter" — immutable objects, vs append-only streamable logs, vs multi-reader multi-writer random-access disks; and where all three of these states of data-buffer can be consumed interchangeably by a consumer through the same set of APIs — where these APIs might work most efficiently for data buffers that are in the right state, but they still work for data buffers that aren't. Sadly, Azure's implementation of its blob-store, is kind of underwhelming — especially for any kind of infrastructure-level use-cases. For example, while there is a change feed for blob events, akin to S3 lifecycle event notifications, it stops exactly where S3's API stops; so there is no event generated by an append to an appendable blob, nor a write to a page in a page blob or a block in a block blob. (And even if there were, they make no guarantees of the change feed being linearized — saying that changes to some resources might arrive out-of-order or not at all; and that if you want a linearized change feed, you need to read it out of a log-multiplexer, which puts a several-minute delay and multi-minute step-granularity on reads from it.) As such, you can't use Append Blobs as the storage layer for a Kafka-alike; and nor can you use Page Blobs as the transport to enable an embedded LMDB-alike to be network-replicated. (Or rather, you can, but in both cases you won't receive timely notifications that new data has been added / that pages have been invalidated at the origin, so unless you're operating with zero caching, your cache will end up stale and your state from successive reads will end up incoherent.)
amzn-throw 2y ago
Sounds good, but not true: https://www.awsgeek.com/AWS-History/ https://www.awsgeek.com/AWS-History/ S3 launched in 2006. EC2 didn't go GA until 2008. Elastic Block Store (a proper block store) showed up in 2008 too, actually went GA before even EC2. S3 was for the longest time pitched as storage for the internet, and remains so today. I super doubt these limitations are there because of any EC2 requirements.
- easton 2y ago
  I think OP isn’t saying S3 was created to service EC2, but people deploying EC2 used it as a substitute for EC2 having certain features.
- KaiserPro 2y ago
  > I super doubt these limitations are there because of any EC2 requirements. I think you are misreading me, but to be fair, I was being vague about timelines. my main thrust is that S3 is dominant because EC2 was/is lacking. S3 is optimised for uptime and consistency, which means that its brilliant at mostly static file hosting. It will work for more dynamic state type stuff, its just not really designed for it (see https://xeiaso.net/blog/anything-message-queue/ https://xeiaso.net/blog/anything-message-queue/) making your own object store that is fast, durable and available and has another feature is really really hard to do at scale. Its far easier to put up with s3 than make your own.
RachelF 2y ago
Yes, it can't write to parts of a file. It also can't read (seek) into the file (or couldn't when I last looked at it). It is not a file system, but is often mistaken for one.
- th0ma5 2y ago
  I've been able to pass byte offsets and ranges to S3 requests.
- hipadev23 2y ago
  You can request specific byte ranges from S3 objects. It’s had this feature since at least 2012. https://stackoverflow.com/questions/26964626/specify-byte-range-via-query-string-in-get-object-s3-request https://stackoverflow.com/questions/26964626/specify-byte-ra...
  - Proven 2y ago
    [dead]
  - RachelF 2y ago
    Thanks for that.
harrison_clarke 2y ago
you can get append or edit-like behavior if you use S3 to store B-trees or something, rather than flat files i wish it had a CAS mechanism, though. (google's does, with the x-goog-if-generation-match header)
- xyzzy_plugh 2y ago
  It is crazy that S3 still does not have a comparable header. It's the single biggest miss they're sitting on right now as far as I'm concerned.
- robertlagrant 2y ago
  > x-goog-if-generation-match Is that an ETag-alike?
  - harrison_clarke 2y ago
    yes. i don't really understand why they don't call it an ETag maybe because you're more likely to have the same generation for different paths, because it's not a hash? an ETag also isn't guaranteed to be a hash, though, so idk
    - robertlagrant 2y ago
      Yeah true - an ETag is just matches/doesn't match.
simonebrunozzi 2y ago
This is a really smart comment, and I generally agree with your conclusions (disc: I was early at AWS). Adding to the thread: EBS was kind of created to make up for EC2's ephemeral storage issue, but in 2024 I'm pretty sure AWS would design S3 + EBS + EC2 in a very different way.

cduzz 2y ago

At scale, running on commodity servers, why would want those features even today?

kbolino 2y ago
Partial updates could be useful for certain kinds of (mostly binary) files, but block storage is going to handle that much better in general than object storage. The concurrency and consistency guarantees are quite different between object and block storage. Making partial updates atomic would be quite difficult in general in S3, though simple preconditions like compare-and-swap (which is sorely needed anyway) might be sufficient to make it possible for certain use cases.
- sillysaurusx 2y ago
  I think gcloud has CAS by comparing etags. Surprised to hear S3 can’t.
  - nathants 2y ago
    the common approach is cas dynamodb pointers to uuid named objects in a named s3 prefix. would it be better to merge ddb and s3? maybe, maybe not. gcp/azure/etc are there to provide fancy cloud solutions. no need for aws to serve that market as well.
    - kbolino 2y ago
      One of the important premises of object storage is that if your PutObject or multipart upload succeed, the entire object is atomically replaced. It is eventually consistent, so you may not immediately retrieve the just-uploaded object with GetObject, but you should see the new version eventually, and never see part of one version mixed with part of another. This should natively support compare-and-swap: "hey if the existing etag is what I expect, apply my change, otherwise ignore my change and tell me so". This has nothing to do with DynamoDB and is not reimplementing its feature set. It is just a natural extension of how the service already works (from an API consumer perspective, not necessarily an implementation perspective).
- skrtskrt 2y ago
  The paradigm can be flipped now for these distributed storage systems. The blocks of a filesystem can now be objects, replicated or erasure-coded - like Ceph running filesystems on top of its low-level object storage protocol, which is done on raw disks, not a filesystem. This can't be done for something like Minio just running on your filesystem, but if you're building the storage system from the ground up it can. We will see more and more of these products appearing. Vast Data is another interesting one. Global deduplication and compression for your data, with an S3, block, or NFS interface. Storing differential backups for thousands or millions of VMs? You'll only store the data from base Ubuntu image once.
- cduzz 2y ago
  I was referring more to the list of hypervisor features not implemented: >> Because EC2's hypervisor was (when it was launched) lacking features (no hot swap, not shared block storage, no host movements, no online backup/clone, no live recovery, no Highly available host failover ) S3 had to step in to pick up some of the slack that proper block or file storage would have taken. I don't want any of that nonsense in my compute layer or an application (at scale) that relies on shared block storage or host movements or live recovery. I'm sure S3 append would be super handy.
KaiserPro 2y ago
At scale? transparent HA means that I can fail over services to other regions without having to get the programmers to think about it. Most of the busy work at scale is managing state or, more correctly recovering state from broken machines. If I can make something else do that reliably, rather than engineer it myself, thats a win. So much of the work of standing up a cluster (be it k8s or something else) is getting to the point where you can arbitrarily kill a datastore and it self heal. If you're talking about s3 partial updates, its about cost/and or performance. If you dealing with megabyte chunks, and you want to flip a few bytes over hundreds of thousands, thats going to eat into transfer costs. Sure you could chunk up the files even smaller, but then you hit into access latency (s3 aint that fast. )
- cduzz 2y ago
  I was referring to the notion that "failings" in the hypervisor layer like "hot swap, shared block storage, host movements, online backup/clone, live recovery, Highly available host failover" are a problem. At scale, I don't want my application to rely on any of that magic. Reliability is always your problem not something to be punted to another layer of the stack that lets you pretend stuff doesn't go wrong.
  - KaiserPro 2y ago
    > Reliability is always your problem yup, which is why relying on devs to engineer it is a pain in the arse. Having online migration is such a useful tool to avoid accidental overloads when doing maintenance, its also a grat tool to have when testing config changes. Currently I work at a place that has its own container spec and scheduler. This makes sense because we have literal millions of machines to manage. but thats an edge case. For something like a global newspaper (when I used to work) it would be a massive overkill, we spent far too long making K8s act like a mainframe, when we could have bought one 20 times over, and still have change for a good party every week. or, just used hosted databases and liberal caches.
    - cduzz 2y ago
      Oh sure -- for piddly enterprise nonsense, having some VM yeeting magic to HA a thing that's not HA is .... yeah, I guess. Ideally in combination with tested backups for when the HA magic corrupts instead of protects, but such is life. But that's not "at scale" that's just some great plains accounting app that's been dragged from one pickle jar to another.

vlovich123 2y ago

> The biggest drawback that is still has (and will likely always have) is that you can't write parts to a file. its either replace or nothing. You can emulate it though via UploadPartCopy in many cases, no?

KaiserPro 2y ago
I mean kinda, but you're still doing a copy on write, its just (from my understanding so I could easily be wrong) you are telling S3 copy the parts of file you don't upload. If you are spooling through a file and changing a few chars at a time, you'll be generating a boat load of writes (and or reading from the wrong file too. )

rowanseymour 2y ago

Sigh append would be such game changer for us. Pulling a big S3 object over the wire to append to it and send it back is far from ideal.

afavour 2y ago
Try Azure’s equivalent. I’ve already forgotten what it’s called but it has append operations.
- magicalhippo 2y ago
  So simple it's hard: append blobs[1]. [1]: https://learn.microsoft.com/en-us/rest/api/storageservices/understanding-block-blobs--append-blobs--and-page-blobs#about-append-blobs https://learn.microsoft.com/en-us/rest/api/storageservices/u...
- derefr 2y ago
  The design of Azure's blob-store abstraction is great progress toward the ideal form of an object-store: a serverless managed store for all types of data buffers, where handles can be efficiently transformed/exchanged between between different "states of matter" — immutable objects, vs append-only streamable logs, vs multi-reader multi-writer random-access disks; and where all three of these states of data-buffer can be consumed interchangeably by a consumer through the same set of APIs — where these APIs might work most efficiently for data buffers that are in the right state, but they still work for data buffers that aren't. Sadly, Azure's implementation of its blob-store, is kind of underwhelming — especially for any kind of infrastructure-level use-cases. For example, while there is a change feed for blob events, akin to S3 lifecycle event notifications, it stops exactly where S3's API stops; so there is no event generated by an append to an appendable blob, nor a write to a page in a page blob or a block in a block blob. (And even if there were, they make no guarantees of the change feed being linearized — saying that changes to some resources might arrive out-of-order or not at all; and that if you want a linearized change feed, you need to read it out of a log-multiplexer, which puts a several-minute delay and multi-minute step-granularity on reads from it.) As such, you can't use Append Blobs as the storage layer for a Kafka-alike; and nor can you use Page Blobs as the transport to enable an embedded LMDB-alike to be network-replicated. (Or rather, you can, but in both cases you won't receive timely notifications that new data has been added / that pages have been invalidated at the origin, so unless you're operating with zero caching, your cache will end up stale and your state from successive reads will end up incoherent.)

amzn-throw 2y ago

Sounds good, but not true: https://www.awsgeek.com/AWS-History/ https://www.awsgeek.com/AWS-History/ S3 launched in 2006. EC2 didn't go GA until 2008. Elastic Block Store (a proper block store) showed up in 2008 too, actually went GA before even EC2. S3 was for the longest time pitched as storage for the internet, and remains so today. I super doubt these limitations are there because of any EC2 requirements.

easton 2y ago
I think OP isn’t saying S3 was created to service EC2, but people deploying EC2 used it as a substitute for EC2 having certain features.
KaiserPro 2y ago
> I super doubt these limitations are there because of any EC2 requirements. I think you are misreading me, but to be fair, I was being vague about timelines. my main thrust is that S3 is dominant because EC2 was/is lacking. S3 is optimised for uptime and consistency, which means that its brilliant at mostly static file hosting. It will work for more dynamic state type stuff, its just not really designed for it (see https://xeiaso.net/blog/anything-message-queue/ https://xeiaso.net/blog/anything-message-queue/) making your own object store that is fast, durable and available and has another feature is really really hard to do at scale. Its far easier to put up with s3 than make your own.

RachelF 2y ago

Yes, it can't write to parts of a file. It also can't read (seek) into the file (or couldn't when I last looked at it). It is not a file system, but is often mistaken for one.

th0ma5 2y ago
I've been able to pass byte offsets and ranges to S3 requests.
hipadev23 2y ago
You can request specific byte ranges from S3 objects. It’s had this feature since at least 2012. https://stackoverflow.com/questions/26964626/specify-byte-range-via-query-string-in-get-object-s3-request https://stackoverflow.com/questions/26964626/specify-byte-ra...
- Proven 2y ago
  [dead]
- RachelF 2y ago
  Thanks for that.

harrison_clarke 2y ago

you can get append or edit-like behavior if you use S3 to store B-trees or something, rather than flat files i wish it had a CAS mechanism, though. (google's does, with the x-goog-if-generation-match header)

xyzzy_plugh 2y ago
It is crazy that S3 still does not have a comparable header. It's the single biggest miss they're sitting on right now as far as I'm concerned.
robertlagrant 2y ago
> x-goog-if-generation-match Is that an ETag-alike?
- harrison_clarke 2y ago
  yes. i don't really understand why they don't call it an ETag maybe because you're more likely to have the same generation for different paths, because it's not a hash? an ETag also isn't guaranteed to be a hash, though, so idk
  - robertlagrant 2y ago
    Yeah true - an ETag is just matches/doesn't match.

simonebrunozzi 2y ago

This is a really smart comment, and I generally agree with your conclusions (disc: I was early at AWS). Adding to the thread: EBS was kind of created to make up for EC2's ephemeral storage issue, but in 2024 I'm pretty sure AWS would design S3 + EBS + EC2 in a very different way.