4 ms·

Related to the overall topic but not Spanner specifically: it's interesting to compare the consistency models of S3, GCS and similar offerings: * S3: https://d

Related to the overall topic but not Spanner specifically: it's interesting to compare the consistency models of S3, GCS and similar offerings: * S3: https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyModel https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction... * GCS: https://cloud.google.com/storage/docs/consistency https://cloud.google.com/storage/docs/consistency * Azure: exercise left to the reader The best usages of S3 I've seen are designed to only use the one major form of strong-consistency that S3 guarantees: if you write a key that has never been written to or read from, and then do a read, it will always work. Sticking to this guarantee will avoid nasty difficult/impossible-to-diagnose bugs. To take advantage of this you do things like: A. Use unpredictable keys (e.g. prefix with a GUID) to guarantee that you don't get EC behaviour, communicate those keys to other services/users after the upload has succeeded B. Store immutable/content-addressable data. Use new keys for new versions of the data (don't ever over-write a key.) If necessary, use an external (strongly-consistent) index to map identifiers to S3 keys. C. Avoid "upload if not exists" in favour of just uploading (e.g. in the case of a possible failure just reupload, don't check if the key exists on S3. Uploads are atomic.) GCS offers a far greater amount of strong-consistency and this allows makes it much more flexible. The key to scalability/performance/simplicity of services like S3 is lack of transactions between keys, not eventual consistency. It's informative to start from Paxos and figure out how you'd build an S3 and why it might exhibit EC behaviour like S3 (hint: a caching front-end) and question what value that really brings vs. the extreme decrease in usability (consider that a caching front-end doesn't give you scale-out in the way that lack of cross-key transactions does.)

k__ 9y ago
How to get rid of old data if you (basically) treat S3 as append-new-files-only-store?
- jacobparker 9y ago
  That really depends on the application, who has access to the keys, how they learn about the keys etc. There isn't a one size fits all solution and that's kinda why eventually consistency is a pain in the ass. The "I want strong consistency but have to use S3 because its the cheapest way to store stuff and its great at serving reads and just otherwise great" option is to maintain an external index that maps between persistent ids (e.g. file paths) and S3 keys. We use this for hosting user content at my company. Our index is in MS SQL of all things ;) The MVP option is to not to delete stuff. :) This can work well in some scenarios (e.g. you want to keep all versions around anyway) but be careful to not dig a hole that's hard to get out of. A more esoteric option is lifecycle based cleanup. This can work well for things like a build artifact cache (where an expired key isn't a big deal because you can regenerate it.) Some of this is discussed in this blog post by Netflix: https://medium.com/netflix-techblog/s3mper-consistency-in-the-cloud-b6a1076aa4f8 https://medium.com/netflix-techblog/s3mper-consistency-in-th... (caveat: s3mper is dead(?) and there is an AWS-native solution for HDFS now.)
  - mjn 9y ago
    Possibly an obvious comment, but reading your post it struck me that any way you slice it, you're more or less building a garbage-collection system, with all the various tradeoffs those entail. Different performance issues since it's S3 buckets rather than RAM, but still very dependent on who has references to what, and when you check for it. Your two examples sound to me like: You can have some set of known exhaustive roots and periodically delete anything not still reachable from them. Or you could have a lifecycle option, analogous to arenas (region-based memory management). Etc.
    - deleted 9y ago
      [deleted]
- mcqueenjordan 9y ago
  https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifec... If you treat S3 as a "I only write new files to you and I want my files to have a TTL," Object Lifecycle Management is the feature for you.
majewsky 9y ago
> The best usages of S3 I've seen are designed to only use the one major form of strong-consistency that S3 guarantees: if you write a key that has never been written to or read from, and then do a read, it will always work. The same applies to OpenStack Swift, a similar, but self-hosted product. Like S3, when it comes to the CAP rule, they choose availability and partition tolerance. Source: I'm running Swift clusters at work, and developing applications on top of it for customers and internal use.
- jacobparker 9y ago
  Applying CAP to come up with useful/practical conclusions is a lot harder than it seems. The fact that S3 offers the form of strong-consistency that it does means that S3 is already suffering most of the "problems" that strong-consistency implies. If S3 suffered a particularly bad network split, it may not be able to serve reads because it can't be sure that a write didn't happen elsewhere. This obviously applies to new keys (due to their documented guarantees.) To demonstrate why CAP doesn't say enough to be interesting: imagine building an S3 by starting with etcd (which is based on Raft.) etcd is CP but has scalability/availability issues, so it won't scale out well. Now run multiple independent etcd clusters and shard your keyspace onto those clusters. Now you have arbitrary scalability and your availability is better than a single etcd cluster. Now of course if you were building an S3 you wouldn't actually run it like this (the same process could participate in different Paxos/Raft logs at the same time) but that's the high-level idea. My point: You can get arbitrarily more availability and throughput without sacrificing strong-consistency, except in some really odd scenarios (no machine can talk to another machine, for example, and you must replicate to multiple machines.) CAP is about 100% availability (vs. 99.9999999%) but some scenarios are dumb and 100% isn't usually necessary. The key to this is that keys in S3/GCS are independent; there are no cross-key transactions. Eric Brewer (coined the term CAP) wrote about this with respect to Spanner here: "Spanner, TrueTime & The CAP Theorem" https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45855.pdf https://static.googleusercontent.com/media/research.google.c...
  - jacobparker 9y ago
    I should note that the demonstration is a toy example, you don't actually want/need to store blob data in a paxos-like log directly. For an in-depth look into what it takes to build a strongly-consistent S3-like system (and more) you could check out the paper "Windows Azure Storage: A Highly Available Cloud Storage Service with Strong Consistency". A link to the paper, a talk and slides is on this page: https://blogs.msdn.microsoft.com/windowsazurestorage/2011/11/20/sosp-paper-windows-azure-storage-a-highly-available-cloud-storage-service-with-strong-consistency/ https://blogs.msdn.microsoft.com/windowsazurestorage/2011/11...

k__ 9y ago

How to get rid of old data if you (basically) treat S3 as append-new-files-only-store?

jacobparker 9y ago
That really depends on the application, who has access to the keys, how they learn about the keys etc. There isn't a one size fits all solution and that's kinda why eventually consistency is a pain in the ass. The "I want strong consistency but have to use S3 because its the cheapest way to store stuff and its great at serving reads and just otherwise great" option is to maintain an external index that maps between persistent ids (e.g. file paths) and S3 keys. We use this for hosting user content at my company. Our index is in MS SQL of all things ;) The MVP option is to not to delete stuff. :) This can work well in some scenarios (e.g. you want to keep all versions around anyway) but be careful to not dig a hole that's hard to get out of. A more esoteric option is lifecycle based cleanup. This can work well for things like a build artifact cache (where an expired key isn't a big deal because you can regenerate it.) Some of this is discussed in this blog post by Netflix: https://medium.com/netflix-techblog/s3mper-consistency-in-the-cloud-b6a1076aa4f8 https://medium.com/netflix-techblog/s3mper-consistency-in-th... (caveat: s3mper is dead(?) and there is an AWS-native solution for HDFS now.)
- mjn 9y ago
  Possibly an obvious comment, but reading your post it struck me that any way you slice it, you're more or less building a garbage-collection system, with all the various tradeoffs those entail. Different performance issues since it's S3 buckets rather than RAM, but still very dependent on who has references to what, and when you check for it. Your two examples sound to me like: You can have some set of known exhaustive roots and periodically delete anything not still reachable from them. Or you could have a lifecycle option, analogous to arenas (region-based memory management). Etc.
  - deleted 9y ago
    [deleted]
mcqueenjordan 9y ago
https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifec... If you treat S3 as a "I only write new files to you and I want my files to have a TTL," Object Lifecycle Management is the feature for you.

majewsky 9y ago

> The best usages of S3 I've seen are designed to only use the one major form of strong-consistency that S3 guarantees: if you write a key that has never been written to or read from, and then do a read, it will always work. The same applies to OpenStack Swift, a similar, but self-hosted product. Like S3, when it comes to the CAP rule, they choose availability and partition tolerance. Source: I'm running Swift clusters at work, and developing applications on top of it for customers and internal use.

jacobparker 9y ago
Applying CAP to come up with useful/practical conclusions is a lot harder than it seems. The fact that S3 offers the form of strong-consistency that it does means that S3 is already suffering most of the "problems" that strong-consistency implies. If S3 suffered a particularly bad network split, it may not be able to serve reads because it can't be sure that a write didn't happen elsewhere. This obviously applies to new keys (due to their documented guarantees.) To demonstrate why CAP doesn't say enough to be interesting: imagine building an S3 by starting with etcd (which is based on Raft.) etcd is CP but has scalability/availability issues, so it won't scale out well. Now run multiple independent etcd clusters and shard your keyspace onto those clusters. Now you have arbitrary scalability and your availability is better than a single etcd cluster. Now of course if you were building an S3 you wouldn't actually run it like this (the same process could participate in different Paxos/Raft logs at the same time) but that's the high-level idea. My point: You can get arbitrarily more availability and throughput without sacrificing strong-consistency, except in some really odd scenarios (no machine can talk to another machine, for example, and you must replicate to multiple machines.) CAP is about 100% availability (vs. 99.9999999%) but some scenarios are dumb and 100% isn't usually necessary. The key to this is that keys in S3/GCS are independent; there are no cross-key transactions. Eric Brewer (coined the term CAP) wrote about this with respect to Spanner here: "Spanner, TrueTime & The CAP Theorem" https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45855.pdf https://static.googleusercontent.com/media/research.google.c...
- jacobparker 9y ago
  I should note that the demonstration is a toy example, you don't actually want/need to store blob data in a paxos-like log directly. For an in-depth look into what it takes to build a strongly-consistent S3-like system (and more) you could check out the paper "Windows Azure Storage: A Highly Available Cloud Storage Service with Strong Consistency". A link to the paper, a talk and slides is on this page: https://blogs.msdn.microsoft.com/windowsazurestorage/2011/11/20/sosp-paper-windows-azure-storage-a-highly-available-cloud-storage-service-with-strong-consistency/ https://blogs.msdn.microsoft.com/windowsazurestorage/2011/11...