34 ms·

S3 Strong Consistency

699 points by juliansimioni 6y ago

totaldude87 6y ago
i already had enough "catching-up-with-latest-aws-tech" problem with amazon, though this is a welcome addition, with daily announcements like this, makes life tougher for a AWS architect who likes to be up to date! just today proton was announced and now this! this is going to be a hectic month :|
WrtCdEvrydy 6y ago
Has anyone ever seen S3 behave eventually consistent? I have not seen a lot of eventual consistency in the real world but I wonder if I'm just working on the wrong problems?
- banana_giraffe 6y ago
  Yes, a ton. The big issue for me was Does key exist? -> No -> Upload object with key. Followed by Get key -> Not found. Less rarely, I'd see Upload object followed by an overwrite object, followed by a download download some combination of the two if the download was segmented (like AWS's cli will do). The first one happened often enough to cause problems, the second one was a fairly rare event, but still had to be handled. It might have been more of an issue with "large" buckets. The bucket in particular where I had to dance around the missing object issue had ~100 million fairly small objects. I ended up having to create a database to track if objects exist so no consumer would ever try to see if a non-existent object existed. Time to revisit all that mess, I suppose.
- OJFord 6y ago
  Yes, it caused us some pain with multipart uploads that we sort of proxied, them being multipart from the client to us as well.
- brian_herman 6y ago
  Yeah, I never had that problem either. But I wonder how they are addressing the https://en.wikipedia.org/wiki/CAP_theorem https://en.wikipedia.org/wiki/CAP_theorem
  - content_sesh 6y ago
    Sacrificing availability I reckon.
    - riking 6y ago
      Yup, the correct answer in the face of a network partition is for the PUT to return 500, even though the file really was uploaded. The API handler can't prove success, due to the network partition, so the correct move is to declare failure and let the client activate their contingencies.
- dilatedmind 6y ago
  I think it was already strongly consistent within regions? like if you had tried to read a non-existent key, then wrote to it, it might continue to appear to not exist for a minute?
  - TheDong 6y ago
    Their previous consistency was RAW, but not RAU (for most regions. I think us-east-1 was missing RAW consistency for quite a while). After a write, you would _always_ be able to read the key you just wrote. After an update, you could get a stale copy of the key if your subsequent read hit a different server.
    - heartofgold 6y ago
      Not quite _always_. There were some documented caveats... the one I hit before was: Read nonexistent key, followed by a write of that same key, and then a subsequent read could return a stale read saying it didn’t exist. (even though it had just been written for the first time.) Anyway I am glad to see these gaps and caveats have been closed.
- richwater 6y ago
  Yes. When you're writing terabytes of data files it was very obvious. That's why manifest files became so popular.
- SpicyLemonZest 6y ago
  I've never seen the minutes long delays that you were theoretically supposed to accommodate, but second-scale delays were pretty common in my team's experience. (We're writing a lot of files though, probably hundreds or thousands per second aggregated across all our customers.)
  - WrtCdEvrydy 6y ago
    Yeah, I got seconds but people were saying there were minutes of data... max I ever saw was 3 seconds on a 2gb file...
    - gregw2 6y ago
      I ran across issues where files that we PUT in S3 were still not visible in Hive 6+ hours later. It only happened a few times out of hundreds of jobs per day for jobs running daily the last 2-3 years, but it was annoying to deal with when it happened because there wasn't a lot we could do beyond open tickets with AWS.
- TheDong 6y ago
  Yes. I've observed eventual consistency with S3 prior to this change where it took on the order of several hundred milliseconds to observe an update of an existing key. I've observed this as recently as 2 years ago.
- johncolanduoni 6y ago
  There used to be a behavior with 404 caching that was pretty easy to trigger: if you race requesting an object and creating it, subsequent reads could continue to 404.
- deleted 6y ago
  [deleted]
- rantwasp 6y ago
  yes. i believe newer region had read after write in a while but you needed to use the right endpoint i remember a time when if you were using the us-standard “region” and were unlucky it could take 12 hours for your objects to become visible
- roseway4 6y ago
  Yes. Writing objects from EMR Spark to S3 buckets in a different account. Immediately setting ACLs on the new objects fail as the objects are not yet “there”. Using EMRFS Consistent View was previously the obvious solution here, but it adds significant complexity.
  - enw 6y ago
    EMRFS consistent view is a mess. Changes to S3 files are likely to break EMRFS, the metadata needs to be kept in sync (otherwise the table will keep increasing forever) with tools that are a pain to install, and when I used it to run Hive queries from a script, it didn't work.
    - jamesblonde 6y ago
      This news is the death of EMRFS. The only thing it has left is fast listings of "directories" using DynamoDB.
- allengeorge 6y ago
  Yes, we have. Specifically: 1. LIST 2. PUT 3. LIST would trigger situations where (3) wouldn't include the object inserted in (2). This is well-known however.
  - juancampa 6y ago
    Also: 1. HEAD key -> 404 2. PUT key -> 200 3. GET key -> 404 (what? But I just put it!) This is commonly used for "upload file if it doesn't exist"
    - electrum 6y ago
      Even more confusing, the 404 on GET is caused by doing the HEAD when the object doesn't exist. Without the previous HEAD, the GET would usually succeed.
- macksd 6y ago
  Yes. In fact I spent months working on a system to workaround all the problems it caused for Hadoop: https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/s3guard.html https://hadoop.apache.org/docs/current/hadoop-aws/tools/hado.... We had a lot of automated tests that would fail because of missing (or more recently, out of date) files a significant portion (maybe 20-30%) of the time. We believed that S3 inconsistency was to blame but I always felt like that was the typical "blame flaky infra" cop-out. As soon as S3Guard was in place all those tests were constantly green. It really was S3 inconsistency, routinely. To be fair to S3, Hadoop pretending S3 is a hierarchical filesystem is a bit of a hack. But I had cases where new objects wouldn't be listed for hours. There's only so much you can do to design an application around that, especially when Azure and Google Storage don't have that problem.
- enw 6y ago
  All the time when using EMR with S3. Really frequent, and really annoying to mitigate.
scrollaway 6y ago
That's pretty incredible. With no price increase either. Using s3 as a makeshift database is possible in a lot more scenarios now. I especially like using s3 as a db when dealing with ETLs, where the state of some data is stored in its key prefix. This means that the etl can stop / resume at any point with a single source of truth as the database. The potential drawback is cost of course; moving (renaming) is free but copying is not. S3's biggest price pain is always its PUTs. In many ETLs this is usually a non-issue because you have to do those PUTs regardless, as you probably want the data to be saved at the various stages for future debugging and recovery.
- foxhop 6y ago
  Yup me too! Checkout this state_doc Interface I wrote in Python for building a state machine where the state is stored in S3. I never bothered to solve for the edge case of inconsistent reads because my little state machine only ever measured in hours when it slept waiting for work and then minutes in-between waking up and doing work to verify restores. With the news of S3's strong consistency my program is immediately safer to use for some set of defects, since the underlying datastore properties, S3 has made my codes naive/invalid assumptions, true. https://github.com/remind101/dbsnap/blob/master/dbsnap_verify/README.rst#state_doc https://github.com/remind101/dbsnap/blob/master/dbsnap_verif... I should likely pull out the StateDoc class into it's own module.
jchw 6y ago
I believe this makes Amazon S3 behave more similar to Azure blob storage[1] and Google Cloud Storage[2], which is pretty convenient for folks who are abstracting blob stores across different offerings. For what it’s worth, consistency in S3 was usually pretty good anyways, but I ran into issues where it could vary a bit in the past. If you designed your application with this in mind, of course, it shouldn’t be an issue. In my case I believe we just had to add some retry logic. (And of course, that is no longer necessary.) [1]: https://docs.microsoft.com/en-us/azure/storage/common/storage-concurrency#managing-concurrency-in-blob-storage https://docs.microsoft.com/en-us/azure/storage/common/storag... [2]: https://cloud.google.com/storage/docs/consistency https://cloud.google.com/storage/docs/consistency
- joana035 6y ago
  And Minio too: https://docs.min.io/docs/distributed-minio-quickstart-guide.html https://docs.min.io/docs/distributed-minio-quickstart-guide....
- eloff 6y ago
  And Wasabi as well: https://wasabi-support.zendesk.com/hc/en-us/articles/115001684591-What-data-consistency-model-does-Wasabi-employ- https://wasabi-support.zendesk.com/hc/en-us/articles/1150016...
  - y4m4b4 6y ago
    Wasabi is eventually consistent for overwrites ``eventual consistency for overwrite HTTP PUTs and HTTP DELETEs`` This is not the same.
    - eloff 6y ago
      Yes, I'm aware.
kartayyar 6y ago
This is one reason I have been a big fan of Google Cloud Storage over AWS S3: at a past company AWS consistency was a huge pain, and GCS has had this for years. https://cloud.google.com/blog/products/gcp/how-google-cloud-storage-offers-strongly-consistent-object-listing-thanks-to-spanner https://cloud.google.com/blog/products/gcp/how-google-cloud-...
- aparsons 6y ago
  Azure as well
QuinnyPig 6y ago
This was a problem with data lakes / analytics. Small inconsistencies would trash runs; that problem now goes away and removes a lot of janky half-fixes to work around the issue. For most common use cases it's not really an issue.
- FridgeSeal 6y ago
  Of course the real issue there is those tool treating S3 as something it's not (a filesystem) and building things on it with the expectation of certain guarantees that it explicitly didn't provide before now. I dig the feature (strong consistency) but in some ways it just enables tools that were abusing it to just do so more easily.
  - staticassertion 6y ago
    Like RedShift? AWS uses S3 as a database, seems reasonable that their customers would.
    - FridgeSeal 6y ago
      That doesn't mean it blindly "mounts" it as a disk and attempts to use it as a filesystem. They're almost certainly using it in a manner that is aware of the specificities of how S3 works.
      - foxhop 6y ago
        What if the blind mount works 99.9% of the time? I knew what I was doing was risky but well with in my tolerance for pain. Now that we have a strong consistency my code is fixed. That's pretty impressive.
  - Decker87 6y ago
    Isn't that the point? To provide some useful functionality for others to build on (and pay for)?
damon_c 6y ago
The more I learn about S3 the more I feel the same awe I feel when looking at the Pyramids or Ankor Wat. > You can send 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per prefix in an S3 bucket.
- dheera 6y ago
  Side question -- why is s3fs so slow? I get only about 1 put per 10 seconds with s3fs. One would hope that you could use something like s3fs as a makeshift S3 API, but its performance is really horrible.
  - rovr138 6y ago
    I can't answer this directly, but I thought I'd chime in with a possible alternative, or at least something else to test. I'm using rclone with S3[1] (and others). There are commands[2] where you can use to just sync/copy/ files but it also has a mount option[3]. It also has caching[4] and other things that might help. [1] https://rclone.org/s3/ https://rclone.org/s3/ [2] https://rclone.org/commands/ https://rclone.org/commands/ [3] https://rclone.org/commands/rclone_mount/ https://rclone.org/commands/rclone_mount/ [4] https://rclone.org/cache/ https://rclone.org/cache/
  - banana_giraffe 6y ago
    One thing I noticed when I went digging through the s3fs source trying to answer that myself is that it's uploader is single threaded. Part of what makes AWS's cli reasonably fast is that it uses several threads to make sure to use as much bandwidth as possible. If I limit AWS's tool to one thread on a box with plenty of bandwidth, it turns my upload down from around 135 MiB/s to around 20 MiB/s Not sure if that's all of it, or even the majority of the slowness.
    - gaul 6y ago
      Could you expand on this comment? s3fs uses multiple threads for uploading and populating readdir metadata via S3fsMultiCurl::MultiPerform. Earlier versions used curl_multi_perform which may have hidden some of the parallelism from you.
mholt 6y ago
Would this enable atomic operations with S3 (e.g. "put object only if it does not exist")?
- OJFord 6y ago
  Could you not already do that by specifying ETag?
  - rcaught 6y ago
    No, because the object may not be there (but already uploaded) with eventual consistency.
    - sterwill 6y ago
      Before this update, S3 provided "read-after-write" consistency for object creation at a unique path[0] with PUT and subsequent GET operations. Replacement wasn't consistent. So I think you could have worked out some kind of atomic pattern with S3 objects, using new keys for each state change. [0] https://github.com/awsdocs/amazon-s3-developer-guide/blob/41df5c1f948b64e26d5f2583d4f97d47568a7baa/doc_source/Introduction.md#ConsistencyModel https://github.com/awsdocs/amazon-s3-developer-guide/blob/41...
  - scarface74 6y ago
    The Etag is not a reliable hash of the file contents. It will be different if the file was uploaded in multiple parts like using the CLI than if it was moved as one operation like copying from one bucket to another. You can tell the difference because the Etag will have a “-“ in it if it was uploaded as a multipart upload.
- ryanworl 6y ago
  I also want the answer to this, but to the best of my knowledge it does not.
- leakybucket 6y ago
  The copy object operation has options to control if the copy occurs based on examining source metadata, see "x-amz-copy-source-if-match" at https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObje... . With the consistency change, those might be useful as the basis for atomic operations.
deleted 6y ago
[deleted]
yellow_lead 6y ago
What's up with all the new AWS features in the past few days? Coincidence or not?
- banana_giraffe 6y ago
  AWS Re:Invent is going on. It's virtual this year, but they're still using it to unveil a bunch of new stuff.
- kommissar 6y ago
  No coincidence. It’s reinvent season. https://reinvent.awsevents.com/ https://reinvent.awsevents.com/
  - yellow_lead 6y ago
    Thanks, I was totally out of the loop there.
    - kommissar 6y ago
      It’s free this year, too. Might as well register if you’re interested.
- deleted 6y ago
  [deleted]
yodsanklai 6y ago
Could someone describe a few real-life scenarios where this is useful and noticeable?
- PetahNZ 6y ago
  Our image processing worker queues/servers write an S3 object and dispatch a follow up job. Currently we have to delay the next job (we use 5 seconds) otherwise then next job may start processing before the S3 object is available (it 404's if the next job is run straight away).
  - ramraj07 6y ago
    That sounds _very_ wrong, what type of throughput are we talking about here?
    - riking 6y ago
      The story wasn't wrong, until today. That's why people are celebrating.
    - PetahNZ 6y ago
      Up to hundreds of images per second.
      - ramraj07 6y ago
        Okay that makes sense, when we have very low throughput it looks like consistency wasn't an issue but at that rate (especially if it occurs in bursts) I can see how the prior S3 implementation could fail to scale fast enough
  - mcqueenjordan 6y ago
    Out of curiosity, did you have retries on the GETs?
    - PetahNZ 6y ago
jz-amz 6y ago
Some related reading - * Jeff Barr's blog post on this topic, has an interesting case study: https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-read-after-write-consistency/ https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-rea... * Collaboration with Hadoop/S3A maintainers leading to this release: https://aws.amazon.com/blogs/opensource/community-collaboration-the-s3a-story/ https://aws.amazon.com/blogs/opensource/community-collaborat...
- yftsui 6y ago
  I am curious about how much Dropbox pays for data ingress/egress, they migrated storage to their own on premise data center, then now moving data back to S3 for the data lake.
  - Scaevolus 6y ago
    That's analytics data, not file storage. If they have 34PB of analytics, they certainly have far more than that of blob storage.
    - SteveNuts 6y ago
      I thought they were almost entirely on prem for the actual storage
      - ddorian43 6y ago
        See https://github.com/rockset/rocksdb-cloud https://github.com/rockset/rocksdb-cloud
    - deleted 6y ago
      [deleted]
    - ashish-gandhi 6y ago
      Definitely much more of user data storage. More about that https://dropbox.tech/infrastructure/inside-the-magic-pocket https://dropbox.tech/infrastructure/inside-the-magic-pocket. Happy to take other questions.
juliansimioni 6y ago
Another, perhaps better link for the announcement: https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-read-after-write-consistency/ https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-rea...
throwaway_qcjal 6y ago
Congratulations to the S3 team from a friendly cloud storage neighbor in SLU :-)
kleebeesh 6y ago
Fantastic. Any write-ups or descriptions for how they made it happen?
- hoytschermerhrn 6y ago
  Seconded, I'd love to read a whitepaper if anyone is able to provide one. I'm assuming this operates on some variant of Google's TrueTime[0]. [0] https://cloud.google.com/spanner/docs/true-time-external-consistency https://cloud.google.com/spanner/docs/true-time-external-con...
  - foota 6y ago
    Not necessarily, I think Spanner's magic is most useful for bounded staleness reads and scaling reads through non-leader replicas. Otherwise I think you're looking at normal "NewSQL" guarantees.
deleted 6y ago
[deleted]
boulos 6y ago
Disclosure: I work on Google Cloud. This is super awesome for customers. I am also beyond excited for all the open-source connectors to finally be simplified so that they don't have to deal with the "oh right, gotta be careful because of list consistency". It was super awesome when we were able to delete a huge chunk of the GCS Connector for Hadoop, and I hope to see the same across the S3-focused connectors. I'm now much more inclined to make my GCS FUSE rewrite in Rust also support S3 directly :).
- basicneo 6y ago
  > It was super awesome when we were able to delete a huge chunk of the GCS Connector for Hadoop Is this public?
  - ceocoder 6y ago
    Yes, https://github.com/GoogleCloudDataproc/hadoop-connectors https://github.com/GoogleCloudDataproc/hadoop-connectors disclosure: I work at Google as well.
    - boulos 6y ago
      Right, https://github.com/GoogleCloudDataproc/hadoop-connectors/releases/tag/v1.9.0 https://github.com/GoogleCloudDataproc/hadoop-connectors/rel... was apparently the release: > Delete metadata cache functionality because Cloud Storage has strong native list operation consistency already. If folks are actually interested in these connectors, I'd also recommend this blogpost from last year: https://cloud.google.com/blog/products/data-analytics/new-release-of-cloud-storage-connector-for-hadoop-improving-performance-throughput-and-more https://cloud.google.com/blog/products/data-analytics/new-re... because even with consistency, GCS and S3 still aren't filesystems :).
      - basicneo 6y ago
        Do you have a link to the commits that removed the code. It'd be good to see what sort of complexity this sort of strong consistency can make redundant.
gfody 6y ago
we use s3 in a repo pattern pretty heavily and I had no idea they didn't have read-after-write consistency. I've noticed s3 will hold connections open by sending back blank lines and always assumed it was synchronizing for consistency.
k__ 6y ago
It's interesting to read all these comments here that talk about the eventual consistency like it was some kind of bug.
- howlgarnish 6y ago
  Are there any scenarios where you would want eventual consistency over strong consistency? (Assuming pricing, performance, replication etc are the same.)
  - crazyjncsu 6y ago
    No.
  - nhumrich 6y ago
    At the benefit of added availability? absolutely. This is the entire premise of mongodb.
    - nullsense 6y ago
      MongoDB is webscale.
      - castlecrasher2 6y ago
        I want the one with the bigger GBs.
        nullsense 6y ago
        Better yet, just pipe all your data to /dev/null it supports sharding. Sharding is the secret in the webscale sauce.
    - oblio 6y ago
      Availability? I thought Mongo was fast and scalable, I don't remember anyone recommending it for higher availability.
deleted 6y ago
[deleted]
psanford 6y ago
Neat! Would this make it possible in the future for S3 to support CAS like operations for PutObject? Something like supporting `if-match` and `if-none-match` headers on the etag of an object would be so useful.
The_rationalist 6y ago
Noob question: when to use S3 vs HDFS?
- jamesblonde 6y ago
  For cost on AWS, use S3. AWS price local storage on VMs to make HDFS not competitive. If you want the HDFS API but S3 storage costs on AWS, use HopsFS-S3 - https://www.logicalclocks.com/blog/hopsfs-100x-times-faster-than-aws-s3 https://www.logicalclocks.com/blog/hopsfs-100x-times-faster-... (disclosure: work on HopsFS). HDFS provides POSIX-like API, and now has atomic metadata operations that S3 doesn't (mv/rename, chown, chmod) and append for files.
cash22 6y ago
How do they accomplish strong consistency without any hit to availability?
- YZF 6y ago
  While I don't know the specifics I can say that availability is down to engineering practices. Let's say that their consistency model is achieved via quorum, that is writes write to a quorum of nodes while reads read from a quorum of nodes (of their metadata database) then this guarantees read after write consistency. The availability aspect of this is just engineering, making sure you're never down to less than a quorum of nodes. E.g. by having redundant power supplies, generators, networks, or whatever engineering it takes to reduce the probability of failure. There's other aspects, e.g. latency, that may suffer, but again this is solved via engineering. Just throw more hardware at it to bring the latency down. The only time where you absolutely can't solve it is if you provide strong consistency across geographical regions that are far apart, there's no way then not to pay that latency. This is just another example of why the CAP theorem isn't really as useful to determining limitations of practical systems as it may seem at first site.
  - beoberha 6y ago
    I hope they write a paper or at least do a talk about how they went about it. Usually these things are more involved than an outsider would think. Agreed that you can beat most of the CAP tradeoffs with significant engineering effort, but it’s usually pretty interesting how they accomplish it. Especially at S3’s scale where the COGs impact of “throw more hardware at it” can cost on the order of tens or hundreds of millions.
  - valenterry 6y ago
    > (of their metadata database) With that you have just moved the question to: how do they ensure that the metadata database is available _and_ strongly consistent at the same time for all the requests? Because the metadata database is certainly also a distributed one, hence you need to query _all_ the nodes or can end up in a split-brain situation and lose consistency (or availability if you choose to down the system or a part of the system).
hendry 6y ago
I wonder if this would make S3 a host for sqlite!
- foxhop 6y ago
  Yup, I think it would.
  - sudeepj 6y ago
    I have started working on this actually (the vfs being developed in Rust). The design was complex but now this simplifies a lot. The vfs will also have snapshotting capabilities i.e. you can have multiple versions of sqlite db.
    - js4ever 6y ago
      How are you handling latency (up to 800ms on S3) or the fact that you have to write the whole file each time?
      - sudeepj 6y ago
        > you have to write the whole file each time Will not write the whole file. Instead each page of the database (default = 4KB) will be separate put. The use case is: one-writer multiple readers. The readers will read an old snapshot of the database (from s3) and writer will write to the active one. This way a lot of readers can open the DB in read-only mode. As for writer, each page will be a separate put. The writes will be appended to a log(s) locally and the page will be synced in the background. This helps to exploit lot of parallel PUTs to S3. Once the sync happens the disk space for that page will be reclaimed. For the writer: v = open_new_version() //inserts + updates //commit close_version(v) For readers: open_version_for_read(v) //select
      - ComodoHacker 6y ago
        Obviously, a sane implementation would divide db file into blocks and write them into separate objects. Not as small as traditional FS blocks, 1-4 Mb, perhaps.
gaul 6y ago
This is great news for application developers! And probably hard work for the AWS implementors. I will try to update this over the weekend: https://github.com/gaul/are-we-consistent-yet https://github.com/gaul/are-we-consistent-yet
stubish 6y ago
Once libraries and tools start relying on this, it is going to make life interesting to the S3-compatible players. The API remains the same, but the behavior is quite different.
- sleepydog 6y ago
  Google Cloud storage has behaved this way for years. According to other comments, so does Azure.
  - stubish 6y ago
    It is the smaller players I'm thinking of, which needed S3 compatibility for uptake. Openstack Swift, Backblaze, Ceph... I am not sure if any of these suddenly stopped being S3 compatible due to this announcement.
    - neverartful 6y ago
      Ceph has strong consistency by design. S3 compatibility in Ceph comes via the Rados Gateway (RGW) which just sits atop normal Ceph. This means that the strong consistency is carried over to RGW. So, no change for Ceph. OpenStack Swift has an eventually consistent design. If the Swift community desires to maintain strict compatibility with S3 regarding consistency model, it would be a challenge. Despite being a challenge, I would never bet against the Swift developers. If they want to do it, I'm quite sure that they can. Having said all of that, I doubt that the Swift community would consider this a burning issue that needs to be solved. They've existed for over 10 years with an eventually consistent model and their user base is well aware of it.
marwan-nwh 6y ago
What about CAP theorem?
- danenania 6y ago
  The CAP theorem doesn't preclude strong consistency. It just means that at a certain scale, you'll pay for it with extra latency.
  - marwan-nwh 6y ago
    And that latency means that some reads won't see the latest write, right? Do I understand this correctly?
    - jabl 6y ago
      If you want consistency, you have to pay for it with latency, since the system has to do work in the background to ensure that the data you read is really the latest. If you care more about latency than consistency you have an 'eventually consistent' system, where a write will eventually propagate, but a read might get stale data.
      - bxji 6y ago
        > If you care more about latency than consistency you have an 'eventually consistent' system, where a write will eventually propagate, but a read might get stale data. Not just stale data, you can also have states which never actually existed. I'll steal the example from Doug Terry's paper "Replicated Data Consistency Explained Through Baseball" because it's really good. Linked below. Say you have a baseball game which is scored by innings. It's the middle of the 7th inning, and the true write log for the state of the game is as follows: Write ("home", 1) Write ("visitors", 1) Write ("home", 2) Write ("home", 3) Write ("visitors", 2) Write ("home", 4) Write ("home", 5) If you were to read the score at this point in time, and your system is strongly consistent, the score can only be 2-5 or a refusal to serve the request. If your system is eventually consistent, the score can be any of the following: 0-0, 0-1, 0-2, 0-3, 0-4, 0-5, 1-0, 1-1, 1-2, 1-3, 1-4, 1-5, 2-0, 2-1,2-2, 2-3, 2-4, 2-5. Source paper: https://www.microsoft.com/en-us/research/wp-content/uploads/2011/10/ConsistencyAndBaseballReport.pdf https://www.microsoft.com/en-us/research/wp-content/uploads/...
whateveracct 6y ago
This is really something, especially for small apps. S3 is now the easiest KV store I can imagine.
- sudeepj 6y ago
  > small apps Even the big ones. A lot of applications (e.g backup vendors) had to resort to use EBS volumes because of lack of read-after-update consistency. The barrier to entry for lot of developers has been reduced. My prediction: A lot of storage infra (FS, DB) with native support for S3 will get commoditised now.
leakybucket 6y ago
With this change, is there a way to implement atomic operations like put-if-absent?
ChuckMcM 6y ago
This is awesome, I could have used it for a project we did a while back. I'm sure that somewhere there is a Product Manager at Amazon who talks to customers they find out are using Google Cloud or Azure and asks them "Why not use AWS?" and they mumble some feature of the other service that they need. Sometime later, said service shows up as part of AWS. :-) For me this is the best part of tech rivalries, whether they are in CPU performance/architecture, programming language features/tools, or network services. Pressure to improve.
ultimoo 6y ago
> S3 consistency is available at no additional cost and removes the need for additional third-party, services, and complex architecture. I wonder whether any companies/businesses solely depended on offering eventual consistency workarounds that probably now need to pivot.
- pas 6y ago
  The hadoop companies spent quite some bucks on developing S3Guard, and selling it. https://hadoop.apache.org/docs/r3.0.3/hadoop-aws/tools/hadoop-aws/s3guard.html https://hadoop.apache.org/docs/r3.0.3/hadoop-aws/tools/hadoo...
david-cako 6y ago
big news. this was a pretty big issue for a customer at AWS that was using S3 for their analytics platform.
eggie5 6y ago
does this mean the TF S3 plugin will stop cluttering my logs w/ s3 retries?
ed25519FUUU 6y ago
The most devious s3 consistency issue I encountered was that S3 bucket configs are actually stored in S3, and were not read-after-write consistent. It was bad. In order to "update" a bucket config programmatically you need to read the entire existing config, make an update, and then PUT it back (overwriting what's there). The problem is when you went to read the config it's possibly 15 minutes old or more, and when you put it back you overwrite any changes that may not be consistent. I had to work around the issue by storing a high-water mark in the config ID, and then poll an external system to find the expected high water mark before I knew I could safely read and update.
jamesblonde 6y ago
Listing and update consistency is great, but the ACID SQL-on-S3 frameworks (Apache Hudi, Dela Lake) also require atomic rename to support transactions. HopsFS-S3 solves the ACID SQL-on-S3 problem discussed here on HN last week ( https://news.ycombinator.com/item?id=25149154 https://news.ycombinator.com/item?id=25149154 ) by adding a metadata layer over S3 and providing a POSIX-like HDFS API to clients. Operations like rename, mv, chown, chmod are metadata operations - rename is not copy and delete. Disclosure: I work on HopsFS
iblaine 6y ago
AWS Re:invent talked about improvements to S3 earlier today. You can see that talk here https://virtual.awsevents.com/media/1_usujusng https://virtual.awsevents.com/media/1_usujusng For any AWS EMR/hadoop users out there, this means the end of emrfs. For the uninitiated, emrfs recognizes consistency problems but does not fix them. It'll throw an exception if there's a consistency problem, even give you some tools to fix the problem, but the tools may give false positives. The result is you've got to fix some consistency problems yourself, parse items out of the emrfs DynamoDB catalog, match them up to s3, then make adjustments where needed. It's an unnecessary chore. It surprises me that this issue has not got more attention over the years and thankfully it'll be solved soon.
leowoo91 6y ago
Interesting, I was always versioning file names as they upload. That seems to be mostly for file updates rather.
Aeolun 6y ago
Oh, so this is why the images I uploaded from my chat app sometimes just didn’t show up from certain locations. Glad it’s solved now :)
tutfbhuf 6y ago
Ask HN: I have a question regarding strong consistency. The question is related to all geo distributed strong consistent storages, but let me pick Google Spanner as example. Spanner has a maximum upper bound of 7 milliseconds clock offset between nodes, thx to TrueTime. To achieve external consistency, Spanner simply waits 7 ms before committing a transaction. After that amount of time the window of uncertainty is over and we can be sure, that every transaction happens to be in the right order. So far so good, but how does Spanner deal with cross data center network latency? Let's say I commit a transaction to Spanner in Canada and after 7 ms, I get my confirmation, but now in Australia someone also does a conflicting transaction and also get his confirmation after 7 ms. Spanner however, bound to the laws of physics, can only see this conflict after 100+ ms delay, due to network latency between the datacenters. How is that problem solved?
- throwaway_dcnt 6y ago
  Partitioning.
- foota 6y ago
  This documentation talks about it in brief: https://cloud.google.com/spanner/docs/whitepapers/life-of-reads-and-writes https://cloud.google.com/spanner/docs/whitepapers/life-of-re.... You can read the spanner paper for more detail: https://static.googleusercontent.com/media/research.google.com/en//archive/spanner-osdi2012.pdf https://static.googleusercontent.com/media/research.google.c... The simple answer is that there are round trips required between datacenters when you have a quorum that spans data centers. Additionally, one replica of a split is the leader, and the leader holds write locks. So already you have to talk to the leader replica, even if it's out of the DC you're in. Getting the write lock overlaps with the transaction commit though. So for your example if we say the leader is in Canada and the client is in Australia, and we're doing a write to row 'Foo' without first reading (a so called blind write): Client (Australia) -> Leader (Canada): Take a write write lock on 'Foo' and try to commit transaction Leader -> other replicas: Start commit, timestamp 123 Other replicas -> Leader: Ready for commit Leader waits for majority ready for commit and for timestamp 123 to be before the current truetime interval Leader -> Other replicas: Commit transaction, and in parallel replies to client. Of course there are things you can do to mitigate this depending on your needs, but there's no free lunch. If you have a client in Australia and a client in Canada writing to the same data you're going to pay for round trips for transactions.
tutfbhuf 6y ago
If someone is wondering why this is huge. It's because of the problems you face, when you don't have strong consistency. See this picture to get an understanding of what kind of problems you might run into with eventual consistency: https://i.ibb.co/DtxrRH3/eventual-consistency.png https://i.ibb.co/DtxrRH3/eventual-consistency.png
- pachico 6y ago
  You just made my day :)
- rafaelturk 6y ago
  I shall print this..
  - freemint 6y ago
    Please support the original artist and make sure you have a license to print it before you do.I
    - homarp 6y ago
      the artist is Pascal Jousselin http://pjousselin.free.fr/ http://pjousselin.free.fr/ it's called 'Imbattable' or Mister Invincible
      - memiux 6y ago
        Ok that explains the Baguette
    - st1ck 6y ago
      So you're concerned about printing 1 copy for home or office, but not about sharing this image on the internet?
    - fuzzer37 6y ago
      Please do whatever you want with an image posted to an internet forum
toolslive 6y ago
atomic multipart uploads, here we come ;)
awssucs 6y ago
“After a successful write of a new object or an overwrite of an existing object, any subsequent read request immediately receives the latest version of the object.” - being able to issue a read after a write has been the case in any system ever. AWS solving a fabricated issue, and the crowd cheering it as progress. Have to love herd mentality.
- jamesblonde 6y ago
  Previously, update-then-read was not guaranteed to be consistent. Now it is consistent.
  - awssucs 6y ago
    yes for aws it was an issue, what i meant is that aws solved an issue that it itself created.
    - p0rkbelly 6y ago
      14 years ago. That is called tech debt. Not an issue. They designed it for the use case at the time. Nobody knew what the cloud was becoming.
      - awssucs 6y ago
        I agree nobody knew what the cloud was becoming. But it feels like aws is selling an unfinished product and people are wasting money and time working around non issues. Aws is great for medium sized projects but as soon as your project grows, you bump into silly issues like this one. Given that s3 is more or less a file storage you don’t really expect issues such as “weak” consistency. You expect a file to be committed once written (yes aws can do queueing behind the scenes) but when it returns a 200 ok it means the file was stored. Otherwise the good folks at aws should return a 202 accepted response as all properly designed apis do and let the user know that a delay in reading is expected.
- swyx 6y ago
  if you've ever worked on distributed storage you'll know that this isn't fabricated. it just takes effort to fix it. and S3 is pretty much the oldest distributed store in existence, lots of tech debt to overcome.
mikaeluman 6y ago
I thought we had worked out in 2010-2012 that the CAP theorem, consistency, availability and partition tolerance (https://en.wikipedia.org/wiki/CAP_theorem#History https://en.wikipedia.org/wiki/CAP_theorem#History) made this impossible? Where is the mistake.
- chillydawg 6y ago
  I would guess write times are slightly up to cover the cost of the new replication completion guarantees.
- johncolanduoni 6y ago
  S3 is almost certainly not fully partition tolerant at the node level and requires some sort of quorum. Other “magical” data stores like Spanner also retain this limitation, they just have very reliable replication strategies.
  - pricechild 6y ago
    "almost certainly not fully partition tolerant at the node level" Whoa hang on! You can't just say you're not tolerant of partitions... I'm struggling to find the best post on aphyr.com about this but https://aphyr.com/posts/325-comments-on-you-do-it-to https://aphyr.com/posts/325-comments-on-you-do-it-to is a good one, specifically the line: "CP and AP are upper bounds: systems can provide C or A during a partition, but might provide neither." In short, Partitions happen, no matter what. No really, when you have multiple nodes they happen, you must tolerate them. You can't have CA.
    - johncolanduoni 6y ago
      I’m not claiming it’s a CA system, and the terminology “partition intolerant” is not verboten by Kyle Kingsbury. From your link: “Specifically, partition-intolerant systems must sacrifice invariants when partitions occur. Which invariants?” The answer in this case is that availability is sacrificed, unless Amazon is making a very misleading claim of strong consistency (per the submission title/link). So it’s CP. In case “not fully partition tolerant” is what is throwing you off, all I mean by that is that there are likely network partitions of the S3 nodes that will not cause outward failure (e.g. partition of a single node), though some will (e.g. too many partitions to form a quorum, or whatever their underlying consensus implementation relies on).
intricatedetail 6y ago
Is "strong" redundant? Something can be either consistent or not. Seems like developers asked marketing for a catchy name, but that looks comical.
bob1029 6y ago
This is really cool. We actually got burned on weak S3 consistency a few weeks ago when generating public download links for a customer. Took us a few hours of troubleshooting to realize they had downloaded a cached/older version of the software we had uploaded to the same URL just a few minutes prior. Resolution was to use unique paths per version to guarantee we were talking to the right files each time. One potentially related item I was thinking about - How does HN feel about the idea of a system that has eventual durability guarantees which can be inspected by the caller? I.e. Call CloudService.WriteObject(). It writes the object to the local datacenter and asynchronously sends it to the remote(s). It returns some transaction id. You can then pass this id to CloudService.TransactionStatus() to determine if a Durable flag is set true. Or, have a convenience method like CloudService.AwaitDurabilityAsync(txnid). In the most disastrous of circumstances (asteroid hits datacenter 1ms after write returns to caller), you would get an exception on this await call after a locally-configured timeout in which case you can assume you should probably retry the write operation. I was thinking this might be a way to give the application a way to decide how to deal with the concept of latency WRT cross-site replication. You could not await the durability for 4 nines or wait the additional 0-150 ms to see more nines. I wonder how this risk profile sits with people on the business side. I feel like having a window of reduced durability per object that is only ~150ms wide and up-front can be safely ignored. Especially considering the hypothetical ways in which you could almost instantaneously interrupt the subsequent activities/processing with the feedback mechanism proposed above.
- scrollaway 6y ago
  It would be a cool interface, I would love to see that.
- 0xbadcafebee 6y ago
  > they had downloaded a cached/older version of the software we had uploaded to the same URL This is a great example (to me) of how weak consistency is useful. It exposed a problem you otherwise wouldn't notice. You were deploying offline artifacts to customers without giving the customer the version of the new artifact. Regardless of backend consistency, the customer will not be able to tell (from the URL, anyway) what version they are downloading. Nor will they be able to, say, download the old version if the new version introduces bugs. By changing your deployment method to use unique URLs per version, your customer gains useful functionality and you avoid having to depend on strong consistency (which actually removes expensive requirements). If you design for weak consistency, your system ends up more resilient to multiple problems.
marius_k 6y ago
I was naive to think that these type of consistencies were already in place.
miohtama 6y ago
Is Dropbox still using AWS? They had a testimonial on the page.
- fcantournet 6y ago
  I think they use cloud providers in regions where they don't have physical footprints, to offer data locality guarantees. But the testimonial only talks about Dropbox's data lake. So this is not about dropbox's main product storage, it's "just" their data lake for analytics. (which is apparently 32PB !)
amuraruhn 6y ago
One capability left to implement now: make `move` operations in S3 O(1)
FirstLvR 6y ago
so many posts about amazon, i like AWS but this looks pretty much like advertising haha
- awithrow 6y ago
  The annual AWS re:Invent conference started this week hence the flood of announcements
2color 6y ago
How does this compare to Cloudflare's durable objects?
finikytou 6y ago
does it mean s3 could be used efficiently as a file system? Would that compete with FSX?
brodouevencode 6y ago
In the early days of S3 this was a real problem. Over the years they've beefed up the backend so objects become became consistent faster, but this is a huge leap.
ris 6y ago
If only there were a way of atomically setting a object tag instead of doing a read-modify-write of the whole tagset...

699 points by juliansimioni 6y ago

i already had enough "catching-up-with-latest-aws-tech" problem with amazon, though this is a welcome addition, with daily announcements like this, makes life tougher for a AWS architect who likes to be up to date! just today proton was announced and now this! this is going to be a hectic month :|

WrtCdEvrydy 6y ago

Has anyone ever seen S3 behave eventually consistent? I have not seen a lot of eventual consistency in the real world but I wonder if I'm just working on the wrong problems?

banana_giraffe 6y ago
Yes, a ton. The big issue for me was Does key exist? -> No -> Upload object with key. Followed by Get key -> Not found. Less rarely, I'd see Upload object followed by an overwrite object, followed by a download download some combination of the two if the download was segmented (like AWS's cli will do). The first one happened often enough to cause problems, the second one was a fairly rare event, but still had to be handled. It might have been more of an issue with "large" buckets. The bucket in particular where I had to dance around the missing object issue had ~100 million fairly small objects. I ended up having to create a database to track if objects exist so no consumer would ever try to see if a non-existent object existed. Time to revisit all that mess, I suppose.
OJFord 6y ago
Yes, it caused us some pain with multipart uploads that we sort of proxied, them being multipart from the client to us as well.
brian_herman 6y ago
Yeah, I never had that problem either. But I wonder how they are addressing the https://en.wikipedia.org/wiki/CAP_theorem https://en.wikipedia.org/wiki/CAP_theorem
- content_sesh 6y ago
  Sacrificing availability I reckon.
  - riking 6y ago
    Yup, the correct answer in the face of a network partition is for the PUT to return 500, even though the file really was uploaded. The API handler can't prove success, due to the network partition, so the correct move is to declare failure and let the client activate their contingencies.
dilatedmind 6y ago
I think it was already strongly consistent within regions? like if you had tried to read a non-existent key, then wrote to it, it might continue to appear to not exist for a minute?
- TheDong 6y ago
  Their previous consistency was RAW, but not RAU (for most regions. I think us-east-1 was missing RAW consistency for quite a while). After a write, you would _always_ be able to read the key you just wrote. After an update, you could get a stale copy of the key if your subsequent read hit a different server.
  - heartofgold 6y ago
    Not quite _always_. There were some documented caveats... the one I hit before was: Read nonexistent key, followed by a write of that same key, and then a subsequent read could return a stale read saying it didn’t exist. (even though it had just been written for the first time.) Anyway I am glad to see these gaps and caveats have been closed.
richwater 6y ago
Yes. When you're writing terabytes of data files it was very obvious. That's why manifest files became so popular.
SpicyLemonZest 6y ago
I've never seen the minutes long delays that you were theoretically supposed to accommodate, but second-scale delays were pretty common in my team's experience. (We're writing a lot of files though, probably hundreds or thousands per second aggregated across all our customers.)
- WrtCdEvrydy 6y ago
  Yeah, I got seconds but people were saying there were minutes of data... max I ever saw was 3 seconds on a 2gb file...
  - gregw2 6y ago
    I ran across issues where files that we PUT in S3 were still not visible in Hive 6+ hours later. It only happened a few times out of hundreds of jobs per day for jobs running daily the last 2-3 years, but it was annoying to deal with when it happened because there wasn't a lot we could do beyond open tickets with AWS.
TheDong 6y ago
Yes. I've observed eventual consistency with S3 prior to this change where it took on the order of several hundred milliseconds to observe an update of an existing key. I've observed this as recently as 2 years ago.
johncolanduoni 6y ago
There used to be a behavior with 404 caching that was pretty easy to trigger: if you race requesting an object and creating it, subsequent reads could continue to 404.
deleted 6y ago
[deleted]
rantwasp 6y ago
yes. i believe newer region had read after write in a while but you needed to use the right endpoint i remember a time when if you were using the us-standard “region” and were unlucky it could take 12 hours for your objects to become visible
roseway4 6y ago
Yes. Writing objects from EMR Spark to S3 buckets in a different account. Immediately setting ACLs on the new objects fail as the objects are not yet “there”. Using EMRFS Consistent View was previously the obvious solution here, but it adds significant complexity.
- enw 6y ago
  EMRFS consistent view is a mess. Changes to S3 files are likely to break EMRFS, the metadata needs to be kept in sync (otherwise the table will keep increasing forever) with tools that are a pain to install, and when I used it to run Hive queries from a script, it didn't work.
  - jamesblonde 6y ago
    This news is the death of EMRFS. The only thing it has left is fast listings of "directories" using DynamoDB.
allengeorge 6y ago
Yes, we have. Specifically: 1. LIST 2. PUT 3. LIST would trigger situations where (3) wouldn't include the object inserted in (2). This is well-known however.
- juancampa 6y ago
  Also: 1. HEAD key -> 404 2. PUT key -> 200 3. GET key -> 404 (what? But I just put it!) This is commonly used for "upload file if it doesn't exist"
  - electrum 6y ago
    Even more confusing, the 404 on GET is caused by doing the HEAD when the object doesn't exist. Without the previous HEAD, the GET would usually succeed.
macksd 6y ago
Yes. In fact I spent months working on a system to workaround all the problems it caused for Hadoop: https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/s3guard.html https://hadoop.apache.org/docs/current/hadoop-aws/tools/hado.... We had a lot of automated tests that would fail because of missing (or more recently, out of date) files a significant portion (maybe 20-30%) of the time. We believed that S3 inconsistency was to blame but I always felt like that was the typical "blame flaky infra" cop-out. As soon as S3Guard was in place all those tests were constantly green. It really was S3 inconsistency, routinely. To be fair to S3, Hadoop pretending S3 is a hierarchical filesystem is a bit of a hack. But I had cases where new objects wouldn't be listed for hours. There's only so much you can do to design an application around that, especially when Azure and Google Storage don't have that problem.
enw 6y ago
All the time when using EMR with S3. Really frequent, and really annoying to mitigate.

scrollaway 6y ago

That's pretty incredible. With no price increase either. Using s3 as a makeshift database is possible in a lot more scenarios now. I especially like using s3 as a db when dealing with ETLs, where the state of some data is stored in its key prefix. This means that the etl can stop / resume at any point with a single source of truth as the database. The potential drawback is cost of course; moving (renaming) is free but copying is not. S3's biggest price pain is always its PUTs. In many ETLs this is usually a non-issue because you have to do those PUTs regardless, as you probably want the data to be saved at the various stages for future debugging and recovery.

foxhop 6y ago
Yup me too! Checkout this state_doc Interface I wrote in Python for building a state machine where the state is stored in S3. I never bothered to solve for the edge case of inconsistent reads because my little state machine only ever measured in hours when it slept waiting for work and then minutes in-between waking up and doing work to verify restores. With the news of S3's strong consistency my program is immediately safer to use for some set of defects, since the underlying datastore properties, S3 has made my codes naive/invalid assumptions, true. https://github.com/remind101/dbsnap/blob/master/dbsnap_verify/README.rst#state_doc https://github.com/remind101/dbsnap/blob/master/dbsnap_verif... I should likely pull out the StateDoc class into it's own module.

jchw 6y ago

I believe this makes Amazon S3 behave more similar to Azure blob storage[1] and Google Cloud Storage[2], which is pretty convenient for folks who are abstracting blob stores across different offerings. For what it’s worth, consistency in S3 was usually pretty good anyways, but I ran into issues where it could vary a bit in the past. If you designed your application with this in mind, of course, it shouldn’t be an issue. In my case I believe we just had to add some retry logic. (And of course, that is no longer necessary.) [1]: https://docs.microsoft.com/en-us/azure/storage/common/storage-concurrency#managing-concurrency-in-blob-storage https://docs.microsoft.com/en-us/azure/storage/common/storag... [2]: https://cloud.google.com/storage/docs/consistency https://cloud.google.com/storage/docs/consistency

joana035 6y ago
And Minio too: https://docs.min.io/docs/distributed-minio-quickstart-guide.html https://docs.min.io/docs/distributed-minio-quickstart-guide....
eloff 6y ago
And Wasabi as well: https://wasabi-support.zendesk.com/hc/en-us/articles/115001684591-What-data-consistency-model-does-Wasabi-employ- https://wasabi-support.zendesk.com/hc/en-us/articles/1150016...
- y4m4b4 6y ago
  Wasabi is eventually consistent for overwrites ``eventual consistency for overwrite HTTP PUTs and HTTP DELETEs`` This is not the same.
  - eloff 6y ago
    Yes, I'm aware.

kartayyar 6y ago

This is one reason I have been a big fan of Google Cloud Storage over AWS S3: at a past company AWS consistency was a huge pain, and GCS has had this for years. https://cloud.google.com/blog/products/gcp/how-google-cloud-storage-offers-strongly-consistent-object-listing-thanks-to-spanner https://cloud.google.com/blog/products/gcp/how-google-cloud-...

aparsons 6y ago
Azure as well

QuinnyPig 6y ago

This was a problem with data lakes / analytics. Small inconsistencies would trash runs; that problem now goes away and removes a lot of janky half-fixes to work around the issue. For most common use cases it's not really an issue.

FridgeSeal 6y ago
Of course the real issue there is those tool treating S3 as something it's not (a filesystem) and building things on it with the expectation of certain guarantees that it explicitly didn't provide before now. I dig the feature (strong consistency) but in some ways it just enables tools that were abusing it to just do so more easily.
- staticassertion 6y ago
  Like RedShift? AWS uses S3 as a database, seems reasonable that their customers would.
  - FridgeSeal 6y ago
    That doesn't mean it blindly "mounts" it as a disk and attempts to use it as a filesystem. They're almost certainly using it in a manner that is aware of the specificities of how S3 works.
    - foxhop 6y ago
      What if the blind mount works 99.9% of the time? I knew what I was doing was risky but well with in my tolerance for pain. Now that we have a strong consistency my code is fixed. That's pretty impressive.
- Decker87 6y ago
  Isn't that the point? To provide some useful functionality for others to build on (and pay for)?

damon_c 6y ago

The more I learn about S3 the more I feel the same awe I feel when looking at the Pyramids or Ankor Wat. > You can send 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per prefix in an S3 bucket.

dheera 6y ago
Side question -- why is s3fs so slow? I get only about 1 put per 10 seconds with s3fs. One would hope that you could use something like s3fs as a makeshift S3 API, but its performance is really horrible.
- rovr138 6y ago
  I can't answer this directly, but I thought I'd chime in with a possible alternative, or at least something else to test. I'm using rclone with S3[1] (and others). There are commands[2] where you can use to just sync/copy/ files but it also has a mount option[3]. It also has caching[4] and other things that might help. [1] https://rclone.org/s3/ https://rclone.org/s3/ [2] https://rclone.org/commands/ https://rclone.org/commands/ [3] https://rclone.org/commands/rclone_mount/ https://rclone.org/commands/rclone_mount/ [4] https://rclone.org/cache/ https://rclone.org/cache/
- banana_giraffe 6y ago
  One thing I noticed when I went digging through the s3fs source trying to answer that myself is that it's uploader is single threaded. Part of what makes AWS's cli reasonably fast is that it uses several threads to make sure to use as much bandwidth as possible. If I limit AWS's tool to one thread on a box with plenty of bandwidth, it turns my upload down from around 135 MiB/s to around 20 MiB/s Not sure if that's all of it, or even the majority of the slowness.
  - gaul 6y ago
    Could you expand on this comment? s3fs uses multiple threads for uploading and populating readdir metadata via S3fsMultiCurl::MultiPerform. Earlier versions used curl_multi_perform which may have hidden some of the parallelism from you.

mholt 6y ago

Would this enable atomic operations with S3 (e.g. "put object only if it does not exist")?

OJFord 6y ago
Could you not already do that by specifying ETag?
- rcaught 6y ago
  No, because the object may not be there (but already uploaded) with eventual consistency.
  - sterwill 6y ago
    Before this update, S3 provided "read-after-write" consistency for object creation at a unique path[0] with PUT and subsequent GET operations. Replacement wasn't consistent. So I think you could have worked out some kind of atomic pattern with S3 objects, using new keys for each state change. [0] https://github.com/awsdocs/amazon-s3-developer-guide/blob/41df5c1f948b64e26d5f2583d4f97d47568a7baa/doc_source/Introduction.md#ConsistencyModel https://github.com/awsdocs/amazon-s3-developer-guide/blob/41...
- scarface74 6y ago
  The Etag is not a reliable hash of the file contents. It will be different if the file was uploaded in multiple parts like using the CLI than if it was moved as one operation like copying from one bucket to another. You can tell the difference because the Etag will have a “-“ in it if it was uploaded as a multipart upload.
ryanworl 6y ago
I also want the answer to this, but to the best of my knowledge it does not.
leakybucket 6y ago
The copy object operation has options to control if the copy occurs based on examining source metadata, see "x-amz-copy-source-if-match" at https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObje... . With the consistency change, those might be useful as the basis for atomic operations.

deleted 6y ago

[deleted]

yellow_lead 6y ago

What's up with all the new AWS features in the past few days? Coincidence or not?

banana_giraffe 6y ago
AWS Re:Invent is going on. It's virtual this year, but they're still using it to unveil a bunch of new stuff.
kommissar 6y ago
No coincidence. It’s reinvent season. https://reinvent.awsevents.com/ https://reinvent.awsevents.com/
- yellow_lead 6y ago
  Thanks, I was totally out of the loop there.
  - kommissar 6y ago
    It’s free this year, too. Might as well register if you’re interested.
deleted 6y ago
[deleted]

yodsanklai 6y ago

Could someone describe a few real-life scenarios where this is useful and noticeable?

PetahNZ 6y ago
Our image processing worker queues/servers write an S3 object and dispatch a follow up job. Currently we have to delay the next job (we use 5 seconds) otherwise then next job may start processing before the S3 object is available (it 404's if the next job is run straight away).
- ramraj07 6y ago
  That sounds _very_ wrong, what type of throughput are we talking about here?
  - riking 6y ago
    The story wasn't wrong, until today. That's why people are celebrating.
  - PetahNZ 6y ago
    Up to hundreds of images per second.
    - ramraj07 6y ago
      Okay that makes sense, when we have very low throughput it looks like consistency wasn't an issue but at that rate (especially if it occurs in bursts) I can see how the prior S3 implementation could fail to scale fast enough
- mcqueenjordan 6y ago
  Out of curiosity, did you have retries on the GETs?
  - PetahNZ 6y ago

jz-amz 6y ago

Some related reading - * Jeff Barr's blog post on this topic, has an interesting case study: https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-read-after-write-consistency/ https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-rea... * Collaboration with Hadoop/S3A maintainers leading to this release: https://aws.amazon.com/blogs/opensource/community-collaboration-the-s3a-story/ https://aws.amazon.com/blogs/opensource/community-collaborat...

yftsui 6y ago
I am curious about how much Dropbox pays for data ingress/egress, they migrated storage to their own on premise data center, then now moving data back to S3 for the data lake.
- Scaevolus 6y ago
  That's analytics data, not file storage. If they have 34PB of analytics, they certainly have far more than that of blob storage.
  - SteveNuts 6y ago
    I thought they were almost entirely on prem for the actual storage
    - ddorian43 6y ago
      See https://github.com/rockset/rocksdb-cloud https://github.com/rockset/rocksdb-cloud
  - deleted 6y ago
    [deleted]
  - ashish-gandhi 6y ago
    Definitely much more of user data storage. More about that https://dropbox.tech/infrastructure/inside-the-magic-pocket https://dropbox.tech/infrastructure/inside-the-magic-pocket. Happy to take other questions.

juliansimioni 6y ago

Another, perhaps better link for the announcement: https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-read-after-write-consistency/ https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-rea...

throwaway_qcjal 6y ago

Congratulations to the S3 team from a friendly cloud storage neighbor in SLU :-)

kleebeesh 6y ago

Fantastic. Any write-ups or descriptions for how they made it happen?

hoytschermerhrn 6y ago
Seconded, I'd love to read a whitepaper if anyone is able to provide one. I'm assuming this operates on some variant of Google's TrueTime[0]. [0] https://cloud.google.com/spanner/docs/true-time-external-consistency https://cloud.google.com/spanner/docs/true-time-external-con...
- foota 6y ago
  Not necessarily, I think Spanner's magic is most useful for bounded staleness reads and scaling reads through non-leader replicas. Otherwise I think you're looking at normal "NewSQL" guarantees.

deleted 6y ago

[deleted]

boulos 6y ago

Disclosure: I work on Google Cloud. This is super awesome for customers. I am also beyond excited for all the open-source connectors to finally be simplified so that they don't have to deal with the "oh right, gotta be careful because of list consistency". It was super awesome when we were able to delete a huge chunk of the GCS Connector for Hadoop, and I hope to see the same across the S3-focused connectors. I'm now much more inclined to make my GCS FUSE rewrite in Rust also support S3 directly :).

basicneo 6y ago
> It was super awesome when we were able to delete a huge chunk of the GCS Connector for Hadoop Is this public?
- ceocoder 6y ago
  Yes, https://github.com/GoogleCloudDataproc/hadoop-connectors https://github.com/GoogleCloudDataproc/hadoop-connectors disclosure: I work at Google as well.
  - boulos 6y ago
    Right, https://github.com/GoogleCloudDataproc/hadoop-connectors/releases/tag/v1.9.0 https://github.com/GoogleCloudDataproc/hadoop-connectors/rel... was apparently the release: > Delete metadata cache functionality because Cloud Storage has strong native list operation consistency already. If folks are actually interested in these connectors, I'd also recommend this blogpost from last year: https://cloud.google.com/blog/products/data-analytics/new-release-of-cloud-storage-connector-for-hadoop-improving-performance-throughput-and-more https://cloud.google.com/blog/products/data-analytics/new-re... because even with consistency, GCS and S3 still aren't filesystems :).
    - basicneo 6y ago
      Do you have a link to the commits that removed the code. It'd be good to see what sort of complexity this sort of strong consistency can make redundant.

gfody 6y ago

we use s3 in a repo pattern pretty heavily and I had no idea they didn't have read-after-write consistency. I've noticed s3 will hold connections open by sending back blank lines and always assumed it was synchronizing for consistency.

k__ 6y ago

It's interesting to read all these comments here that talk about the eventual consistency like it was some kind of bug.

howlgarnish 6y ago
Are there any scenarios where you would want eventual consistency over strong consistency? (Assuming pricing, performance, replication etc are the same.)
- crazyjncsu 6y ago
  No.
- nhumrich 6y ago
  At the benefit of added availability? absolutely. This is the entire premise of mongodb.
  - nullsense 6y ago
    MongoDB is webscale.
    - castlecrasher2 6y ago
      I want the one with the bigger GBs.
      - nullsense 6y ago
        Better yet, just pipe all your data to /dev/null it supports sharding. Sharding is the secret in the webscale sauce.
  - oblio 6y ago
    Availability? I thought Mongo was fast and scalable, I don't remember anyone recommending it for higher availability.

deleted 6y ago

[deleted]

psanford 6y ago

Neat! Would this make it possible in the future for S3 to support CAS like operations for PutObject? Something like supporting `if-match` and `if-none-match` headers on the etag of an object would be so useful.

The_rationalist 6y ago

Noob question: when to use S3 vs HDFS?

jamesblonde 6y ago
For cost on AWS, use S3. AWS price local storage on VMs to make HDFS not competitive. If you want the HDFS API but S3 storage costs on AWS, use HopsFS-S3 - https://www.logicalclocks.com/blog/hopsfs-100x-times-faster-than-aws-s3 https://www.logicalclocks.com/blog/hopsfs-100x-times-faster-... (disclosure: work on HopsFS). HDFS provides POSIX-like API, and now has atomic metadata operations that S3 doesn't (mv/rename, chown, chmod) and append for files.

cash22 6y ago

How do they accomplish strong consistency without any hit to availability?

YZF 6y ago
While I don't know the specifics I can say that availability is down to engineering practices. Let's say that their consistency model is achieved via quorum, that is writes write to a quorum of nodes while reads read from a quorum of nodes (of their metadata database) then this guarantees read after write consistency. The availability aspect of this is just engineering, making sure you're never down to less than a quorum of nodes. E.g. by having redundant power supplies, generators, networks, or whatever engineering it takes to reduce the probability of failure. There's other aspects, e.g. latency, that may suffer, but again this is solved via engineering. Just throw more hardware at it to bring the latency down. The only time where you absolutely can't solve it is if you provide strong consistency across geographical regions that are far apart, there's no way then not to pay that latency. This is just another example of why the CAP theorem isn't really as useful to determining limitations of practical systems as it may seem at first site.
- beoberha 6y ago
  I hope they write a paper or at least do a talk about how they went about it. Usually these things are more involved than an outsider would think. Agreed that you can beat most of the CAP tradeoffs with significant engineering effort, but it’s usually pretty interesting how they accomplish it. Especially at S3’s scale where the COGs impact of “throw more hardware at it” can cost on the order of tens or hundreds of millions.
- valenterry 6y ago
  > (of their metadata database) With that you have just moved the question to: how do they ensure that the metadata database is available _and_ strongly consistent at the same time for all the requests? Because the metadata database is certainly also a distributed one, hence you need to query _all_ the nodes or can end up in a split-brain situation and lose consistency (or availability if you choose to down the system or a part of the system).

hendry 6y ago

I wonder if this would make S3 a host for sqlite!

foxhop 6y ago
Yup, I think it would.
- sudeepj 6y ago
  I have started working on this actually (the vfs being developed in Rust). The design was complex but now this simplifies a lot. The vfs will also have snapshotting capabilities i.e. you can have multiple versions of sqlite db.
  - js4ever 6y ago
    How are you handling latency (up to 800ms on S3) or the fact that you have to write the whole file each time?
    - sudeepj 6y ago
      > you have to write the whole file each time Will not write the whole file. Instead each page of the database (default = 4KB) will be separate put. The use case is: one-writer multiple readers. The readers will read an old snapshot of the database (from s3) and writer will write to the active one. This way a lot of readers can open the DB in read-only mode. As for writer, each page will be a separate put. The writes will be appended to a log(s) locally and the page will be synced in the background. This helps to exploit lot of parallel PUTs to S3. Once the sync happens the disk space for that page will be reclaimed. For the writer: v = open_new_version() //inserts + updates //commit close_version(v) For readers: open_version_for_read(v) //select
    - ComodoHacker 6y ago
      Obviously, a sane implementation would divide db file into blocks and write them into separate objects. Not as small as traditional FS blocks, 1-4 Mb, perhaps.

gaul 6y ago

This is great news for application developers! And probably hard work for the AWS implementors. I will try to update this over the weekend: https://github.com/gaul/are-we-consistent-yet https://github.com/gaul/are-we-consistent-yet

stubish 6y ago

Once libraries and tools start relying on this, it is going to make life interesting to the S3-compatible players. The API remains the same, but the behavior is quite different.

sleepydog 6y ago
Google Cloud storage has behaved this way for years. According to other comments, so does Azure.
- stubish 6y ago
  It is the smaller players I'm thinking of, which needed S3 compatibility for uptake. Openstack Swift, Backblaze, Ceph... I am not sure if any of these suddenly stopped being S3 compatible due to this announcement.
  - neverartful 6y ago
    Ceph has strong consistency by design. S3 compatibility in Ceph comes via the Rados Gateway (RGW) which just sits atop normal Ceph. This means that the strong consistency is carried over to RGW. So, no change for Ceph. OpenStack Swift has an eventually consistent design. If the Swift community desires to maintain strict compatibility with S3 regarding consistency model, it would be a challenge. Despite being a challenge, I would never bet against the Swift developers. If they want to do it, I'm quite sure that they can. Having said all of that, I doubt that the Swift community would consider this a burning issue that needs to be solved. They've existed for over 10 years with an eventually consistent model and their user base is well aware of it.

marwan-nwh 6y ago

What about CAP theorem?

danenania 6y ago
The CAP theorem doesn't preclude strong consistency. It just means that at a certain scale, you'll pay for it with extra latency.
- marwan-nwh 6y ago
  And that latency means that some reads won't see the latest write, right? Do I understand this correctly?
  - jabl 6y ago
    If you want consistency, you have to pay for it with latency, since the system has to do work in the background to ensure that the data you read is really the latest. If you care more about latency than consistency you have an 'eventually consistent' system, where a write will eventually propagate, but a read might get stale data.
    - bxji 6y ago
      > If you care more about latency than consistency you have an 'eventually consistent' system, where a write will eventually propagate, but a read might get stale data. Not just stale data, you can also have states which never actually existed. I'll steal the example from Doug Terry's paper "Replicated Data Consistency Explained Through Baseball" because it's really good. Linked below. Say you have a baseball game which is scored by innings. It's the middle of the 7th inning, and the true write log for the state of the game is as follows: Write ("home", 1) Write ("visitors", 1) Write ("home", 2) Write ("home", 3) Write ("visitors", 2) Write ("home", 4) Write ("home", 5) If you were to read the score at this point in time, and your system is strongly consistent, the score can only be 2-5 or a refusal to serve the request. If your system is eventually consistent, the score can be any of the following: 0-0, 0-1, 0-2, 0-3, 0-4, 0-5, 1-0, 1-1, 1-2, 1-3, 1-4, 1-5, 2-0, 2-1,2-2, 2-3, 2-4, 2-5. Source paper: https://www.microsoft.com/en-us/research/wp-content/uploads/2011/10/ConsistencyAndBaseballReport.pdf https://www.microsoft.com/en-us/research/wp-content/uploads/...

whateveracct 6y ago

This is really something, especially for small apps. S3 is now the easiest KV store I can imagine.

sudeepj 6y ago
> small apps Even the big ones. A lot of applications (e.g backup vendors) had to resort to use EBS volumes because of lack of read-after-update consistency. The barrier to entry for lot of developers has been reduced. My prediction: A lot of storage infra (FS, DB) with native support for S3 will get commoditised now.

leakybucket 6y ago

With this change, is there a way to implement atomic operations like put-if-absent?

ChuckMcM 6y ago

This is awesome, I could have used it for a project we did a while back. I'm sure that somewhere there is a Product Manager at Amazon who talks to customers they find out are using Google Cloud or Azure and asks them "Why not use AWS?" and they mumble some feature of the other service that they need. Sometime later, said service shows up as part of AWS. :-) For me this is the best part of tech rivalries, whether they are in CPU performance/architecture, programming language features/tools, or network services. Pressure to improve.

ultimoo 6y ago

> S3 consistency is available at no additional cost and removes the need for additional third-party, services, and complex architecture. I wonder whether any companies/businesses solely depended on offering eventual consistency workarounds that probably now need to pivot.

pas 6y ago
The hadoop companies spent quite some bucks on developing S3Guard, and selling it. https://hadoop.apache.org/docs/r3.0.3/hadoop-aws/tools/hadoop-aws/s3guard.html https://hadoop.apache.org/docs/r3.0.3/hadoop-aws/tools/hadoo...

david-cako 6y ago

big news. this was a pretty big issue for a customer at AWS that was using S3 for their analytics platform.

eggie5 6y ago

does this mean the TF S3 plugin will stop cluttering my logs w/ s3 retries?

ed25519FUUU 6y ago

The most devious s3 consistency issue I encountered was that S3 bucket configs are actually stored in S3, and were not read-after-write consistent. It was bad. In order to "update" a bucket config programmatically you need to read the entire existing config, make an update, and then PUT it back (overwriting what's there). The problem is when you went to read the config it's possibly 15 minutes old or more, and when you put it back you overwrite any changes that may not be consistent. I had to work around the issue by storing a high-water mark in the config ID, and then poll an external system to find the expected high water mark before I knew I could safely read and update.

jamesblonde 6y ago

Listing and update consistency is great, but the ACID SQL-on-S3 frameworks (Apache Hudi, Dela Lake) also require atomic rename to support transactions. HopsFS-S3 solves the ACID SQL-on-S3 problem discussed here on HN last week ( https://news.ycombinator.com/item?id=25149154 https://news.ycombinator.com/item?id=25149154 ) by adding a metadata layer over S3 and providing a POSIX-like HDFS API to clients. Operations like rename, mv, chown, chmod are metadata operations - rename is not copy and delete. Disclosure: I work on HopsFS

iblaine 6y ago

AWS Re:invent talked about improvements to S3 earlier today. You can see that talk here https://virtual.awsevents.com/media/1_usujusng https://virtual.awsevents.com/media/1_usujusng For any AWS EMR/hadoop users out there, this means the end of emrfs. For the uninitiated, emrfs recognizes consistency problems but does not fix them. It'll throw an exception if there's a consistency problem, even give you some tools to fix the problem, but the tools may give false positives. The result is you've got to fix some consistency problems yourself, parse items out of the emrfs DynamoDB catalog, match them up to s3, then make adjustments where needed. It's an unnecessary chore. It surprises me that this issue has not got more attention over the years and thankfully it'll be solved soon.

leowoo91 6y ago

Interesting, I was always versioning file names as they upload. That seems to be mostly for file updates rather.

Aeolun 6y ago

Oh, so this is why the images I uploaded from my chat app sometimes just didn’t show up from certain locations. Glad it’s solved now :)

tutfbhuf 6y ago

Ask HN: I have a question regarding strong consistency. The question is related to all geo distributed strong consistent storages, but let me pick Google Spanner as example. Spanner has a maximum upper bound of 7 milliseconds clock offset between nodes, thx to TrueTime. To achieve external consistency, Spanner simply waits 7 ms before committing a transaction. After that amount of time the window of uncertainty is over and we can be sure, that every transaction happens to be in the right order. So far so good, but how does Spanner deal with cross data center network latency? Let's say I commit a transaction to Spanner in Canada and after 7 ms, I get my confirmation, but now in Australia someone also does a conflicting transaction and also get his confirmation after 7 ms. Spanner however, bound to the laws of physics, can only see this conflict after 100+ ms delay, due to network latency between the datacenters. How is that problem solved?

throwaway_dcnt 6y ago
Partitioning.
foota 6y ago
This documentation talks about it in brief: https://cloud.google.com/spanner/docs/whitepapers/life-of-reads-and-writes https://cloud.google.com/spanner/docs/whitepapers/life-of-re.... You can read the spanner paper for more detail: https://static.googleusercontent.com/media/research.google.com/en//archive/spanner-osdi2012.pdf https://static.googleusercontent.com/media/research.google.c... The simple answer is that there are round trips required between datacenters when you have a quorum that spans data centers. Additionally, one replica of a split is the leader, and the leader holds write locks. So already you have to talk to the leader replica, even if it's out of the DC you're in. Getting the write lock overlaps with the transaction commit though. So for your example if we say the leader is in Canada and the client is in Australia, and we're doing a write to row 'Foo' without first reading (a so called blind write): Client (Australia) -> Leader (Canada): Take a write write lock on 'Foo' and try to commit transaction Leader -> other replicas: Start commit, timestamp 123 Other replicas -> Leader: Ready for commit Leader waits for majority ready for commit and for timestamp 123 to be before the current truetime interval Leader -> Other replicas: Commit transaction, and in parallel replies to client. Of course there are things you can do to mitigate this depending on your needs, but there's no free lunch. If you have a client in Australia and a client in Canada writing to the same data you're going to pay for round trips for transactions.

tutfbhuf 6y ago

If someone is wondering why this is huge. It's because of the problems you face, when you don't have strong consistency. See this picture to get an understanding of what kind of problems you might run into with eventual consistency: https://i.ibb.co/DtxrRH3/eventual-consistency.png https://i.ibb.co/DtxrRH3/eventual-consistency.png

pachico 6y ago
You just made my day :)
rafaelturk 6y ago
I shall print this..
- freemint 6y ago
  Please support the original artist and make sure you have a license to print it before you do.I
  - homarp 6y ago
    the artist is Pascal Jousselin http://pjousselin.free.fr/ http://pjousselin.free.fr/ it's called 'Imbattable' or Mister Invincible
    - memiux 6y ago
      Ok that explains the Baguette
  - st1ck 6y ago
    So you're concerned about printing 1 copy for home or office, but not about sharing this image on the internet?
  - fuzzer37 6y ago
    Please do whatever you want with an image posted to an internet forum

toolslive 6y ago

atomic multipart uploads, here we come ;)

awssucs 6y ago

“After a successful write of a new object or an overwrite of an existing object, any subsequent read request immediately receives the latest version of the object.” - being able to issue a read after a write has been the case in any system ever. AWS solving a fabricated issue, and the crowd cheering it as progress. Have to love herd mentality.

jamesblonde 6y ago
Previously, update-then-read was not guaranteed to be consistent. Now it is consistent.
- awssucs 6y ago
  yes for aws it was an issue, what i meant is that aws solved an issue that it itself created.
  - p0rkbelly 6y ago
    14 years ago. That is called tech debt. Not an issue. They designed it for the use case at the time. Nobody knew what the cloud was becoming.
    - awssucs 6y ago
      I agree nobody knew what the cloud was becoming. But it feels like aws is selling an unfinished product and people are wasting money and time working around non issues. Aws is great for medium sized projects but as soon as your project grows, you bump into silly issues like this one. Given that s3 is more or less a file storage you don’t really expect issues such as “weak” consistency. You expect a file to be committed once written (yes aws can do queueing behind the scenes) but when it returns a 200 ok it means the file was stored. Otherwise the good folks at aws should return a 202 accepted response as all properly designed apis do and let the user know that a delay in reading is expected.
swyx 6y ago
if you've ever worked on distributed storage you'll know that this isn't fabricated. it just takes effort to fix it. and S3 is pretty much the oldest distributed store in existence, lots of tech debt to overcome.

mikaeluman 6y ago

I thought we had worked out in 2010-2012 that the CAP theorem, consistency, availability and partition tolerance (https://en.wikipedia.org/wiki/CAP_theorem#History https://en.wikipedia.org/wiki/CAP_theorem#History) made this impossible? Where is the mistake.

chillydawg 6y ago
I would guess write times are slightly up to cover the cost of the new replication completion guarantees.
johncolanduoni 6y ago
S3 is almost certainly not fully partition tolerant at the node level and requires some sort of quorum. Other “magical” data stores like Spanner also retain this limitation, they just have very reliable replication strategies.
- pricechild 6y ago
  "almost certainly not fully partition tolerant at the node level" Whoa hang on! You can't just say you're not tolerant of partitions... I'm struggling to find the best post on aphyr.com about this but https://aphyr.com/posts/325-comments-on-you-do-it-to https://aphyr.com/posts/325-comments-on-you-do-it-to is a good one, specifically the line: "CP and AP are upper bounds: systems can provide C or A during a partition, but might provide neither." In short, Partitions happen, no matter what. No really, when you have multiple nodes they happen, you must tolerate them. You can't have CA.
  - johncolanduoni 6y ago
    I’m not claiming it’s a CA system, and the terminology “partition intolerant” is not verboten by Kyle Kingsbury. From your link: “Specifically, partition-intolerant systems must sacrifice invariants when partitions occur. Which invariants?” The answer in this case is that availability is sacrificed, unless Amazon is making a very misleading claim of strong consistency (per the submission title/link). So it’s CP. In case “not fully partition tolerant” is what is throwing you off, all I mean by that is that there are likely network partitions of the S3 nodes that will not cause outward failure (e.g. partition of a single node), though some will (e.g. too many partitions to form a quorum, or whatever their underlying consensus implementation relies on).

intricatedetail 6y ago

Is "strong" redundant? Something can be either consistent or not. Seems like developers asked marketing for a catchy name, but that looks comical.

bob1029 6y ago

This is really cool. We actually got burned on weak S3 consistency a few weeks ago when generating public download links for a customer. Took us a few hours of troubleshooting to realize they had downloaded a cached/older version of the software we had uploaded to the same URL just a few minutes prior. Resolution was to use unique paths per version to guarantee we were talking to the right files each time. One potentially related item I was thinking about - How does HN feel about the idea of a system that has eventual durability guarantees which can be inspected by the caller? I.e. Call CloudService.WriteObject(). It writes the object to the local datacenter and asynchronously sends it to the remote(s). It returns some transaction id. You can then pass this id to CloudService.TransactionStatus() to determine if a Durable flag is set true. Or, have a convenience method like CloudService.AwaitDurabilityAsync(txnid). In the most disastrous of circumstances (asteroid hits datacenter 1ms after write returns to caller), you would get an exception on this await call after a locally-configured timeout in which case you can assume you should probably retry the write operation. I was thinking this might be a way to give the application a way to decide how to deal with the concept of latency WRT cross-site replication. You could not await the durability for 4 nines or wait the additional 0-150 ms to see more nines. I wonder how this risk profile sits with people on the business side. I feel like having a window of reduced durability per object that is only ~150ms wide and up-front can be safely ignored. Especially considering the hypothetical ways in which you could almost instantaneously interrupt the subsequent activities/processing with the feedback mechanism proposed above.

scrollaway 6y ago
It would be a cool interface, I would love to see that.
0xbadcafebee 6y ago
> they had downloaded a cached/older version of the software we had uploaded to the same URL This is a great example (to me) of how weak consistency is useful. It exposed a problem you otherwise wouldn't notice. You were deploying offline artifacts to customers without giving the customer the version of the new artifact. Regardless of backend consistency, the customer will not be able to tell (from the URL, anyway) what version they are downloading. Nor will they be able to, say, download the old version if the new version introduces bugs. By changing your deployment method to use unique URLs per version, your customer gains useful functionality and you avoid having to depend on strong consistency (which actually removes expensive requirements). If you design for weak consistency, your system ends up more resilient to multiple problems.

marius_k 6y ago

I was naive to think that these type of consistencies were already in place.

miohtama 6y ago

Is Dropbox still using AWS? They had a testimonial on the page.

fcantournet 6y ago
I think they use cloud providers in regions where they don't have physical footprints, to offer data locality guarantees. But the testimonial only talks about Dropbox's data lake. So this is not about dropbox's main product storage, it's "just" their data lake for analytics. (which is apparently 32PB !)

amuraruhn 6y ago

One capability left to implement now: make `move` operations in S3 O(1)

FirstLvR 6y ago

so many posts about amazon, i like AWS but this looks pretty much like advertising haha

awithrow 6y ago
The annual AWS re:Invent conference started this week hence the flood of announcements

2color 6y ago

How does this compare to Cloudflare's durable objects?

finikytou 6y ago

does it mean s3 could be used efficiently as a file system? Would that compete with FSX?

brodouevencode 6y ago

In the early days of S3 this was a real problem. Over the years they've beefed up the backend so objects become became consistent faster, but this is a huge leap.

ris 6y ago

If only there were a way of atomically setting a object tag instead of doing a read-modify-write of the whole tagset...