3 ms·

Could someone describe a few real-life scenarios where this is useful and noticeable?

PetahNZ 6y ago
Our image processing worker queues/servers write an S3 object and dispatch a follow up job. Currently we have to delay the next job (we use 5 seconds) otherwise then next job may start processing before the S3 object is available (it 404's if the next job is run straight away).
- ramraj07 6y ago
  That sounds _very_ wrong, what type of throughput are we talking about here?
  - riking 6y ago
    The story wasn't wrong, until today. That's why people are celebrating.
  - PetahNZ 6y ago
    Up to hundreds of images per second.
    - ramraj07 6y ago
      Okay that makes sense, when we have very low throughput it looks like consistency wasn't an issue but at that rate (especially if it occurs in bursts) I can see how the prior S3 implementation could fail to scale fast enough
- mcqueenjordan 6y ago
  Out of curiosity, did you have retries on the GETs?
  - PetahNZ 6y ago
    No, we didn't try retrying (well actually if the job failed because of a 404 it would retry the whole job based on exponential backoff). When we encountered the error we simply added the delay as it was an easy config change to our SQS queues and doesn't meaningfully affect our use case.
- random5634 6y ago
  Same issues updating a file - you’d update on s3, then trigger processing and even though it was 50ms later - you’d get OLD version. I didn’t know gcp didn’t have this issue - would have been a big selling point
juliansimioni 6y ago
One very common one is for situations where you might have a multi-step pipeline to process data - step 1 generates/processes data, stores it in S3, overwriting the previous copy. triggers step 2 to run - step 2 runs, fetches the data from s3 for its own processing. However, because only a few seconds have elapsed, step 2 fetches the old version of data from the S3 bucket You can work around this by, for example, always using unique S3 object keys, but then you have to coordinate across the data processing steps, and it becomes harder to manage things like storing only the 10 latest versions. The Argo workflow tool (https://argoproj.github.io/ https://argoproj.github.io/) is one example of a tool that can suffer from this problem.
Macha 6y ago
You have a processing job which dumps a bunch of output files in a directory. A downstream job uses these files as input, sees a new directory, and pulls all the files in the new directory. Because s3 was not strongly consistent, you would have the downstream job see a arbitrary subset of the files for a short while after creation, and not just the oldest files. This could cause your job to skip processing input files unless you provided some sort of manifest of all the files it would expect in that batch. So then you'd have to load the manifest, then keep retrying until all the input files showed up in whatever s3 node you were hitting.
- mullen 6y ago
  If this is an issue, you wait X amount of seconds after the file has been created and then start processing it. This would allow the file to be consistent before processing it. A better idea would be triggering Lambda jobs which either directly processes the files as they are added to S3 or trigger Lambda jobs which add the files to SQS and each job in the SQS is processed by another Lambda job.
  - Macha 6y ago
    The issues with a sleep and pray strategy are: 1. No amount of time is provably enough 2. If you just take the maximum recorded time and say add 20% padding, then waiting that amount of time to process every dataset could be detrimental to performance. The example I gave happened to the team I was on in 2017/2018. We had 1000s of files totaling terabytes of data in a given batch. The 90th percentile time for consistency was the low 10s of seconds, the 99th percentile was measured in minutes. The manifest and retry not yet present method avoids having to put in a sleep(5 minutes) for the 1% of cases.
HiJon89 6y ago
We used to use S3 for Maven artifact storage. This is mostly an append-only workload, however Maven updates maven-metadata.xml files in place. These files contain info about what versions exist, when they were updated, what the latest snapshot version is, etc. We would see issues where a Maven build publishes to S3, and then a downstream build would read an out-of-date maven-metadata.xml and blow up. Or worse, it could silently use an older build and cause a nasty surprise when you deploy. It only happened a small percentage of the time, but when you’re doing tens of thousands of builds per day it ends up happening every day. We switched to GCS for our Maven artifacts and the problem went away.
- riking 6y ago
  To be clear: the "blowing up" would occur when a client observed the new maven-metadata.xml file, but old ("does-not-exist") records for the newly uploaded artifact, correct? With this update, ordering the metadata update after the artifact upload means this failure is now impossible.
  - HiJon89 6y ago
    For some context, S3 used to provide read-after-write consistency on new objects, but only if you didn't do a GET or HEAD to the key before it was created. So this access pattern is all good: PUT new-key 200 GET new-key 200 However, this access pattern would be unreliable: GET new-key 404 PUT new-key 200 GET new-key 200/404 This last GET could return a 200, or it could return a cached 404. Unfortunately, Maven uses this access pattern when publishing maven-metadata.xml files, because it needs to know whether it should update an existing maven-metadata.xml file or create one if it doesn't exist yet. So when publishing a new version, it does something like this: GET com.example/example-artifact/1.0-SNAPSHOT/maven-metadata.xml 404 PUT com.example/example-artifact/1.0-SNAPSHOT/maven-metadata.xml 200 And then downstream builds would try to resolve version 1.0-SNAPSHOT, and the first thing Maven does is fetch the maven-metadata.xml: GET com.example/example-artifact/1.0-SNAPSHOT/maven-metadata.xml 404 So when publishing a new version, you could get a cached 404 and fail loudly. However, when updating an existing maven-metadata.xml file you could silently read an old maven-metadata.xml, and end up using an out-of-date artifact (which is even more concerning). Here's what one of the maven-metadata.xml files looks like: https://oss.sonatype.org/content/repositories/snapshots/org/openapitools/openapi-generator-core/5.0.0-SNAPSHOT/maven-metadata.xml https://oss.sonatype.org/content/repositories/snapshots/org/... Because updating an existing object in S3 didn't have read-after-write consistency, we could have a publishing flow that looked like: GET com.example/example-artifact/1.0-SNAPSHOT/maven-metadata.xml 200 (v1) PUT com.example/example-artifact/1.0-SNAPSHOT/maven-metadata.xml 200 (v2) And then downstream builds would fetch the maven-metadata.xml: GET com.example/example-artifact/1.0-SNAPSHOT/maven-metadata.xml 200 (v1) So downstream builds could read a stale maven-metadata.xml file, which results in silently using an out-of-date artifact. We ended up just switching to GCS because it was relatively straight-forward and gave us the consistency guarantees we want.

Our image processing worker queues/servers write an S3 object and dispatch a follow up job. Currently we have to delay the next job (we use 5 seconds) otherwise then next job may start processing before the S3 object is available (it 404's if the next job is run straight away).

ramraj07 6y ago
That sounds _very_ wrong, what type of throughput are we talking about here?
- riking 6y ago
  The story wasn't wrong, until today. That's why people are celebrating.
- PetahNZ 6y ago
  Up to hundreds of images per second.
  - ramraj07 6y ago
    Okay that makes sense, when we have very low throughput it looks like consistency wasn't an issue but at that rate (especially if it occurs in bursts) I can see how the prior S3 implementation could fail to scale fast enough
mcqueenjordan 6y ago
Out of curiosity, did you have retries on the GETs?
- PetahNZ 6y ago
  No, we didn't try retrying (well actually if the job failed because of a 404 it would retry the whole job based on exponential backoff). When we encountered the error we simply added the delay as it was an easy config change to our SQS queues and doesn't meaningfully affect our use case.
random5634 6y ago
Same issues updating a file - you’d update on s3, then trigger processing and even though it was 50ms later - you’d get OLD version. I didn’t know gcp didn’t have this issue - would have been a big selling point

juliansimioni 6y ago

One very common one is for situations where you might have a multi-step pipeline to process data - step 1 generates/processes data, stores it in S3, overwriting the previous copy. triggers step 2 to run - step 2 runs, fetches the data from s3 for its own processing. However, because only a few seconds have elapsed, step 2 fetches the old version of data from the S3 bucket You can work around this by, for example, always using unique S3 object keys, but then you have to coordinate across the data processing steps, and it becomes harder to manage things like storing only the 10 latest versions. The Argo workflow tool (https://argoproj.github.io/ https://argoproj.github.io/) is one example of a tool that can suffer from this problem.

Macha 6y ago

You have a processing job which dumps a bunch of output files in a directory. A downstream job uses these files as input, sees a new directory, and pulls all the files in the new directory. Because s3 was not strongly consistent, you would have the downstream job see a arbitrary subset of the files for a short while after creation, and not just the oldest files. This could cause your job to skip processing input files unless you provided some sort of manifest of all the files it would expect in that batch. So then you'd have to load the manifest, then keep retrying until all the input files showed up in whatever s3 node you were hitting.

mullen 6y ago
If this is an issue, you wait X amount of seconds after the file has been created and then start processing it. This would allow the file to be consistent before processing it. A better idea would be triggering Lambda jobs which either directly processes the files as they are added to S3 or trigger Lambda jobs which add the files to SQS and each job in the SQS is processed by another Lambda job.
- Macha 6y ago
  The issues with a sleep and pray strategy are: 1. No amount of time is provably enough 2. If you just take the maximum recorded time and say add 20% padding, then waiting that amount of time to process every dataset could be detrimental to performance. The example I gave happened to the team I was on in 2017/2018. We had 1000s of files totaling terabytes of data in a given batch. The 90th percentile time for consistency was the low 10s of seconds, the 99th percentile was measured in minutes. The manifest and retry not yet present method avoids having to put in a sleep(5 minutes) for the 1% of cases.

HiJon89 6y ago

We used to use S3 for Maven artifact storage. This is mostly an append-only workload, however Maven updates maven-metadata.xml files in place. These files contain info about what versions exist, when they were updated, what the latest snapshot version is, etc. We would see issues where a Maven build publishes to S3, and then a downstream build would read an out-of-date maven-metadata.xml and blow up. Or worse, it could silently use an older build and cause a nasty surprise when you deploy. It only happened a small percentage of the time, but when you’re doing tens of thousands of builds per day it ends up happening every day. We switched to GCS for our Maven artifacts and the problem went away.

riking 6y ago
To be clear: the "blowing up" would occur when a client observed the new maven-metadata.xml file, but old ("does-not-exist") records for the newly uploaded artifact, correct? With this update, ordering the metadata update after the artifact upload means this failure is now impossible.
- HiJon89 6y ago
  For some context, S3 used to provide read-after-write consistency on new objects, but only if you didn't do a GET or HEAD to the key before it was created. So this access pattern is all good: PUT new-key 200 GET new-key 200 However, this access pattern would be unreliable: GET new-key 404 PUT new-key 200 GET new-key 200/404 This last GET could return a 200, or it could return a cached 404. Unfortunately, Maven uses this access pattern when publishing maven-metadata.xml files, because it needs to know whether it should update an existing maven-metadata.xml file or create one if it doesn't exist yet. So when publishing a new version, it does something like this: GET com.example/example-artifact/1.0-SNAPSHOT/maven-metadata.xml 404 PUT com.example/example-artifact/1.0-SNAPSHOT/maven-metadata.xml 200 And then downstream builds would try to resolve version 1.0-SNAPSHOT, and the first thing Maven does is fetch the maven-metadata.xml: GET com.example/example-artifact/1.0-SNAPSHOT/maven-metadata.xml 404 So when publishing a new version, you could get a cached 404 and fail loudly. However, when updating an existing maven-metadata.xml file you could silently read an old maven-metadata.xml, and end up using an out-of-date artifact (which is even more concerning). Here's what one of the maven-metadata.xml files looks like: https://oss.sonatype.org/content/repositories/snapshots/org/openapitools/openapi-generator-core/5.0.0-SNAPSHOT/maven-metadata.xml https://oss.sonatype.org/content/repositories/snapshots/org/... Because updating an existing object in S3 didn't have read-after-write consistency, we could have a publishing flow that looked like: GET com.example/example-artifact/1.0-SNAPSHOT/maven-metadata.xml 200 (v1) PUT com.example/example-artifact/1.0-SNAPSHOT/maven-metadata.xml 200 (v2) And then downstream builds would fetch the maven-metadata.xml: GET com.example/example-artifact/1.0-SNAPSHOT/maven-metadata.xml 200 (v1) So downstream builds could read a stale maven-metadata.xml file, which results in silently using an out-of-date artifact. We ended up just switching to GCS because it was relatively straight-forward and gave us the consistency guarantees we want.