4 ms·

> write your app against S3 and call it a day. ... and then later you notice that you need a special SLA and Amazon Redshift to guarantee that a read from S3 w

> write your app against S3 and call it a day. ... and then later you notice that you need a special SLA and Amazon Redshift to guarantee that a read from S3 will return the same value as the last write. Even S3 is only eventually consistent and especially if a US user uploads images into your US bucket and then you serve the URLs to EU users, you might have loading problems. The correct solution, according to our support contact, is that we wait a second after uploading to S3 to make sure at least the metadata has replicated to the EU before we post the image url. That, or pay for Redshift to tell us how long to wait more precisely. Because contrary to what everyone expects, S3 requests to US buckets might be served by delayed replicas in other countries if the request comes from outside the US.

Galanwe 6y ago
> The correct solution, according to our support contact, is that we wait a second after uploading to S3 I'm shocked that you got this answer. This is definitely not how you are supposed to operate. If you need to ensure the sequantiality of a write followed by a read on S3, the idiomatic way is to enable versioning on your bucket, issue a write, and provide the version ID to whoever need to read after that write. Not only will that transparently ensure that you will not read deprecated data, but it will even ensure that you actually read the result of that particular write, and not any consecutive write that could have happened in between. This pattern is very easy to implement for sequential operations in a single function, like: version = s3.put_object(...) data = s3.get_object(version_id=version, ...) I agree that when you deal with various processes it can become messy. In practice it never bothered me too much though. I prefer having explicit synchronization through versions, rather than having a blocking write waiting for all caches to be synchronized. Also, this should only be necessary if you write to an already existing object. New keys will always trigger a cache invalidation from my understanding.
- fxtentacle 6y ago
  Depending on your write throughput, versioning can be quite expensive. Also, we were already at a throughput where the s3 metadata nodes would sometimes go offline, so I'm not sure putting more work on them would have improved our overall situation.
  - Galanwe 6y ago
    Interesting, do you have more information on how you did monitors these S3 metadata nodes? Never heard of that. Why would an unversioned put be less costly than a versioned put? To me it should be pretty much the same. I would almost suspect unversioned puts are hacked around versioned ones. Out of curiosity what kind of workload did you perform? I have been moving terabytes of files (some very small 1kB files, some huge 150GB files) and never noticed a single blip of change in behavior from S3 (and certainly not anything going "off")
    - fxtentacle 6y ago
      > Interesting, do you have more information on how you did monitors these S3 metadata nodes? Never heard of that. Pretty simple, we started a bulk import, and then HTTPS requests to amazonaws.com started timing out... Later, support told us to spread out our requests to alleviate the load spike. What we did was process roughly 50mio 1MB JPEG images from the full-res S3 bucket into an derived-data S3 bucket.
dividuum 6y ago
> S3 requests to US buckets might be served by delayed replicas in other countries if the request comes from outside the US What? That makes no sense. Do you have a source for that? I thought the explicit choice of region when creating a bucket limits where data is located. Why would the give you geo-replication for free? Also: "Amazon S3 creates buckets in a Region you specify. To optimize latency, minimize costs, or address regulatory requirements, choose any AWS Region that is geographically close to you." - https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.... Also the consistency when creating objects is described here: https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyModel https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.... For new objects, you get read-after-write consistency and the example you gave contradicts that.
- Dunedan 6y ago
  > For new objects, you get read-after-write consistency and the example you gave contradicts that. Mind the documented caveat for this case: > The caveat is that if you make a HEAD or GET request to a key name before the object is created, then create the object shortly after that, a subsequent GET might not return the object due to eventual consistency.
  - dividuum 6y ago
    Indeed, although that seems irrelevant for the mentioned use case. After all, why would the client request an image before it's uploaded. It seems like a straight forward "server side upload->generate url and send it to client->client requests image" flow. It's perfectly covered by the consistency provided.
    - fxtentacle 6y ago
      That's what we did. Server side upload, link into websocket, client downloads image. It did work 99.99% of all requests.
      - 6y ago

Galanwe 6y ago

> The correct solution, according to our support contact, is that we wait a second after uploading to S3 I'm shocked that you got this answer. This is definitely not how you are supposed to operate. If you need to ensure the sequantiality of a write followed by a read on S3, the idiomatic way is to enable versioning on your bucket, issue a write, and provide the version ID to whoever need to read after that write. Not only will that transparently ensure that you will not read deprecated data, but it will even ensure that you actually read the result of that particular write, and not any consecutive write that could have happened in between. This pattern is very easy to implement for sequential operations in a single function, like: version = s3.put_object(...) data = s3.get_object(version_id=version, ...) I agree that when you deal with various processes it can become messy. In practice it never bothered me too much though. I prefer having explicit synchronization through versions, rather than having a blocking write waiting for all caches to be synchronized. Also, this should only be necessary if you write to an already existing object. New keys will always trigger a cache invalidation from my understanding.

fxtentacle 6y ago
Depending on your write throughput, versioning can be quite expensive. Also, we were already at a throughput where the s3 metadata nodes would sometimes go offline, so I'm not sure putting more work on them would have improved our overall situation.
- Galanwe 6y ago
  Interesting, do you have more information on how you did monitors these S3 metadata nodes? Never heard of that. Why would an unversioned put be less costly than a versioned put? To me it should be pretty much the same. I would almost suspect unversioned puts are hacked around versioned ones. Out of curiosity what kind of workload did you perform? I have been moving terabytes of files (some very small 1kB files, some huge 150GB files) and never noticed a single blip of change in behavior from S3 (and certainly not anything going "off")
  - fxtentacle 6y ago
    > Interesting, do you have more information on how you did monitors these S3 metadata nodes? Never heard of that. Pretty simple, we started a bulk import, and then HTTPS requests to amazonaws.com started timing out... Later, support told us to spread out our requests to alleviate the load spike. What we did was process roughly 50mio 1MB JPEG images from the full-res S3 bucket into an derived-data S3 bucket.

dividuum 6y ago

> S3 requests to US buckets might be served by delayed replicas in other countries if the request comes from outside the US What? That makes no sense. Do you have a source for that? I thought the explicit choice of region when creating a bucket limits where data is located. Why would the give you geo-replication for free? Also: "Amazon S3 creates buckets in a Region you specify. To optimize latency, minimize costs, or address regulatory requirements, choose any AWS Region that is geographically close to you." - https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.... Also the consistency when creating objects is described here: https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyModel https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.... For new objects, you get read-after-write consistency and the example you gave contradicts that.

Dunedan 6y ago
> For new objects, you get read-after-write consistency and the example you gave contradicts that. Mind the documented caveat for this case: > The caveat is that if you make a HEAD or GET request to a key name before the object is created, then create the object shortly after that, a subsequent GET might not return the object due to eventual consistency.
- dividuum 6y ago
  Indeed, although that seems irrelevant for the mentioned use case. After all, why would the client request an image before it's uploaded. It seems like a straight forward "server side upload->generate url and send it to client->client requests image" flow. It's perfectly covered by the consistency provided.
  - fxtentacle 6y ago
    That's what we did. Server side upload, link into websocket, client downloads image. It did work 99.99% of all requests.
    - 6y ago