2 ms·

To be clear: the "blowing up" would occur when a client observed the new maven-metadata.xml file, but old ("does-not-exist") records for the newly uploaded arti

• HiJon89 6y ago
  For some context, S3 used to provide read-after-write consistency on new objects, but only if you didn't do a GET or HEAD to the key before it was created. So this access pattern is all good: PUT new-key 200 GET new-key 200 However, this access pattern would be unreliable: GET new-key 404 PUT new-key 200 GET new-key 200/404 This last GET could return a 200, or it could return a cached 404. Unfortunately, Maven uses this access pattern when publishing maven-metadata.xml files, because it needs to know whether it should update an existing maven-metadata.xml file or create one if it doesn't exist yet. So when publishing a new version, it does something like this: GET com.example/example-artifact/1.0-SNAPSHOT/maven-metadata.xml 404 PUT com.example/example-artifact/1.0-SNAPSHOT/maven-metadata.xml 200 And then downstream builds would try to resolve version 1.0-SNAPSHOT, and the first thing Maven does is fetch the maven-metadata.xml: GET com.example/example-artifact/1.0-SNAPSHOT/maven-metadata.xml 404 So when publishing a new version, you could get a cached 404 and fail loudly. However, when updating an existing maven-metadata.xml file you could silently read an old maven-metadata.xml, and end up using an out-of-date artifact (which is even more concerning). Here's what one of the maven-metadata.xml files looks like: https://oss.sonatype.org/content/repositories/snapshots/org/openapitools/openapi-generator-core/5.0.0-SNAPSHOT/maven-metadata.xml https://oss.sonatype.org/content/repositories/snapshots/org/... Because updating an existing object in S3 didn't have read-after-write consistency, we could have a publishing flow that looked like: GET com.example/example-artifact/1.0-SNAPSHOT/maven-metadata.xml 200 (v1) PUT com.example/example-artifact/1.0-SNAPSHOT/maven-metadata.xml 200 (v2) And then downstream builds would fetch the maven-metadata.xml: GET com.example/example-artifact/1.0-SNAPSHOT/maven-metadata.xml 200 (v1) So downstream builds could read a stale maven-metadata.xml file, which results in silently using an out-of-date artifact. We ended up just switching to GCS because it was relatively straight-forward and gave us the consistency guarantees we want.