4 ms·
Our image processing worker queues/servers write an S3 object and dispatch a follow up job. Currently we have to delay the next job (we use 5 seconds) otherwise
by PetahNZ 6y ago
Our image processing worker queues/servers write an S3 object and dispatch a follow up job. Currently we have to delay the next job (we use 5 seconds) otherwise then next job may start processing before the S3 object is available (it 404's if the next job is run straight away).
- ramraj07 6y agoThat sounds _very_ wrong, what type of throughput are we talking about here?
- riking 6y agoThe story wasn't wrong, until today. That's why people are celebrating.
- PetahNZ 6y agoUp to hundreds of images per second.
- ramraj07 6y agoOkay that makes sense, when we have very low throughput it looks like consistency wasn't an issue but at that rate (especially if it occurs in bursts) I can see how the prior S3 implementation could fail to scale fast enough
- mcqueenjordan 6y agoOut of curiosity, did you have retries on the GETs?
- PetahNZ 6y agoNo, we didn't try retrying (well actually if the job failed because of a 404 it would retry the whole job based on exponential backoff). When we encountered the error we simply added the delay as it was an easy config change to our SQS queues and doesn't meaningfully affect our use case.
- random5634 6y agoSame issues updating a file - you’d update on s3, then trigger processing and even though it was 50ms later - you’d get OLD version. I didn’t know gcp didn’t have this issue - would have been a big selling point