3 ms·
Just serve and not transcode? Because it sounds like the Amazon Prime microservice was doing some heavy processing of uploaded video - breaking it into frames a
by zamnos 3y ago
Just serve and not transcode? Because it sounds like the Amazon Prime microservice was doing some heavy processing of uploaded video - breaking it into frames and applying some ML algorithms to them, and also the same with the audio. Sounds like even if you can fit in under the 10 min lamda execution timeout (assuming internal teams are limited to the same), that AWS Lambda/Step Functions isn't a generic job queue handler and is not infinitely parallel like AWS would want you to believe.
It's a bitl ike using an SQL database for streamed append-only logging. It'll work in test, with test amounts of data, but doesn't actually scale.
- emodendroket 3y agoWriting every frame to S3 and reading them back in other steps was probably a major factor.
- traspler 3y agoThere is not much of an option with step functions, outputs/inputs can be max 256KB so especially if you go into or out of a map/parallel state you can‘t do much more than pass references to the data and not the data itself.
- emodendroket 3y agoSure but I guess I mean the nature of that task probably made them the wrong architectural choice.