5 ms·
We do this all the time already when we write various interface services to access a common data store. Each service provides whatever interface is appropriate
by blunte 6y ago
We do this all the time already when we write various interface services to access a common data store. Each service provides whatever interface is appropriate and performs any desired transformations.
It's not clear to me what the real benefits of Object Lambda provides. Indeed it could move your transformation code closer to its data, but the cost of doing that is the management and complexity of the lambdas. I am curious what a best use case example would be for this.
- chrisandchris 6y agoFor my use-cases, this would be a huge benefit: > Resizing and watermarking images on the fly using caller-specific details, such as the user who requested the object. Now we need to either do rescale directly after storing or need to use another service (on EC2) to do the resize. With Object Lamda, we could do it on-the-fly as the user calls the image. This is operational a lot easier.
- ec109685 6y agoEven without this feature, you could stand up a lambda and do the resize on the fly. Advantage here is that clients can just continue to use their existing S3 interface yet your lambda is still invoked.
- captn3m0 6y agoDoes this get triggered over the S3-over-HTTPS (the one used for websites) served objects? (instead of S3 APIs)
- deleted 6y ago[deleted]
- matt-attack 6y agoHow are parameters passed? That is how does Lambda get its resize parameters from just an S3 GET? Are they query params?
- jon-wood 6y agoIn the example use case for that they suggested baking the size into the filename, like image_500_240.jpg
- skynet-9000 6y agoYou might want to build some rate limiting into your lambda as well!
- ralusek 6y agoAt the moment this is currently handled with lambda @ edge with Cloudfront. The benefit with Cloudfront over S3 lambda is that the image is automatically cached after the resize is computed.
- mwarkentin 6y agoYou could cache the image with s3 lambda too, no? Edit: I guess you meant that cloudfront caches it for you automatically.
- ralusek 6y agoExactly, yup. Whatever it returns is automatically cached at the key.
- joshuanapoli 6y agoThe benefit of S3 Object Lambda is that your function is adapted to the S3 API. If you already have an application integrated with S3, then you can use Object Lambda to provide some new behavior to the data without changing the application.
- barbazoo 6y agoYour example is way better than what AWS has on their page (marketing, analytics) which made me think immediately: you shouldn't use a file store for that but a database.
- lambda_obrien 6y agoS3 is basically a database. You can store everything there and interact with it via SQL with Athena, for example. I could see utilizing all of Amazon's "infinitely scalable" services to build very simple and powerful software really easily.
- barbazoo 6y agoI think Athena has to get a lot less expensive for that to become true.
- kryptk 6y agoFor small numbers of large queries, Athena has been an incredible technology stack .. I pay pennies a month to run my analytics jobs on it (aggregates across a few hundred million rows, joins with tens of millions of rows into hundreds of thousands of rows). I start my data as .csv.gz but the first step is a CTAS to extract columns and convert to compressed parquet. This step basically costs the most but gives a 10x data size reduction to downstream steps. Athena does not work at all if you perform large numbers of small indexed read queries, definitely use a traditional database for that.
- lambda_obrien 6y agoYes, it's great if you setup the data structures properly, we pay a few bucks a month to run a lot of infra on it to query telemetry data over a few hours to a few days at a time. I think an equivalent cost in just a database server instance to run the same load would be at least 25 to 50 a month. You're not really locked into athena, either, it just takes standard sql, so you could easily transition to a dedicated box later.
- SoapSeller 6y agoIf S3 is database - Object Lambda is stored procedure? I guess that for some systems that are heavily S3-based it will help adding new features without needing to rewrite the interface or duplicate the functionality. It's nice to know that AWS achieved such lock-in with some customers.
- kbumsik 6y agoOur team write complex code for Spark to sanitize/anonymize personal data before processing. With this we might simplify the pipeline because we can keep the same S3 interface.
- mxz3000 6y agoThis. This allows sensitive data to never leave the storage layer.
- lettergram 6y ago> the cost of doing that is the management and complexity of the lambdas The lock-in mechanism. Imo this isn’t more functional than other approaches, it’s a way to ensure you don’t switch services.
- mr_toad 6y ago> It's not clear to me what the real benefits of Object Lambda provides. It’s basically doing the same thing as an SQL view. It just hides and abstracts away some complexity.