4 ms·

Has anyone replicated a sqllite db to Cloudfront cache and used that as an input to their Lambda@Edge ? I have been thinking about this for a while as it would

Has anyone replicated a sqllite db to Cloudfront cache and used that as an input to their Lambda@Edge ? I have been thinking about this for a while as it would avoid having cloudfront making a origin call to the backend server for db reads on data that does not change frequently. I am trying to figure out why this is a bad idea .

snissn 6y ago
do it! Years ago my team messed around with using lambdas to pull/push data from sqlite into s3 and it worked pretty well for certain regime sizes and workloads! Then we partitioned the data to filter out the slow zones into more robust/long running things like a job queue and postgres
jedberg 6y ago
Why not just include the sqlite DB as part of your lambda deployment package?
- diroussel 6y ago
  As long as you can fit it in your 1MB zip file, for viewer-request events. For origin-requests you get 50MB. See https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/cloudfront-limits.html#limits-lambda-at-edge https://docs.aws.amazon.com/AmazonCloudFront/latest/Develope...
  - deleted 6y ago
    [deleted]
  - jedberg 6y ago
    You only have five seconds to process the entire request. If your lambda has to make a request to download a sqlite database every time, and the process it, it probably needs to be small enough to fit into 1MB anyway.
    - freemium413 6y ago
      AWS Lambdas can run up to 15 minutes actually
      - shree_ch 6y ago
        Lambda functions running @Edge have different limits compared to vanilla AWS Lambda. You get 30 seconds to respond if it's on a cache miss (Origin Request/Response), or 5 seconds if it's on a cache hit (Viewer Request/Response): https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-requirements-limits.html https://docs.aws.amazon.com/AmazonCloudFront/latest/Develope...
  - cobookman 6y ago
    Lambda (at least the non-@edge version) comes with 512MiB of /tmp directory storage And "The same Lambda execution environment may be reused by multiple Lambda invocations to optimize performance. The /tmp area is preserved for the lifetime of the execution environment and provides a transient cache for data between invocations. Each time a new execution environment is created, this area is deleted." https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-... https://aws.amazon.com/blogs/compute/choosing-between-aws-lambda-data-storage-options-in-web-apps/ https://aws.amazon.com/blogs/compute/choosing-between-aws-la...
- recuter 6y ago
  I'm playing around with just that. Regular Lambda (not edge) are just container images up to 10 GB in size. There's also the layers thing. A function can use up to five layers at a time. The total unzipped size of the function and all layers can't exceed the unzipped deployment package size limit of 250 MB. Can update separately from the container, almost as fast as a regular fs. And finally it has access to EFS which is faster than S3.
ignoramous 6y ago
We replicate sharded, versioned, csv-files on-demand from S3 to Cloudflare edge for code run by workers.dev. Cache invalidation is tricky. Avoid it by ttl-expiring caches as frequently as budget allows. We, instead, employ a (probably over-engineered) hand-rolled set-reconciliation algorithm to match our read/write patterns.

snissn 6y ago

do it! Years ago my team messed around with using lambdas to pull/push data from sqlite into s3 and it worked pretty well for certain regime sizes and workloads! Then we partitioned the data to filter out the slow zones into more robust/long running things like a job queue and postgres

jedberg 6y ago

Why not just include the sqlite DB as part of your lambda deployment package?

diroussel 6y ago
As long as you can fit it in your 1MB zip file, for viewer-request events. For origin-requests you get 50MB. See https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/cloudfront-limits.html#limits-lambda-at-edge https://docs.aws.amazon.com/AmazonCloudFront/latest/Develope...
- deleted 6y ago
  [deleted]
- jedberg 6y ago
  You only have five seconds to process the entire request. If your lambda has to make a request to download a sqlite database every time, and the process it, it probably needs to be small enough to fit into 1MB anyway.
  - freemium413 6y ago
    AWS Lambdas can run up to 15 minutes actually
    - shree_ch 6y ago
      Lambda functions running @Edge have different limits compared to vanilla AWS Lambda. You get 30 seconds to respond if it's on a cache miss (Origin Request/Response), or 5 seconds if it's on a cache hit (Viewer Request/Response): https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-requirements-limits.html https://docs.aws.amazon.com/AmazonCloudFront/latest/Develope...
- cobookman 6y ago
  Lambda (at least the non-@edge version) comes with 512MiB of /tmp directory storage And "The same Lambda execution environment may be reused by multiple Lambda invocations to optimize performance. The /tmp area is preserved for the lifetime of the execution environment and provides a transient cache for data between invocations. Each time a new execution environment is created, this area is deleted." https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-... https://aws.amazon.com/blogs/compute/choosing-between-aws-lambda-data-storage-options-in-web-apps/ https://aws.amazon.com/blogs/compute/choosing-between-aws-la...
recuter 6y ago
I'm playing around with just that. Regular Lambda (not edge) are just container images up to 10 GB in size. There's also the layers thing. A function can use up to five layers at a time. The total unzipped size of the function and all layers can't exceed the unzipped deployment package size limit of 250 MB. Can update separately from the container, almost as fast as a regular fs. And finally it has access to EFS which is faster than S3.

ignoramous 6y ago

We replicate sharded, versioned, csv-files on-demand from S3 to Cloudflare edge for code run by workers.dev. Cache invalidation is tricky. Avoid it by ttl-expiring caches as frequently as budget allows. We, instead, employ a (probably over-engineered) hand-rolled set-reconciliation algorithm to match our read/write patterns.