5 ms·

The article discusses the specific use case of serverless computing, e.g. AWS Lambda, and how a central database doesn't always work well with apps constructed

by efitz 2y ago

The article discusses the specific use case of serverless computing, e.g. AWS Lambda, and how a central database doesn't always work well with apps constructed in a serverless fashion. I was immediately interested in this post because 6-7 years ago I worked on this very problem- I needed to ingest a set of complex hierarchical files that could change at any time, and I needed to "query" them to extract particular information. FaaS is expensive for computationally expensive tasks, and it also didn't make sense to load big XML files and parse them every time I needed to do a lookup in any instance of my Lambda function. My solution was to have a central function on a timer that read and parsed the files every couple of minutes, loaded the data into a SQLite database, indexed it, and put the file in S3. Now my functions just downloaded the file from S3, if it was newer than the local copy or on a cold start, and did the lookup. Blindingly fast and no duplication of effort. One of the things that is not immediately obvious from Lambda is that it has a local /tmp directory that you can read from and write to. Also the Python runtime includes SQLite; no need to upload code besides your function. I'm excited that work is going on that might make such solutions even faster; I think it's a very useful pattern for distributed computing.

deleted 2y ago
[deleted]
rmbyrro 2y ago
> Now my functions just downloaded the file from S3, if it was newer than the local copy if you have strong consistency requirements, this doesn't work. synchronizing clocks reliably between different servers is surprisingly hard. you might end up working with stale data. might work for use cases that can accept eventual consistency.
- 66yatman 2y ago
  This shouldn't depend on clocks, just tracking Etag is more consistency proof.
- Spivak 2y ago
  I don't really know if that matters for this use case. Just by the very nature of source_data -> processing -> dest_data taking nonzero time anything consuming dest_data must already be tolerant of some amount of lag. And how it's coded guarantees you can never observe dest_data going new -> old -> new.
- jedberg 2y ago
  One of the announcements from AWS this year at Re:invent is that they now can guarantee that the instances clocks are synced within microseconds of each other. Close enough that you can rely on it for distributed timekeeping.
- Dylan16807 2y ago
  If you have strong consistency requirements, then it doesn't work by the very nature of making multiple copies of the database. Even if the clocks are perfect. (Though the clocks are probably close enough that it doesn't matter.)
- efitz 2y ago
  I versioned the databases and my use case didn’t require strong consistency.
- iskela 2y ago
  Wouldnt e-tag version numbers also work? Or just havkng .jsom with version metadata next to the db blob? No need to sync clocks. Just GET the small db-ver.json and compare version details?
viccis 2y ago
> One of the things that is not immediately obvious from Lambda is that it has a local /tmp directory that you can read from and write to. The other big thing a lot of people don't know about Python on Lambda is that your global scope is also persisted for that execution context's lifetime like /tmp is. I ran into issues at one point with Lambdas that processed a high volume of data getting intermittent errors connecting to S3. An AWS engineer told me to cache my boto3 stuff (session, client, resources, etc.) in the global namespace, and that solved the problem overnight.
moduspol 2y ago
We have that issue at work, though I solved it by including the sqlite database within the container image that we use. We then deploy the new container image (with the same code as before, but with a revised database file) at most every fifteen minutes. This gives you an atomic point at which you are 100% confident all instances are using the same database, and by provisioning concurrency, you can also avoid a "thundering herd" of instances all fetching from the file on S3 at startup (which can otherwise lead to throttling). Of course, that's only feasible if it's acceptable that your data can be stale for some number of minutes, but if you're caching the way you are, and periodically checking S3 for an updated database, it probably is.
- efitz 2y ago
  I actually versioned my database file - I had a small metadata table with version number and creation time. Then in the output from each of my other functions, I included the database version number. So all my output could be subsequently normalized by re-running the same input versus an arbitrary version of the database file.
  - bob1029 2y ago
    Have you looked at the user_version pragma? I've been able to avoid use of special metadata tables with this.
- jrochkind1 2y ago
  > "thundering herd" of instances all fetching from the file on S3 at startup (which can otherwise lead to throttling). Have any "thundering herd" problems with S3, including throttling, actually been seen? I think S3 is advertised to have no concurrent connection limit, and support up to at least 5,500 GETs per second (per "prefix", which I'm confused about what that means exactly in practice). I don't think S3 ever applies intentional throttling, although of course if you exceed it's capacity to deliver data you will see "natural" throttling. Do you have a fleet big enough that you might be exceeding those limits, or have people experienced problems even well under these limits, or is it just precautionary?
up2isomorphism 2y ago
It is so painful to see so many people make wrong use of S3.
- lcnPylGDnU4H9OF 2y ago
  Is there a list of correct uses of s3 we can all follow?
  - akira2501 2y ago
    Yep. Right here. [0]. Generally people ignore the per PUT and GET pricing on S3 along with the higher latency since it's a "global" service. If your objects are small then you're almost always benefited from using DynamoDB as the GET pricing and latency are far more favorable, as long as you don't mind the region dependency or the multi region setup. [0]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimi...
  - compootr 2y ago
    storing massive amounts of unstructured data
avinassh 2y ago
> My solution was to have a central function on a timer that read and parsed the files every couple of minutes, loaded the data into a SQLite database, indexed it, and put the file in S3. I really love using S3 as a storage backend!

by efitz 2y ago

deleted 2y ago

[deleted]

rmbyrro 2y ago

> Now my functions just downloaded the file from S3, if it was newer than the local copy if you have strong consistency requirements, this doesn't work. synchronizing clocks reliably between different servers is surprisingly hard. you might end up working with stale data. might work for use cases that can accept eventual consistency.

66yatman 2y ago
This shouldn't depend on clocks, just tracking Etag is more consistency proof.
Spivak 2y ago
I don't really know if that matters for this use case. Just by the very nature of source_data -> processing -> dest_data taking nonzero time anything consuming dest_data must already be tolerant of some amount of lag. And how it's coded guarantees you can never observe dest_data going new -> old -> new.
jedberg 2y ago
One of the announcements from AWS this year at Re:invent is that they now can guarantee that the instances clocks are synced within microseconds of each other. Close enough that you can rely on it for distributed timekeeping.
Dylan16807 2y ago
If you have strong consistency requirements, then it doesn't work by the very nature of making multiple copies of the database. Even if the clocks are perfect. (Though the clocks are probably close enough that it doesn't matter.)
efitz 2y ago
I versioned the databases and my use case didn’t require strong consistency.
iskela 2y ago
Wouldnt e-tag version numbers also work? Or just havkng .jsom with version metadata next to the db blob? No need to sync clocks. Just GET the small db-ver.json and compare version details?

viccis 2y ago

> One of the things that is not immediately obvious from Lambda is that it has a local /tmp directory that you can read from and write to. The other big thing a lot of people don't know about Python on Lambda is that your global scope is also persisted for that execution context's lifetime like /tmp is. I ran into issues at one point with Lambdas that processed a high volume of data getting intermittent errors connecting to S3. An AWS engineer told me to cache my boto3 stuff (session, client, resources, etc.) in the global namespace, and that solved the problem overnight.

moduspol 2y ago

We have that issue at work, though I solved it by including the sqlite database within the container image that we use. We then deploy the new container image (with the same code as before, but with a revised database file) at most every fifteen minutes. This gives you an atomic point at which you are 100% confident all instances are using the same database, and by provisioning concurrency, you can also avoid a "thundering herd" of instances all fetching from the file on S3 at startup (which can otherwise lead to throttling). Of course, that's only feasible if it's acceptable that your data can be stale for some number of minutes, but if you're caching the way you are, and periodically checking S3 for an updated database, it probably is.

efitz 2y ago
I actually versioned my database file - I had a small metadata table with version number and creation time. Then in the output from each of my other functions, I included the database version number. So all my output could be subsequently normalized by re-running the same input versus an arbitrary version of the database file.
- bob1029 2y ago
  Have you looked at the user_version pragma? I've been able to avoid use of special metadata tables with this.
jrochkind1 2y ago
> "thundering herd" of instances all fetching from the file on S3 at startup (which can otherwise lead to throttling). Have any "thundering herd" problems with S3, including throttling, actually been seen? I think S3 is advertised to have no concurrent connection limit, and support up to at least 5,500 GETs per second (per "prefix", which I'm confused about what that means exactly in practice). I don't think S3 ever applies intentional throttling, although of course if you exceed it's capacity to deliver data you will see "natural" throttling. Do you have a fleet big enough that you might be exceeding those limits, or have people experienced problems even well under these limits, or is it just precautionary?

up2isomorphism 2y ago

It is so painful to see so many people make wrong use of S3.

lcnPylGDnU4H9OF 2y ago
Is there a list of correct uses of s3 we can all follow?
- akira2501 2y ago
  Yep. Right here. [0]. Generally people ignore the per PUT and GET pricing on S3 along with the higher latency since it's a "global" service. If your objects are small then you're almost always benefited from using DynamoDB as the GET pricing and latency are far more favorable, as long as you don't mind the region dependency or the multi region setup. [0]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimi...
- compootr 2y ago
  storing massive amounts of unstructured data

avinassh 2y ago

> My solution was to have a central function on a timer that read and parsed the files every couple of minutes, loaded the data into a SQLite database, indexed it, and put the file in S3. I really love using S3 as a storage backend!