3 ms·
We have that issue at work, though I solved it by including the sqlite database within the container image that we use. We then deploy the new container image (
by moduspol 2y ago
We have that issue at work, though I solved it by including the sqlite database within the container image that we use. We then deploy the new container image (with the same code as before, but with a revised database file) at most every fifteen minutes.
This gives you an atomic point at which you are 100% confident all instances are using the same database, and by provisioning concurrency, you can also avoid a "thundering herd" of instances all fetching from the file on S3 at startup (which can otherwise lead to throttling).
Of course, that's only feasible if it's acceptable that your data can be stale for some number of minutes, but if you're caching the way you are, and periodically checking S3 for an updated database, it probably is.
- efitz 2y agoI actually versioned my database file - I had a small metadata table with version number and creation time. Then in the output from each of my other functions, I included the database version number. So all my output could be subsequently normalized by re-running the same input versus an arbitrary version of the database file.
- bob1029 2y agoHave you looked at the user_version pragma? I've been able to avoid use of special metadata tables with this.
- jrochkind1 2y ago> "thundering herd" of instances all fetching from the file on S3 at startup (which can otherwise lead to throttling). Have any "thundering herd" problems with S3, including throttling, actually been seen? I think S3 is advertised to have no concurrent connection limit, and support up to at least 5,500 GETs per second (per "prefix", which I'm confused about what that means exactly in practice). I don't think S3 ever applies intentional throttling, although of course if you exceed it's capacity to deliver data you will see "natural" throttling. Do you have a fleet big enough that you might be exceeding those limits, or have people experienced problems even well under these limits, or is it just precautionary?
- finnh 2y agoYes, I've been throttled many times by S3. My largest database is ingesting ~5PB/day and that turns into a lot of files in S3. At one point we changed our S3 key scheme to not have hashes up front, which unlocked some simplicity in control plane operations like deleting old files; we did this on the strength of the announcement from AWS that you no longer needed to get clever with prefixes. This was incorrect at our scale, and we had to switch back.
- moduspol 2y agoSorry--the throttling was at the AWS Lambda layer, not S3. We were being throttled because we'd deploy a new container image and suddenly thousands of new containers are all simultaneously trying to pull the database file from S3. We aim to return a response in the single digit milliseconds and sometimes get tens of thousands of requests per second, so even if it only takes a second or two to fetch that file from S3, the request isn't getting served while it's happening, and new requests are coming in. You very quickly hit your Lambda concurrency limit and get throttled just waiting for your instances to fetch the file, even though logically you're doing exactly what you planned to. By having the file exist already in the container image, you lean on AWS's existing tools for a phased rollout to replace portions of your deployment at a time, and every one is responding in single digit milliseconds from its very first request. EDIT: The same technique could be applied for other container management systems, but for stuff like Kubernetes or ECS, it might be simpler to use OP's method with a readiness check that only returns true if you fetched the file successfully. And maybe some other logic to do something if your file gets too stale, or you're failing to fetch updates for some reason.