4 ms·

> 20-30 Lambda functions for different services Yes. This is the basis of privilege separation and differential rollouts. If you collapse all this down into

> 20-30 Lambda functions for different services Yes. This is the basis of privilege separation and differential rollouts. If you collapse all this down into a single server or even lambda you lose that. Once your service sees load you will want this badly. > SQS and various background jobs backed by Lambda Yes. This is the basis of serverless. The failure of one server is no longer a material concern to your operation. Well done you. > Logs scattered across CloudWatch Okay. I can't lie. CloudWatch is dogturds. There is no part of the service that is redeemable. I created a DyanmoDB table and created a library which puts log lines collected into "task records" into the table paritioned by lambda name and sorted by record time. Each lambda can configure the logging environment or use default which include a log entry expiration time. Then I created a command line utility which can query and or "tail" this table. This work took me 3 days. It's paid off 1000x fold since I did it. You do sometimes have to roll your own out here. CloudWatch is strictly about logging cold start times now. > Could this have been simplified to a single NodeJS container or Python Flask/FastAPI app with Redis for background tasks? Absolutely. Could this have been simplified into something far more fragile than what is described? Absolutely. Why you'd want this is entirely beyond me.

throwawaythekey 2y ago
I agree that cloudwatch is dogturds, but want to dive deeper for illustrative purposes: Your dynamodb solution isn't foolproof. It has throughput limited to the partition granularity -> in your case the lambda name. It's also relatively expensive and fairly slow to query in bulk (DDB is designed for OLTP). I don't have direct experience here, but I expect slapping grafana on top of any disk basked source is likely to be cheaper, faster, and have better ergonomics. Once your logging is too much for a disk to handle (this will be later than you would've outgrown ddb, but before you would've outgrown cloudwatch) then you can bring something fancy in.
- akira2501 2y ago
  > has throughput limited The event volume is not particularly large as we tend to process things in batch and rarely on the edge of an event. I also wouldn't, for example, log API requests using this mechanism. We're nowhere near this being an issue as 20-30 lambdas is not a particular problem for us. Choose a good naming convention and build your own deployment infrastructure and it's no sweat. > relatively expensive Large object compression and/or offload to s3 is baked into our dynamodb interface library. Not that this matters as almost all log records end up being less than 4kb anyways. > slow to query in bulk Which is why time is part of the key. You're not often looking back more than an hour. There's bulk export back onto campus servers if you wanted that anyways. TTL is default 1 day. Running a "tail" is absurdly cheap, much cheaper than CloudWatch's laughable rate for their similar feature, a miss is 1/2 a read unit, and a hit is almost never more than 2. > slapping grafana I didn't need "observability." I need current state and recent deltas. This is particularly true when any changes are made. Otherwise my logs are pure annoyance and don't generally provide value. We optimized for the exceptionally narrow case we felt the cloud underserved in and left it at that.
senko 2y ago
> Once your service sees load you will want this badly. > [...] > The failure of one server is no longer a material concern to your operation. Elsewhere in thread you say: > The event volume is not particularly large as we tend to process things in batch and rarely on the edge of an event. So the service is not actually under load, and it runs in batches so (temporary) failure is not actually a concern. > This work took me 3 days. It's paid off 1000x fold since I did it. Since Lambda was introduced less than 10 years ago, what you're saying here is that it'd be full time job for you for the past 10 years to maintain this (3000 days instead of three) if you have not gone the serverless way, which I find doubtful. > Could this have been simplified into something far more fragile than what is described? Absolutely. Considering the hyperboles in the rest of your comment, this sounds more like snark than a considered opinion.
marcosdumay 2y ago
> privilege separation and differential rollouts What relation does any of those have with load? (And also, why are people so kin on doing privilege separation by giving full privilege to a 3rd party and asking it to limit what each piece of code can do?)

throwawaythekey 2y ago

I agree that cloudwatch is dogturds, but want to dive deeper for illustrative purposes: Your dynamodb solution isn't foolproof. It has throughput limited to the partition granularity -> in your case the lambda name. It's also relatively expensive and fairly slow to query in bulk (DDB is designed for OLTP). I don't have direct experience here, but I expect slapping grafana on top of any disk basked source is likely to be cheaper, faster, and have better ergonomics. Once your logging is too much for a disk to handle (this will be later than you would've outgrown ddb, but before you would've outgrown cloudwatch) then you can bring something fancy in.

akira2501 2y ago
> has throughput limited The event volume is not particularly large as we tend to process things in batch and rarely on the edge of an event. I also wouldn't, for example, log API requests using this mechanism. We're nowhere near this being an issue as 20-30 lambdas is not a particular problem for us. Choose a good naming convention and build your own deployment infrastructure and it's no sweat. > relatively expensive Large object compression and/or offload to s3 is baked into our dynamodb interface library. Not that this matters as almost all log records end up being less than 4kb anyways. > slow to query in bulk Which is why time is part of the key. You're not often looking back more than an hour. There's bulk export back onto campus servers if you wanted that anyways. TTL is default 1 day. Running a "tail" is absurdly cheap, much cheaper than CloudWatch's laughable rate for their similar feature, a miss is 1/2 a read unit, and a hit is almost never more than 2. > slapping grafana I didn't need "observability." I need current state and recent deltas. This is particularly true when any changes are made. Otherwise my logs are pure annoyance and don't generally provide value. We optimized for the exceptionally narrow case we felt the cloud underserved in and left it at that.

senko 2y ago

> Once your service sees load you will want this badly. > [...] > The failure of one server is no longer a material concern to your operation. Elsewhere in thread you say: > The event volume is not particularly large as we tend to process things in batch and rarely on the edge of an event. So the service is not actually under load, and it runs in batches so (temporary) failure is not actually a concern. > This work took me 3 days. It's paid off 1000x fold since I did it. Since Lambda was introduced less than 10 years ago, what you're saying here is that it'd be full time job for you for the past 10 years to maintain this (3000 days instead of three) if you have not gone the serverless way, which I find doubtful. > Could this have been simplified into something far more fragile than what is described? Absolutely. Considering the hyperboles in the rest of your comment, this sounds more like snark than a considered opinion.

marcosdumay 2y ago

> privilege separation and differential rollouts What relation does any of those have with load? (And also, why are people so kin on doing privilege separation by giving full privilege to a 3rd party and asking it to limit what each piece of code can do?)