3 ms·

The UI for Cloudwatch Logs completely crumbles if you use it for Lambda functions, especially if you have a lot of executions. We run ETL pipelines and have la

The UI for Cloudwatch Logs completely crumbles if you use it for Lambda functions, especially if you have a lot of executions. We run ETL pipelines and have lambda as a component of that, maybe 250k-500k invocations a day. If you want to search through the logs for a particular invocation at a particular time (e.g. when an error happened) the UI just cannot handle it. I think part of the reason is Lambda creates a new "log stream" every time a new container is created, which I'm guessing is how they shard the logs. So you end up with hundreds/thousands of log streams within your lambda log group and the backend has to search them to find the particular search pattern you are requesting. Most of the time we have to export the logs to S3, download them and do some manual grepping.

zedpm 8y ago
I'd recommend against searching via the web UI. I use awslogs[0] daily and find it an effective way to search and filter logs. [0] https://github.com/jorgebastida/awslogs https://github.com/jorgebastida/awslogs
scarface74 8y ago
For Python I use WatchTower to specify the name of the log group/log stream. For C#, I use Serilog to specify the log group and stream. Another benefit of using Serilog, is that you get structured JSON logging with name/value pairs that you can use the JSON filters in cloudwatch abd you can send them to a NoSQL data store like Mongo and ElasticSearch and do more advanced queries than just regex expressions.
chair6 8y ago
Yeah, don't use the web UI. An alternative to awslogs is saw, https://github.com/TylerBrock/saw https://github.com/TylerBrock/saw. Me likey..
niklasrde 8y ago
We move our Logstreams to our ElastiCache cluster to make them somewhat usable - using another lambda which receives subscription filters that are attached to a log group. The processor also periodically scans new log groups under the /aws/lambda/ prefix, and creates subscriptions for them if they don't exist already. Speaking about, this may be a good candidate for a more in-depth blogpost
- shshhdhs 8y ago
  Yes, I’d love to see the blog post!

zedpm 8y ago

I'd recommend against searching via the web UI. I use awslogs[0] daily and find it an effective way to search and filter logs. [0] https://github.com/jorgebastida/awslogs https://github.com/jorgebastida/awslogs

scarface74 8y ago

For Python I use WatchTower to specify the name of the log group/log stream. For C#, I use Serilog to specify the log group and stream. Another benefit of using Serilog, is that you get structured JSON logging with name/value pairs that you can use the JSON filters in cloudwatch abd you can send them to a NoSQL data store like Mongo and ElasticSearch and do more advanced queries than just regex expressions.

chair6 8y ago

Yeah, don't use the web UI. An alternative to awslogs is saw, https://github.com/TylerBrock/saw https://github.com/TylerBrock/saw. Me likey..

niklasrde 8y ago

We move our Logstreams to our ElastiCache cluster to make them somewhat usable - using another lambda which receives subscription filters that are attached to a log group. The processor also periodically scans new log groups under the /aws/lambda/ prefix, and creates subscriptions for them if they don't exist already. Speaking about, this may be a good candidate for a more in-depth blogpost

shshhdhs 8y ago
Yes, I’d love to see the blog post!