6 ms·
Why not go for App -> AWS Firehose -> S3 architecture. Firehose will automatically take care of batching logs and putting them together. Also it can create part
by apexkid 9y ago
Why not go for App -> AWS Firehose -> S3 architecture. Firehose will automatically take care of batching logs and putting them together. Also it can create partitions. No need to use Lambda at all.
- guitarbill 9y agoPossibly because the app is running on Lambda? Cloudwatch logging is not great for Lambdas.
- szalonypiekarz 9y agoIn that case, why not Lambda -> Cloudwatch Logs -> Firehose -> S3 ? No bespoke logging solution is needed.
- joemaller1 9y agoDon’t assume the author made an informed but poor choice, they likely made something work with the tools they know or found. It’s really difficult to keep up with all the various AWS offerings if your not part of a team that’s immersed in it. Constructively suggesting alternates is welcome, I’ve got some new tools to read up on this morning.
- jcims 9y agoWhy is that?
- throwaway2016a 9y ago> Cloudwatch logging is not great for Lambdas. I run some Lambda systems at scale and CloudWatch Logs has worked excellent for me. It was completely painless and CloudWatch logs has a lot of tooling that comes with it (backups, streaming, indexing with Elastic Search, etc). What part of it has not worked well for you?
- guitarbill 9y agoMainly searching and exporting. Exporting logs to S3 manually for debugging/post-mortem is harder than it needs to be - maybe this isn't a huge use-case, but we like to attach logs to tickets. And searching is very limited. Yes, you can stream them to ElasticSearch => LogStash/Kibana, etc but that's extra steps, and I can do similar things completely without CloudWatch Logs, e.g. Sentry and SumoLogic integrate fine with Lambdas also. I'm not hating on CloudWatch Logs, and alarms are good. It's just not a fully featured logging solution, and it sounds like you agree with that?
- throwaway2016a 9y agoI agree you need more tools around it to make it a full logging solution. Doing data visualization is more or less impossible without extra tools. But I've overall liked it. Search is powerful enough to do something like say, "get me all messages with this trace GUID" and S3 exports can be automated[1] and you could probably use the "task-name" to associate with your ticketing system. [1] http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/S3ExportTasks.html http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/S3Ex...
- ellimilial 9y agoThank you for the feedback. I was considering Kinesis but my use case (multiple small apps not running on AWS sending medium-sized messages a few times per user session) values simple components and keeping the long-term maintenance effort to the minimum.
- weego 9y agoI'm not sure what your understanding of Kinesis Firehose is but, in this situation it would literally remove some moving parts, it would not add complexity. Also potentially cheaper than SQS anyway.
- ellimilial 9y agoYup, makes sense, looks like it's a decomposition of it. Might have been the talk of installing agents, sharding etc that put me off track.