4 ms·
Good read, except the part where the author says there are no existing solutions for processing logs. There are quite a few robust scalable ones. syslog-ng, lo
by user5994461 6y ago
Good read, except the part where the author says there are no existing solutions for processing logs. There are quite a few robust scalable ones.
syslog-ng, logstash or fluentd on the host to collect and aggregate logs. (logstash/fluentd can parse text messages with regex and handle a hundred different things like s3/kafka/http but they are much more resource intensive).
kibana or graylog to centralize logs and search, the storage is elasticsearch.
A simple syslog-ng on the devices could probably do the job. Little known fact about syslog, it can reliably forward messages over TCP, logs are numbered, have retries and syslog-ng can do DNS load balancing.
- camdencheek 6y agoThe flexibility of Fluentd and the community around it are great, but ultimately the resource intensity proved to be too high and too unpredictable for our use case. We ended up building our own general purpose log parser in Go to support our hosted log monitoring platform. Though it's still somewhat young, performance is great (beats Fluent Bit in most of our benchmarks), and it's nearly as flexible as Fluentd in terms of configuration. If you want to check it out, we recently open-sourced it and are always looking for more feedback: https://github.com/observIQ/stanza https://github.com/observIQ/stanza
- d4rti 6y agoelasticsearch would not cope with that volume of data on the hardware described.
- user5994461 6y agoWell, the article doesn't mention the hardware. Only one machine for 5 TB a day which doesn't add up as far as I am concerned. If they were storing everything in S3, it's possible to do something similar with ElasticSearch for the same order of costs (maybe three times?), the money going to EBS storage instead. ElasticSearch allows to query and visualize logs, which plain S3 storage doesn't, it's worth a bit more IMO.
- Spivak 6y agoI think the author's math tracks. Our pipeline looks something like hosts -> rsyslog collector -> kafka -> custom crunching -> elastic which is a bit inefficient and we try to go hosts -> kafka when we can but a lot of stuff only supports rsyslog so it's there and simple enough. The rsyslog collector and our crunchers are teeeny tiny compared to the rest of the pipeline and can chew through up to a week of backlog in a few hours. The bottleneck is the network for us and if we upped the pipe to 10G we could probably get away with a single host.
- kakoni 6y agoWell there is clickhouse. And projects like this https://github.com/flant/loghouse https://github.com/flant/loghouse
- cbsmith 6y agoI missed where a volume of data was described that was outside elastic search's capacity. Seems unlikely that it "would not cope" regardless of configuration.
- Joker_vD 6y agoYeah, fluentd... I had to look into its in_stdin plugin, and I can tell you one thing: it simply doesn't handle regexes properly. In fact, none of its input plugins can correctly handle the situation where a multiline regex doesn't match the whole line, unless they do what in_tail did (split the lines yourself and disallow the multiline regexes). Edit: Sorry, I meant "Fluent bit". No idea how fluentd handle this scenario, but I was told it was too slow (being written in Ruby) so that's why the switch to fluentbit was made.
- user5994461 6y agoI've done multiline processing with logstash. It works but it's not necessarily efficient or simple. Depends a lot on the pattern and whether the multiline stack traces you want to extract turns out to be 700 lines long sometimes (yes, speaking from experience). I'd personally recommend to not bother with processing multiline output into a single message. Lots of trouble for no benefits. It's just a stream of lines at the end of the day, it will look the same in tail and kibana.
- taneliv 6y agoLast time I tried, fluent-bit did not support IPv6. Maybe it does now? (Maybe it does not: https://github.com/fluent/fluent-bit/issues/2056 https://github.com/fluent/fluent-bit/issues/2056 has been opened recently). Anyway, compared to the solution in the article, it seemed way too fanciful. Fluentd would not just be slow, but also run out of memory. I am no ruby-head, but a then-colleague of mine helped configure it. It still ran out of memory. I had no patience for a log system that did not work out of the box on one computer basically just logging failed ssh login attempts.
- cbsmith 6y agoThere's also good ol' dirty syslog.