5 ms·

ElasticSearch was never really meant for log storage anyway. It’s a full text engine, and just happened to work reasonably well for that purpose at lower volume

ElasticSearch was never really meant for log storage anyway. It’s a full text engine, and just happened to work reasonably well for that purpose at lower volume. ELK ran with it in an attempt to go after Splunk, but it is phenomenally difficult to scale an indexing pipeline like ELK to high volume. There are far better ways to handle log analysis, particularly when your primary query is counting things that happen over T instead of finding log entries that match a query (which it always is) — streaming analysis is a much better fit than indexing, just lesser known.

amenod 9y ago
What are the better alternatives for handling log analysis? (I didn't downvote you btw)
- jsmthrowaway 9y ago
  “Dataflow” and the open source ecosystem in that neighborhood (Flink, Spark, Beam, Kafka, that family of stuff) is a much more powerful way to look at logs in real time, rather than indexing them into storage and then querying. There just isn’t something off the shelf as easy to deploy as ElasticSearch with that architecture, that I’m aware of. (There should be!) When you jump the mental gap of events being identical to logs, then start looking at fun architectures like event sourcing, you start realizing streaming analysis is a pretty powerful way of thinking. I’ve extracted insight from millions of log records per second on a single node with a similar setup, in real time, with much room to scale. The key to scaling log analysis is to get past textual parsing, which means using something structured, which usually negates the reason you were using ElasticSearch in the first place. Google’s first Dataflow talk from I/O and the paper should give you an idea of what log analysis can look like when you get past the full text indexing mindset. Note that there’s nothing wrong with ELK, but you will run into scaling difficulty far sooner than you’d expect trying to index every log event as it comes. It’s also bad to discover this when you get slashdotted, and your ES cluster whimpers trying to keep up. One thing streaming often gets you is latency in that situation instead of death, since you’re probably building atop a distributed log (and falling behind is less bad than falling over). The key here is: are you looking for specific log events or are you counting? You’re almost always counting. Why index individual events, then?
  - yehosef 9y ago
    I don't think this is "always" true. The power of having the data in records is that you can trivially pivot and slice and dice the data in different ways. When it's aggregated to a count - I don't have that ability. When trying to debug something that's happening, I find it far easier to have the entire records. As for scaling, it scales very well (you can read through the elastic blog/use cases to see plenty of examples.) That's not to say there aren't levels of scaling it won't handle. But I would venture to say that for 99% of the people out there, it will solve there problems very well.
  - bb01100100 9y ago
    Would it be accurate to say that your ability to count something new, starting from a previous point in time (i.e. not just from 'now') is dependent upon how easily you can replay your stream (I'm thinking in Kafka terms); or is there something in your architecture that allows you to consume from the stream from a (relative? absolute?) point in time (again, Kafka thinking leaking into my question)? Thanks.
dozzie 9y ago
> ElasticSearch was never really meant for log storage anyway. Indeed, it wasn't designed for log storage, though it happened to match this use scenario well (now less so with every release). > There are far better ways to handle log analysis, particularly when your primary query is counting things that happen over T instead of finding log entries that match a query (which it always is) Oh? This is the first time I hear that my use case (storing logs from syslog for diagnostics at a later time) counts things over time. Good to know. I may ask you later for more insight about my environment. > streaming analysis is a much better fit than indexing, just lesser known. Well, I do this, too, and not only from logs. I still need arbitrary term search for logs.
- jsmthrowaway 9y ago
  The snark is totally unnecessary, since the vast majority of people deploy ELK to do reporting. Full term search is achievable with grep; what does ES give you for your non-reporting use case, since troubleshooting is an extremely low frequency event? Are you primarily leaning on its clustering and replication? Genuinely curious. The overhead per log record, building multiple indexes at log line rate, there’s just so many reasons not to do your use case in ES that I don’t even think about it. I think it’s a poorer fit than reporting, to be honest.
  - dozzie 9y ago
    > The snark is totally unnecessary, since the vast majority of people deploy ELK to do reporting. You shouldn't have used authoritatively universal quantifier. There are plenty of sysadmins who use ES for this case, you apparently just happened to only be exposed to using it with websites. Then, what ES+Kibana give me over grep? Search over specific field (my logs are parsed to a proper data structure), which includes type of event (obviously, different types for different daemons), a query language, and a frontend with histograms. Mind you, troubleshooting around a specific event is but one of the things sysadmins do with logs. There are also other uses, all landing in the realm of post-hoc analysis.
    - jsmthrowaway 9y ago
unethical_ban 9y ago
As someone who knows of several places using it for multi-petabyte log retention and log analysis, what is Elastic used for, if it's not good at log indexing and text search? If I need to find logs based on one snippet of data, then pivot on different fields to analyze events, what should I be using?

amenod 9y ago

What are the better alternatives for handling log analysis? (I didn't downvote you btw)

jsmthrowaway 9y ago
“Dataflow” and the open source ecosystem in that neighborhood (Flink, Spark, Beam, Kafka, that family of stuff) is a much more powerful way to look at logs in real time, rather than indexing them into storage and then querying. There just isn’t something off the shelf as easy to deploy as ElasticSearch with that architecture, that I’m aware of. (There should be!) When you jump the mental gap of events being identical to logs, then start looking at fun architectures like event sourcing, you start realizing streaming analysis is a pretty powerful way of thinking. I’ve extracted insight from millions of log records per second on a single node with a similar setup, in real time, with much room to scale. The key to scaling log analysis is to get past textual parsing, which means using something structured, which usually negates the reason you were using ElasticSearch in the first place. Google’s first Dataflow talk from I/O and the paper should give you an idea of what log analysis can look like when you get past the full text indexing mindset. Note that there’s nothing wrong with ELK, but you will run into scaling difficulty far sooner than you’d expect trying to index every log event as it comes. It’s also bad to discover this when you get slashdotted, and your ES cluster whimpers trying to keep up. One thing streaming often gets you is latency in that situation instead of death, since you’re probably building atop a distributed log (and falling behind is less bad than falling over). The key here is: are you looking for specific log events or are you counting? You’re almost always counting. Why index individual events, then?
- yehosef 9y ago
  I don't think this is "always" true. The power of having the data in records is that you can trivially pivot and slice and dice the data in different ways. When it's aggregated to a count - I don't have that ability. When trying to debug something that's happening, I find it far easier to have the entire records. As for scaling, it scales very well (you can read through the elastic blog/use cases to see plenty of examples.) That's not to say there aren't levels of scaling it won't handle. But I would venture to say that for 99% of the people out there, it will solve there problems very well.
- bb01100100 9y ago
  Would it be accurate to say that your ability to count something new, starting from a previous point in time (i.e. not just from 'now') is dependent upon how easily you can replay your stream (I'm thinking in Kafka terms); or is there something in your architecture that allows you to consume from the stream from a (relative? absolute?) point in time (again, Kafka thinking leaking into my question)? Thanks.

dozzie 9y ago

> ElasticSearch was never really meant for log storage anyway. Indeed, it wasn't designed for log storage, though it happened to match this use scenario well (now less so with every release). > There are far better ways to handle log analysis, particularly when your primary query is counting things that happen over T instead of finding log entries that match a query (which it always is) Oh? This is the first time I hear that my use case (storing logs from syslog for diagnostics at a later time) counts things over time. Good to know. I may ask you later for more insight about my environment. > streaming analysis is a much better fit than indexing, just lesser known. Well, I do this, too, and not only from logs. I still need arbitrary term search for logs.

jsmthrowaway 9y ago
The snark is totally unnecessary, since the vast majority of people deploy ELK to do reporting. Full term search is achievable with grep; what does ES give you for your non-reporting use case, since troubleshooting is an extremely low frequency event? Are you primarily leaning on its clustering and replication? Genuinely curious. The overhead per log record, building multiple indexes at log line rate, there’s just so many reasons not to do your use case in ES that I don’t even think about it. I think it’s a poorer fit than reporting, to be honest.
- dozzie 9y ago
  > The snark is totally unnecessary, since the vast majority of people deploy ELK to do reporting. You shouldn't have used authoritatively universal quantifier. There are plenty of sysadmins who use ES for this case, you apparently just happened to only be exposed to using it with websites. Then, what ES+Kibana give me over grep? Search over specific field (my logs are parsed to a proper data structure), which includes type of event (obviously, different types for different daemons), a query language, and a frontend with histograms. Mind you, troubleshooting around a specific event is but one of the things sysadmins do with logs. There are also other uses, all landing in the realm of post-hoc analysis.
  - jsmthrowaway 9y ago

unethical_ban 9y ago

As someone who knows of several places using it for multi-petabyte log retention and log analysis, what is Elastic used for, if it's not good at log indexing and text search? If I need to find logs based on one snippet of data, then pivot on different fields to analyze events, what should I be using?