6 ms·

Serious question: does indexing Logstash/JSON logs really need to take gigabytes of memory + disk and sharding?

DmitryOlshansky 7y ago
(Setting aside the fact that Logstash is JRuby/Java app easily eating said gigabytes of heap) Do JSON logs take gigabytes? If they do for you then yes, gigabytes of disk and memory are pretty much guaranteed. Also things tend to pile up with time (even on a few weeks horizon). In all honesty, I believe a finely crafted native code solution for this problem could achieve x3 less ram usage and x2-3 indexing/search performance. Going beyond that is also possible but will take remarkable engineering. Update: to expand on the last point, c++ solutions are typically closed sourced. Rust and Go both have interesting open-source full text engines: https://github.com/tantivy-search/tantivy https://github.com/tantivy-search/tantivy https://github.com/blevesearch/bleve https://github.com/blevesearch/bleve In near future I totally see someone producing a great open-source distributed search project that is at least on par with today’s ES core feature set.
- geggam 7y ago
  Not sure if you are aware of this. We ran this at Y! https://vespa.ai/ https://vespa.ai/
  - DmitryOlshansky 7y ago
    Seen that and starred probably half a year ago. Never had the time to dig around and see what it’s like in perf, operations and scalability.
    - geggam 7y ago
      Saw 4 petabyes in it. Cant remember the hosts number exactly but it was several hundred bare metal servers of various sizes Behind Y! Groups
  - emmelaich 7y ago
    Looks awesome. ES is good but it takes a small army to keep on top of the performance and management. And then there's the upgrades which will fix a few bugs and introduce new ones too.
  - otisg 7y ago
    How would you compare Vespa to ES when it comes to CPU/memory requirements and performance for something like log indexing and search use case?
- zmmmmm 7y ago
  > I believe a finely crafted native code solution for this problem could achieve x3 less ram usage and x2-3 indexing/search performance Many people believe that about everything written in Java. The story often ends up a lot more complicated though. In the domains where it shines Java is surprisingly hard to beat without very significant effort. And then you must also contemplate what they same significant effort would achieve if directed at the Java solution or cherry picking particular performance hotspots out of it.
  - DmitryOlshansky 7y ago
    I’m basing it on my limited experience of rewriting things from optimized but messy C++/D to JVM. Yes, the end result is much simpler but is memory hungry and is ~2x slower (after optimizations and tuning). Sometimes you can fit Java in less then ~2x of original footprint but at the cost of burning CPU on frequent GC cycles. Not every application is the same but the moment it involves manipulating a lot of data in Java you fight the platform to get back the control you require for those things. And by the end of day there is a limit at which reasonable people stop and just give up dodging memory allocations, creatively reusing objects, wrangling off-heap memory without the help of type system.
jrockway 7y ago
No. ELK is a slow and expensive way to store and retrieve logs. The reason people use it is that nothing else exists. (I was blown away when I started using it at my last job. I used the fluentd Kubernetes daemonset to extract logs from k8s and into ES... and the "cat a file and send it over the network" thing uses 300+MB of RAM per node. There is an alternate daemon that can be used now, but wow. 300MB to tail some files and parse JSON.) I think a better strategy is to store logs in flat files with several replicas. Do your metric generation in realtime, regexing a bunch of logs on a bunch of workers as they come in. (I handled > 250MB/s on less than one Google production machine, though did eventually shard it up for better schedulability and disaster resilience. Also those 10Gb NICs start to feel slow when a bunch of log sources come back after a power outage!) For simple lookups like "show me all the logs in the last 5 minutes", you can maintain an index of timestamp -> log file in a standard database, and do additional filtering on whatever is retrieving the files. You can also probably afford to index other things, like x-request-id and maybe a trigram index of messages, and actually be able to debug full request cycles in a handful of milliseconds when necessary. For complicated queries, you can just mapreduce it. After using ES, you will also be impressed at how fast grepping a flat file for your search term is. The problem is, the machinery to do this easily doesn't exist. Everything is designed for average performance at large scale, instead of great performance at medium scale. Someday I plan to fix this, but I just don't see a business case, so it's low priority. Would you fund someone who can make your log queries faster? Nope. "Next time we won't have a production issue that the logs will help debug." And so, there's nothing good.
- cheeze 7y ago
  Flat files and grep works to a level. For huge datasets it can be _really_ hard to answer some questions using pipes, grep, awk, etc. that something like a structured query makes pretty simple. We tried using CloudWatch logs insights at work and I was blown away by how fast it was on indexed data (we saw 10+ GB/s searches across a few hours of logs. Only problem is that it was prohibitively expensive so we ended up not going with it. Biggest thing for me is that I don't want to own a log searching service/software. My customers don't see any benefit from me investing my time in a better log searching platform than what is already available out there IMO. I want to let someone who is an expert on querying logs to solve that problem so that I can solve my specific problem.
cpitman 7y ago
One project worth keeping an eye on is Loki (https://github.com/grafana/loki https://github.com/grafana/loki), which eschews full text search for more basic indexing off of "labels", ie it works a lot like prometheus. There's a writeup on the differences with the EFK stack here: https://github.com/grafana/loki/blob/master/docs/overview/comparisons.md https://github.com/grafana/loki/blob/master/docs/overview/co... After working with a client for multiple years continually hitting bottlenecks and complexity with the EFK stack, I'm really looking forward to something different.
- sandstrom 7y ago
  Downside is that Loki can't handle high-cardinality labels, such as a `Request-Id`, IP-addresses or user-ids. I'd be happy to move from ELK to Loki, but no option to filter on IP-address or User-IDs is a big drawback. It's an open issue though, so may be added in the future: https://github.com/grafana/loki/issues/91 https://github.com/grafana/loki/issues/91

(Setting aside the fact that Logstash is JRuby/Java app easily eating said gigabytes of heap) Do JSON logs take gigabytes? If they do for you then yes, gigabytes of disk and memory are pretty much guaranteed. Also things tend to pile up with time (even on a few weeks horizon). In all honesty, I believe a finely crafted native code solution for this problem could achieve x3 less ram usage and x2-3 indexing/search performance. Going beyond that is also possible but will take remarkable engineering. Update: to expand on the last point, c++ solutions are typically closed sourced. Rust and Go both have interesting open-source full text engines: https://github.com/tantivy-search/tantivy https://github.com/tantivy-search/tantivy https://github.com/blevesearch/bleve https://github.com/blevesearch/bleve In near future I totally see someone producing a great open-source distributed search project that is at least on par with today’s ES core feature set.

geggam 7y ago
Not sure if you are aware of this. We ran this at Y! https://vespa.ai/ https://vespa.ai/
- DmitryOlshansky 7y ago
  Seen that and starred probably half a year ago. Never had the time to dig around and see what it’s like in perf, operations and scalability.
  - geggam 7y ago
    Saw 4 petabyes in it. Cant remember the hosts number exactly but it was several hundred bare metal servers of various sizes Behind Y! Groups
- emmelaich 7y ago
  Looks awesome. ES is good but it takes a small army to keep on top of the performance and management. And then there's the upgrades which will fix a few bugs and introduce new ones too.
- otisg 7y ago
  How would you compare Vespa to ES when it comes to CPU/memory requirements and performance for something like log indexing and search use case?
zmmmmm 7y ago
> I believe a finely crafted native code solution for this problem could achieve x3 less ram usage and x2-3 indexing/search performance Many people believe that about everything written in Java. The story often ends up a lot more complicated though. In the domains where it shines Java is surprisingly hard to beat without very significant effort. And then you must also contemplate what they same significant effort would achieve if directed at the Java solution or cherry picking particular performance hotspots out of it.
- DmitryOlshansky 7y ago
  I’m basing it on my limited experience of rewriting things from optimized but messy C++/D to JVM. Yes, the end result is much simpler but is memory hungry and is ~2x slower (after optimizations and tuning). Sometimes you can fit Java in less then ~2x of original footprint but at the cost of burning CPU on frequent GC cycles. Not every application is the same but the moment it involves manipulating a lot of data in Java you fight the platform to get back the control you require for those things. And by the end of day there is a limit at which reasonable people stop and just give up dodging memory allocations, creatively reusing objects, wrangling off-heap memory without the help of type system.

jrockway 7y ago

No. ELK is a slow and expensive way to store and retrieve logs. The reason people use it is that nothing else exists. (I was blown away when I started using it at my last job. I used the fluentd Kubernetes daemonset to extract logs from k8s and into ES... and the "cat a file and send it over the network" thing uses 300+MB of RAM per node. There is an alternate daemon that can be used now, but wow. 300MB to tail some files and parse JSON.) I think a better strategy is to store logs in flat files with several replicas. Do your metric generation in realtime, regexing a bunch of logs on a bunch of workers as they come in. (I handled > 250MB/s on less than one Google production machine, though did eventually shard it up for better schedulability and disaster resilience. Also those 10Gb NICs start to feel slow when a bunch of log sources come back after a power outage!) For simple lookups like "show me all the logs in the last 5 minutes", you can maintain an index of timestamp -> log file in a standard database, and do additional filtering on whatever is retrieving the files. You can also probably afford to index other things, like x-request-id and maybe a trigram index of messages, and actually be able to debug full request cycles in a handful of milliseconds when necessary. For complicated queries, you can just mapreduce it. After using ES, you will also be impressed at how fast grepping a flat file for your search term is. The problem is, the machinery to do this easily doesn't exist. Everything is designed for average performance at large scale, instead of great performance at medium scale. Someday I plan to fix this, but I just don't see a business case, so it's low priority. Would you fund someone who can make your log queries faster? Nope. "Next time we won't have a production issue that the logs will help debug." And so, there's nothing good.

cheeze 7y ago
Flat files and grep works to a level. For huge datasets it can be _really_ hard to answer some questions using pipes, grep, awk, etc. that something like a structured query makes pretty simple. We tried using CloudWatch logs insights at work and I was blown away by how fast it was on indexed data (we saw 10+ GB/s searches across a few hours of logs. Only problem is that it was prohibitively expensive so we ended up not going with it. Biggest thing for me is that I don't want to own a log searching service/software. My customers don't see any benefit from me investing my time in a better log searching platform than what is already available out there IMO. I want to let someone who is an expert on querying logs to solve that problem so that I can solve my specific problem.

cpitman 7y ago

One project worth keeping an eye on is Loki (https://github.com/grafana/loki https://github.com/grafana/loki), which eschews full text search for more basic indexing off of "labels", ie it works a lot like prometheus. There's a writeup on the differences with the EFK stack here: https://github.com/grafana/loki/blob/master/docs/overview/comparisons.md https://github.com/grafana/loki/blob/master/docs/overview/co... After working with a client for multiple years continually hitting bottlenecks and complexity with the EFK stack, I'm really looking forward to something different.

sandstrom 7y ago
Downside is that Loki can't handle high-cardinality labels, such as a `Request-Id`, IP-addresses or user-ids. I'd be happy to move from ELK to Loki, but no option to filter on IP-address or User-IDs is a big drawback. It's an open issue though, so may be added in the future: https://github.com/grafana/loki/issues/91 https://github.com/grafana/loki/issues/91