4 ms·

Does Elasticsearch need to be as complicated as it is? I was surprised to find there wasn't an Elasticsearch + Kibana competitor that is "simpler". I just wan

Does Elasticsearch need to be as complicated as it is? I was surprised to find there wasn't an Elasticsearch + Kibana competitor that is "simpler". I just want to be able to store JSON logs with a timestamp + a bunch of fields then search them in a nice little UI later. Apparently, that's pretty hard to do right.

mountaineer 7y ago
> JSON logs with a timestamp + a bunch of fields then search them There is S3+Athena for this with AWS and Google can store/query JSON with BigQuery. The nice little UI doesn't come with it, but at least you don't have to spin up an Elastic cluster.
- liveoneggs 7y ago
  aws cloudwatch log insights can do a lot natively without the s3/athena step now
- PretzelFisch 7y ago
  Do you have a S3 + Athena example because when I tried it didn't seem to query adhock json file but rather the S3 file needs contain a array of json documents.
fizx 7y ago
I mean, that product is called LogDNA. But Elasticsearch has evolved into a whole bunch of things to meet everyone's needs. There's a way to do what you want and simply, but you have to find the simple path in the middle of the big product.
inertiatic 7y ago
I think the question you should be asking is, does my use case require a tool as powerful as ES? And only you can answer that fully.
softwaredoug 7y ago
I mean, the same reason that SQL isn't any simpler? It takes arbitrary data, and you can filter and aggregate it in arbitrary ways? People aren't just doing log data... it's just been heavily adopted for that purpose I think the "simpler" version of ES+Kibana is probably a spreadsheet.
chasers 7y ago
I built https://logflare.app https://logflare.app for exactly this.
jrudolph 7y ago
My team is using Loki + Grafana and we're pretty happy with it. It's pretty basic but it does what you expect it to do just fine. We ditched Elastic as it was a super massive PITA to operate (& a resource hog at that). I'll admit I'm not an expert at ELK at all, but tbh I was absolutely surprised just how bad Elastic + Kibana was for our basic log uses when they tout it as one of their mainstays. Or we were just exceptionally stupid, who knows. In any case, the experience we had with it didn't motivate us to become ELK experts at all. Our pet peeves: - The Kibana UI needlessly wastes tons screen for whitespace - makes it hard to dig down into logs - never seems to find exact string matches when we wanted it to and instead returns "helpful" fuzzy matches - Kibana has no qualms sending requests to Elastic that will happily kill your node instead of applying sensible paging / query timeouts. I mean that's why I'm using Kibana and not writing my own elastic frontend...
- cpitman 7y ago
  How has your experience with Loki been? I keep an eye on it, but haven't deployed it since it seems to be maturing still.
  - nerdbaggy 7y ago
    Running Loki with multiple instances is insanely hard
itronitron 7y ago
Solr is quite a bit more straightforward than ElasticSearch, but still a bit complicated in my opinion.
dijit 7y ago
ElasticSearch is really just clustered Lucene with some nice features wrapping it. You can probably get away with something that also wraps lucene. Though elasticsearch has a dominant position precisely because it is quite full featured and easy to run.
- nirmalc 7y ago
  Im afraid its a bit over simplification . For example - Aggregations in elasticsearch are not using lucene facets, but it just leverages the basic lucene mechanisms ( collector) .
rb808 7y ago
if the logs are only a few GB you can still write to files and fgrep.
miguel_rdp 7y ago
Splunk is that product, but it’s only free up to 500MB/day of indexing

mountaineer 7y ago

> JSON logs with a timestamp + a bunch of fields then search them There is S3+Athena for this with AWS and Google can store/query JSON with BigQuery. The nice little UI doesn't come with it, but at least you don't have to spin up an Elastic cluster.

liveoneggs 7y ago
aws cloudwatch log insights can do a lot natively without the s3/athena step now
PretzelFisch 7y ago
Do you have a S3 + Athena example because when I tried it didn't seem to query adhock json file but rather the S3 file needs contain a array of json documents.

fizx 7y ago

I mean, that product is called LogDNA. But Elasticsearch has evolved into a whole bunch of things to meet everyone's needs. There's a way to do what you want and simply, but you have to find the simple path in the middle of the big product.

inertiatic 7y ago

I think the question you should be asking is, does my use case require a tool as powerful as ES? And only you can answer that fully.

softwaredoug 7y ago

I mean, the same reason that SQL isn't any simpler? It takes arbitrary data, and you can filter and aggregate it in arbitrary ways? People aren't just doing log data... it's just been heavily adopted for that purpose I think the "simpler" version of ES+Kibana is probably a spreadsheet.

chasers 7y ago

I built https://logflare.app https://logflare.app for exactly this.

jrudolph 7y ago

My team is using Loki + Grafana and we're pretty happy with it. It's pretty basic but it does what you expect it to do just fine. We ditched Elastic as it was a super massive PITA to operate (& a resource hog at that). I'll admit I'm not an expert at ELK at all, but tbh I was absolutely surprised just how bad Elastic + Kibana was for our basic log uses when they tout it as one of their mainstays. Or we were just exceptionally stupid, who knows. In any case, the experience we had with it didn't motivate us to become ELK experts at all. Our pet peeves: - The Kibana UI needlessly wastes tons screen for whitespace - makes it hard to dig down into logs - never seems to find exact string matches when we wanted it to and instead returns "helpful" fuzzy matches - Kibana has no qualms sending requests to Elastic that will happily kill your node instead of applying sensible paging / query timeouts. I mean that's why I'm using Kibana and not writing my own elastic frontend...

cpitman 7y ago
How has your experience with Loki been? I keep an eye on it, but haven't deployed it since it seems to be maturing still.
- nerdbaggy 7y ago
  Running Loki with multiple instances is insanely hard

itronitron 7y ago

Solr is quite a bit more straightforward than ElasticSearch, but still a bit complicated in my opinion.

dijit 7y ago

ElasticSearch is really just clustered Lucene with some nice features wrapping it. You can probably get away with something that also wraps lucene. Though elasticsearch has a dominant position precisely because it is quite full featured and easy to run.

nirmalc 7y ago
Im afraid its a bit over simplification . For example - Aggregations in elasticsearch are not using lucene facets, but it just leverages the basic lucene mechanisms ( collector) .

rb808 7y ago

if the logs are only a few GB you can still write to files and fgrep.

miguel_rdp 7y ago

Splunk is that product, but it’s only free up to 500MB/day of indexing