3 ms·

> While ELK was better at performing queries like COUNT, SigNoz is 13x faster than ELK for aggregate queries. The author should also mention how much ES was fa

by dig1 4y ago

> While ELK was better at performing queries like COUNT, SigNoz is 13x faster than ELK for aggregate queries. The author should also mention how much ES was faster against SigNoz with trace_id fetches (137x) and fetching first 100 logs (14x). Aggregating queries is known pain point for ES and will always be, due to ES design. People use additional tools for this, like Kafka Streams or Spark. > ClickHouse provides various codecs and compression mechanism which is used by SigNoz for storing logs What was "index.codec" for ES? Unfortunately, the default value does not provide the best compression ratio. I won't say that ES (or OpenSearch) is perfect, but I was surprised it holds well here, considering ES was run in (I'll presume) a non-optimal environment. First, put Kafka instead of Logstash (or in front of Logstash), and your ingestion rate will skyrocket. The second, learn how to tune JVM. Also, the author should use OpenSearch [1] because that is the place where all open-source development is happening now. [1] https://opensearch.org https://opensearch.org

nullify88 4y ago
What about index mapping, how many primaries, how many replicas, index rollover. Is your hot tier optimised for ingest, warm tier optimised for querying, and cold tier optimised for storage? There's so much to think about to get Elasticsearch running "optimally" and to keep it that way. It highlights the operational cost of running Elasticsearch.
- EdwardDiego 4y ago
  Any tool handling large amounts of data has an operational cost.
  - nullify88 4y ago
    Sure but in the end it boils down to whether that operational cost is worth it for the value received. Tools like Loki, are worthwhile alternatives for centralising infrastructure logs with lower operational costs.
    - dig1 4y ago
      ES isn't cheap to start with, and I agree with you on that, but it is straightforward to scale after you go above 3-6 nodes. ClickHouse is easy to start with (a single server), but not so much with unobtrusive scaling up or down. > What about index mapping, how many primaries, how many replicas, index rollover. Is your hot tier optimised for ingest, warm tier optimised for querying, and cold tier optimised for storage? Yes, there are details in this, but like every truly distributed system, you can't just plug it in and hope it works in the most optimal way. Also, regarding hot/cold storage, AFAIK, ES can do it after the fact, but with CH, you need to plan it in. > Sure but in the end it boils down to whether that operational cost is worth it for the value received. And as ELK is commonly used to centralise infrastructure logs, tools like Loki, SigNoz are becoming worthwhile alternatives. Actually, it boils down to whether you plan to grow or not, and tools like SigNoz or Loki has their place for sure. For example, for centralized logging, if you have a few servers and keep it that way for the next N years, ELK might not be for you. But, if you suddenly end up with 100 servers, ML team and would like to drill through logs and other data to get more insight on everything, moving to ELK will be way pricier than starting with it.
- nityananda123 4y ago
  Hi, I think this question is pointed towards Elasticsearch, But here are some points for SigNoz. ( I am one of the maintainers at SigNoz) > Directly ingesting to disk(hot tier) is faster than directly ingesting to s3(cold storage) > The query results were an average of cold + hot run ( for elk as well ). We didn’t have an explicit concept of warm storage for SigNoz in our benchmark. > The query perf for logs with cold storage is almost similar to hot storage, but the operational cost will reduce with cold storage. So ingesting to host storage and moving to cold storage after a certain amount of time is a good option for Signoz.
pranay01 4y ago
Thanks for the feedback. We chose Elasticsearch as in our experience it is still the default tool for people to get started with logs. But do understand that may be opensearch is also catching up now. >The author should also mention how much ES was faster against SigNoz with trace_id fetches (137x) and fetching first 100 logs (14x). We didn't mention this in the summary as for the scale we tested at the difference would not have been perceived by a user. e.g. For getting logs corresponding to a trace_id (log corresponding to high cardinality field), SigNoz tool 0.137s, and Elastic took 0.001s I think we read somewhere (will try to find source) that anything below 200ms in server response is not perceived by user
- Aeolun 4y ago
  That sounds like an excuse. I do agree they’d likely both feel more or less instant though.

by dig1 4y ago

nullify88 4y ago

What about index mapping, how many primaries, how many replicas, index rollover. Is your hot tier optimised for ingest, warm tier optimised for querying, and cold tier optimised for storage? There's so much to think about to get Elasticsearch running "optimally" and to keep it that way. It highlights the operational cost of running Elasticsearch.

EdwardDiego 4y ago
Any tool handling large amounts of data has an operational cost.
- nullify88 4y ago
  Sure but in the end it boils down to whether that operational cost is worth it for the value received. Tools like Loki, are worthwhile alternatives for centralising infrastructure logs with lower operational costs.
  - dig1 4y ago
    ES isn't cheap to start with, and I agree with you on that, but it is straightforward to scale after you go above 3-6 nodes. ClickHouse is easy to start with (a single server), but not so much with unobtrusive scaling up or down. > What about index mapping, how many primaries, how many replicas, index rollover. Is your hot tier optimised for ingest, warm tier optimised for querying, and cold tier optimised for storage? Yes, there are details in this, but like every truly distributed system, you can't just plug it in and hope it works in the most optimal way. Also, regarding hot/cold storage, AFAIK, ES can do it after the fact, but with CH, you need to plan it in. > Sure but in the end it boils down to whether that operational cost is worth it for the value received. And as ELK is commonly used to centralise infrastructure logs, tools like Loki, SigNoz are becoming worthwhile alternatives. Actually, it boils down to whether you plan to grow or not, and tools like SigNoz or Loki has their place for sure. For example, for centralized logging, if you have a few servers and keep it that way for the next N years, ELK might not be for you. But, if you suddenly end up with 100 servers, ML team and would like to drill through logs and other data to get more insight on everything, moving to ELK will be way pricier than starting with it.
nityananda123 4y ago
Hi, I think this question is pointed towards Elasticsearch, But here are some points for SigNoz. ( I am one of the maintainers at SigNoz) > Directly ingesting to disk(hot tier) is faster than directly ingesting to s3(cold storage) > The query results were an average of cold + hot run ( for elk as well ). We didn’t have an explicit concept of warm storage for SigNoz in our benchmark. > The query perf for logs with cold storage is almost similar to hot storage, but the operational cost will reduce with cold storage. So ingesting to host storage and moving to cold storage after a certain amount of time is a good option for Signoz.

pranay01 4y ago

Thanks for the feedback. We chose Elasticsearch as in our experience it is still the default tool for people to get started with logs. But do understand that may be opensearch is also catching up now. >The author should also mention how much ES was faster against SigNoz with trace_id fetches (137x) and fetching first 100 logs (14x). We didn't mention this in the summary as for the scale we tested at the difference would not have been perceived by a user. e.g. For getting logs corresponding to a trace_id (log corresponding to high cardinality field), SigNoz tool 0.137s, and Elastic took 0.001s I think we read somewhere (will try to find source) that anything below 200ms in server response is not perceived by user

Aeolun 4y ago
That sounds like an excuse. I do agree they’d likely both feel more or less instant though.