3 ms·

Splunk is a great tool but expensive. I like splunk's aggregation feature very much. If it is server logs, it can aggregate and tell me how many http 500 errors

Splunk is a great tool but expensive. I like splunk's aggregation feature very much. If it is server logs, it can aggregate and tell me how many http 500 errors I have, how many requests resulted in 404 etc. It can tell me top IP addresses where I am getting requests from, etc. I want to take a CSV file and provide same functionality. Eg. Give user information on how many times each field occurs. For example, if it is a CSV file with cities, countries, continents, I want to aggregate and tell how many cities are in each country and how many countries are in each continent. Is there an open source version of splunk I can modify? I tried logstash but it is not straight forward to work with. It still needs me to define schema everytime. Thx!

stevewatson301 4y ago
> Is there an open source version of splunk I can modify? https://github.com/grafana/loki https://github.com/grafana/loki might work for you. It’s not a drop in replacement for Splunk, FWIW.
- thayne 4y ago
  Is there any way to do subqueries (or some kind of join) with loki? That is one feature of splunk I haven't seen elsewhere, open source or not.
  - Lightbody 4y ago
    New Relic’s NRQL can do sub queries.
sofixa 4y ago
What you're describing sounds like Loki (Grafana's Prometheus inspired logging tool, which is super fast and cheap/easy, even though it sacrifices some flexibility to get there) Metric Queries: https://grafana.com/docs/loki/latest/logql/metric_queries/ https://grafana.com/docs/loki/latest/logql/metric_queries/
wizwit999 4y ago
We're building Matano (https://github.com/matanolabs/matano https://github.com/matanolabs/matano), an open source security lake platform. It's a different approach since we normalize logs from JSON, csv, etc, and ingest them into Apache Iceberg tables, but it allows for massive scale and joins, aggregations, etc using SQL.

stevewatson301 4y ago

> Is there an open source version of splunk I can modify? https://github.com/grafana/loki https://github.com/grafana/loki might work for you. It’s not a drop in replacement for Splunk, FWIW.

thayne 4y ago
Is there any way to do subqueries (or some kind of join) with loki? That is one feature of splunk I haven't seen elsewhere, open source or not.
- Lightbody 4y ago
  New Relic’s NRQL can do sub queries.

sofixa 4y ago

What you're describing sounds like Loki (Grafana's Prometheus inspired logging tool, which is super fast and cheap/easy, even though it sacrifices some flexibility to get there) Metric Queries: https://grafana.com/docs/loki/latest/logql/metric_queries/ https://grafana.com/docs/loki/latest/logql/metric_queries/

wizwit999 4y ago

We're building Matano (https://github.com/matanolabs/matano https://github.com/matanolabs/matano), an open source security lake platform. It's a different approach since we normalize logs from JSON, csv, etc, and ingest them into Apache Iceberg tables, but it allows for massive scale and joins, aggregations, etc using SQL.