3 ms·

I would try first with these columns based databases but I think you should consider saving less data. Doing what you want is definitely possible but it may be

I would try first with these columns based databases but I think you should consider saving less data. Doing what you want is definitely possible but it may be expensive, and if it's only because one non technical co-worker would love to query so much data for his reports, perhaps you should try to show him the cost once you tried a few technologies. Querying less data but saving everything is also an alternative. With Clickhouse you can specify a sampling rate for example.

psankar 6y ago
This turned out to be a good advice. I am evaluating if it is possible to somehow intelligently sample the data when it comes to ELK. Something like an average of 1min data via a logstash filter. We could do this from our backend programs too, but for now, I am trying to do this in ELK during the writes. Also we do not use Clickhouse, but I will see if I can somehow do a sampling rate in the reads, during my read queries. Thanks.