In our case, what we do is use Grafana Loki for logging, which emphasizes on a small amount of indexes, but great support for “sequential scans” of log files.

In our case, what we do is use Grafana Loki for logging, which emphasizes on a small amount of indexes, but great support for “sequential scans” of log files. We send the logs in semi-structured format (logfmt), and we have no real performance problems with this. The trick is that you want to optimize for “easy to use regexes”, which we did not have with json. Logfmt gives this to us, while still being easy to read with the eyes. One of the key decisions we made, however, was to not use any cloud provider for this and run it ourselves. It gave us much more freedom with various ways to interact with our metrics / data, without the risk of getting huge bills (I’m looking at you Datadog, your billing practices are terrible).