3 ms·
The good thing about this is that the grepping can be parallelised and distributed on to several nodes. Having said that, once you select the relevant metadata
by gouthamve 8y ago
The good thing about this is that the grepping can be parallelised and distributed on to several nodes. Having said that, once you select the relevant metadata right, you should be able to narrow it down enough for the queries to be snappy enough.
While this will definitely be slower than something that indexes the contents, you'll be able to store much more in Loki at much lower costs.
- GordonS 8y agoYeah, I am thinking about the worst case here, but never underestimate the power of your users to perform very silly queries! For your hosted service, will you put in place any restrictions on, for example, the size of the time range that can be queried? Also for your hosted service, will the degree of parallelisation vary by pricing tier?
- zacmps 8y agoWhat are you using to run the regex? ripgrep could make up for some of the loss from not having it indexed.
- ecnahc515 8y agohttps://github.com/grafana/loki/blob/master/pkg/iter/iterator.go#L252:6 https://github.com/grafana/loki/blob/master/pkg/iter/iterato... Looks like the Go regex lib, which isn't super performant, so it could potentially be improved if it ends up being an issue.