4 ms·

It’s defiantly something that’s missing from the readme, and perhaps not that obvious in the grafana explore view either - but it is there! You can push a regex

It’s defiantly something that’s missing from the readme, and perhaps not that obvious in the grafana explore view either - but it is there! You can push a regexp match server side and have that distributed to each Loki node, giving you distributed grep. Will make it more obvious. Davkals has an iteration of the UI that makes it a separate field, which will also help.

danlimerick 8y ago
There is some documentation about the Loki search syntax in the Grafana docs: http://docs.grafana.org/features/explore/#logs-integration-loki-specific-features http://docs.grafana.org/features/explore/#logs-integration-l...
samstave 8y ago
Please make it more obvious with exact examples of how to do this.
GordonS 8y ago
Maybe I'm not understanding this - the docs say Loki is all about storing compressed log data with metadata, such that only the metadata is indexed. Are you saying you can search the compressed, unindexed data using regex? If so, wouldn't that potentially be incredibly slow?
- gouthamve 8y ago
  The good thing about this is that the grepping can be parallelised and distributed on to several nodes. Having said that, once you select the relevant metadata right, you should be able to narrow it down enough for the queries to be snappy enough. While this will definitely be slower than something that indexes the contents, you'll be able to store much more in Loki at much lower costs.
  - GordonS 8y ago
    Yeah, I am thinking about the worst case here, but never underestimate the power of your users to perform very silly queries! For your hosted service, will you put in place any restrictions on, for example, the size of the time range that can be queried? Also for your hosted service, will the degree of parallelisation vary by pricing tier?
  - zacmps 8y ago
    What are you using to run the regex? ripgrep could make up for some of the loss from not having it indexed.
    - ecnahc515 8y ago
      https://github.com/grafana/loki/blob/master/pkg/iter/iterator.go#L252:6 https://github.com/grafana/loki/blob/master/pkg/iter/iterato... Looks like the Go regex lib, which isn't super performant, so it could potentially be improved if it ends up being an issue.

danlimerick 8y ago

There is some documentation about the Loki search syntax in the Grafana docs: http://docs.grafana.org/features/explore/#logs-integration-loki-specific-features http://docs.grafana.org/features/explore/#logs-integration-l...

samstave 8y ago

Please make it more obvious with exact examples of how to do this.

GordonS 8y ago

Maybe I'm not understanding this - the docs say Loki is all about storing compressed log data with metadata, such that only the metadata is indexed. Are you saying you can search the compressed, unindexed data using regex? If so, wouldn't that potentially be incredibly slow?

gouthamve 8y ago
The good thing about this is that the grepping can be parallelised and distributed on to several nodes. Having said that, once you select the relevant metadata right, you should be able to narrow it down enough for the queries to be snappy enough. While this will definitely be slower than something that indexes the contents, you'll be able to store much more in Loki at much lower costs.
- GordonS 8y ago
  Yeah, I am thinking about the worst case here, but never underestimate the power of your users to perform very silly queries! For your hosted service, will you put in place any restrictions on, for example, the size of the time range that can be queried? Also for your hosted service, will the degree of parallelisation vary by pricing tier?
- zacmps 8y ago
  What are you using to run the regex? ripgrep could make up for some of the loss from not having it indexed.
  - ecnahc515 8y ago
    https://github.com/grafana/loki/blob/master/pkg/iter/iterator.go#L252:6 https://github.com/grafana/loki/blob/master/pkg/iter/iterato... Looks like the Go regex lib, which isn't super performant, so it could potentially be improved if it ends up being an issue.