7 ms·

I'm curious about this, because at Square we maintain our own homegrown log aggregation system, and it's not really a core competency. While a lot of our loggi

• netingle 8y ago
  It’s defiantly something that’s missing from the readme, and perhaps not that obvious in the grafana explore view either - but it is there! You can push a regexp match server side and have that distributed to each Loki node, giving you distributed grep. Will make it more obvious. Davkals has an iteration of the UI that makes it a separate field, which will also help.
  • danlimerick 8y ago
    There is some documentation about the Loki search syntax in the Grafana docs: http://docs.grafana.org/features/explore/#logs-integration-loki-specific-features http://docs.grafana.org/features/explore/#logs-integration-l...
  • samstave 8y ago
    Please make it more obvious with exact examples of how to do this.
  • GordonS 8y ago
    Maybe I'm not understanding this - the docs say Loki is all about storing compressed log data with metadata, such that only the metadata is indexed. Are you saying you can search the compressed, unindexed data using regex? If so, wouldn't that potentially be incredibly slow?
    • gouthamve 8y ago
      The good thing about this is that the grepping can be parallelised and distributed on to several nodes. Having said that, once you select the relevant metadata right, you should be able to narrow it down enough for the queries to be snappy enough. While this will definitely be slower than something that indexes the contents, you'll be able to store much more in Loki at much lower costs.
      • GordonS 8y ago
        Yeah, I am thinking about the worst case here, but never underestimate the power of your users to perform very silly queries! For your hosted service, will you put in place any restrictions on, for example, the size of the time range that can be queried? Also for your hosted service, will the degree of parallelisation vary by pricing tier?
      • zacmps 8y ago
        What are you using to run the regex? ripgrep could make up for some of the loss from not having it indexed.
        • ecnahc515 8y ago
          https://github.com/grafana/loki/blob/master/pkg/iter/iterator.go#L252:6 https://github.com/grafana/loki/blob/master/pkg/iter/iterato... Looks like the Go regex lib, which isn't super performant, so it could potentially be improved if it ends up being an issue.
• programd 8y ago
  I was going to suggest you look at OKlog [0] but it seems that the project is recently archived and no longer maintained. A pity because I really like its simple and minimalistic design. It does still work of course and the code is all there. Loki would do well to look to it for some inspiration. I particularly like the super simple install, and lack if the need for some clunky UI to tail and grep your distributed logs. [0] https://github.com/oklog/oklog https://github.com/oklog/oklog
  • netingle 8y ago
    Agreed - Loki was heavily influenced by OKLog. We really like its ease of use and simplicity - hopefully we managed to get some of that right with Loki. OTOH we felt like we needed a little bit of metadata and index to help find the right logs - the brute force approach of only having grep doesn't seem like the right tradeoff to me. We use a minimal label index to narrow down the search space + then let you do distributed grep... There is a lightweight CLI for Loki too, you don't have to use the Grafana UI: https://github.com/grafana/loki/blob/master/docs/logcli.md https://github.com/grafana/loki/blob/master/docs/logcli.md
    • sandstrom 8y ago
      That's great. With a lightweight CLI you can run this in development and just use the CLI. Then add Grafana to your production environment.
  • zellyn 8y ago
    Thanks! Always love Peter's stuff. We actually have a perfectly viable working system internally. We're just (always) considering alternatives :-)