5 ms·

You cannot do much analytics on Loki, there is no index so each query is costly

buro9 5y ago
You can do a surprisingly lot with Loki. To get the performance run more queriers (horizontally scaled read path that is in front of the object storage). We're improving performance constantly, and we already run Loki at a very large scale (multi-region, multi-tenant, etc) with aggressive internal SLOs. We see customers doing network analytics, log analytics, line-of-business data analytics all on Loki and it works really well. Customers also use it as part of Cloud hosted ETL processes and to drive alerts. What we run in the Cloud today is the OSS version on k8s. That said... if we're missing some use-case or just a scenario that you feel should work and it doesn't work for you, then we really want to learn about it. If there's an out of the box experience that isn't great, then let us know. Feel free to ping me any details you're not willing to put on HN at david.kitchen@grafana.com but I'm very cool with public conversations too.
- jarpineh 5y ago
  Hi. I have a question. I have put some time trying to learn the Grafana stack. Heaviness of elastic both self hosted and cloud version has led me to seek for alternatives. Is there a way to get logs into Loki that can be run offline, when jobs have finished and hosts have been shutdown? It seems Loki recommends Promtail agent which looks a bit heavy-handed (https://grafana.com/docs/loki/latest/getting-started/get-logs-into-loki/ https://grafana.com/docs/loki/latest/getting-started/get-log... ) I have relaxed requirements on gathering logs for analytics. I’d much rather just run a tool that goes through log files and ingests them into Loki. Results of several jobs would be compared as a whole. From what I’ve seen Loki with Grafana can do a lot. For example this video: https://m.youtube.com/watch?v=7zmRhHd-ohk https://m.youtube.com/watch?v=7zmRhHd-ohk Thank you!
  - buro9 5y ago
    One of the engineers has suggested trying: ```cat <files> | promtail -stdin``` Though we like the idea as a feature request... "Have promtail support reading a file or directory and exiting once complete". So we've put that on the plan.
    - jarpineh 5y ago
      Ah, my thanks. I had somehow understood Promtail wrong. Its name, mention of Prometheus, scraping and such first tells me it's closer to something like node exporter. I had watched Loki release presentation and few others, but managed to still misunderstand. Oh well. Now, things make more sense. Moving forward with the installation... :)
      - buro9 5y ago
        Promtail is a process that watches a directory and effectively tails the files and sends them to Loki. Agree that the name isn't ideal as it's not clear that this is a "tail and send text files" process. But yup, Promtail is what you want.
- airocker 5y ago
  I wonder the Loki designers thought about the features that ES provides before rewriting the whole thing in a new way? Search and Analyze is pretty cool. Almost all of Loki can be an analzer plugin. Automatic schema detection is cool. ITs not that crazy to administer.

You can do a surprisingly lot with Loki. To get the performance run more queriers (horizontally scaled read path that is in front of the object storage). We're improving performance constantly, and we already run Loki at a very large scale (multi-region, multi-tenant, etc) with aggressive internal SLOs. We see customers doing network analytics, log analytics, line-of-business data analytics all on Loki and it works really well. Customers also use it as part of Cloud hosted ETL processes and to drive alerts. What we run in the Cloud today is the OSS version on k8s. That said... if we're missing some use-case or just a scenario that you feel should work and it doesn't work for you, then we really want to learn about it. If there's an out of the box experience that isn't great, then let us know. Feel free to ping me any details you're not willing to put on HN at david.kitchen@grafana.com but I'm very cool with public conversations too.

jarpineh 5y ago
Hi. I have a question. I have put some time trying to learn the Grafana stack. Heaviness of elastic both self hosted and cloud version has led me to seek for alternatives. Is there a way to get logs into Loki that can be run offline, when jobs have finished and hosts have been shutdown? It seems Loki recommends Promtail agent which looks a bit heavy-handed (https://grafana.com/docs/loki/latest/getting-started/get-logs-into-loki/ https://grafana.com/docs/loki/latest/getting-started/get-log... ) I have relaxed requirements on gathering logs for analytics. I’d much rather just run a tool that goes through log files and ingests them into Loki. Results of several jobs would be compared as a whole. From what I’ve seen Loki with Grafana can do a lot. For example this video: https://m.youtube.com/watch?v=7zmRhHd-ohk https://m.youtube.com/watch?v=7zmRhHd-ohk Thank you!
- buro9 5y ago
  One of the engineers has suggested trying: ```cat <files> | promtail -stdin``` Though we like the idea as a feature request... "Have promtail support reading a file or directory and exiting once complete". So we've put that on the plan.
  - jarpineh 5y ago
    Ah, my thanks. I had somehow understood Promtail wrong. Its name, mention of Prometheus, scraping and such first tells me it's closer to something like node exporter. I had watched Loki release presentation and few others, but managed to still misunderstand. Oh well. Now, things make more sense. Moving forward with the installation... :)
    - buro9 5y ago
      Promtail is a process that watches a directory and effectively tails the files and sends them to Loki. Agree that the name isn't ideal as it's not clear that this is a "tail and send text files" process. But yup, Promtail is what you want.
airocker 5y ago
I wonder the Loki designers thought about the features that ES provides before rewriting the whole thing in a new way? Search and Analyze is pretty cool. Almost all of Loki can be an analzer plugin. Automatic schema detection is cool. ITs not that crazy to administer.