4 ms·

We love JSON logs and previously just sent most of it to systemd's journald and use a custom tool to view them. But maybe a year ago Grafana released https://gi

We love JSON logs and previously just sent most of it to systemd's journald and use a custom tool to view them. But maybe a year ago Grafana released https://github.com/grafana/loki https://github.com/grafana/loki and we've been using it on https://oya.to/ https://oya.to/ ever since. IIRC, the recommended way to integrate it with Grafana is via promtail but we weren't too keen on the added complexity of yet-another service in the middle so we developed a custom client library in Go to just send the logs straight to Loki (which we should probably open source at some point). I don't think there's any fancy graph integration yet, but the Grafana explore tab with log level/severity and label filtering works well enough esp. since they introduced support for pretty printed JSON log payloads.

slimsag 7y ago
> we developed a custom client library in Go to just send the logs straight to Loki (which we should probably open source at some point). I'd be interested in seeing this, we're likely to start using Loki at Sourcegraph soon and would likely want this approach as well.
- WinonaRyder 7y ago
  I'll push it to https://github.com/oyato/promqueen https://github.com/oyato/promqueen when it's ready. Hopefully I will have time to work on it over the next week or two.
  - slimsag 7y ago
    Awesome! Much appreciated :)
thojest 7y ago
I have tried out loki, too. But I was not satisfied because you have to run an extra server for it. I have only a very small app so I was searching for a much simpler solution and found https://goaccess.io/ https://goaccess.io/ . The nice thing is that it is very flexible (you can pipe your logs in command line, but also run as a server) and if you are using standard tools like e.g. nginx or apache, the setup only takes an evening :) Here are a few more monitoring tools listed https://turtle.community/topic/monitoring https://turtle.community/topic/monitoring It is always a question of complexity and how much time you want to spend :) At first I used Multitail to peek into server logs via ssh. Then I switched to GoAccess and if you really have a greater infrastructure I would maybe switch to Loki or ELK.
- omegabravo 7y ago
  I thought goaccess (which is spectacular) was for HTTP logs, and not a general purpose logging solution
- WinonaRyder 7y ago
  I've used GoAccess in the past, but didn't find it a good fit esp. since I prefer JSON logs. I don't know if it supports this - maybe I didn't read enough docs - but at the time, I remember it was easier to spend an hour whipping up a tool in Go that did exactly what I wanted. As for loki, it's a separate server but the setup takes maybe 10-30 minutes of reading some docs, maybe changing some config files and the systemd unit file to keep it up and running is less than 10 lines (most of which is boilerplate). Of-course I have the benefit of a client library so I can just call a function on a struct at the end of a request with no need to worry about serializing the relevant data into some predetermined format, compression, etc.
  - darekkay 7y ago
    > I've used GoAccess in the past, but didn't find it a good fit esp. since I prefer JSON logs. I don't know if it supports this Yes, GoAccess supports HTML, JSON and CSV: goaccess --output=json

slimsag 7y ago

> we developed a custom client library in Go to just send the logs straight to Loki (which we should probably open source at some point). I'd be interested in seeing this, we're likely to start using Loki at Sourcegraph soon and would likely want this approach as well.

WinonaRyder 7y ago
I'll push it to https://github.com/oyato/promqueen https://github.com/oyato/promqueen when it's ready. Hopefully I will have time to work on it over the next week or two.
- slimsag 7y ago
  Awesome! Much appreciated :)

thojest 7y ago

I have tried out loki, too. But I was not satisfied because you have to run an extra server for it. I have only a very small app so I was searching for a much simpler solution and found https://goaccess.io/ https://goaccess.io/ . The nice thing is that it is very flexible (you can pipe your logs in command line, but also run as a server) and if you are using standard tools like e.g. nginx or apache, the setup only takes an evening :) Here are a few more monitoring tools listed https://turtle.community/topic/monitoring https://turtle.community/topic/monitoring It is always a question of complexity and how much time you want to spend :) At first I used Multitail to peek into server logs via ssh. Then I switched to GoAccess and if you really have a greater infrastructure I would maybe switch to Loki or ELK.

omegabravo 7y ago
I thought goaccess (which is spectacular) was for HTTP logs, and not a general purpose logging solution
WinonaRyder 7y ago
I've used GoAccess in the past, but didn't find it a good fit esp. since I prefer JSON logs. I don't know if it supports this - maybe I didn't read enough docs - but at the time, I remember it was easier to spend an hour whipping up a tool in Go that did exactly what I wanted. As for loki, it's a separate server but the setup takes maybe 10-30 minutes of reading some docs, maybe changing some config files and the systemd unit file to keep it up and running is less than 10 lines (most of which is boilerplate). Of-course I have the benefit of a client library so I can just call a function on a struct at the end of a request with no need to worry about serializing the relevant data into some predetermined format, compression, etc.
- darekkay 7y ago
  > I've used GoAccess in the past, but didn't find it a good fit esp. since I prefer JSON logs. I don't know if it supports this Yes, GoAccess supports HTML, JSON and CSV: goaccess --output=json