4 ms·

You are underestimating the value of indexed logs and a Kibana view into them. For example, if you wanted a histogram of error logs for the last 12 days, and th

• travbrack 7y ago
  You can do this with graylog
• fxtentacle 7y ago
  You are underestimating how much costs the issues caused by Kibana and a slow ES can produce. I used to have these nice Kibana diagrams set up with auth and a https proxy so that our team could easily check if the server error rates looked normal. I quickly found out two things: 1. Most of our employees never ever looked at it, dismissing it as "too technical" just from seeing a screenshot. 2. The ELK stack uses backpressure, meaning the servers running filebeat run out of HDD space. BTW, with a bit of grep in front, the actual per-service working data sets are small enough so that I can load them into R on a 256GB RAM server and then produce nice diagrams. You can also script R from the command line and then send the diagrams around as a cronjob.
  • pepemon 7y ago
    Have you used the file input? With such throughput you don't need it at all, it hangs both of its host because of iowait and remote endpoint too if beats proto is used. All of my log traffic is sent as GELF UDP or syslog UDP.
• eecc 7y ago
  Sure it’s nice and maybe it’s worth 50k/month if you’re running Uber... but grep and sed together can get you quite far. Admittedly for the histogram you’d have to cut and paste into a spreadsheet but yeah, hardly something you need in real time (probably more for a postmortem presentation to management.)