24 ms·

Fetching logs regularly sounds hard? Wouldn't you need to keep track of the position of all files, with heuristics around file rotations? And if something catas

Fetching logs regularly sounds hard? Wouldn't you need to keep track of the position of all files, with heuristics around file rotations? And if something catastrophic happens, the most interesting data would be in that last block which couldn't be polled? Normally you'd avoid all that complexity by shipping logs the other way, sending from each machine. That way you can keep state locally should you need to. All unix-like systems do this out of the box, and almost all software supports the syslog protocol to directly stream logs. But you can also use something like filebeat and a bunch of other modern alternatives. The analyzer can then run locally on the log server and a whole lot of complexity just disappears.

eigenvalue 3y ago
I considered doing it the way you described, but then you need to deploy software on every single one of your machines and make sure it's running, that it's not accidentally using up 99% of your CPU (I've had bad experiences with the monitoring agents for Splunk and Netdata misbehaving and slowing down the machines and causing problems), etc. Whereas with the "pull" approach I used in my tool, you don't need to deploy ANY software to the machines you are monitoring-- you just connect with SSH and grab the files you need and do all the work on your control node.
- xorcist 3y ago
  One way or the other, your hosts are running your application, and you are already deploying software on every single host. But I hear you with some of the agents. That's why I mentioned syslog. It's already there, it's supported by most logging packages, and it's dead simple. No additional software required. All text. What it doesn't do is structured logging, but analyzing on the log host is often enough. Agents aren't all that bad however, and you're likely already running some agent like icinga or zabbix for regular monitoring.
- hjvalente 3y ago
  having a Netdata agent taking your machine's CPU to 99% shouldn't happen, not sure when was the list time you tried it but a lot of recent improvements have been done on the Netdata Agent also, with Netdata you can achieve the same architecture design using a Netdata Parent that could be your "control node" and to where you stream the metrics of the nodes you want to keep running with as less load as possible - you can even offload the health engine and the machine learning take a look at https://learn.netdata.cloud/docs/streaming/ https://learn.netdata.cloud/docs/streaming/ and https://learn.netdata.cloud/docs/configuring/how-to-optimize-the-netdata-agent-s-performance https://learn.netdata.cloud/docs/configuring/how-to-optimize...
  - eigenvalue 3y ago
    I hear you, but once something like that burns me, I am very loathe to risk it again if the cost/benefit ratio seems unfavorable to me. While it's nice to have the pretty dashboards for Netdata, it's not worth even a small risk of it breaking or degrading my systems.
johnthescott 3y ago
> Wouldn't you need to keep track of the position of all files rsync --append is your friend.
- xorcist 3y ago
  That doesn't take neither rotated nor truncated log files into account. The easiest way, and what most log shippers do, is following inodes. It's also not as effective as streaming them to their intended target directly. Syslog can write a complementary local copy too should you wish to keep one. Logs has been a thing since the past forty years. In order to reinvent it, it is good to be acquainted with the standard systems.

eigenvalue 3y ago

I considered doing it the way you described, but then you need to deploy software on every single one of your machines and make sure it's running, that it's not accidentally using up 99% of your CPU (I've had bad experiences with the monitoring agents for Splunk and Netdata misbehaving and slowing down the machines and causing problems), etc. Whereas with the "pull" approach I used in my tool, you don't need to deploy ANY software to the machines you are monitoring-- you just connect with SSH and grab the files you need and do all the work on your control node.

xorcist 3y ago
One way or the other, your hosts are running your application, and you are already deploying software on every single host. But I hear you with some of the agents. That's why I mentioned syslog. It's already there, it's supported by most logging packages, and it's dead simple. No additional software required. All text. What it doesn't do is structured logging, but analyzing on the log host is often enough. Agents aren't all that bad however, and you're likely already running some agent like icinga or zabbix for regular monitoring.
hjvalente 3y ago
having a Netdata agent taking your machine's CPU to 99% shouldn't happen, not sure when was the list time you tried it but a lot of recent improvements have been done on the Netdata Agent also, with Netdata you can achieve the same architecture design using a Netdata Parent that could be your "control node" and to where you stream the metrics of the nodes you want to keep running with as less load as possible - you can even offload the health engine and the machine learning take a look at https://learn.netdata.cloud/docs/streaming/ https://learn.netdata.cloud/docs/streaming/ and https://learn.netdata.cloud/docs/configuring/how-to-optimize-the-netdata-agent-s-performance https://learn.netdata.cloud/docs/configuring/how-to-optimize...
- eigenvalue 3y ago
  I hear you, but once something like that burns me, I am very loathe to risk it again if the cost/benefit ratio seems unfavorable to me. While it's nice to have the pretty dashboards for Netdata, it's not worth even a small risk of it breaking or degrading my systems.

johnthescott 3y ago

> Wouldn't you need to keep track of the position of all files rsync --append is your friend.

xorcist 3y ago
That doesn't take neither rotated nor truncated log files into account. The easiest way, and what most log shippers do, is following inodes. It's also not as effective as streaming them to their intended target directly. Syslog can write a complementary local copy too should you wish to keep one. Logs has been a thing since the past forty years. In order to reinvent it, it is good to be acquainted with the standard systems.