6 ms·

All these graphs are never really actionable and are only of interest for a short period of time and you won't be looking at it after a while because they don't

All these graphs are never really actionable and are only of interest for a short period of time and you won't be looking at it after a while because they don't mean anything unless you know where and when the problem is. A sever admin wants "Incident" panel that only shows anomaly components at the top coupled with adjustable alerting mechanism and not just a dump of all the data there is blindly. There are so many tools that does this and pretend it's impressive including ELK but whether it's Grafana or Kibana, you need a lot of manual tweaking to make the dashboards actually useful.

xPaw 6y ago
Netdata does have alarms/alerts, and comes with default ones. https://learn.netdata.cloud/guides/step-by-step/step-05 https://learn.netdata.cloud/guides/step-by-step/step-05
- odyslam 6y ago
  Actually, we give a lot of thought in defining sane default alarms for most of the data sources that we have. We want our users to get 80% of the value with 20% of the effort. It's an opinionated approach that liberates a lot of users from having to setup and maintain everything.
  - mdip 6y ago
    I think you took a lot of flack in the original comment[0] The alarms in netdata resolved a long-standing network issue on one of my boxes, and have variously alerted me to problems I could resolve with storage which greatly improved performance on my largest volume. On my other box, one look at the graphs alerted me to the fact that the entire SSD for my bcache volume was going unused[1]. I then used them while altering configuration and working with the drive to ensure the cache was being filled in a manner consistent with what the volume stored/how it was used. The more I think about it, I might not have been as enthusiastic in my original comment as I should have been. It's been very helpful to me. I don't usually keep things like this running for very long (it wastes cycles on aging hardware...that isn't heavily used, but hey, it's the principal!) but I've kept this around because every time I've thought about removing it, I've visited the dashboard one last time and found something there that made me keep it. [0] Though, as I mentioned, I'm not a sysadmin; I have a lab that might indicate otherwise, but I don't get paid for it. [1] I had reloaded the machine/redone a previous configuration that included bcache and it screamed; I knew my new setup was much slower but I had forgotten about it until netdata made it obvious, again. I can't remember what I had to do to fix it, but it had something to do with the policy used to determine if a file should be put into the cache, and I think it was related to the fact that the cache was added to a volume with data present that rarely changed.
andrewm4894 6y ago
Disclaimer - i work at Netdata Cloud on ML. This is one of the things i am focusing on most - how to package and then surface up "anomaly events" to the user that the user can then quickly digest and decide if they are or are not something that could represent an "incident". So human in the loop sort of ML to help assist and lower the cognitive load of all the charts. We have a first step on this ladder via the python based anomalies collector that you could play around with to see if any use: https://learn.netdata.cloud/docs/agent/collectors/python.d.plugin/anomalies https://learn.netdata.cloud/docs/agent/collectors/python.d.p... That will give two summary aggregation type charts based on anomaly scores built off all the system.* charts by default (but can be configured however you want) - so, at each second, an anomaly probability for each chart and an anomaly flag if the model thinks that chart looked anomalous at that time. We are also working on some other related projects to build on this capability out more and do it at the edge, in C++ or Go as opposed to Python, (or via a parent node) as cheap as possible so minimal, ideally negligible, impact on the agent itself. We should have some more features related to this in coming months as we are just trying to dogfood them internally a little first. Anyone interested i'd love to hear some more feedback here or in this community megathread here: https://community.netdata.cloud/t/anomalies-collector-feedback-megathread/767 https://community.netdata.cloud/t/anomalies-collector-feedba... or just email me at analytics-ml-team@netdata.cloud
- lima 6y ago
  What's the argument for anomaly detection - it's an obvious thing to do that has been tried many times, but doesn't actually seem to provide much value in practice (especially at large scale, where you'll get spurious correlations). What would you need it for? Once you defined your SLOs, either your service meets them or not. What's the value in alerting someone that "this graph looks funny"?
  - laminatedsmore 6y ago
    Do 4xx responses count against your SLO? For me they don't, but an abnormal increase might still signify that something is actually wrong. (I haven't yet found a useful tool for highlighting this kind of abnormality though)
unixhero 6y ago
I do not necessarily disagree with you regarding what a server admin / ops personnel needs, however; I for one deeply enjoy interacting with my Netdata dashboard whenever I want to deep dive into my servers resources and behaviors. For me it fits a purpose and if I ever were to run a company that hosted things, I would want it and I would want to pay for it. I am a huge fan and a long time homelab user of Netdata.
tyingq 6y ago
I agree with this, and it's interesting how many open source tools there are that create these graphs and charts, store tons of data, etc. All with a mostly "eyes on glass" bent, which doesn't scale terribly well. When, really, what's more important is actionable events, correlation, duplicate suppression, escalating notifications, etc. Something like what "Netcool Omnibus" and other commercial software does. Isolate actionable problems and make sure somebody owns the problem. But for reasons I don't understand, there isn't much in the open source world in that space.
mdip 6y ago
> never really actionable ... only of interest for a short period ... you know where and when the problem is. I'm not a sysadmin of a large shop (I did that for a short bit, but prior to this existing), so I can only speak as a guy who runs a few big linux servers/virtuals. I've had netdata installed on my home severs for quite some time. And yes, the graphs were really cool, at first, and kinda went into the background. Here's the thing: when something isn't right with those boxes, that's become the first place I visit. Since I had some franken-boxes with a bunch of storage, it's often related to the array, or btrfs. When I hop in there I'll notice an alert or two, gooble it, alter something and never see it again. It helped me solve some network issues. I don't know, short of it being a more busy process than I wish it were on my server (only a little and I'm running a few plug-ins on the one that I'm unhappy with), it's been helpful.
- fatlasp 6y ago
  Same. I'm not a fully qualified sys admin but I do have access to a number of our servers (I'm more of a full stack generalist than an expert at anything) and I immediately go to netdata when one of my services isn't acting right. For me its a nice 'system at a glance' where I can check on the host and then alert someone more knowledgeable than myself if there's something that looks off
ktsaou 6y ago
Thank you for this feedback. I am the founder of Netdata. Netdata is about making our lives easier. If you need to tweak Netdata, please open a github issue to let us know. It is a bug. Netdata should provide the best possible dashboards and alerts out of the box. If it does not for you, we missed something and we need your help to fix it, so please open a github issue to let us know of your use case. We want Netdata to be installed and effectively used with zero configuration, even mid-crisis, so although tweaking is possible and we support plenty of it, it should not be required. An "incident" is a way to organize people, an issue management tool for monitoring, a collaboration feature. Netdata's primary goal however, is about exploring and understanding our infrastructure. We are trying to be amazingly effective in this by providing unlimited high resolution metrics, real-time dashboards and battle tested alarms. In our roadmap we have many features that we believe will change the way we understand monitoring. We are changing even the most fundamental features of a chart. Of course at the same time we are trying to improve collaboration. This is why Netdata.Cloud, our free-forever SaaS offering that complements the open-source agent to provide out of the box infrastructure-level monitoring along side several convenience features, organizes our infra in war-rooms. In these war-rooms we have added metrics correlation tools that can help us find the most relevant metrics for something that got our attention, an alarm, a spike or a dive on a chart. For Netdata, the high level incident panel you are looking for, will be based on a mix of charts and alarms. And we hope it is going to be also fully automated, autodetected and provided with zero configuration and tweaking. Stay tuned. We are baking it...
- Croftengea 6y ago
  > our free-forever SaaS offering that complements the open-source agent How do you make or plan to make money?
  - nawgz 6y ago
    This is a good question, their website doesn't seem to have any "Pricing" information anywhere and everything is "get now" and "sign up for free"...
  - sdesol 6y ago
goodpoint 6y ago
Netdata is focused on short-term, real-time metrics. I use it often during development.
chrisandchris 6y ago
So true. But netdata also supports alerts by e-mail or HTTP-based (e.g. for Slack), so why not just turn on notifications and live from them?
jeffbee 6y ago
Based on my experience a dashboard that you only use when you know where and when the problem occurred is incredibly useful, and the lack of one can be very frustrating. While you of course need a systematic approach to incident detection, you also need comprehensive eyes-on-glass dashboards during your investigations. "Anomaly detection" is much spoken of but generalized anomaly detection doesn't exist. You still need skilled operators to just have a look around in many cases. An example, drawn from several major incidents in my career. You get an alert, you narrow it down to a process or machine, you evict the machine from your serving population to remediate the incident, but how do you keep it from recurring? The anomalous thing isn't apparent in your monitoring data, so it must be among the bazillion statistics that a running system exposes, but which you can't afford to collect and monitor on a per-host, per-container, per-process level of detail. That's when you want something exactly like netdata!
toomanybeersies 6y ago
There's value in having large dashboards that contain a bunch of non-prioritised graphs and gauges. I've managed to find a fair few problems by scrolling through such dashboards. Usually it's due to poorly configured monitors/alerts, but sometimes I'll spot things that you wouldn't reasonably expect an algorithm to pick up. Plus it's good fun to look at a big dashboard and pretend you're Homer Simpson at the Springfield Nuclear Power Plant.
nednar 6y ago
You seem to have a very specific vision. Could you mock it somehow? HTML, Figma, Paint, Powerpoint, whatever? Quite curious about ideas.
ohthehugemanate 6y ago
> A sever admin wants "Incident" panel that only shows anomaly components at the top coupled with adjustable alerting mechanism and not just a dump of all the data there is blindly. Netdata does this too, with a ton of thresholds already set up by default. The list of active alerts is at the top, with a badge and everything. Notifications use a hook system, so you can use whatever mechanism you like. Personally I get emails for medium level alerts, SMS for high and above, and wall posts/notifications on my primary machine for crits. It took some tuning to get the thresholds right for me, all perfectly easy to do. I agree I would prefer to have the active warnings more visible than the graphs, but one click away really isn't bad.
ARandomerDude 6y ago
I completely disagree. I work on a system with multiple servers communicating with each other and billions of events per day. Watching the meters wiggle on all the servers simultaneously is a really important debugging tool. If something slows down or goes down, and you know what it's connected to, it's pretty easy to troubleshoot what the cause is at an incredibly specific level just by looking at the meters. I think it's likely one's take on whether these tools are useful is very dependent on the system architecture.

xPaw 6y ago

Netdata does have alarms/alerts, and comes with default ones. https://learn.netdata.cloud/guides/step-by-step/step-05 https://learn.netdata.cloud/guides/step-by-step/step-05

odyslam 6y ago
Actually, we give a lot of thought in defining sane default alarms for most of the data sources that we have. We want our users to get 80% of the value with 20% of the effort. It's an opinionated approach that liberates a lot of users from having to setup and maintain everything.
- mdip 6y ago
  I think you took a lot of flack in the original comment[0] The alarms in netdata resolved a long-standing network issue on one of my boxes, and have variously alerted me to problems I could resolve with storage which greatly improved performance on my largest volume. On my other box, one look at the graphs alerted me to the fact that the entire SSD for my bcache volume was going unused[1]. I then used them while altering configuration and working with the drive to ensure the cache was being filled in a manner consistent with what the volume stored/how it was used. The more I think about it, I might not have been as enthusiastic in my original comment as I should have been. It's been very helpful to me. I don't usually keep things like this running for very long (it wastes cycles on aging hardware...that isn't heavily used, but hey, it's the principal!) but I've kept this around because every time I've thought about removing it, I've visited the dashboard one last time and found something there that made me keep it. [0] Though, as I mentioned, I'm not a sysadmin; I have a lab that might indicate otherwise, but I don't get paid for it. [1] I had reloaded the machine/redone a previous configuration that included bcache and it screamed; I knew my new setup was much slower but I had forgotten about it until netdata made it obvious, again. I can't remember what I had to do to fix it, but it had something to do with the policy used to determine if a file should be put into the cache, and I think it was related to the fact that the cache was added to a volume with data present that rarely changed.

andrewm4894 6y ago

Disclaimer - i work at Netdata Cloud on ML. This is one of the things i am focusing on most - how to package and then surface up "anomaly events" to the user that the user can then quickly digest and decide if they are or are not something that could represent an "incident". So human in the loop sort of ML to help assist and lower the cognitive load of all the charts. We have a first step on this ladder via the python based anomalies collector that you could play around with to see if any use: https://learn.netdata.cloud/docs/agent/collectors/python.d.plugin/anomalies https://learn.netdata.cloud/docs/agent/collectors/python.d.p... That will give two summary aggregation type charts based on anomaly scores built off all the system.* charts by default (but can be configured however you want) - so, at each second, an anomaly probability for each chart and an anomaly flag if the model thinks that chart looked anomalous at that time. We are also working on some other related projects to build on this capability out more and do it at the edge, in C++ or Go as opposed to Python, (or via a parent node) as cheap as possible so minimal, ideally negligible, impact on the agent itself. We should have some more features related to this in coming months as we are just trying to dogfood them internally a little first. Anyone interested i'd love to hear some more feedback here or in this community megathread here: https://community.netdata.cloud/t/anomalies-collector-feedback-megathread/767 https://community.netdata.cloud/t/anomalies-collector-feedba... or just email me at analytics-ml-team@netdata.cloud

lima 6y ago
What's the argument for anomaly detection - it's an obvious thing to do that has been tried many times, but doesn't actually seem to provide much value in practice (especially at large scale, where you'll get spurious correlations). What would you need it for? Once you defined your SLOs, either your service meets them or not. What's the value in alerting someone that "this graph looks funny"?
- laminatedsmore 6y ago
  Do 4xx responses count against your SLO? For me they don't, but an abnormal increase might still signify that something is actually wrong. (I haven't yet found a useful tool for highlighting this kind of abnormality though)

unixhero 6y ago

I do not necessarily disagree with you regarding what a server admin / ops personnel needs, however; I for one deeply enjoy interacting with my Netdata dashboard whenever I want to deep dive into my servers resources and behaviors. For me it fits a purpose and if I ever were to run a company that hosted things, I would want it and I would want to pay for it. I am a huge fan and a long time homelab user of Netdata.

tyingq 6y ago

I agree with this, and it's interesting how many open source tools there are that create these graphs and charts, store tons of data, etc. All with a mostly "eyes on glass" bent, which doesn't scale terribly well. When, really, what's more important is actionable events, correlation, duplicate suppression, escalating notifications, etc. Something like what "Netcool Omnibus" and other commercial software does. Isolate actionable problems and make sure somebody owns the problem. But for reasons I don't understand, there isn't much in the open source world in that space.

mdip 6y ago

> never really actionable ... only of interest for a short period ... you know where and when the problem is. I'm not a sysadmin of a large shop (I did that for a short bit, but prior to this existing), so I can only speak as a guy who runs a few big linux servers/virtuals. I've had netdata installed on my home severs for quite some time. And yes, the graphs were really cool, at first, and kinda went into the background. Here's the thing: when something isn't right with those boxes, that's become the first place I visit. Since I had some franken-boxes with a bunch of storage, it's often related to the array, or btrfs. When I hop in there I'll notice an alert or two, gooble it, alter something and never see it again. It helped me solve some network issues. I don't know, short of it being a more busy process than I wish it were on my server (only a little and I'm running a few plug-ins on the one that I'm unhappy with), it's been helpful.

fatlasp 6y ago
Same. I'm not a fully qualified sys admin but I do have access to a number of our servers (I'm more of a full stack generalist than an expert at anything) and I immediately go to netdata when one of my services isn't acting right. For me its a nice 'system at a glance' where I can check on the host and then alert someone more knowledgeable than myself if there's something that looks off

ktsaou 6y ago

Thank you for this feedback. I am the founder of Netdata. Netdata is about making our lives easier. If you need to tweak Netdata, please open a github issue to let us know. It is a bug. Netdata should provide the best possible dashboards and alerts out of the box. If it does not for you, we missed something and we need your help to fix it, so please open a github issue to let us know of your use case. We want Netdata to be installed and effectively used with zero configuration, even mid-crisis, so although tweaking is possible and we support plenty of it, it should not be required. An "incident" is a way to organize people, an issue management tool for monitoring, a collaboration feature. Netdata's primary goal however, is about exploring and understanding our infrastructure. We are trying to be amazingly effective in this by providing unlimited high resolution metrics, real-time dashboards and battle tested alarms. In our roadmap we have many features that we believe will change the way we understand monitoring. We are changing even the most fundamental features of a chart. Of course at the same time we are trying to improve collaboration. This is why Netdata.Cloud, our free-forever SaaS offering that complements the open-source agent to provide out of the box infrastructure-level monitoring along side several convenience features, organizes our infra in war-rooms. In these war-rooms we have added metrics correlation tools that can help us find the most relevant metrics for something that got our attention, an alarm, a spike or a dive on a chart. For Netdata, the high level incident panel you are looking for, will be based on a mix of charts and alarms. And we hope it is going to be also fully automated, autodetected and provided with zero configuration and tweaking. Stay tuned. We are baking it...

Croftengea 6y ago
> our free-forever SaaS offering that complements the open-source agent How do you make or plan to make money?
- nawgz 6y ago
  This is a good question, their website doesn't seem to have any "Pricing" information anywhere and everything is "get now" and "sign up for free"...
- sdesol 6y ago

goodpoint 6y ago

Netdata is focused on short-term, real-time metrics. I use it often during development.

chrisandchris 6y ago

So true. But netdata also supports alerts by e-mail or HTTP-based (e.g. for Slack), so why not just turn on notifications and live from them?

jeffbee 6y ago

Based on my experience a dashboard that you only use when you know where and when the problem occurred is incredibly useful, and the lack of one can be very frustrating. While you of course need a systematic approach to incident detection, you also need comprehensive eyes-on-glass dashboards during your investigations. "Anomaly detection" is much spoken of but generalized anomaly detection doesn't exist. You still need skilled operators to just have a look around in many cases. An example, drawn from several major incidents in my career. You get an alert, you narrow it down to a process or machine, you evict the machine from your serving population to remediate the incident, but how do you keep it from recurring? The anomalous thing isn't apparent in your monitoring data, so it must be among the bazillion statistics that a running system exposes, but which you can't afford to collect and monitor on a per-host, per-container, per-process level of detail. That's when you want something exactly like netdata!

toomanybeersies 6y ago

There's value in having large dashboards that contain a bunch of non-prioritised graphs and gauges. I've managed to find a fair few problems by scrolling through such dashboards. Usually it's due to poorly configured monitors/alerts, but sometimes I'll spot things that you wouldn't reasonably expect an algorithm to pick up. Plus it's good fun to look at a big dashboard and pretend you're Homer Simpson at the Springfield Nuclear Power Plant.

nednar 6y ago

You seem to have a very specific vision. Could you mock it somehow? HTML, Figma, Paint, Powerpoint, whatever? Quite curious about ideas.

ohthehugemanate 6y ago

> A sever admin wants "Incident" panel that only shows anomaly components at the top coupled with adjustable alerting mechanism and not just a dump of all the data there is blindly. Netdata does this too, with a ton of thresholds already set up by default. The list of active alerts is at the top, with a badge and everything. Notifications use a hook system, so you can use whatever mechanism you like. Personally I get emails for medium level alerts, SMS for high and above, and wall posts/notifications on my primary machine for crits. It took some tuning to get the thresholds right for me, all perfectly easy to do. I agree I would prefer to have the active warnings more visible than the graphs, but one click away really isn't bad.

ARandomerDude 6y ago

I completely disagree. I work on a system with multiple servers communicating with each other and billions of events per day. Watching the meters wiggle on all the servers simultaneously is a really important debugging tool. If something slows down or goes down, and you know what it's connected to, it's pretty easy to troubleshoot what the cause is at an incredibly specific level just by looking at the meters. I think it's likely one's take on whether these tools are useful is very dependent on the system architecture.