3 ms·

One thing that I was not aware prior to working as an SRE is how much they rely on statistics. This approach that you can use stats to determine what is a norma

One thing that I was not aware prior to working as an SRE is how much they rely on statistics. This approach that you can use stats to determine what is a normal or abnormal level for a particular metric (like packet loss for example) became pretty useful through my career. A quick example, I was called into a meeting at a company where I worked in a non-SRE role, and the team explained to me that they are not able to identify what is wrong but their cluster is misbehaving and there is a node that gets kicked out of it regularly. I pulled up a console and started to compare OS level metrics across the cluster. The sysadmin team thought I am stupid because they explicitly told me which node was in trouble. After the third metrics I checked I found out that the node in question was doing 5000x more packet loss than the second worst in the cluster. It was a faulty NIC at that time. The sysadmin team was checking all of the metrics on the broken node but never compared the results to the healthy ones.

graycat 10y ago
Yup, such things are some of why, when I did work in anomaly detection, I wanted it to be multi-dimensional. What you observed is the same song, second verse of: There was a cluster doing transaction processing. One of the computers in the cluster got a little sick and was throwing all its incoming transactions into the bit bucket. The load leveling for the cluster sent the next transaction to the least busy computer in the cluster, and the sick computer, not doing any real work, was usually the least busy so got nearly all the incoming transactions. So, really the one sick computer was throwing away nearly all the incoming transactions of the whole cluster, made the whole cluster look sick. So, sure, some anomaly detection that takes as its input data, say, the CPU busy of each of the computers in the cluster should see that the one sick computer was comparatively low on CPU busy, call that an anomaly, raise an alarm, and let diagnosis begin. Sure, if do have such an anomaly detector, very much want good control over false alarm rate. So, can see what I posted five days ago (gee, suddenly anomaly detection is getting popular on HN?) in https://news.ycombinator.com/item?id=14119753 https://news.ycombinator.com/item?id=14119753 and https://news.ycombinator.com/user?id=graycat https://news.ycombinator.com/user?id=graycat

graycat 10y ago

Yup, such things are some of why, when I did work in anomaly detection, I wanted it to be multi-dimensional. What you observed is the same song, second verse of: There was a cluster doing transaction processing. One of the computers in the cluster got a little sick and was throwing all its incoming transactions into the bit bucket. The load leveling for the cluster sent the next transaction to the least busy computer in the cluster, and the sick computer, not doing any real work, was usually the least busy so got nearly all the incoming transactions. So, really the one sick computer was throwing away nearly all the incoming transactions of the whole cluster, made the whole cluster look sick. So, sure, some anomaly detection that takes as its input data, say, the CPU busy of each of the computers in the cluster should see that the one sick computer was comparatively low on CPU busy, call that an anomaly, raise an alarm, and let diagnosis begin. Sure, if do have such an anomaly detector, very much want good control over false alarm rate. So, can see what I posted five days ago (gee, suddenly anomaly detection is getting popular on HN?) in https://news.ycombinator.com/item?id=14119753 https://news.ycombinator.com/item?id=14119753 and https://news.ycombinator.com/user?id=graycat https://news.ycombinator.com/user?id=graycat