3 ms·
It is possible for data sources to be so noisy in such unpredictable and pathological ways that there is no practical way to extract signal. I expect this would
by jerf 4y ago
It is possible for data sources to be so noisy in such unpredictable and pathological ways that there is no practical way to extract signal. I expect this would be one of those cases.
As a simple for instance, out of what would be thousands of interacting and correlated issues, if some new line of RAM is put out that has more errors than before, your detector would have a hard time not seeing that as an increase in cosmic rays. And the real world deals out correlations that are very difficult to deal with. They could creep out slowly, or, AWS might order a whackload of these over the course of weeks and turn them all on at Tuesday at 9am. Trying to filter all those out is not necessarily mathematically possible.
- sliken 4y agoDunno. I've heard that phone accelometers can help detect earth quakes early to allow earlier warnings. Certainly that's a noisier signal than ECC errors. Just aggregate the memory errors so you get a baseline for N machines and if that baseline goes to 10N then you likely had something interesting happen ... hopefully not WW3.
- jerf 4y ago"such unpredictable and pathological ways" is not an extraneous phrase. This is simply a mathematical fact; the perversity of the universe is greater than your mathematical toolkit can handle. There is no guarantee that you can extract signal from a stream, even if you "know" it must be there in some sense. This is one of the great machine learning delusions.
- sliken 4y agoRight, but ECC errors are rare. If your base rate is 20 per hour for 100,000 devices, and you see 1,000,000 there's likely something going on. My example was just to show that even MUCH noisier dataset (like phone motion) have been useful for detecting earthquakes that ECC should be MUCH easier.