3 ms·
At Google, we couldn't have read every single error that eg our Google Photos servers produce, even if we wanted to. At that scale, you get some errors even pu
by eru 4y ago
At Google, we couldn't have read every single error that eg our Google Photos servers produce, even if we wanted to.
At that scale, you get some errors even purely from bits flipping randomly in memory due to cosmic rays. (The latter is speculation. I don't remember how many of those we got, or whether we used enough error correcting memory. It's been a while.)
- mteigers 4y agoDitto, my managed service at Google threw close to 3k 500s / second when I was still there. Anything from cosmic rays, faulty hardware bit flipping, hard drive failures. We did, however, aggregate and group similar 500s and those did get looked at, but no way could we have looked at all errors. The other thing, is that with resilient infrastructure, who cares about an occasional 500. Back off and retry. No harm done.
- willcipriano 4y ago3k? What percentage of requests is that?
- samus 4y agoUser experience might indeed not be influenced by these errors, but errors of a less stochastic nature will impact it. The former obscure visibility of the latter, and that's probably the point of TA.
- MichaelZuo 4y agoIt's likely some fraction of those errors would have had some tangible impact on user experience.
- beachy 4y agoSurely you would have wanted to know how many bit flip errors you were getting - surely you attempted to isolate them, count them?
- eru 4y agoSure, you do statistics on them and you look at random samples in detail. But that's different from 'Reading Every Single Error' like the submitted article suggests.
- cortesoft 4y agoCounting and grouping them is the opposite of what the article is arguing for. The whole point of the article is that you shouldn't be doing any aggregation and should be investigating each one individually.
- samus 4y agoBut TA also argues that people reach for these aggregation tools way too early. I'd say the author is honest enough to refrain from statement about Google scale because they haven't experienced it. There is value in keeping error rates down even at that level.
- cortesoft 4y agoSure, but the comment I was responding to was missing the point, because the article wasn't trying to convince people that ignore errors to record and aggregate them, they are arguing that people who record and aggregate errors should instead look at each one individually. Pointing out that errors should be counted is tangential to the argument at hand.
- bobbylarrybobby 4y agoCurious whether there were any attempts to “read” them with AI. Is this something a LLM could do these days?
- samus 4y agoThe article admits that it might not be feasible at Google scale, and that it makes sense to aggregate errors at that point. But surely that works better the fewer uninvestigated classes of errors there are, no?