4 ms·

>To the engineering team, that's 1 error out of hundreds of thousands of requests per day. This is low severity. I think ignoring these edge cases is a mistake

>To the engineering team, that's 1 error out of hundreds of thousands of requests per day. This is low severity. I think ignoring these edge cases is a mistake. In statistics, we are often most interested in the tails of our curves. That is where the interesting stuff happens. P-values disprove null hypothesises, things fall outside confidence intervals, medium importance individuals suffer a failure and cause a PR disaster that diverts millions of dollars in revenue from your company. I wonder if big (or even small) companies could benefit from this "attention to the tails" thinking. It would be the kind of long term thinking and planning that keeps them on top for a long time instead of crumpling at the first sign of disruption. Of course this subscribes too the idea that anyone in the money-side of the company even cares when mostly they are just fighting for power in a zero sum game. I am worried this doesn't make sense, but I hope it provides insight :)

didibus 7y ago
Hum, I don't know much about statistics, so I'm not totally following, but this got me interested to read more about this "tails of our curve". I feel common metrics applied to software systems arn't very elaborate. Like I said, you might look at percentage of errors over total. Then you might take a goal like ok, let's try and have it less then 1%. The number chosen is arbitrary and has no rational, could have been 2%, or 0.5 but 1 sounded good right now. Every metric seem mostly linear and arbitrary, nothing fancy. I wonder if applying something more thorough like what is done in research and leveraging better statistic could make things better.
- ImaCake 7y ago
  Hmm I definitely think using some basic statistical theory could improve your choice of numbers. A bit of stats provides a lot more information from which you can draw more informed decisions. And of course there are lots of results that are highly unintuitive which have real world applications. I think this would fall under "risk engineering" and it is most relevant there. For an example, say you are interested in how frequently a particular function fails. You are especially worried that a first time user will have this function fail on the first attempt and then disengage entirely from your product. So you want to know how likely it is that your function will fail on the first try. You can run an experiment where you call the function repeatedly until it fails. You find it fails on the 10th attempt. You can use knowledge of probability distributions and a trick for calculating the most likely probability given the experiment you conducted to conclude the obvious: that it has a 1/10 chance of failing on the first go. What the long way gets you, is the chance for it failing after x attempts. So a bit of knowledge gets you a lot more information to make informed decisions with. You could always simulate this experiment in a script to estimate these numbers as well, which I think could be useful to someone with a software engineering background but without the time or resources to learn statistics. 0.https://en.wikipedia.org/wiki/Geometric_distribution https://en.wikipedia.org/wiki/Geometric_distribution 1.Probability dist for the example: https://www.wolframalpha.com/input/?i=geometric+distribution+p%3D0.1 https://www.wolframalpha.com/input/?i=geometric+distribution...

didibus 7y ago

Hum, I don't know much about statistics, so I'm not totally following, but this got me interested to read more about this "tails of our curve". I feel common metrics applied to software systems arn't very elaborate. Like I said, you might look at percentage of errors over total. Then you might take a goal like ok, let's try and have it less then 1%. The number chosen is arbitrary and has no rational, could have been 2%, or 0.5 but 1 sounded good right now. Every metric seem mostly linear and arbitrary, nothing fancy. I wonder if applying something more thorough like what is done in research and leveraging better statistic could make things better.

ImaCake 7y ago
Hmm I definitely think using some basic statistical theory could improve your choice of numbers. A bit of stats provides a lot more information from which you can draw more informed decisions. And of course there are lots of results that are highly unintuitive which have real world applications. I think this would fall under "risk engineering" and it is most relevant there. For an example, say you are interested in how frequently a particular function fails. You are especially worried that a first time user will have this function fail on the first attempt and then disengage entirely from your product. So you want to know how likely it is that your function will fail on the first try. You can run an experiment where you call the function repeatedly until it fails. You find it fails on the 10th attempt. You can use knowledge of probability distributions and a trick for calculating the most likely probability given the experiment you conducted to conclude the obvious: that it has a 1/10 chance of failing on the first go. What the long way gets you, is the chance for it failing after x attempts. So a bit of knowledge gets you a lot more information to make informed decisions with. You could always simulate this experiment in a script to estimate these numbers as well, which I think could be useful to someone with a software engineering background but without the time or resources to learn statistics. 0.https://en.wikipedia.org/wiki/Geometric_distribution https://en.wikipedia.org/wiki/Geometric_distribution 1.Probability dist for the example: https://www.wolframalpha.com/input/?i=geometric+distribution+p%3D0.1 https://www.wolframalpha.com/input/?i=geometric+distribution...