5 ms·

I don't think this is player stupidity at all. Actually, I suspect it's perfectly rational. The game is so noisy that no one can really predict what will happe

by Gunax 5y ago

I don't think this is player stupidity at all. Actually, I suspect it's perfectly rational. The game is so noisy that no one can really predict what will happen. If I hit the grue with my sword--it could reasonably do 10% of damage or 40%--and both seem plausible. In a statistical sense, the greater the standard deviation is, the more difficult it is to detect a sample population that differs from the established distribution. In a bayesian sense, the players may have been perfectly rational--the likelihood of doubling or tripling your winrate is just not very unlikely when you're playing a game which is very random. Perhaps players can rationally conclude they were in the top 1% of luck at the moment, and that the chance of a major bug is less than that. As evidenced by the referenced posts, it only becomes obvious when the sample size is huge (eg. the sample of all players reading the subreddit).

ajuc 5y ago
Yes. It's very similar to when you encounter compiler or operating system or hardware bug as a programmer. You are trained for years to NEVER BLAME THE LOWER LAYERS because that makes you a worse programmer. So when it is actually their fault - you might look stupid for exhausting every other possibility before checking these basic assumptions. But this is still a good rule, because it breaks very rarely.
- Retric 5y ago
  Reminds me of the utter hell it was to discover the test machine had bad RAM.
  - ajuc 5y ago
    I've had a bug in linux driver for a very shitty touchscreen we used in our embedded devices. It needed manual calibration and you had to save the calibration matrix in a config file read on system startup. They used locale-aware input function in a driver so when the locale was German (or other with "," as a decimal separator) it broke and users got kernel panic at startup :) It bricked all our devices in Germany. We worked around it, notified the device driver developers and forgot about it. A year later we shipped with new version of kernel and it broke again - because the bug was fixed in kernel and our workaround was now causing the bug :)
- hinkley 5y ago
  This is part of what drove people to CI/CD. If your code is broken in the morning and the build is green, it’s probably you. If it’s red, maybe you should check other things sooner?
eutectic 5y ago
Assuming your win rate goes from 2% -> 4%, you need around 200 samples to be 90% confident that something is up, 270 to be 95% confident, and 450 to be 99% confident, if you start with a Beta(2, 98) prior. Obviously results will change somewhat depending on the model.
- gnramires 5y ago
  From the article: > Parthenocarpy: > Well that explains me going from a 2% winrate to 17.95% in the span of two weeks In that case, as soon as you got more than 1 win you should notice an abnormality (according to back-of-the napkin 8 trials to perceive the difference, using frequentist stat). Also, that assumes you only get information from win/loss. In reality, every interaction (monster damage) should provide information (too easy kills, abnormal damage).
  - eutectic 5y ago
    For that drastic a difference you need to play about 30 games on average to be 99% sure that p_win>0.02 in the Bayesian model. Agreed that win rates are not the only data available, and that players probably should have noticed the difference. Attributing it to a bug is arguably a harder problem.
jerf 5y ago
One of the most interesting insights I retained from my machine learning course from a couple of decades ago is that you can put a mathematical bound on how quickly you can learn from a given set of data, and how much you can learn. In the situations presented by the real world this is often not very interesting because we can only process such a vanishing fraction of the data available that we're so far below this theoretical optimum that it's uninteresting, but in more constrained situations it can be relevant. In this case, while I agree that there was enough data to have a good guess that something had gone wrong after a certain period of time, it definitely is the sort of noisy data that would take some non-trivial aggregation to be sure, and even in hindsight I'm not so sure the signal was so strong that it's worth metaphorically beating anyone up about. Moreover, only in hindsight is it obvious that the culprit was a doubling of melee damage. Prior to knowing that, the learning rate was constrained by the multiplicity of possible theories that could have explained this. It's not like it was simply a matter of "Either 1. the game is unchanged and people have just Got Gud or 2. the melee damage rate has been doubled", the span of possibilities was much larger than that. Had an oracle come down from whereever and offered that choice to the devs, sure, gathering enough data to resolve that one bit's worth of information would have been easy. But presented with the full, massive array of possibilities, it's a much harder task to even determine that there is a problem, let alone what it is. Upshot, don't beat yourselves up too much. It may be obvious in hindsight but these are really quite hard problems to solve in the deliberate absence of information.

by Gunax 5y ago

ajuc 5y ago

Yes. It's very similar to when you encounter compiler or operating system or hardware bug as a programmer. You are trained for years to NEVER BLAME THE LOWER LAYERS because that makes you a worse programmer. So when it is actually their fault - you might look stupid for exhausting every other possibility before checking these basic assumptions. But this is still a good rule, because it breaks very rarely.

Retric 5y ago
Reminds me of the utter hell it was to discover the test machine had bad RAM.
- ajuc 5y ago
  I've had a bug in linux driver for a very shitty touchscreen we used in our embedded devices. It needed manual calibration and you had to save the calibration matrix in a config file read on system startup. They used locale-aware input function in a driver so when the locale was German (or other with "," as a decimal separator) it broke and users got kernel panic at startup :) It bricked all our devices in Germany. We worked around it, notified the device driver developers and forgot about it. A year later we shipped with new version of kernel and it broke again - because the bug was fixed in kernel and our workaround was now causing the bug :)
hinkley 5y ago
This is part of what drove people to CI/CD. If your code is broken in the morning and the build is green, it’s probably you. If it’s red, maybe you should check other things sooner?

eutectic 5y ago

Assuming your win rate goes from 2% -> 4%, you need around 200 samples to be 90% confident that something is up, 270 to be 95% confident, and 450 to be 99% confident, if you start with a Beta(2, 98) prior. Obviously results will change somewhat depending on the model.

gnramires 5y ago
From the article: > Parthenocarpy: > Well that explains me going from a 2% winrate to 17.95% in the span of two weeks In that case, as soon as you got more than 1 win you should notice an abnormality (according to back-of-the napkin 8 trials to perceive the difference, using frequentist stat). Also, that assumes you only get information from win/loss. In reality, every interaction (monster damage) should provide information (too easy kills, abnormal damage).
- eutectic 5y ago
  For that drastic a difference you need to play about 30 games on average to be 99% sure that p_win>0.02 in the Bayesian model. Agreed that win rates are not the only data available, and that players probably should have noticed the difference. Attributing it to a bug is arguably a harder problem.

jerf 5y ago

One of the most interesting insights I retained from my machine learning course from a couple of decades ago is that you can put a mathematical bound on how quickly you can learn from a given set of data, and how much you can learn. In the situations presented by the real world this is often not very interesting because we can only process such a vanishing fraction of the data available that we're so far below this theoretical optimum that it's uninteresting, but in more constrained situations it can be relevant. In this case, while I agree that there was enough data to have a good guess that something had gone wrong after a certain period of time, it definitely is the sort of noisy data that would take some non-trivial aggregation to be sure, and even in hindsight I'm not so sure the signal was so strong that it's worth metaphorically beating anyone up about. Moreover, only in hindsight is it obvious that the culprit was a doubling of melee damage. Prior to knowing that, the learning rate was constrained by the multiplicity of possible theories that could have explained this. It's not like it was simply a matter of "Either 1. the game is unchanged and people have just Got Gud or 2. the melee damage rate has been doubled", the span of possibilities was much larger than that. Had an oracle come down from whereever and offered that choice to the devs, sure, gathering enough data to resolve that one bit's worth of information would have been easy. But presented with the full, massive array of possibilities, it's a much harder task to even determine that there is a problem, let alone what it is. Upshot, don't beat yourselves up too much. It may be obvious in hindsight but these are really quite hard problems to solve in the deliberate absence of information.