5 ms·

Seems like having tests would've easily solved this issue.

Supermancho 5y ago
The question is "why didn't the players notice" not why the developers didn't notice. The article leads with a clickbait question. The reason they didn't notice is because the stats aren't readily available and public to everyone, to see the sharp uptick in wins paired with the game being filled with opaque systems. No player playing for the first time would notice or have a chance to notice. To put it another way, this is an article about a (secret and mistaken) massive change to a very difficult game that made it slightly less difficult. Same article turns around and asks why nobody noticed, like that's an interesting question.
- hnxs 5y ago
  “Slightly less difficult” is a huge understatement. I don’t know specific stats, but I’d say the average player had their winrates at least double (eg 2% to 4%)
  - cortesoft 5y ago
    It is not easy to notice a change of 2% success to 4% success without a lot of samples.
- djmips 5y ago
  I thought it was an interesting question followed by an interesting take. There was a bit of ego salve in the conclusion but it seemed plausible.
- thaumasiotes 5y ago
  > The question is "why didn't the players notice" not why the developers didn't notice. No, lots of players noticed. The question is phrased as "how did the players miss this?", but that's not what the author means - what he means is "why didn't the community come to a consensus on what had happened within two weeks of the introduction of the change?" Which is a much stupider question.
  - mandmandam 5y ago
    > Which is a much stupider question. I don't think it's a stupid question at all; not when the effect is so large. It was literally double what it should be, leading to a 3x overall win rate. Coming from Dota 2, if there was a bug like this it would be caught within minutes, if not seconds, and patched within an hour or two. - Yes, the Dota player base is orders of magnitude larger, and the damage numbers are easily available. Also I have never played Crawl.
  - tsimionescu 5y ago
    It's not a stupid question at all, it's actually a good pretext to discuss the difficulties inherent in reasoning about an opaque system you are interacting with. The article also brings evidence that many players didn't notice or believe that anything changed, they specifically believed that nothing did change, which is a surprising belief for such a huge change - but understandable in the context of of such an opaque system. I also believe that other players noticed something had made the game easier, but no one realized what specifically had changed, or most likely suspected how big of a change it was.
    - thaumasiotes 5y ago
      > The article also brings evidence that many players didn't notice or believe that anything changed, they specifically believed that nothing did change, which is a surprising belief for such a huge change Not really. The article cites responses in online discussion to that effect. But there is no evidence that those responses came from someone who had ever played the altered version of the game. With certainty, many of them didn't. You don't have to do a playthrough on the latest version before leaving a new comment in a forum thread. It's a safe bet that the majority of the player base never encountered the bug at all, since it was only available if you downloaded the game within a two-week window.
- kibwen 5y ago
  > the stats aren't readily available and public to everyone The stats have always been public and instantly available, impressively so. Here's the bot command to query the stats in question: !lg * start>"2015-03-06 00:00:00" start<"2015-03-21 00:00:00" cv=0.16 / won Output: 1180/39437 games for * (start>'2015-03-06 00:00:00' start<'2015-03-21 00:00:00' cv=0.16): N=1180/39437 (2.99%) So that's a 3% winrate for the major version in question during the dates the bug existed. Using another query to determine the all-time winrate across all versions: !lg * / won We see the historic winrate to be about 1%. The listgame interface is documented at https://github.com/crawl/sequell/blob/master/docs/listgame.md https://github.com/crawl/sequell/blob/master/docs/listgame.m... , and the bots can be found on IRC and Discord.
muzani 5y ago
This is an interesting idea. Let's say there's a new feature where electric damage and cold damage ignores metal armor. But suddenly there's a bug where if the weapon does both electric and cold damage, the physical damage also ignores armor. How would someone design tests that work well for roguelikes? It's not simply Human strikes Orc for 10 damage. Damage range may be a bell curve of 5-15. There's misses and hits. There's little bonuses that increase the miss probability, probably to ridiculous levels. There's armor, calculated from things like race, constitution, magic, divine bonuses, class mastery bonus. And then you have procedurally generated... stuff. Equipment are the easiest to control. Some games have procedurally generated monsters, some have procedurally generated deities. To be able to reduce this stuff to tests kind of defeats the purpose of doing it, which is to be unpredictable. Sometimes you have a combo of things that add up to 40% damage resistance. Is 40% too much or too little? After all this, how do you detect that the bugged spear is doing an unreasonable amount of damage?
- personjerry 5y ago
  In cases like this we use mock functions to replace the rng dependent functions during testing. https://en.m.wikipedia.org/wiki/Mock_object https://en.m.wikipedia.org/wiki/Mock_object
  - deleted 5y ago
    [deleted]
  - muzani 5y ago
    This won't catch the procedurally generated edge case though unless it mocks every possible combination. It could well run into millions of scenarios just to test damage.
- a_e_k 5y ago
  (a) Set a deterministic RNG seed before each test. and/or (b) Have each test loop enough to get a good estimate of the probability distribution.
  - 5y ago
soared 5y ago
> try to design an experiment that would prove whether the value written in the system is correct. If designing the experiment is very hard, that should be interpreted as a risk factor - if no one can check that the system is doing what it ought to, then maybe it really is wrong!
nimih 5y ago
It sure would be inconvenient for your hypothesis if the crawl project had a `test` directory which predates the bug in question...

The question is "why didn't the players notice" not why the developers didn't notice. The article leads with a clickbait question. The reason they didn't notice is because the stats aren't readily available and public to everyone, to see the sharp uptick in wins paired with the game being filled with opaque systems. No player playing for the first time would notice or have a chance to notice. To put it another way, this is an article about a (secret and mistaken) massive change to a very difficult game that made it slightly less difficult. Same article turns around and asks why nobody noticed, like that's an interesting question.

hnxs 5y ago
“Slightly less difficult” is a huge understatement. I don’t know specific stats, but I’d say the average player had their winrates at least double (eg 2% to 4%)
- cortesoft 5y ago
  It is not easy to notice a change of 2% success to 4% success without a lot of samples.
djmips 5y ago
I thought it was an interesting question followed by an interesting take. There was a bit of ego salve in the conclusion but it seemed plausible.
thaumasiotes 5y ago
> The question is "why didn't the players notice" not why the developers didn't notice. No, lots of players noticed. The question is phrased as "how did the players miss this?", but that's not what the author means - what he means is "why didn't the community come to a consensus on what had happened within two weeks of the introduction of the change?" Which is a much stupider question.
- mandmandam 5y ago
  > Which is a much stupider question. I don't think it's a stupid question at all; not when the effect is so large. It was literally double what it should be, leading to a 3x overall win rate. Coming from Dota 2, if there was a bug like this it would be caught within minutes, if not seconds, and patched within an hour or two. - Yes, the Dota player base is orders of magnitude larger, and the damage numbers are easily available. Also I have never played Crawl.
- tsimionescu 5y ago
  It's not a stupid question at all, it's actually a good pretext to discuss the difficulties inherent in reasoning about an opaque system you are interacting with. The article also brings evidence that many players didn't notice or believe that anything changed, they specifically believed that nothing did change, which is a surprising belief for such a huge change - but understandable in the context of of such an opaque system. I also believe that other players noticed something had made the game easier, but no one realized what specifically had changed, or most likely suspected how big of a change it was.
  - thaumasiotes 5y ago
    > The article also brings evidence that many players didn't notice or believe that anything changed, they specifically believed that nothing did change, which is a surprising belief for such a huge change Not really. The article cites responses in online discussion to that effect. But there is no evidence that those responses came from someone who had ever played the altered version of the game. With certainty, many of them didn't. You don't have to do a playthrough on the latest version before leaving a new comment in a forum thread. It's a safe bet that the majority of the player base never encountered the bug at all, since it was only available if you downloaded the game within a two-week window.
kibwen 5y ago
> the stats aren't readily available and public to everyone The stats have always been public and instantly available, impressively so. Here's the bot command to query the stats in question: !lg * start>"2015-03-06 00:00:00" start<"2015-03-21 00:00:00" cv=0.16 / won Output: 1180/39437 games for * (start>'2015-03-06 00:00:00' start<'2015-03-21 00:00:00' cv=0.16): N=1180/39437 (2.99%) So that's a 3% winrate for the major version in question during the dates the bug existed. Using another query to determine the all-time winrate across all versions: !lg * / won We see the historic winrate to be about 1%. The listgame interface is documented at https://github.com/crawl/sequell/blob/master/docs/listgame.md https://github.com/crawl/sequell/blob/master/docs/listgame.m... , and the bots can be found on IRC and Discord.

muzani 5y ago

This is an interesting idea. Let's say there's a new feature where electric damage and cold damage ignores metal armor. But suddenly there's a bug where if the weapon does both electric and cold damage, the physical damage also ignores armor. How would someone design tests that work well for roguelikes? It's not simply Human strikes Orc for 10 damage. Damage range may be a bell curve of 5-15. There's misses and hits. There's little bonuses that increase the miss probability, probably to ridiculous levels. There's armor, calculated from things like race, constitution, magic, divine bonuses, class mastery bonus. And then you have procedurally generated... stuff. Equipment are the easiest to control. Some games have procedurally generated monsters, some have procedurally generated deities. To be able to reduce this stuff to tests kind of defeats the purpose of doing it, which is to be unpredictable. Sometimes you have a combo of things that add up to 40% damage resistance. Is 40% too much or too little? After all this, how do you detect that the bugged spear is doing an unreasonable amount of damage?

personjerry 5y ago
In cases like this we use mock functions to replace the rng dependent functions during testing. https://en.m.wikipedia.org/wiki/Mock_object https://en.m.wikipedia.org/wiki/Mock_object
- deleted 5y ago
  [deleted]
- muzani 5y ago
  This won't catch the procedurally generated edge case though unless it mocks every possible combination. It could well run into millions of scenarios just to test damage.
a_e_k 5y ago
(a) Set a deterministic RNG seed before each test. and/or (b) Have each test loop enough to get a good estimate of the probability distribution.
- 5y ago

soared 5y ago

> try to design an experiment that would prove whether the value written in the system is correct. If designing the experiment is very hard, that should be interpreted as a risk factor - if no one can check that the system is doing what it ought to, then maybe it really is wrong!

nimih 5y ago

It sure would be inconvenient for your hypothesis if the crawl project had a `test` directory which predates the bug in question...