3 ms·

He actually says " I'm not advocating that you shouldn't put in tests. [ The reason we can get away with this ] is that we have a great community. " I take thi

He actually says " I'm not advocating that you shouldn't put in tests. [ The reason we can get away with this ] is that we have a great community. " I take this to mean that he feels that StackOverflow doesn't need tests. Not that tests are useless.

kmontrose 13y ago
That's an accurate read. - Stack Exchange employee
BrandonY 13y ago
User community as testers presents some interesting pros and cons. Pros: * Tests are self-updating. Add a new feature: tests come in for free. Change a feature: tests automatically update. Fail to document a change: tests fail. * Tests are unusually thorough * Eventually consistent testing. If nobody ever complains, it probably wasn't a bug worth fixing. Cons: * Tests cannot be run offline. Feature must be committed and deployed before tests can be run. * Potentially large quantity of false positives (bad bug reports) * Potentially large quantity of false negatives (nobody notices particular bug, release considered good) * Does not work for non-user-visible features So basically you trade the reliability of your tests for a substantial build/release speedup. Some users experience each bug, but they are the users who are actively using the meta-community and have signed up to experience more bugs. Still, lack of pre-release unit testing must radically increase the importance of VERY careful code reviews. Not the decision I would have made, but definitely has the sorts of advantages that a small team of engineers drool would drool over.
- sklivvz1971 13y ago
  Remember that our community writes bug reports but also vets bug reports. We rarely have to deal with bad reports. Interestingly, large quantities of false negatives are a non-issue.
  - welegan 13y ago
    Presumably the same reason why they don't have a ton of bad questions on stack overflow: their community scoring would apply just as much to bug reports

kmontrose 13y ago

That's an accurate read. - Stack Exchange employee

BrandonY 13y ago

User community as testers presents some interesting pros and cons. Pros: * Tests are self-updating. Add a new feature: tests come in for free. Change a feature: tests automatically update. Fail to document a change: tests fail. * Tests are unusually thorough * Eventually consistent testing. If nobody ever complains, it probably wasn't a bug worth fixing. Cons: * Tests cannot be run offline. Feature must be committed and deployed before tests can be run. * Potentially large quantity of false positives (bad bug reports) * Potentially large quantity of false negatives (nobody notices particular bug, release considered good) * Does not work for non-user-visible features So basically you trade the reliability of your tests for a substantial build/release speedup. Some users experience each bug, but they are the users who are actively using the meta-community and have signed up to experience more bugs. Still, lack of pre-release unit testing must radically increase the importance of VERY careful code reviews. Not the decision I would have made, but definitely has the sorts of advantages that a small team of engineers drool would drool over.

sklivvz1971 13y ago
Remember that our community writes bug reports but also vets bug reports. We rarely have to deal with bad reports. Interestingly, large quantities of false negatives are a non-issue.
- welegan 13y ago
  Presumably the same reason why they don't have a ton of bad questions on stack overflow: their community scoring would apply just as much to bug reports