4 ms·
https://astralcodexten.substack.com/p/heuristics-that-almost-always-work https://astralcodexten.substack.com/p/heuristics-that-almost...
by ssivark 5y ago
https://astralcodexten.substack.com/p/heuristics-that-almost-always-work https://astralcodexten.substack.com/p/heuristics-that-almost...
- nikanj 5y agoI was just coming here to post the same link. The 95% accuracy should be compared to the general approval rate - does the fancy AI beat a simple return true?
- zwaps 5y agoThe logistic regression would likely pick up on that if it were a stratification issue.
- kqr 5y agoYup. I always advocate that the first model one should build is the one that constantly returns the most common answer. That way, you can compare the more fancy stuff to something to see whether you're really improving. You also get to evaluate the economic gains of the more sophisticated model against the development, maintenance, and data costs of it, compared to the dumb one. (Another good baseline is returning a random historic result in proportion to how often it occurs. It sometimes helps against exploit attempts at the expense of data.)