3 ms·
We estimate it based on when we start being evenly-matched against teams with a given average MMR. (As you might expect, our usual pattern is to lose to a test
by gdb 8y ago
We estimate it based on when we start being evenly-matched against teams with a given average MMR. (As you might expect, our usual pattern is to lose to a test team consistently, then start being evenly matched, then consistently beat them.) The version of this diagram on our Five page also include bars indicating when we started and stopped testing against various teams: https://openai.com/five/ https://openai.com/five/.
We don't claim this is perfect, which is why we label the chart "estimated MMR".
- shawn 8y agoAh, that seems like an excellent system. It removes all of the guesswork of some formula-based solution. Empirical observations tend to be more accurate. It might be worth hiring one of the top dota pros to coach OpenAI's bots: they could point out each small mistake that the bots made which pros wouldn't make. That might make it more tractable to beat the top team next year. (The actual dota coaches might be even better for this purpose than the pros, too.) Looking forward to the outcome! The goal of "beat the top dota team" is one of those ambitious ones that few companies take on. It's really interesting to see the incremental progress.
- Buge 8y agoHow would the pro point out the mistakes to the bot? The bot cannot understand human language.
- shawn 8y agoNot to the bot, but rather write down the mistakes and decide (as an AI programmer) how best to address them. The solutions would probably take the form of adjusting weights, adding additional dimensions, removing or refactoring existing signals, or deciding that extra training would naturally solve the problem.
- deleted 8y ago[deleted]