3 ms·
The most interesting thing here to me is the leaderboard, because they actually included the estimated price per game. Gemini gets the highest score with a fai
by jedberg 8mo ago
The most interesting thing here to me is the leaderboard, because they actually included the estimated price per game. Gemini gets the highest score with a fairly reasonable cost (about 1/3 of the way down).
- GregorStocks 8mo agoTo be clear, that's not estimated price, it's actual price I paid across all the real games. My hope is you'll see it trend down over time as I find more ways to make the harness token-efficient :)
- jedberg 8mo agoThat's even more interesting then! It would be cool if you added a price to performance column. Even if it's just for this one task, it's still interesting.
- GregorStocks 8mo agoPerformance is tricky to measure. Right now the best measure of performance I've got is the "blunder index", but that's currently flagging a lot of stuff that I really don't consider to be true blunders - I think my top priority for the next few evenings is going to be iterating on the blunder-annotator, and that'll help me identify what issues in the actual gameplay code to focus on. And the blunder index isn't really defined in such a way that you can do arithmetic on it meaningfully :)