4 ms·

OpenAI codenamed one of their models "Project Strawberry" and IIRC, Sam Altman himself was taking a victory lap that it can count the number of "r"s in "strawbe

OpenAI codenamed one of their models "Project Strawberry" and IIRC, Sam Altman himself was taking a victory lap that it can count the number of "r"s in "strawberry". Which I think goes to show that it's hard to distinguish between LLMs getting genuinely better at a class of problems versus just being fine-tuned for a particular benchmark that's making rounds.

zahlman 1y ago
See also the various wolf/goat/cabbage benchmarks, or the crossing a bridge at various speeds with limited light sources benchmarks.
KeplerBoy 1y ago
It gets strawberry right though, so I guess we are only one project blueberry from getting one step closer to AGI.