2 ms·
> In Greek mythology, who was Jason's maternal great-grandfather? These tests are getting ridiculous (especially when you take into account the name they choos
by hdjjhhvvhga 2y ago
> In Greek mythology, who was Jason's maternal great-grandfather?
These tests are getting ridiculous (especially when you take into account the name they choose for them...).
For me things are quite simple. I witness the limitations of LLMs every day. What this benchmark checks if what data it was trained on, that's pretty much it. No need to get "humanity" involved into this.
- baal80spam 2y agoTo me, comments like yours prove that goal posts are constantly moved every day.
- sottol 2y agoBut there's also been substantially more investment now than say 2 years ago (AI + AI-adjecent companies have, what, 4-5 trillion USD valuation now?). CEOs tout how they stopped hiring. And then these benchmarks having names like "humanity's last exam". All that implies LLMs are a direct replacement for human workers so imo it's fair to measure LLMs against the hype/promises. So I would say the goalposts are moving just in step with the market expectations/hype. Nobody's disputing that LLMs are amazing at generating useful text anymore.