3 ms·

Humanity's Last Exam

• deleted 2y ago
  [deleted]
• deleted 2y ago
  [deleted]
• hdjjhhvvhga 2y ago
  > In Greek mythology, who was Jason's maternal great-grandfather? These tests are getting ridiculous (especially when you take into account the name they choose for them...). For me things are quite simple. I witness the limitations of LLMs every day. What this benchmark checks if what data it was trained on, that's pretty much it. No need to get "humanity" involved into this.
  • baal80spam 2y ago
    To me, comments like yours prove that goal posts are constantly moved every day.
    • sottol 2y ago
      But there's also been substantially more investment now than say 2 years ago (AI + AI-adjecent companies have, what, 4-5 trillion USD valuation now?). CEOs tout how they stopped hiring. And then these benchmarks having names like "humanity's last exam". All that implies LLMs are a direct replacement for human workers so imo it's fair to measure LLMs against the hype/promises. So I would say the goalposts are moving just in step with the market expectations/hype. Nobody's disputing that LLMs are amazing at generating useful text anymore.