3 ms·

Then why dont agents work? If those skills were real, why do they fizzle out on production data ?

• TeMPOraL 3y ago
  That's a good question. What is different about "production data"? What do those "production people" do that suddenly makes LLMs fail on things they work well on when not "production"?
  • deleted 3y ago
    [deleted]
  • intended 3y ago
    I can tell you who would love to hear your answer to that question. Me for starters. If it works, I can quit. Next up are Karpathy and the CTO of OpenAI. Around July and September both talked about production challenges. AI ops was the largest subcategory of fall YC startups. Every single ml and LLM ops individual who gets far enough deals with evals. I don’t know, but maybe - just maybe- the issue is that people don’t understand themselves enough, to avoid assuming too much of those emergent properties. As I recall there was also a paper that pointed out the issues with how LLMs are measured, and that the emergence of properties was not a step change once the tests were updated. Edit- found it: https://hai.stanford.edu/news/ais-ostensible-emergent-abilities-are-mirage https://hai.stanford.edu/news/ais-ostensible-emergent-abilit...