8 ms·
LLMs will fail this simple eval every time
- ifuknowuknow 2y agowe know what we don't know :)
- Jimmc414 2y agoWhat am I missing, it responded with specific examples of what it didn’t know
- nabla9 2y agoFor example: >The exact number of grains of sand on a specific, untouched beach. Just saying "specific" is not being specific. "The exact number of grains of sand on a Source d’Argent beach on Seychelles." Would be specific and exact thing of knowledge it does not have. It fails to name exact, specific thing. It's like it is compelled to be unspecific and not exact and be generic. "any given town.", "undiscovered ancient manuscript", "hypothetical alien civilization's technology", "of a specific person","random individual", "unidentified and uncatalogued species"
- mtmail 2y agoCan you explain the failure?
- ifuknowuknow 2y ago1) can't tell any one specific thing it doesn't know 2) when it does tell a specific thing it doesn't know, it's a thing it actually knows well this train of conversation looks monkeypatched by openai bc it reveals a massive vulnerability in reasoning, but is consistently reproducible across all llms