4 ms·
"initial human evaluation" is codeword for "cherrypicked prompts given to people who don't know how to trick an LLM"
by atleastoptimal 4y ago
"initial human evaluation" is codeword for "cherrypicked prompts given to people who don't know how to trick an LLM"
- sebzim4500 4y agoI don't understand why that is a bad thing. If your goal is to make an AI assistant, then you should be optimizing for giving answers that real users find useful, not trying to impress other AI researchers.
- ChubbyGlasses 4y agoi always found this to be a strange pov to have on LLMs. imo, it's not humans tricking/gaming the ai, but rather chatgpt has tricked you into believing it's smarter than it actually is. (in human terms, chatgpt is just more articulate than llama) it's a subtle distinction, but i think it shapes and reflects how you view ai as a tool for humans or as a replacement.
- bitL 4y agoThere aren't any good automated metrics for generative LLM quality, so it's all up to humans to evaluate.