3 ms·
Seems like if you want to argue that LLMs have an intrinsic failure mode you should find an example using the best model available, not the 4th best.
by sebzim4500 3y ago
Seems like if you want to argue that LLMs have an intrinsic failure mode you should find an example using the best model available, not the 4th best.
- godelski 3y agoWhen you can tell me what the best model available is I'll do that. But as an ML researcher, I can't even tell you that. There is no best, but there are "better at certain tasks."
- sebzim4500 3y agoGPT-4 is by far the best 'general purpose' LLM and, at a mimimum, any claim about instrinsic flaws in LLMs absolutely needs to use this as a test case. If you want to go above that low bar then you could also test with the #2 and #3 model according to the chat LLM arena [1], currently claude v1 and claude v1 instant. In a perfect world you would also test against med-palm-2, which ostensibly is the least likely to hallucinate but unfortunately none of us have access to this. [1] https://chat.lmsys.org/?arena https://chat.lmsys.org/?arena
- godelski 3y agoI'm not sure why you felt the need to link that. I'm aware of it as well as that GPT-4 isn't on the list and that the top is a LLaMA tuned model. But I'm not going to get into the rabbit hole of the difficulties of evaluation. Especially when arena exists due to this difficulty and even writes about such. We're going to just have to agree to disagree I guess because I'm not going to agree that there's a definitive singular metric that determines "best." The intrinsic flaws I'm discussing are well known in the research community and not something we really debate about. That only seems to happen when discussing in more public settings.