3 ms·
I've never had these retrieval demos work well. Either the LLM hallucinates content that could plausibly be in the document, or it's `reasoning` is so hamstrun
by vagabund 4y ago
I've never had these retrieval demos work well. Either the LLM hallucinates content that could plausibly be in the document, or it's `reasoning` is so hamstrung by the constraint that it misunderstands the meaning of the specific quote it's referencing.
I just tried this one on the NBA CBA, which I would think is an ideal use case, and it didn't answer a single question correctly. Hopefully we find a more clever way to use LLMs in conjunction with a knowledge-base.
- freediver 4y agoCan you share the link you used and a few questions/correct answers?
- vagabund 4y agoPDF link: https://cosmic-s3.imgix.net/3c7a0a50-8e11-11e9-875d-3d44e94ae33f-2017-NBA-NBPA-Collective-Bargaining-Agreement.pdf https://cosmic-s3.imgix.net/3c7a0a50-8e11-11e9-875d-3d44e94a... "What's a mid-level exception?" (Initially stated the info wasn't in the provided context, but upon asking again, it got it correctly) "How long does a team have to wait after receiving a player in a trade before trading the player again?" (It's 60 days under certain circumstances, it told me 6 months) "What's a traded player exception?" (Claims isn't in the document) "How are traded player exceptions created?" (Answers correctly) So after playing with it a bit more I think I understand the sorts of questions it does and doesn't like to answer, but still, I would think fuzzy matches for section titles should work.