3 ms·
I don't see a description on the GitHub page of any experiments performed showing the model can pass an exam. Where are these details, and how was the testing p
by ubj 3y ago
I don't see a description on the GitHub page of any experiments performed showing the model can pass an exam. Where are these details, and how was the testing performed?
EDIT: The git commit was 4c6d52a when I originally posted this comment.
- bomewish 3y agoAgree. Seems highly irresponsible to make such claims without demonstrating and validating them. Would also like to see how much it hallucinates or how closely the generations are to source data.
- avereveard 3y agoin the notebook there is a passage where it goes trough the exam dataset and use an embedding model to compare answer. I don't think that's the accepted methodology (I think they ought to compare the logist weight on the answers keys), on top of that, the embedding model they use for similarity is somewhat weak. also passing in their own graph means 61% right or something like that.