3 ms·
Can anyone answer the chance that example tests of these questions were in its training set? And it's just regurgitating the answers someone else wrote? As I
by mattmanser 4y ago
Can anyone answer the chance that example tests of these questions were in its training set?
And it's just regurgitating the answers someone else wrote?
As I imagine it's a very high chance given how much uni lecturers recycle exam questions.
When I was at uni you could just get the last 5 years worth of questions from the library for almost any subject and guess what the questions were probably going to be. Often they just changed a few numbers.
Teaching undergrads is like a sausage factory, the actual intellectual value for undergrads is in the seminars, the practical value in the labs. The rest is showing you can regurgitate what you've been told.
Which ChatGPT excels at.
- skepticATX 4y agoI don’t know anything about quantum computing. I successfully answered a few true/false questions by pasting them into Google. The exact questions aren’t in the top results, but the answers are.
- sudosysgen 4y agoYes, the vast majority of these questions are standard known problems it definitely already saw with a slightly different formulation.
- moonchrome 4y agoYou can try phrasing the question in a way that it wouldn't be phrased but would still demonstrate understanding of concept. I remember Yann LeCun gave an interview and he came up with some random question like "If I'm holding a peace of paper with both of my hands above the desk and I release one what would happen". His point was that since the LLM doesn't have a world model it wouldn't be able to answer these trivial intuitive questions unless it saw something similar in the training set. And then the interviewer tried it and it failed. That was 3.5. I've tried many variation of that class of problem with 4 and it seems to generalize basic physics concepts quite well. So maybe 4 learned basic physics ? Why couldn't it learn QM theory as well ?
- M4v3R 4y agoFrom the article: > To the best of my knowledge—and I double-checked—this exam has never before been posted on the public Internet, and could not have appeared in GPT-4’s training data.
- sudosysgen 4y agoThe exam, no, but most of the questions most certainly are. I know this because I've done extremely similar problems for homework and checked my answers online.