5 ms·
Passing the LSAT with no time limit and a copy of the training material in front of you is not an achievement. Anybody here could have written code to pass the
by phphphphp 4y ago
Passing the LSAT with no time limit and a copy of the training material in front of you is not an achievement. Anybody here could have written code to pass the LSAT. Standardised tests are only hard to solve with technology if you add a bunch of constraints! Standardised tests are not a test of intelligence, they’re a test of information retention — something that technology has been able to out perform humans on for decades. LLMs are a bridge between human-like behaviour and long established technology.
- scotty79 4y agoWhy don't you show your program then that does 90% on LSAT?
- phphphphp 4y agoSend me the answer key and I’ll write you the necessary =VLOOKUP().
- speedgoose 4y agoYour program has to figure it out.
- awestroke 4y agoConsidering your username, I'm not surprised that you have completely misunderstood what an LLM is. There is no material or data stored in the model, just weights in a network
- kurisufag 4y agoweights are data relationships made totally quantitative. imagine claiming the human brain doesn't hold data simply because it's not in readable bit form.
- phphphphp 4y agoI know what an LLM is. My point is that “doesn’t have the data in memory” is a completely meaningless and arbitrary constraint when considering the ability to use technology to pass a standardised test. If you can explain why weights in a network is a unique threat to standardised tests, compared to, say, a spreadsheet, please share.
- Auracle 4y agoIt's not that standardized tests are under threat. It's that those weights in a network are significantly more similar to how our brains work than a spreadsheet and similarly flexible.
- chairhairair 4y agoYou honestly believe you could hand write code to pass an arbitrary LSAT-level exam?
- phphphphp 4y agoYou’ve added a technical constraint. I didn’t say arbitrary. Standardised tests are standard. The point is that a simple lookup is all you need. There’s lots of interesting aspects to LLMs but their ability to pass standardised tests means nothing for standardised tests.
- chairhairair 4y agoYou think that it’s being fed questions that it has a lookup table for? Have you used these models? They can answer arbitrary new questions. This newest model was tested against tests it hasn’t seen before. You understand that that isn’t a lookup problem, right?
- phphphphp 4y agoThe comment I replied to suggested that the author was fearful of what LLMs meant for the future because they can pass standardised tests. The point I’m making is that standardised tests are literally standardised for a reason: to test information retention in a standard way, they do not test intelligence. Information retention and retrieval is a long solved problem in technology, you could pass a standardised test using technology in dozens of different ways, from a lookup table to Google searches. The fact that LLMs can complete a standardised test is interesting because it’s a demonstration of what they can do but it has not one iota of impact on standardised testing! Standardised tests have been “broken” for decades, the tests and answers are often kept under lock and key because simply having access to the test in advance can make it trivial to pass. A standardised test is literally an arbitrary list of questions. You’re arguing a completely different point.
- chairhairair 4y agoI have no idea what you are talking about now. You claimed to be able to write a program that can pass the LSAT. Now it sounds like you think the LSAT is a meaningless test because it... has answers? I suspect that your own mind is attempting to do a lookup on a table entry that doesn't exist.