3 ms·
Why are you acting like the digital equivalent of rote memorization is an impressive feat? There are things that are impressive about generative AI, but gettin
by devmor 1y ago
Why are you acting like the digital equivalent of rote memorization is an impressive feat?
There are things that are impressive about generative AI, but getting correct questions on what is essentially a school test that is available online is not one of them.
- AuthAuth 1y agoYeah its not impressive to serve questions to an AI that has the answers to those questions within its dataset.
- scotty79 1y agoPick a single question from math olympiad. Solve it using any online resource beyond the solution to this specific problem itself, AI or service of smart people. Let us know how it went.
- devmor 1y agoWhy would they exclude the solution to the problem and give themselves a handicap that the AI does not have?
- scotty79 1y agoBecause AI didn't have solutions for this year's problem in its training materials. Same way that students participating in this year's Math Olympiad didn't. Guys, don't you know anything about how such competitions work? You get limited time and a set of problems to solve. New problems, that weren't available anywhere before they were given to you. That might be the issue. How can you appreciate what the AI achieved if you don't know anything about it beyond the name?
- devmor 1y agoPer Gregor Dolinar, President of the IMO: > “It is very exciting to see progress in the mathematical capabilities of AI models, but we would like to be clear that the IMO cannot validate the methods, including the amount of compute used or whether there was any human involvement, or whether the results can be reproduced. What we can say is that correct mathematical proofs, whether produced by the brightest students or AI models, are valid.”
- fragmede 1y agoWhich is to say, they didn't give the AI model the answer, the AI model produced one without having seen it, and that it's valid. It might have taken extra time or had outside help, or otherwise cheated, but it didn't have the answer given to it to merely reproduce.
- devmor 1y agoNo such stipulation or claim is made. The only verified claim is that the proof is valid.
- fragmede 1y agobut also no stipulation otherwise is made by the contest runners. As the runners of a contest, I would presume they didn't give the answer to any of the test takers before they took the test because that would defeat the whole point of having the contest! If you think people/machines are cheating absent an explicit claim as such, I can't help you, but that seems unlikely to me. It's entirely possible that both Google and OpenAI independently bribed a question writer for the answers and fed that into the LLM to be used as training data, and have not been discovered doing so, and that only then was the LLM able to generate the correct answer, but that seems a bit far fetched to me.
- scotty79 1y agoSo earlier you said, AI answered with scraped answers. Since you now realize that was not the case, what's your belief now? That OpenAI (and Google) did a full on mechanical turk scam? Because that's what the fragment you cited is saying. That MO can't assure that the answers provided by AI companies were really generated by AI (because they have no way of knowing that).
- thrown-0825 1y agoI think you underestimate how much of what we call intelligence is just memorization of some basic principles and skills.
- devmor 1y agoThere is no “we”, here. What you and other AI hype proponents call intelligence and what I call intelligence are not the same thing.
- thrown-0825 1y agoIntelligence is defined as the ability to acquire, understand, and use knowledge in The American Heritage® Dictionary of the English Language, 5th Edition. Based on that definition I think we are well on out way to building an artificial intelligence that is more capable than the overwhelming majority of the planet.
- devmor 1y agoI don’t believe that you understand what an LLM is - that or you have a problem separating science fiction from reality.
- thrown-0825 1y agoYou should let my employers know, they are paying my salary based on the assumption that I do.
- throwawayoldie 1y agoAnd after all a company has never done anything foolish.
- scotty79 1y agoHow much, do you think, of rote memorization competition level math is? > There are things that are impressive about generative AI, but getting correct questions on what is essentially a school test that is available online is not one of them. Nevermind. Try to solve one. You'll be enlightened.
- devmor 1y agoI don’t need to try to solve one. The answers are available online. That is my entire point.
- scotty79 1y agoYou need to solve one to put yourself in the shoes of the AI model to understand its achievement. When the models and students got the puzzles the answers weren't available online yet.