8 ms·
It can even solve IQ tests...I mean, how much further are we moving the goal post? Is there a model that can solve differential equations symbolically and nume
by ducktective 4y ago
It can even solve IQ tests...I mean, how much further are we moving the goal post?
Is there a model that can solve differential equations symbolically and numerically? Most of modern engineering just boils down to diff.eqs whether ordinary or partial. It's our current best method to reason about stuff and control them.
- scotty79 4y agoYeah, that could be good. I think LLMs will start to be really useful when they start to do math at human level. When this happens, the sky is the limit.
- Y_Y 4y agoIt's pretty big by any standards, but you may find the work of Gradshteyn and Ryzhik solves this problem nicely.
- didntreadarticl 4y agoIts crap at the visual Raven IQ test though, it scores 22% vs an algorithm that takes random guesses scoring 17%.
- kalium-xyz 4y agoBet you 5 bucks I can train one that gets 100%. Just gotta train it on the ravens answer key.
- thenaturalist 4y agoI'd be cautiuous with such general statements given the rapid pace of development in this area. Benchmark shelf lives aren't that long. You ommitted the fact that tuning bumped it to 26% vs random. Sure, questionable what effort is involved in that step, but at the same time, that hints to me that tuning will be the new baseline within the next 12-24 months.
- didntreadarticl 4y agoSure I would expect it to improve. But it was a bit fishy how 'it took an IQ test!' is in all the highlights but then they mumble quietly about the score that it actually got and hope no-one is listening to that bit. Its notable that it was able to attempt it at all I suppose.
- nmarinov 4y agoSemi related, there's a (pretty good) course at OMSCS where the main project is building an agent to solve RPM problems: https://lucylabs.gatech.edu/kbai/spring-2023/project-overview/ https://lucylabs.gatech.edu/kbai/spring-2023/project-overvie... And quite a lot of papers about that: https://scholar.google.com/scholar?q=%22raven%27s+progressive+matrices%22+%22artificial+intelligence%22&hl=en&as_sdt=0,11 https://scholar.google.com/scholar?q=%22raven%27s+progressiv...
- seydor 4y agoI prefer not focusing on games and benchmarks. Hopefully we ll get to robotics soon
- moffkalast 4y agoRoger roger.
- brookst 4y agoArnold was great in that documentary!
- mhh__ 4y agoWriting down the equations is the task for the AI.
- bootsmann 4y ago> It can even solve IQ tests...I mean, how much further are we moving the goal post? The problem with test like this is that when trained on the existing big datasets (commoncrawl etc.), chances are the test is already in the input so the validation is not proper. Its the same thing with all the "AI beats SAT" headlines. The exercises for those very tests exist all over the internet already.
- nayroclade 4y agoIt's well documented that these models can solve variations of questions that are not found anywhere in their training set, and even entirely novel problems invented by prompters. Not with 100% success, but they can do it far with a rate far better than chance, so the idea that they're pulling responses from their training data is simply not correct.
- 331c8c71 4y agoWell, my experience was that ChatGPT produced utter nonsense including literally "1/19+1/19=2/19=1" when asked a simple probability question.
- shaunsingh0207 4y agoIQ test, the sat, and exact mathematics are very different fields though. ChatGPT etc. are very good at "emulating" by sheer force and size logic, exact math doesn't work with "emulation"
- nayroclade 4y agoTranscript of a conversation I just had with ChatGPT: Prompt: I'm talking with a user named "331c8c71" on Hacker News. According to statistics I found by Googling, the number of Hacker News readers living in the USA is 36%. Assuming this is correct, what is the probability that 331c8c71 does not reside in the USA? Please explain your reasoning. Response: If we assume that the statistic you found is accurate, then we know that 36% of Hacker News readers live in the USA. Therefore, the probability that any given Hacker News reader does not live in the USA is 100% - 36% = 64%. However, we do not know if 331c8c71 is a typical Hacker News reader. They could be part of a demographic that is more or less likely to live in the USA. Without more information, we cannot say for certain what the probability is that 331c8c71 does not reside in the USA. So, the answer is 64% assuming that 331c8c71 is a typical Hacker News reader, but it may not be an accurate estimate depending on additional information about the user. --- Like I said, these models are far from 100% accurate, and there is lots they get wrong, but they clearly are capable of some kind of reasoning that goes beyond simple text substitution of training data.
- RivieraKid 4y agoWhat goalpost specifically?
- ducktective 4y agoSolving IQ tests which measure quantitative reasoning.
- RivieraKid 4y agoThere was no movement of this goalpost.
- espadrine 4y ago> I mean, how much further are we moving the goal post? Look at it this way: humans don’t have BPE-encoded text as input to their brain. It is ALL visual input. For AGI, you would at least need to add audio input as well. And be driven by action and reward. The learning capabilities of the brain are currently beyond the processing capabilities of current architectures. Just the notion of a model receiving only pixel data that contains a question and being able to output voice data that produces a correct answer, using no partial model trained on another corpus, is probably not tractable without significant improvements. But the models can be very useful without being AGI!
- pmontra 4y ago> It is ALL visual input. And sound, taste, smell, touch.
- konfusinomicon 4y agodon't forget humor
- KineticLensman 4y agophlegmatic, choleric, sanguine or melancholic?
- lupire 4y agoand doom, awe, timing, a word, absurd, in, horse, common, wonder, beauty, honor, duty, doubt, danger, fear, style, worth, direction, dread, non. https://m.youtube.com/watch?v=SKnnF5qA31I https://m.youtube.com/watch?v=SKnnF5qA31I
- konfusinomicon 4y agoTIL horse sense. I was thinking what on earth is the sense of horse?
- sillysaurusx 4y ago
- Tepix 4y agoIt wasn't very good at the IQ test. But yes, it is promising. "Although there is still a large performance gap between the current model and the average level of adults, KOSMOS-1 demonstrates the potential of MLLMs to perform zero-shot nonverbal reasoning by aligning perception with language models."
- Jevon23 4y ago>how much further are we moving the goal post? My goalpost for AGI is when Microsoft can fire their entire engineering staff, replace them with AI, and not notice any decrease in productivity or quality of output. This test is empirically verifiable (in principle). No need to argue over whether the AI scoring X% on Y assessment task is “truly” impressive or not.
- staticautomatic 4y agoYou mean that isn’t the Teams origin story?
- akiselev 4y agoTeams is the projection of a fourth dimension ancient Eldritch horror onto the mortal plane. AI had nothing to do with it.
- gfodor 4y agoYou’re confusing the goal - the goal here isn’t about the finish line but the point where people all concede that the finish line is actually reachable without any major, presently unthinkable advances.
- Jevon23 4y agoSurely anyone familiar with software engineering knows that the finish line is reached when the finish line is reached. And no sooner. Physicists at the beginning of the 20th century also thought that the finish line of physics was in sight and all that remained was tightening a few constants. Look how that turned out.
- gfodor 4y agoThis perspective is too reductionist - we make predictions of success all the time based on first principles reasoning. It’s perfectly sane to try to predict, and make good arguments, if AGI is possible to achieve without new breakthroughs.
- lumost 4y agoChatGPT does a great job on symbolic manipulation. You have to prompt it to show derivations however vs. discussing the topic at a high level.
- trobertson 4y agoThere's no goalpost to move. Psychologists have been saying that IQ tests are of limited scope and utility for decades. Specifically, it is widely agreed that an IQ test is not a valid way "to assess intelligence in a broader sense". https://en.wikipedia.org/wiki/Intelligence_quotient#Validity_as_a_measure_of_intelligence https://en.wikipedia.org/wiki/Intelligence_quotient#Validity...
- pillarofkiller 4y agoyeah there is a model that can do differential equations