9 ms·
>But arguably GPT-4 is already at that level There really is no arguably about it. There isn't a verifiable definition/criteria for general intelligence that G
by famouswaffles 4y ago
>But arguably GPT-4 is already at that level
There really is no arguably about it. There isn't a verifiable definition/criteria for general intelligence that GPT-4 fails that a significant chunk of the human population doesn't also fail.
GPT-4 performs nearly all tasks it's given at at least the average human level, usually well above the average baseline. In some cases (nlp, analogical reasoning - https://arxiv.org/abs/2212.09196 https://arxiv.org/abs/2212.09196, and some others it's top %)
You basically have to make up your own definition that can't be substantiated (don't worry, plenty seem to do this). It almost always boils down "Well it's not "real" understanding, the difference is huge, i can't show you this supposed huge difference but trust me bro, it's not real"
- hooloovoo_zoo 4y agoAny artificial neural net has a finite set of inputs and a finite set of outputs. That makes it equivalent to some lookup table. You must therefore claim a lookup table has the ability to reason.
- cjbprime 4y agoAnd what, humans are made out of an infinite number of neurons?
- hooloovoo_zoo 4y agoOne neuron has an infinite number of possible states (as far as we know).
- cjbprime 4y agoGPT-4 thinks you should provide a citation: == Me: Is it true that, in humans, one neuron has an infinite number of possible states (as far as we know)? GPT-4: It is not accurate to say that a single neuron in humans has an infinite number of possible states. Neurons are complex cells that can exhibit a wide range of behaviors and responses, but the number of states is not infinite. Neurons communicate with each other through electrical and chemical signals, and their primary functional unit is the synapse, where communication between neurons occurs. The strength and patterns of these connections can vary, resulting in a large number of potential states for a single neuron. However, this number is still finite, albeit extremely large. Moreover, the activity of a neuron can be described in terms of its membrane potential, which is a continuous variable ranging from hyperpolarized to depolarized states. While there are many possible values for the membrane potential, it is still a bounded range, not infinite. So, while the number of possible states for a single neuron is vast and complex, it is not infinite. The complexity and diversity of neuronal states contribute to the rich repertoire of cognitive and behavioral functions that humans possess.
- hooloovoo_zoo 4y agoThat implies a lookup table requires me to provide a citation :) In fact, assuming space is not discrete, not only can a neuron be in any of infinite states, per quantum mechanics it can be in multiple at the same time.
- cjbprime 4y agoThe lookup table says I should tell you that quantum mechanics applies to subatomic particles, not very large objects like neurons. (Would it bother you to be corrected by a lookup table?)
- hooloovoo_zoo 4y agoSchrödinger's cat will be so relieved!
- jachee 4y agoHow does that hold in the case that connections among neurons exchange state information via subatomic particles?
- famouswaffles 4y agoquantum mechanics ? you think that is applicable for neurons ?
- akomtu 4y agoNeurons apparently contain microscopic tubes that make use of some quantum effects: https://phys.org/news/2014-01-discovery-quantum-vibrations-microtubules-corroborates.html https://phys.org/news/2014-01-discovery-quantum-vibrations-m...
- famouswaffles 4y agoThe user was implying neurons could be in multiple states at the same time because...quantum mechanics. Wrong conclusion even from this link.
- airgapstopgap 4y agoWe don't know that in any way that is relevant. A biological neuron is complex and made of lots of elements where very fine interactions can plausibly have effect, so in a sense, there is a combinatorially infinite space of neuron-states, but there's no evidence its upper-level computational function that leads to learning and producing adaptive outputs in response to inputs is not discrete. Indeed it seems that most of the complexity [that isn't just "piping" for life functions – don't forget these things aren't semiconductors in metallic sheets fabricated by TSMC, they have to migrate to their sites and branch out on their own, they consume, they excrete, they function for many decades!] is redundant and serves to compensate for various biological shortcomings like low transmission speed, limited bandwidth, noise and stochasticity, developmental abnormalities, cell death etc. Analog computers also don't have infinite states, because of the noise floor. A synapse can encode at most some low tens of states[1], synapses are discrete, spiking is discrete, receptor density is discrete, each neurotransmitter release is made up of discrete number of vesicles with a discrete number of molecules, even epigenetic expression is discrete. That it's not all perfectly algorithmically executed is not key to the function being performed – and anyway, at the limits of high-performance compute we also start to deal with esoteric processes, and ways of suppressing them. We didn't design the brain nor do we have perfect ways of observing it in vivo, so we can't always neatly distinguish the wheat and the chaff; but this cannot be seriously taken as a cause to think it's all wheat. Non-human animals show that it's not magic either; even apes, with brains architecturally almost (not wholly) identical to ours but smaller, cannot learn any of the hard symbol-manipulating stuff that we or later GPTs can and that we colloquially refer to as intelligent behavior and reward economically. This, I think, is a very strong prior for that sort of intelligence not being dependent on some low-level expressivity of biological neural computational substrate. Penrose-Hamerroff's "quantum effects in microtubules" thesis is crank science not supported by any direct evidence. 1: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5247597/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5247597/
- deleted 4y ago[deleted]
- doctor_eval 4y agoOne of the reasons I think LLMs are somehow analogous to brains is that they scale based on size. This seems true for animal brains as well. But you seem to be suggesting that all animal brains are equally powerful, because they have infinite neuron states, quantum superpositioned neurons, etc. As far as I can tell, you seem to be suggesting that my old cat’s brain has the same complexity as my brain - right? So, I’m interested in understanding why my cat is demonstrably stupider than I am, despite having infinite complexity, and why whatevrr you come up with doesn’t also apply to LLM size.
- hooloovoo_zoo 4y agoNo, I’m saying having a finite number of neurons doesn’t imply a particular number of brain states. Scaling laws actually don’t apply very strictly to brain size. For example, whales have very large brains but aren’t generally thought to be more capable of reasoning than humans. Women generally have smaller brains than men but aren’t thought to be less capable of reasoning.
- SideburnsOfDoom 4y ago> No, I’m saying having a finite number of neurons doesn’t imply a particular number of brain states. But of course it does imply that. It's vast but necessarily finite number of "brain states" however that's defined, which is itself a subset of the vast but also finite number of states of the finite number of component particles.
- dragonwriter 4y agoThat clearly isn’t true, since a neuron that could be in infinite states would thereby be capable of storing infinite information, and (as the neuron itself is finite in size) have infinite information density, but there is an established physical upper limit on information density. https://en.wikipedia.org/wiki/Bekenstein_bound https://en.wikipedia.org/wiki/Bekenstein_bound
- hooloovoo_zoo 4y agoMm not sure that bound applies because neurons are neither of fixed size nor fixed energy. But on the other hand one starts to stretch the definition of neuron.
- SideburnsOfDoom 4y ago> Mm not sure that bound applies because neurons are neither of fixed size nor fixed energy Neurons are not of unbounded size nor energy.
- hooloovoo_zoo 4y agoWhat’s the bound?
- SideburnsOfDoom 4y agoCan you not work this out yourself? Use some common sense. Do you disagree, are you saying that animal body cells can be infinitely large?
- hooloovoo_zoo 4y agoNo bound, got it.
- SideburnsOfDoom 4y agoWhich animal body cells can be infinitely large then? This is foolery. "Neurons are of unbounded size" is a ridiculous statement, we any fool can see it's false. IDK what you're aiming at here, try making a case for something sensible?
- famouswaffles 4y agoNot even going to bother with the weird leap of logic you've made here but Memory augmented LLMs are computationally universal - https://arxiv.org/abs/2301.04589 https://arxiv.org/abs/2301.04589
- hooloovoo_zoo 4y agoThen I won’t bother with your citation that says if you add infinite memory you can achieve infinite states :)
- famouswaffles 4y agoinfinite memory ? guess you can't read then. Well you do you.
- cjbprime 4y ago(I agree with you.)
- riku_iki 4y ago> GPT-4 performs nearly all tasks it's given at at least the average human level yet, you will afraid to trust gpt-4 to file your taxes through installed browser plugin. There is a huge gap between some narrow cherry picked (researchers have strong incentive to report mind boggling results) and diverse and multi modal real life situations
- flangola7 4y agoI wouldn't trust the vast majority of humans to file my taxes either, and I definitely wouldn't trust any below a certain intelligence level. Nevertheless that doesn't mean those humans are completely devoid of any intelligence at all. I don't need to cherry pick, I use GPT4 daily. At this point naysayers can only be willfully ignorant or uncreative in their use of it.
- riku_iki 4y agoI also tried to use gpt4 daily(canceled lately), and limitations of its "intelligence" are very visible to me.
- armchairhacker 4y agoIdk about “above average baseline”. Maybe, because quite a lot of people aren’t very smart. But even GPT4 struggles with complex logic problems, particularly ones with multiple steps, and when it makes logic errors it makes ones a human would spot. It also has trouble counting letters and words, though that’s largely due to its autoregressive nature. It has more knowledge and memory than any human, and I really think it’s emergent, but at its core it doesn’t seem very “intelligent”
- famouswaffles 4y ago>Idk about “above average baseline” That's the minimum. It's far above average for most things. >Maybe, because quite a lot of people aren’t very smart. Lol okay..and ? >But even GPT4 struggles with complex logic problems So does everyone else >and when it makes logic errors it makes ones a human would spot people spot mistakes they're capable of making all the time and so can GPT-4 - https://arxiv.org/abs/2303.11366 https://arxiv.org/abs/2303.11366 https://twitter.com/johnjnay/status/1639362071807549446 https://twitter.com/johnjnay/status/1639362071807549446 >It also has trouble counting letters and words, though that’s largely due to its autoregressive nature. It sees tokens not words or letters and there aren't many instances of counting occurring in text so it doesn't get learnt. >and I really think it’s emergent, but at its core it doesn’t seem very “intelligent” and we're back to the trust me bro argument. It's not intelligent because you say it isn't not because it doesn't perform a great many tasks intelligently. if you say so
- chrisco255 4y agoIf it's so smart already, why hasn't it replaced the most basic functionality in human society yet? Like, can a GPT4 drive a car because most people can drive a car.
- famouswaffles 4y agoCan a blind man drive ? are blind people unintelligent ?
- cjbprime 4y agoBecause we haven't hooked it up to visual input yet?
- breuleux 4y agoThe "basic functionality" may just be more complicated than intelligence. Intelligence and language are relatively new capabilities built on top of older, better honed ones, so it shouldn't necessarily come at a surprise that computers get smart before they can properly deal with the physical world.
- dragonwriter 4y ago> There isn’t a verifiable definition/criteria for general intelligence that GPT-4 fails that a significant chunk of the human population doesn’t also fail. The usual broad indicia of intelligence (which, e.g., IQ is considered as a useful proxy measure for in humans because it predicts/correlates with them) are lifelong success in a wide variety of tasks GPT-4 can’t even begin to approach.
- famouswaffles 4y ago>are lifelong success in a wide variety of tasks GPT-4 can’t even begin to approach. Right. and what are these tasks and why is intelligence the barrier? (meaning don't tell me something like driving, blind people are not unintelligent)
- woolion 4y agoWell I've tried asking for lists of words that satisfy certain simple constraints, like a theme and a given number of syllables. Almost no word satisfied the constraint, despite my many tries. So if you can have a conversation that goes "give me a list of 3 syllables word", you get a list and then you ask for each word "how many syllables is that word?" and get answers like 2, 4, 5, sometimes 3. You can ask the question again and still get the same words with wrong numbers of syllables. Despite a very obvious logical implication of facts here. This sounds quite similar to Bing famously explaining to a user that 2022 is in the future while we are in 2023. It's something you would expect maybe a child to do (it reminds of "Patrick Star's Wallet" meme), but to me that clearly fails the notion of intelligence in a very fundamental way. What would be the counter-argument?
- famouswaffles 4y agoLanguage Models don't see words. They see tokens. So asking them to perform minute operations on letters/syllables/words etc don't work well. Doesn't really have anything to do with Intelligence. Sentences look something like this to the model, [754 76 4752 67 456]
- lucubratory 4y agoThere still isn't any evidence that it can pass novel Winograd schemas (as in, ones it cannot possibly be trained on), which almost every human can pass. The abstract reasoning corpus is more of a "lots of people will fail this too" test, but if a human is approaching it carefully they will be able to do it and there's still no evidence that GPT-4 can do it reliably (I expect that to change once it's multimodal). The systems are very impressive and I think there is a 'there' there, they're clearly intelligent to some significant degree. But it's incorrect to say that they pass every test.
- edouard-harris 4y agoI just tested GPT-4 on the Winograd schemas below, which I made up and (as far as I know) are novel. It scored 3/4, incorrectly answering "the pouch" instead of "the baby kangaroo" for Question 2 of Schema 1. And now that I look at the question, one could easily imagine the kangaroo being afraid to jump into a pouch that's too large! In hindsight, if a human complained to me that this question was ambiguous, I'd have a hard time disputing them. Granted these are just two examples, but they aren't cherry-picked. Moral of the story: before one makes a strong claim such as "there isn't any evidence a model can do X", it's often worth making a minimal effort to investigate one's own claim. ======================================== Winograd Schema 1: Sentence 1: The baby kangaroo was afraid to jump into the pouch because it was too small. Question 1: What was too small? (Correct answer: the pouch) Sentence 2: The baby kangaroo was afraid to jump into the pouch because it was too large. Question 2: What was too large? (Correct answer: the baby kangaroo) Winograd Schema 2: Sentence 1: The gardener couldn't grow the flowers in the greenhouse because it was too hot. Question 1: What was too hot? (Correct answer: the greenhouse) Sentence 2: The gardener couldn't grow the flowers in the greenhouse because they were too delicate. Question 2: What were too delicate? (Correct answer: the flowers)
- bamboozled 4y agoMoral of the story: before one makes a strong claim such as "there isn't any evidence a model can do X", it's often worth making a minimal effort to investigate one's own claim. Do you think this was a strong refutation ? I just flipped a coin for fun and also got 3/4 correct. I understand your point, but I mean, something that's intelligent really should've got the second question in that test correct.
- alasdair_ 4y agoI’ve yet to see an llm score well on non-verbal iq tests like https://www.creativeorgdesign.com/tests/non-verbal-iq-test/ https://www.creativeorgdesign.com/tests/non-verbal-iq-test/