7 ms·
I think GPT4 can converse on any subject at all as well as a (let's say) 80 IQ human. On some subjects it can converse much better. That feels fundamentally di
by drcode 3y ago
I think GPT4 can converse on any subject at all as well as a (let's say) 80 IQ human. On some subjects it can converse much better.
That feels fundamentally different than a calculator.
- skepticATX 3y agoGPT-4 is absolutely more generally knowledgeable than any individual person. Individual humans can still easily beat it when it comes to knowledge of individual subjects. Let’s not conflate knowledge with intelligence though. GPT-4 simply isn’t intelligent.
- MichaelBosworth 3y agoWould be curious to hear an elaboration on this perspective. In your opinion, on which measures of intelligence would GPT-4 fail to out-perform a human with an IQ of 80? Conversely, on which measures do you imagine it would succeed at doing so? Are the latter less significant or valid than the former?
- reso 3y agoHumans handily outperform GPT4 handily on the task of "write a random string of length [x]" for any x > ~25.
- pclmulqdq 3y agoThey have a specific device to do that now. I have tried to say "write a random sentence with 6 words and 2 numbers" and it completely fails, but it can do the straightforward "write a random [x] of length [y]."
- verbify 3y agoI got "Here is a random string of 32 characters: a8Jk5pYr0Dm9Nc1Vz8Qf2Bt6Hg3Lw4Uo"
- reso 3y agoWouldn’t be surprised if it’s good at 32 because it’s a power of 2. I’ve tested this with a wide variety of number inputs and it’s performance is highly variable. Error also increases linearly with strong length.
- olddustytrail 3y agoIf you asked most people to perform that task, they literally wouldn't have a clue what you'd just asked them to do.
- ux-app 3y agoa 4 year old would fail at this task. does a 4 year old have intelligence?
- TeMPOraL 3y agoYup. I think this is the best point of comparison - a 4-6 year old kid. Specifically, one that hasn't gone to school yet. The difference between a typical 6-year old and a typical adult is in big part that the latter spent 10+ years being systematically fine-tuned. Logic, arithmetics, algebra, precisely following steps of an algorithm - those are not skills one "kinda" just "gets" at some point, they're trained by deliberate practice, by solving lots and lots of problems specifically constructed to exercise those skills. Point being, get GPT-4 through school, and then compare with adult performance on math-adjacent tasks. Or at least give it a chance by prompting it to solve it step-by-step as a problem, so it can search closer to the slice of latent space that encodes for relevant examples of similar problems and methods of solving them.
- hammyhavoc 3y agoI started seriously using computers at 2.5, and I started writing and recording songs with a tape recorder at 3, won a local award for one song, and playing chess at 4. I know plenty of people with similar experiences. If you nurture kids and don't treat them like they're stupid, they can do some quite impressive things. Anecdote: admittedly, I'm autistic as are the people I know, so maybe that's not a good sample. I struggle with a lot of basic shit even as an adult. Oh god, I empathize with the hypothetical GPT5.
- reso 3y agoI did not say GPT4 did not have intelligence. I gave an example of a task it fails that is easy for most humans.
- chimprich 3y agoConscious thought. In biological terms it has a superhuman cerebellum but no cerebral cortex at all. It can't assess what it's doing. GPT4 will produce stuff, but only if prodded to do so by a human. I recently asked it to help me write some code for a Garmin smartwatch. The language used for this is MonkeyC, of which there isn't a huge amount of examples on the internet. It confidently provided me with code, but it was terrible. There were gaps with comments suggesting what it should do, bugs, function calls that didn't exist, and many other problems. I pointed out the issues and GPT4 kept apologising and trying new stuff, but without any improvement. There wasn't any intelligence there; the model had just intuited what a program might look like from sparse data, and then kept doing the same thing. It didn't know what it was doing; it just took directions from me. It couldn't suggest ideas when it couldn't map to a concept in memory. A human with an IQ of 80 would know if they didn't know how to code in MonkeyC. If they thought they did, they'd soon adjust their behaviour when they realised they couldn't. They'd know where the limit of their knowledge was. They wouldn't keep trying to guess what functions were available. If they didn't have any examples in memory of what the functions might be like, they might come up with novel workarounds, or they'd appreciate what program I was trying to write and suggest a different approach. Presumably we'll make progress on this at some point, but I think it'll take new breakthroughs, not just throwing more parameters at existing models.
- hammyhavoc 3y agoExactly my experiences. With a fucking NGINX configuration, for which I provided it the documentation, and the URL rewrite lines it would require. I spent days on trying to find the value that other people are claiming it has.
- mise_en_place 3y agoIt's a gradient. You can't be too specific, but you can't be too general either. IME
- hammyhavoc 3y agoStrangely, specificity is exactly what people champion the importance of when it comes to successful prompting.
- Elextric 3y agoSorry for being pedantic. The intelligence of something is inconsequential. What truly matters is its ability to convincingly imitate intelligence.
- byyy 3y agoIf the imitation becomes indistinguishable to the real thing based off of every test that can possibly be generated in the universe then it is an intelligence. In that sense, because we are making progress on producing an indistinguishable imitation... you might as well say we are making progress on an actual sentient intelligence.
- xwdv 3y agoIs GPT more knowledgeable though than an individual person using Google?
- AndrewKemendo 3y agoSo here in this forum right now, convince everyone that you are intelligent. ….
- janalsncm 3y agoIt would be very helpful to define intelligence before asserting that a thing does not have it. A cursory look at the Wikipedia page for the definition of intelligence shows there is no one, agreed-upon definition. In fact some believe that “intelligence” simply means pointing to ourselves.
- cactusplant7374 3y agoBut it can’t make novel discoveries like humans. It would be great if it could discover new uses of mRNA and prototype them.
- cjbprime 3y ago> Individual humans can still easily beat it when it comes to knowledge of individual subjects. What does a phrase like "GPT-4 scores 90th percentile on the Uniform Bar Exam" mean to you, regarding whether humans can easily surpass its knowledge and reasoning? https://www.forbes.com/sites/johnkoetsier/2023/03/14/gpt-4-beats-90-of-lawyers-trying-to-pass-the-bar/ https://www.forbes.com/sites/johnkoetsier/2023/03/14/gpt-4-b...
- dragonwriter 3y ago> What does a phrase like "GPT-4 scores 90th percentile on the Uniform Bar Exam" mean to you, regarding whether humans can easily surpass its knowledge and reasoning? Absolutely nothing, because of construct validity. Those tests measure things that have shown to correlate with abilities of concern in humans, and so are, for their purposes, valid for humans. This hasn’t been demonstrated for LLMs, and the assumption that construct validity can be assumed without being established is begging the question: it is presuming not only that LLMs are general intelligences, but thaf they are general intelligences structurally similar to human intelligences such that the proxy measures for cognitive capacities work similarly.
- comp_throw7 3y agoConstruct validity! I suppose, when GPT-4 writes correctly working code that does what you want on the first try, this says absolutely nothing about its cognitive capacity, because, after all, it's just a proxy measurement for the underlying generative process. (Yes, obviously the cognition is _different_ from what happens in humans. That does not mean that... it isn't intelligence?)
- dragonwriter 3y ago> I suppose, when GPT-4 writes correctly working code that does what you want on the first try, this says absolutely nothing about its cognitive capacity It says something about its ability to write code. Beyond that... its impossible to say. We simply don’t have the information about generative AI models to be able to generalize from limited proxies about them; psychometry is not transferrable from humans to them — or at least, we have neither evidence nor a strong theoretical reason to think it should be.
- mise_en_place 3y agoGreat take. But I think when autonomous agents become good enough, intelligence is certainly possible. Especially when those agents start to interact with the real world.
- staticman2 3y agoDo you frequently talk to people who you know to have an 80 IQ about a range of subjects?
- Kranar 3y agoStatistically, about 16% of the time.
- staticman2 3y agoYou entirely missed my point When you speak to someone with an 80 IQ do they introduce themselves by saying "Hello I have an 80 IQ, nice to meet you." So that, like the person I responded to above, you can compare their conversation skills to the ChatGPT4 conversation skills?
- Kranar 3y agoFirst off, you wouldn't need to do that specifically. You'd only need to know that most of the people you talk to are above an 80 IQ on any given topic, in fact most people are about a 100 IQ on any given topic. So you already have a reasonable baseline for comparison. Secondly, I'd say you're likely the one missing OPs point by trying to take a mostly colloquial statement about how ChatGPT is about as informed as the bottomish X% of the population on any given topic and trying to be pedantic about it. Furthermore the real purpose of OPs point is that the X% is now a lower bound, even if X isn't 16% but 5%, it's only going to go up from here. Yes there's evidence of diminishing returns with the current architectures but there's also a lot of room for growth with newer architectures or multimodal modals. I think most people understand OPs point without having the need to go around asking everyone what their IQ is. There are numerous indicators, both formal and informal, that indicate that ChatGPT is as informed on most any given topic as the bottom 16% of the population. In fact, it's likely much much higher than that.
- deleted 3y ago[deleted]
- deleted 3y ago
- titzer 3y agoI feel like lost in this conversation is that ChatGPT is incredibly good at writing English. It basically never makes grammatical mistakes, it doesn't spew gibberish, and for the most part has extremely well-structured replies. The replies might be bullshit or hallucinations, but it's not gibberish. It's kind of breathtaking that we forgot about that being hard. The goalposts are moving again. BTW, it has passed many standardized tests under the same circumstances as a human.
- dragonwriter 3y ago> BTW, it has passed many standardized tests under the same circumstances as a human. No, it hasn’t, and it is physically impossible for it to. The extent to which the differences are material may be debatable, but this claim is simply false.
- cjbprime 3y agoIt would be a useful contribution to explain what you think the material differences are, rather than referencing them through innuendo, as if anyone knows what you mean.
- jrk 3y agoI am not the original poster, but I assumed they meant as an embodied entity, interacting in the physical world.
- bbor 3y agoTechnically no two sets of circumstances can EVER be the same ;). See https://en.wikipedia.org/wiki/Heraclitus#Panta_rhei https://en.wikipedia.org/wiki/Heraclitus#Panta_rhei
- hammyhavoc 3y agoSome of the replies are gibberish, especially once you get into technical subjects that it has very little training data on. It kitbashes words together that actually mean nothing, which is no surprise given that it's an LLM.
- byyy 3y agoOf course it does. He knows it. Some people just can't bring himself to stare at the reality of it all.
- SanderNL 3y agoI'm curious if you actually ever interacted with IQ 80 humans. They are definitely not on this scale.