7 ms·

On ChatGPT

71 points by picture 4y ago

calaverainfo 4y ago
I like the essay a lot and I find it intellectually amusing that many negative comments here manifest exactly the flaws that could be prevented by learning the essay writing as described in the blog. The article clearly states assumptions, definitions, context and limitations of it's arguments and in that specific area it's very hard to refute them because they are sound. On the other side the gist of many comments here is that article is negative towards GPT and should be positive, i.e. the same generalist aproach that GPT generated text would take and which is hard to argue about. I'm not going to analyse particular flaws of separate comments because the article itself is really comprehensive and answers for many objections are better in original form there than any reproduction I can make here. But the high number of rebuttals that obviously have problems either understanding the essay or formulating a solid counterargument is curious. An obvious explanation is that some people couldn't resist the pleasure of letting AI dismiss the statement about insufficient AI capabilities, but that's a lazy one. Comments saying current AI is comparable to actual human inteligence may be right even though I can clearly see that AI is not performing actions I consider necessary for thinking. It's because I consider my own mind as a model of human inteligence, but as I learned many times before, the thinking process may be hugely different for different people. I have no idea if other people also do need to create an explicit model of something in their head to be able to "think" about it. It would explain A LOT for me if thinking of some other people really works in the same way as in GPT. That situation should however still be viewed not as a technology enhancement so big it reaches human level but reconsideration of what we consider a human level. This happens to me a lot. I want to make a joke, but it gets so long an complex that it becomes an analysis.
LesZedCB 4y ago
it just makes me sad and angry that people discount their pets experience because they are afraid of over-anthropomorphizing. we are animals which evolved just as our nearby mammalian friends. yes their experience is very different but not in any way fundamentally alien. sure they aren't jealous but they could experience anxiety and many other forms of suffering or excitement and pleasure just like we can. why negate that? if we can't understand that, I have a deep horror for the atrocities we may commit if/when we do create machine subjectivity.
- deleted 4y ago
  [deleted]
- quonn 4y ago
  It‘s still not clear to me why any subjective experience would arise in a computing process. We know it happens for humans, we can presume that it happens for mammals due to the biological similarity and common history. Unfortunately we don‘t understand it so far, but until we do, the burden of proof is on those who claim without evidence that there is consciousness in machines.
  - jaggederest 4y ago
    There's no "consciousness" there. There's consciousness in what it is emulating, though, and if you give it enough of that, it will be indistinguishable from an actual consciousness, and so it may very well be that we must treat it as conscious. There's a philosophical concept, "zimboes", which is useful here - philosophical zombies which believe that they are conscious, and believe that they experience subjective understanding. I don't follow all the arguments, but from what I understand, they're used to argue for "if it quacks like a duck" presentations of consciousness, rather than some more abstract criteria.
    - ifdefdebug 4y ago
      > There's consciousness in what it is emulating This begs the question (if you made it to the footnotes of the article discussed here, you will know what I mean). More precisely, you start your argument by begging me to take for granted that it can emulate conscious beings - no, you will have to proof that, and not by making it into an axiom.
      - jaggederest 4y ago
        My argument is that there's no way of distinguishing "behaving like conscious beings" from "being conscious". It does not matter whether it is "genuinely emulating conscious beings" or not. It's a simple extension from "it mimics human text" to "what if it got so much better we can't tell the difference". If we can't tell the difference, definitionally it may as well be conscious. You can disagree with the axiom that it does mimic human text, but generating text with it looks like a pretty good mimicry to me. You may not agree that it will ever get that good - I would also agree with that. I don't think the proposed scenario is at all likely, certainly not in my lifetime, but if it was, I'd be there lobbying for the large language models to have rights.
firatsarlar 4y ago
It may be possible to resist something predicted, expected, feared. I couldn't quite get what the article wanted to reveal, except for the already spoken and known things?
triyambakam 4y ago
> Learn. Don’t cheat. Teach, then? I don't know this professor, but many college professors themselves could be replaces with ChatGPT.
- isaacfrond 4y ago
  Most teachers have a higher uptime than ChatGTP fortunately.
- imiric 4y ago
  Wow. That is all you got from the article? Students will find ways to make their lives "easier" by sabotaging their own education with assistants like AI, regardless of the quality of professors teaching them. Tools like ChatGPT can only regurgitate what was already written, often with confident but wrong takes. They can't offer new perspectives, or inspire students with a different way of thinking about the world. These are not teacher, let alone college professor, replacements.
  - triyambakam 4y ago
    > Tools like ChatGPT can only regurgitate what was already written, often with confident but wrong takes. Yeah, just like many professors
  - chii 4y ago
    > They can't offer new perspectives i think this remains to be seen. At least, if you asked chatGPT for recipes, they can produce recipes that aren't pre-existing. So for at least some small domain, it can offer new perspective. Whether those recipes tastes good is a different story...
    - hgomersall 4y ago
      Just as it can offer "new perspectives" on actual facts.
  - charcircuit 4y ago
    >Tools like ChatGPT can only regurgitate what was already written, often with confident but wrong takes. They can't offer new perspectives, or inspire students with a different way of thinking about the world. That is not how ChatGPT works. It doesn't just regurgitate data it has seen. It its capable of offering new perspectives. Just ask it.
isaacfrond 4y ago
The article starts out in bold with the statement they pretend to be like minds – like human minds. But it is only pretend, there is no mind there and that is the key to understanding what ChatGPT is And surely they have no mind like us. But this obsession with whether or not it is like us, seems to miss the point, doesn't it. Is it useful? Despite the writers insistence that it is not, my children provide my with many examples that it is. For example, my daughter is learning to program and got negative feedback on her commenting style. (which admittedly where mostly absent) So, she asked how to do it better. The teacher gave only non-committal responses, like being helpful to the reader, etc. It is not the kind of instruction that is helpful for her. So she gave the whole program the ChatGTP and asked it to insert comments without changing the code. The next day, she took the output to the teacher and asked if this is what he meant. It was perfect. But now, she knows what is expected of her, and now she can do it without ChatGTP.
- Gigachad 4y ago
  When you drill down these discussion points, it all comes down to the fact we don’t exactly know how the human mind works so we can’t properly answer what is the difference between pretending to understand, and understanding.
  - Jensson 4y ago
    We know that some humans can use information to solve harder problems, while others can not do it. We say the first kind built understanding about the information. You can easily test for this by just throwing hard problems at them that they haven't seen before, Google does that in their interviews for example (but many who do similar just takes problems people definitely have seen). So doing similar kinds of testing on ChatGPT we see that it doesn't understand the information. It can repeat a lot of stuff more or less verbatim, but it cannot apply most of the knowledge it can repeat, so it doesn't understand any of it. The only thing ChatGPT really understands are relationships between words that has been repeated in texts it has seen, that is some level of understanding but that has no relationship to what ChatGPT says about the words. When you ask ChatGPT to describe something it repeats a description it has seen of the word, when you ask it to do something with a word then it fully ignores that description and instead look at the relationships it has seen the word have to other words. So its understanding and its knowledge are completely separate things, so you can say that it doesn't understand anything it knows.
DougBTX 4y ago
I don’t buy this “it is just statistics” dismissal. Practically speaking, any multiple choice question is ultimately a “predict the next character, A, B, or C” challenge. Should we dismiss all multiple choice questions because they are just “predict the next token” tasks? There is a broader question about whether it is possible to learn from reading. Can a blind person ever really understand “blue”? If not, what can be learnt by reading? Why do we rely on reading and writing so heavily for learning? Edit: just want to note that this is a great site, really like the author’s approach and style. And maybe if I’ve not learnt from his writing, at the very least it is a good read :-)
- jaggederest 4y ago
  I agree. Quantum mechanics is "just statistics", and yet it still causes chemistry to happen. I think that the argument for qualia is pretty weak - if we can have a conversation about "blueness" with a blind person, we have to admit the possibility of "understanding" existing in some form in LLMs. Even if it's just a statistical emulation, at what point does a statistical emulation become reality, if it's a good enough emulation down to the metaphorical planck distance.
  - bloppe 4y ago
    Anybody who has ever done arithmetic knows that human logic is more than just statistical pattern prediction. Humans can learn and execute algorithms in precise, deterministic ways, which current LLM's fundamentally cannot do.
    - mola 4y ago
      That's funny. So being intelligent is executing algorithms deterministically and being a machine is making mistakes and being non deterministic? How the tables have turned
    - space_fountain 4y ago
      I’m not sure that’s true. Human make mistakes performing math all the time. Human’s ability to accurately perform multiplication is very much a statistical phenomenon. Now you probably dismiss those mistakes as different than the sort of thing that a llm does when following a rigid set of instructions, but to me they seem different in degree but not kind
Animats 4y ago
He's already a few weeks behind. Bing's new AI can query Bing's search engine. So it hasn't "burned the library". It can reach out for new information. That's going to be really important. Large language models can already summarize documents. How long before you can tell a system "Read this book. Then I will have some questions about it"? Once you have that, you can apply the base model to domain-specific problems.
- imiric 4y ago
  I think you missed the argument about the two Roman Empire books. ChatGPT was clearly trained on them, or at least metadata about them, yet it couldn't produce an intuitive response about their relationship. And it couldn't even return the top search result that explained it. Sure, these issues will be fixed eventually, but the point of the article is that it can't replace the thinking of a student to write a quality essay. It might be able to regurgitate search results which will need to be fact checked, so at best it's useful as a research tool. BTW, Bing's AI is a really poor counterexample, as it can't even return search results reliably.
jaggederest 4y ago
It's a little frustrating when the author calls out "technofetishistic" faith in large language models[1] and then goes on to misunderstand their capabilities in the opposite direction. I'm no "true believer", they're just useful tools for modeling text generation, but to underestimate them is a shame. > Ideally, the essay writer has first observed their subject, then drawn some sort of analytical conclusion about that subject, then organized their evidence in a way that expresses the logical connections between various pieces of evidence, before finally communicating that to a reader in a way that is clear and persuasive. Of the elements discussed about "writing an essay" above, none are clearly impossible for a large language model to do at the present, despite the author's insistence that some of them are impossible. If the subject is something in writing (and you can pass it in the prompt), or is written about in the training set, then the model has indeed "observed" it, and to get it to draw analytical conclusions all you have to do is ask nicely and correctly, maybe using those words. Again, with organizing evidence, you merely must ask for it by name, rather than hoping that the model understands what you mean when you say "write an essay" - ask for "organize the evidence in a way to support/contradict the conclusion X", and finally of course communicating cogently to the reader is something the writer correctly understands is possible even with a trivial prompt. Sure, it moves the challenge from "write an essay" to "design a prompt to get the language model to write an essay", but that's "not writing an essay", or is it? It is immensely frustrating that people feel like a large language model is "cheating", but also don't consider using word processing software with grammar and spelling correction "cheating", even though they use fundamentally the same processes behind the scene and differ only in the user interface, really. [1] I don't like the term AI, I prefer large language model because it is unambiguous about the properties of the system - AI implies thought processes, which is a distraction at best from the useful properties of a complex statistical model.
- chii 4y ago
  > also don't consider using word processing software with grammar and spelling correction "cheating", even though they use fundamentally the same processes behind the scene and differ only in the user interface, really. i think it's a stretch to claim that spelling/grammar correction and large language models are similar or fundamentally the same. that's like saying the horse-drawn carriage is fundamentally the same as a plane. Sure, they both move people around, but the differences are much larger than the similarities.
zxcvbnm 4y ago
So then don't torture students by making them write long essays and waste their valuable time. Make it the task expressing the argument on a back of a napkin. More thought, less time wasted on writing stuff no one ever read properly.
- mola 4y ago
  A lot of time writing is a personal endeavor. A way to communicate with yourself rather than others. It allows you to lay out your chaotic thoughts in a clear way. It's the foundation for critical thinking. It'll be a very sad world where kids don't atleast practice this skill.
zxcvbnm 4y ago
So then don't torture students by making them write long essays and waste their valuable time. Make it the task expressing the argument on a back of a napkin. More thought, less time wasted on writing text-amount. Also less of the teacher's time wasted, do they really read 100x student 3 pages, weekly? Don't they have anything better to do?
- bloppe 4y ago
  It's not fair to assume that's what this teacher asks of their students. Word minimums may be useful for children as a crude deterrent against laziness, but any serious student or teacher would recognize a good essay when they see one, no matter the length (though fitting a good and interesting essay on the back of a napkin would be almost unthinkably prolific). "I have made this letter longer than usual, only because I have not had the time to make it shorter." - Blaise Pascal
  - izacus 4y ago
    There's nothing to assume because he describes what he asks of his students towards the last part of the essay. But people who think ChatGPT is great usually aren't capable of reading that far :P
  - acatnamedjoe 4y ago
    A word count is also a signal to the student of what level of depth the teacher is hoping for. For most interesting questions you could write anything between a few sentences and a full book, depending on how much you develop and defend your arguments, evaluate other possible answers, etc. The word count is a signal on where on that spectrum you should be aiming.
  - firatsarlar 4y ago
    If we see Pascal's quote as a pretext, it could be used to justify writing longer texts, even when brevity is possible and appropriate. However, by interpreting the quote as a genuine apology for the length of his letter, we can recognize the importance of clarity and conciseness in communication, and strive to make our own messages as clear and concise as possible. This highlights the importance of interpreting someone's words in the right context, and not using them to justify behaviors or attitudes that are not aligned with the speaker's true intentions. By being mindful of how we interpret others' words, we can avoid misunderstandings and communicate more effectively.
MrSqueezles 4y ago
Education has always done a terrible job of keeping up with technology. Kids in school now will graduate into a world in which all information will be instantly available and will be presented in whatever format is most suitable. They'll have computers correcting their grammar, improving their ideas, completing their sentences and paragraphs. Instead of learning those technologies themselves and instructing students how to best utilize them, teachers tell kids, "that's cheating". When you graduate and get your first job, your boss isn't going to take away your books, phone, and laptop, and ask you to write a report about a well understood subject. I understand why that's a common practice in education. Maybe it's time to change.
- jacquesm 4y ago
  That would be all fine and good if it were true. But what will happen in practice is that they will think that they have access to all information, that the corrections may be subtly wrong, that their ideas will no longer really be their ideas, and that sentences and paragraphs will be completed with meaningless or faulty junk.
- chii 4y ago
  > I understand why that's a common practice in education. Maybe it's time to change. not that i disagree, but what would you have it change to? The fact that there's no need to memorize anything any more, means that any tests which allows the use of tech (like the internet, or chatGPT) to retrieve data, means that students no longer need to commit things to long term memory. However, having the capability, and be able to recall facts when needed, and at high speed, is something that is foundational to higher creative thoughts. This higher, creative thought, is not really easy to test, so the memorization is the proxy. I do not know what form education (and the testing of it) would take on, if tools like chatGPT is allowed to be used.
  - BlueTemplar 4y ago
    In my experience (from exams allowing full access to computers and the Internet (for information retrieval, not communication with other people)) you need to memorise the most important concepts and patterns anyway, or you will be too slow during exams. YMMV.
hcks 4y ago
ChatGPT, please write a bland uninformed critique of LLMs as a boomer overinvested in education saying there is no way a chatbot can write something as complex as a college essay
- blackbear_ 4y ago
  You demonstrate a pretty superficial understanding of the article. The main point is that writing an essay is just the final step of a process that involves researching a topic, pondering about it, and organizing your thought logically and coherently. The danger of using ChatGPT is that students avoid all these steps, which means they will not learn about the topic, will not think about it, and will not learn to think logically.
avereveard 4y ago
> ChatGPT is, in fact, incapable of knowing anything at all. this is foundamentally wrong. I've been instructing chat gpt about wargame rules, and it's understanding those rules well enough to run rounds of combat. that is far beyond what this article pretends chatgpt to be (a massive pretrained markov chain) as all that learning is happening after training heck, you can make chatgpt pretend to be something that isn't, exhibiting creativity that is well beyond previous llm. > All it knows, all it knows are the statistical relationships of how words appear together again, you can ask nonsense, and it will not spew nonsense back. it has a cursory understanding of what nonsense is, and will not fall for easy traps (i.e. what year apollo 7 landed on the moon) that would trick straighforward probabilistic models all in all, this article toned down my respect for the blog down two pegs, if he's so wrong about this topic, how much was he wrong on other topics where I had no understanding and took his word for good?
- plutonorm 4y ago
  Agree. The guy is in deep cope territory.
- blackbear_ 4y ago
  > if he's so wrong about this topic This sounds very arrogant. What is your experience grading college essays? (In other words, you are talking about completely differet use-cases)
  - avereveard 4y ago
    his cursory understanding of chatgpt is easily falsiable
- mejutoco 4y ago
  Knowing is putting in relationship with, IMHO. You know what a table is, by were it usually is placed, differences with a chair, differences with a cat, materials it can have, usage given by people. In summary, how it exists in the context of something else. If we imagine a universe with only one thing (difficult because we would be there too, but bear with me), what could we know about that one thing? It would be the whole universe, all and nothing at the same time. I would like to propose an alternative take on the same: we also are incapable of knowing. We are just putting in relationship one thing with others, in a similar way as ChatGPT, but on a much bigger scale. And that is fine. We are still special: we have sensors (senses), we can move, feel pain, but from the point of view of knowledge we are still comparing things and putting them in relationship to one another.
nothing0001 4y ago
I asked chatgpt to generate the different orders of 5 words, it failed at it. Then I ask it to generalize the problem and then it refers to permutations (giving the formula). Then I ask it to use that information to generate the differents orders and then the chatgpt answer the original problem correctly. So it can use some of the information that it can't understand.
tragomaskhalos 4y ago
The analogy with alchemy was very on point; this stuff is an Eliza de nos jours, a clever parlour trick and for sure not without its uses, but it is not intelligence in any meaningful definition of that term. A lot of folks are dismissive of these criticisms by claiming that we have no evidence that human cognition is fundamentally different, but it must be - our referents are based in the real world, whereas the LLMs have no such referents, and it is this tie-back to reality that underpins our ability to act intelligently. And the argument that an equivalent understanding can somehow be an emergent property is undermined by the observation that the LLMs have no concept of truth, and no ability to tell whether their output is pure BS.
visarga 4y ago
Funny thing, in 2023 it takes a 8000 word essay (45Kb) to convey why chatGPT's essay is not really an essay. A mere 5 years ago this would have been sci-fi territory. > But it is only pretend, there is no mind there and that is the key to understanding what ChatGPT is (and thus what it is capable of). This is where an educator thinks they know AI better than people who work on it. A-priori decision it can't work. What they miss when they do that is the special quality of the training set. Language has special properties that lead to the emergence of abilities previously only possible for humans. Humans for example like to read fiction. It's all pretend, but enjoyable, maybe also instructive. We can simulate other people and their actions, we can infer their emotional states and motives. But it's still all pretend. What if language models can do the same - infer our emotional states, and respond in kind? It would be like generating a novel, something it has seen plenty of in the training data. The magic dust is not in the model and the next token prediction task, but in what data the model was trained on. Our language, our mind stream, taken from real experiences. Language has the ability to create things like chatGPT, language can create modern capable humans from just babies. The author missed language and only saw flaws on the model. Already, by coupling search with language model like bingChat we see even more extreme cases. Now it knows you said things about it on Twitter. These are already going beyond pure LLM. They are going to have a whole host of apps: calculator, calendar, search engine, python and JS execution engines, simulators, recursive calls, language chains, games, memory modules, episodic memory, knowledge bases, etc. I call this paradigm "language model with toys". And they will have this discussion and all the other discussions about it in the next training corpus, thus gaining a way to create a Self. It will have a self because we talk about it like a human.
- c-cube 4y ago
  > Language has special properties that lead to the emergence of abilities previously only possible for humans. Citation needed. > Humans for example like to read fiction. It's all pretend, but enjoyable, maybe also instructive. We can simulate other people and their actions, we can infer their emotional states and motives. Humans like to read fiction that is extremely rooted in our actual experience of the world. We don't read scifi about 5-dimensional universes with no protagonist that mostly behaves like people. Even on this very blog you'll find excellent articles on why Rings of Powers is bad because it's not believable, and why Lord of the Rings is great because it is (and rooted in much historical knowledge). > Language has the ability to create things like chatGPT, language can create modern capable humans from just babies. No? Babies grow into adults after years of experiencing the real world at every instant, as well as the social world. Language starts as a tool to express basic concepts that are very much grounded in the real world, and abstraction comes years later. It's not language that makes babies into adults!
jhanschoo 4y ago
> In that sense, ChatGPT’s greatest limitation is that it doesn’t know anything about anything; it isn’t storing definitions of words or a sense of their meanings or connections to real world objects or facts to reference about them. > ChatGPT is, in fact, incapable of knowing anything at all. All my knowledge about modern high-energy physics comes from reading about them. There are a great many things that I know of only second-hand, and not direct observation. Theoretically, any AI's knowledge of these things cannot be worse than mine, then.
- bleuarff 4y ago
  Your reading comprehension of these texts is probably more than just the statistical words correlations that LLMs only care about.
_nalply 4y ago
> there is no mind there Yesterday I read Wolfram's piece and he wrote: > I think, is that language is at a fundamental level somehow simpler than it seems. If Wolfram is true we could argument that GPT-3 was successful in capturing an at least an important aspect of language. And if we also presume that language is part of the mind, then we should refute the statement that there is [absolutely] no mind in GPT-3. However, GPT-3's mind, if at all, is not human. And this is difficult for us to keep in mind. After all we anthropomorphize everything and their grandsomething. Even I catch myself having the urge to be polite to ChatGPT and saying thank you. Or, the last time I even told ChatGPT not to worry about something. I know exactly that this is just weird and incongruous, but it did make me feel better, so I just did that. YOLO.

71 points by picture 4y ago

I like the essay a lot and I find it intellectually amusing that many negative comments here manifest exactly the flaws that could be prevented by learning the essay writing as described in the blog. The article clearly states assumptions, definitions, context and limitations of it's arguments and in that specific area it's very hard to refute them because they are sound. On the other side the gist of many comments here is that article is negative towards GPT and should be positive, i.e. the same generalist aproach that GPT generated text would take and which is hard to argue about. I'm not going to analyse particular flaws of separate comments because the article itself is really comprehensive and answers for many objections are better in original form there than any reproduction I can make here. But the high number of rebuttals that obviously have problems either understanding the essay or formulating a solid counterargument is curious. An obvious explanation is that some people couldn't resist the pleasure of letting AI dismiss the statement about insufficient AI capabilities, but that's a lazy one. Comments saying current AI is comparable to actual human inteligence may be right even though I can clearly see that AI is not performing actions I consider necessary for thinking. It's because I consider my own mind as a model of human inteligence, but as I learned many times before, the thinking process may be hugely different for different people. I have no idea if other people also do need to create an explicit model of something in their head to be able to "think" about it. It would explain A LOT for me if thinking of some other people really works in the same way as in GPT. That situation should however still be viewed not as a technology enhancement so big it reaches human level but reconsideration of what we consider a human level. This happens to me a lot. I want to make a joke, but it gets so long an complex that it becomes an analysis.

LesZedCB 4y ago

it just makes me sad and angry that people discount their pets experience because they are afraid of over-anthropomorphizing. we are animals which evolved just as our nearby mammalian friends. yes their experience is very different but not in any way fundamentally alien. sure they aren't jealous but they could experience anxiety and many other forms of suffering or excitement and pleasure just like we can. why negate that? if we can't understand that, I have a deep horror for the atrocities we may commit if/when we do create machine subjectivity.

deleted 4y ago
[deleted]
quonn 4y ago
It‘s still not clear to me why any subjective experience would arise in a computing process. We know it happens for humans, we can presume that it happens for mammals due to the biological similarity and common history. Unfortunately we don‘t understand it so far, but until we do, the burden of proof is on those who claim without evidence that there is consciousness in machines.
- jaggederest 4y ago
  There's no "consciousness" there. There's consciousness in what it is emulating, though, and if you give it enough of that, it will be indistinguishable from an actual consciousness, and so it may very well be that we must treat it as conscious. There's a philosophical concept, "zimboes", which is useful here - philosophical zombies which believe that they are conscious, and believe that they experience subjective understanding. I don't follow all the arguments, but from what I understand, they're used to argue for "if it quacks like a duck" presentations of consciousness, rather than some more abstract criteria.
  - ifdefdebug 4y ago
    > There's consciousness in what it is emulating This begs the question (if you made it to the footnotes of the article discussed here, you will know what I mean). More precisely, you start your argument by begging me to take for granted that it can emulate conscious beings - no, you will have to proof that, and not by making it into an axiom.
    - jaggederest 4y ago
      My argument is that there's no way of distinguishing "behaving like conscious beings" from "being conscious". It does not matter whether it is "genuinely emulating conscious beings" or not. It's a simple extension from "it mimics human text" to "what if it got so much better we can't tell the difference". If we can't tell the difference, definitionally it may as well be conscious. You can disagree with the axiom that it does mimic human text, but generating text with it looks like a pretty good mimicry to me. You may not agree that it will ever get that good - I would also agree with that. I don't think the proposed scenario is at all likely, certainly not in my lifetime, but if it was, I'd be there lobbying for the large language models to have rights.

firatsarlar 4y ago

It may be possible to resist something predicted, expected, feared. I couldn't quite get what the article wanted to reveal, except for the already spoken and known things?

triyambakam 4y ago

> Learn. Don’t cheat. Teach, then? I don't know this professor, but many college professors themselves could be replaces with ChatGPT.

isaacfrond 4y ago
Most teachers have a higher uptime than ChatGTP fortunately.
imiric 4y ago
Wow. That is all you got from the article? Students will find ways to make their lives "easier" by sabotaging their own education with assistants like AI, regardless of the quality of professors teaching them. Tools like ChatGPT can only regurgitate what was already written, often with confident but wrong takes. They can't offer new perspectives, or inspire students with a different way of thinking about the world. These are not teacher, let alone college professor, replacements.
- triyambakam 4y ago
  > Tools like ChatGPT can only regurgitate what was already written, often with confident but wrong takes. Yeah, just like many professors
- chii 4y ago
  > They can't offer new perspectives i think this remains to be seen. At least, if you asked chatGPT for recipes, they can produce recipes that aren't pre-existing. So for at least some small domain, it can offer new perspective. Whether those recipes tastes good is a different story...
  - hgomersall 4y ago
    Just as it can offer "new perspectives" on actual facts.
- charcircuit 4y ago
  >Tools like ChatGPT can only regurgitate what was already written, often with confident but wrong takes. They can't offer new perspectives, or inspire students with a different way of thinking about the world. That is not how ChatGPT works. It doesn't just regurgitate data it has seen. It its capable of offering new perspectives. Just ask it.

isaacfrond 4y ago

The article starts out in bold with the statement they pretend to be like minds – like human minds. But it is only pretend, there is no mind there and that is the key to understanding what ChatGPT is And surely they have no mind like us. But this obsession with whether or not it is like us, seems to miss the point, doesn't it. Is it useful? Despite the writers insistence that it is not, my children provide my with many examples that it is. For example, my daughter is learning to program and got negative feedback on her commenting style. (which admittedly where mostly absent) So, she asked how to do it better. The teacher gave only non-committal responses, like being helpful to the reader, etc. It is not the kind of instruction that is helpful for her. So she gave the whole program the ChatGTP and asked it to insert comments without changing the code. The next day, she took the output to the teacher and asked if this is what he meant. It was perfect. But now, she knows what is expected of her, and now she can do it without ChatGTP.

Gigachad 4y ago
When you drill down these discussion points, it all comes down to the fact we don’t exactly know how the human mind works so we can’t properly answer what is the difference between pretending to understand, and understanding.
- Jensson 4y ago
  We know that some humans can use information to solve harder problems, while others can not do it. We say the first kind built understanding about the information. You can easily test for this by just throwing hard problems at them that they haven't seen before, Google does that in their interviews for example (but many who do similar just takes problems people definitely have seen). So doing similar kinds of testing on ChatGPT we see that it doesn't understand the information. It can repeat a lot of stuff more or less verbatim, but it cannot apply most of the knowledge it can repeat, so it doesn't understand any of it. The only thing ChatGPT really understands are relationships between words that has been repeated in texts it has seen, that is some level of understanding but that has no relationship to what ChatGPT says about the words. When you ask ChatGPT to describe something it repeats a description it has seen of the word, when you ask it to do something with a word then it fully ignores that description and instead look at the relationships it has seen the word have to other words. So its understanding and its knowledge are completely separate things, so you can say that it doesn't understand anything it knows.

DougBTX 4y ago

I don’t buy this “it is just statistics” dismissal. Practically speaking, any multiple choice question is ultimately a “predict the next character, A, B, or C” challenge. Should we dismiss all multiple choice questions because they are just “predict the next token” tasks? There is a broader question about whether it is possible to learn from reading. Can a blind person ever really understand “blue”? If not, what can be learnt by reading? Why do we rely on reading and writing so heavily for learning? Edit: just want to note that this is a great site, really like the author’s approach and style. And maybe if I’ve not learnt from his writing, at the very least it is a good read :-)

jaggederest 4y ago
I agree. Quantum mechanics is "just statistics", and yet it still causes chemistry to happen. I think that the argument for qualia is pretty weak - if we can have a conversation about "blueness" with a blind person, we have to admit the possibility of "understanding" existing in some form in LLMs. Even if it's just a statistical emulation, at what point does a statistical emulation become reality, if it's a good enough emulation down to the metaphorical planck distance.
- bloppe 4y ago
  Anybody who has ever done arithmetic knows that human logic is more than just statistical pattern prediction. Humans can learn and execute algorithms in precise, deterministic ways, which current LLM's fundamentally cannot do.
  - mola 4y ago
    That's funny. So being intelligent is executing algorithms deterministically and being a machine is making mistakes and being non deterministic? How the tables have turned
  - space_fountain 4y ago
    I’m not sure that’s true. Human make mistakes performing math all the time. Human’s ability to accurately perform multiplication is very much a statistical phenomenon. Now you probably dismiss those mistakes as different than the sort of thing that a llm does when following a rigid set of instructions, but to me they seem different in degree but not kind

Animats 4y ago

He's already a few weeks behind. Bing's new AI can query Bing's search engine. So it hasn't "burned the library". It can reach out for new information. That's going to be really important. Large language models can already summarize documents. How long before you can tell a system "Read this book. Then I will have some questions about it"? Once you have that, you can apply the base model to domain-specific problems.

imiric 4y ago
I think you missed the argument about the two Roman Empire books. ChatGPT was clearly trained on them, or at least metadata about them, yet it couldn't produce an intuitive response about their relationship. And it couldn't even return the top search result that explained it. Sure, these issues will be fixed eventually, but the point of the article is that it can't replace the thinking of a student to write a quality essay. It might be able to regurgitate search results which will need to be fact checked, so at best it's useful as a research tool. BTW, Bing's AI is a really poor counterexample, as it can't even return search results reliably.

jaggederest 4y ago

It's a little frustrating when the author calls out "technofetishistic" faith in large language models[1] and then goes on to misunderstand their capabilities in the opposite direction. I'm no "true believer", they're just useful tools for modeling text generation, but to underestimate them is a shame. > Ideally, the essay writer has first observed their subject, then drawn some sort of analytical conclusion about that subject, then organized their evidence in a way that expresses the logical connections between various pieces of evidence, before finally communicating that to a reader in a way that is clear and persuasive. Of the elements discussed about "writing an essay" above, none are clearly impossible for a large language model to do at the present, despite the author's insistence that some of them are impossible. If the subject is something in writing (and you can pass it in the prompt), or is written about in the training set, then the model has indeed "observed" it, and to get it to draw analytical conclusions all you have to do is ask nicely and correctly, maybe using those words. Again, with organizing evidence, you merely must ask for it by name, rather than hoping that the model understands what you mean when you say "write an essay" - ask for "organize the evidence in a way to support/contradict the conclusion X", and finally of course communicating cogently to the reader is something the writer correctly understands is possible even with a trivial prompt. Sure, it moves the challenge from "write an essay" to "design a prompt to get the language model to write an essay", but that's "not writing an essay", or is it? It is immensely frustrating that people feel like a large language model is "cheating", but also don't consider using word processing software with grammar and spelling correction "cheating", even though they use fundamentally the same processes behind the scene and differ only in the user interface, really. [1] I don't like the term AI, I prefer large language model because it is unambiguous about the properties of the system - AI implies thought processes, which is a distraction at best from the useful properties of a complex statistical model.

chii 4y ago
> also don't consider using word processing software with grammar and spelling correction "cheating", even though they use fundamentally the same processes behind the scene and differ only in the user interface, really. i think it's a stretch to claim that spelling/grammar correction and large language models are similar or fundamentally the same. that's like saying the horse-drawn carriage is fundamentally the same as a plane. Sure, they both move people around, but the differences are much larger than the similarities.

zxcvbnm 4y ago

mola 4y ago
A lot of time writing is a personal endeavor. A way to communicate with yourself rather than others. It allows you to lay out your chaotic thoughts in a clear way. It's the foundation for critical thinking. It'll be a very sad world where kids don't atleast practice this skill.

zxcvbnm 4y ago

So then don't torture students by making them write long essays and waste their valuable time. Make it the task expressing the argument on a back of a napkin. More thought, less time wasted on writing text-amount. Also less of the teacher's time wasted, do they really read 100x student 3 pages, weekly? Don't they have anything better to do?

bloppe 4y ago
It's not fair to assume that's what this teacher asks of their students. Word minimums may be useful for children as a crude deterrent against laziness, but any serious student or teacher would recognize a good essay when they see one, no matter the length (though fitting a good and interesting essay on the back of a napkin would be almost unthinkably prolific). "I have made this letter longer than usual, only because I have not had the time to make it shorter." - Blaise Pascal
- izacus 4y ago
  There's nothing to assume because he describes what he asks of his students towards the last part of the essay. But people who think ChatGPT is great usually aren't capable of reading that far :P
- acatnamedjoe 4y ago
  A word count is also a signal to the student of what level of depth the teacher is hoping for. For most interesting questions you could write anything between a few sentences and a full book, depending on how much you develop and defend your arguments, evaluate other possible answers, etc. The word count is a signal on where on that spectrum you should be aiming.
- firatsarlar 4y ago
  If we see Pascal's quote as a pretext, it could be used to justify writing longer texts, even when brevity is possible and appropriate. However, by interpreting the quote as a genuine apology for the length of his letter, we can recognize the importance of clarity and conciseness in communication, and strive to make our own messages as clear and concise as possible. This highlights the importance of interpreting someone's words in the right context, and not using them to justify behaviors or attitudes that are not aligned with the speaker's true intentions. By being mindful of how we interpret others' words, we can avoid misunderstandings and communicate more effectively.

MrSqueezles 4y ago

Education has always done a terrible job of keeping up with technology. Kids in school now will graduate into a world in which all information will be instantly available and will be presented in whatever format is most suitable. They'll have computers correcting their grammar, improving their ideas, completing their sentences and paragraphs. Instead of learning those technologies themselves and instructing students how to best utilize them, teachers tell kids, "that's cheating". When you graduate and get your first job, your boss isn't going to take away your books, phone, and laptop, and ask you to write a report about a well understood subject. I understand why that's a common practice in education. Maybe it's time to change.

jacquesm 4y ago
That would be all fine and good if it were true. But what will happen in practice is that they will think that they have access to all information, that the corrections may be subtly wrong, that their ideas will no longer really be their ideas, and that sentences and paragraphs will be completed with meaningless or faulty junk.
chii 4y ago
> I understand why that's a common practice in education. Maybe it's time to change. not that i disagree, but what would you have it change to? The fact that there's no need to memorize anything any more, means that any tests which allows the use of tech (like the internet, or chatGPT) to retrieve data, means that students no longer need to commit things to long term memory. However, having the capability, and be able to recall facts when needed, and at high speed, is something that is foundational to higher creative thoughts. This higher, creative thought, is not really easy to test, so the memorization is the proxy. I do not know what form education (and the testing of it) would take on, if tools like chatGPT is allowed to be used.
- BlueTemplar 4y ago
  In my experience (from exams allowing full access to computers and the Internet (for information retrieval, not communication with other people)) you need to memorise the most important concepts and patterns anyway, or you will be too slow during exams. YMMV.

hcks 4y ago

ChatGPT, please write a bland uninformed critique of LLMs as a boomer overinvested in education saying there is no way a chatbot can write something as complex as a college essay

blackbear_ 4y ago
You demonstrate a pretty superficial understanding of the article. The main point is that writing an essay is just the final step of a process that involves researching a topic, pondering about it, and organizing your thought logically and coherently. The danger of using ChatGPT is that students avoid all these steps, which means they will not learn about the topic, will not think about it, and will not learn to think logically.

avereveard 4y ago

> ChatGPT is, in fact, incapable of knowing anything at all. this is foundamentally wrong. I've been instructing chat gpt about wargame rules, and it's understanding those rules well enough to run rounds of combat. that is far beyond what this article pretends chatgpt to be (a massive pretrained markov chain) as all that learning is happening after training heck, you can make chatgpt pretend to be something that isn't, exhibiting creativity that is well beyond previous llm. > All it knows, all it knows are the statistical relationships of how words appear together again, you can ask nonsense, and it will not spew nonsense back. it has a cursory understanding of what nonsense is, and will not fall for easy traps (i.e. what year apollo 7 landed on the moon) that would trick straighforward probabilistic models all in all, this article toned down my respect for the blog down two pegs, if he's so wrong about this topic, how much was he wrong on other topics where I had no understanding and took his word for good?

plutonorm 4y ago
Agree. The guy is in deep cope territory.
blackbear_ 4y ago
> if he's so wrong about this topic This sounds very arrogant. What is your experience grading college essays? (In other words, you are talking about completely differet use-cases)
- avereveard 4y ago
  his cursory understanding of chatgpt is easily falsiable
mejutoco 4y ago
Knowing is putting in relationship with, IMHO. You know what a table is, by were it usually is placed, differences with a chair, differences with a cat, materials it can have, usage given by people. In summary, how it exists in the context of something else. If we imagine a universe with only one thing (difficult because we would be there too, but bear with me), what could we know about that one thing? It would be the whole universe, all and nothing at the same time. I would like to propose an alternative take on the same: we also are incapable of knowing. We are just putting in relationship one thing with others, in a similar way as ChatGPT, but on a much bigger scale. And that is fine. We are still special: we have sensors (senses), we can move, feel pain, but from the point of view of knowledge we are still comparing things and putting them in relationship to one another.

nothing0001 4y ago

I asked chatgpt to generate the different orders of 5 words, it failed at it. Then I ask it to generalize the problem and then it refers to permutations (giving the formula). Then I ask it to use that information to generate the differents orders and then the chatgpt answer the original problem correctly. So it can use some of the information that it can't understand.

tragomaskhalos 4y ago

The analogy with alchemy was very on point; this stuff is an Eliza de nos jours, a clever parlour trick and for sure not without its uses, but it is not intelligence in any meaningful definition of that term. A lot of folks are dismissive of these criticisms by claiming that we have no evidence that human cognition is fundamentally different, but it must be - our referents are based in the real world, whereas the LLMs have no such referents, and it is this tie-back to reality that underpins our ability to act intelligently. And the argument that an equivalent understanding can somehow be an emergent property is undermined by the observation that the LLMs have no concept of truth, and no ability to tell whether their output is pure BS.

visarga 4y ago

Funny thing, in 2023 it takes a 8000 word essay (45Kb) to convey why chatGPT's essay is not really an essay. A mere 5 years ago this would have been sci-fi territory. > But it is only pretend, there is no mind there and that is the key to understanding what ChatGPT is (and thus what it is capable of). This is where an educator thinks they know AI better than people who work on it. A-priori decision it can't work. What they miss when they do that is the special quality of the training set. Language has special properties that lead to the emergence of abilities previously only possible for humans. Humans for example like to read fiction. It's all pretend, but enjoyable, maybe also instructive. We can simulate other people and their actions, we can infer their emotional states and motives. But it's still all pretend. What if language models can do the same - infer our emotional states, and respond in kind? It would be like generating a novel, something it has seen plenty of in the training data. The magic dust is not in the model and the next token prediction task, but in what data the model was trained on. Our language, our mind stream, taken from real experiences. Language has the ability to create things like chatGPT, language can create modern capable humans from just babies. The author missed language and only saw flaws on the model. Already, by coupling search with language model like bingChat we see even more extreme cases. Now it knows you said things about it on Twitter. These are already going beyond pure LLM. They are going to have a whole host of apps: calculator, calendar, search engine, python and JS execution engines, simulators, recursive calls, language chains, games, memory modules, episodic memory, knowledge bases, etc. I call this paradigm "language model with toys". And they will have this discussion and all the other discussions about it in the next training corpus, thus gaining a way to create a Self. It will have a self because we talk about it like a human.

c-cube 4y ago
> Language has special properties that lead to the emergence of abilities previously only possible for humans. Citation needed. > Humans for example like to read fiction. It's all pretend, but enjoyable, maybe also instructive. We can simulate other people and their actions, we can infer their emotional states and motives. Humans like to read fiction that is extremely rooted in our actual experience of the world. We don't read scifi about 5-dimensional universes with no protagonist that mostly behaves like people. Even on this very blog you'll find excellent articles on why Rings of Powers is bad because it's not believable, and why Lord of the Rings is great because it is (and rooted in much historical knowledge). > Language has the ability to create things like chatGPT, language can create modern capable humans from just babies. No? Babies grow into adults after years of experiencing the real world at every instant, as well as the social world. Language starts as a tool to express basic concepts that are very much grounded in the real world, and abstraction comes years later. It's not language that makes babies into adults!

jhanschoo 4y ago

> In that sense, ChatGPT’s greatest limitation is that it doesn’t know anything about anything; it isn’t storing definitions of words or a sense of their meanings or connections to real world objects or facts to reference about them. > ChatGPT is, in fact, incapable of knowing anything at all. All my knowledge about modern high-energy physics comes from reading about them. There are a great many things that I know of only second-hand, and not direct observation. Theoretically, any AI's knowledge of these things cannot be worse than mine, then.

bleuarff 4y ago
Your reading comprehension of these texts is probably more than just the statistical words correlations that LLMs only care about.

_nalply 4y ago

> there is no mind there Yesterday I read Wolfram's piece and he wrote: > I think, is that language is at a fundamental level somehow simpler than it seems. If Wolfram is true we could argument that GPT-3 was successful in capturing an at least an important aspect of language. And if we also presume that language is part of the mind, then we should refute the statement that there is [absolutely] no mind in GPT-3. However, GPT-3's mind, if at all, is not human. And this is difficult for us to keep in mind. After all we anthropomorphize everything and their grandsomething. Even I catch myself having the urge to be polite to ChatGPT and saying thank you. Or, the last time I even told ChatGPT not to worry about something. I know exactly that this is just weird and incongruous, but it did make me feel better, so I just did that. YOLO.