6 ms·

> required having a tutor (me) on hand to help make the context make sense. I wonder how good ChatGPT would be as that tutor. You can ask it to “explain like

by acchow 2y ago

> required having a tutor (me) on hand to help make the context make sense. I wonder how good ChatGPT would be as that tutor. You can ask it to “explain like I’m 12”

CamperBob2 2y ago
It's almost unbelievably good as a tutor. Still, you need to check everything it tells you. Treat verification as part of the lesson.
- dgacmu 2y ago
  So, I'm a professor, and I have, um, really strong opinions about this. :-) Perhaps too strong and long for the current forum. But I'll see if I can be brief. ChatGPT is really, really good at providing solid answers of varying levels of detail and complexity to hyper-common questions such as those used in problem sets. This is one part of the skill set of a tutor, and it's a valuable one. When I interview TAs for my classes, however, I actually put a lot more emphasis on a different skill: The ability to get into a student's head and understand where their conceptual difficulty or misunderstanding is. This is a very different skill, and it's one that ChatGPT isn't as good at, because we've gone from "maximum likelihood answers from questions that are in the middle of the distribution" into a wide range of possible sources of confusion, which the student may lack the words to explain in a precise way. In the case of my kid, the PV=nRT question manifested as "I don't get it!" (with more exclamation points). Asking ChatGPT (well, copilot, since I have institutional access to that, but it uses ChatGPT) to help understand the problem: It digressed and introduced Boyle's Law, threw in a new symbol "I" (ok, the 12 year old) had never seen for "proportional to", and ... in some sense just added to the cognitive overload. The human approach was to ask a question: Have you ever seen this equation before? (No) Oh! Well, let's talk a little about gases.. Now, responding to ChatGPT and asking "No, that didn't help. Please ELI5 instead?" actually produced a much better answer: An analogy using a balloon. Which, amusingly, is exactly how I explained the behavior of gases to her. But even here, there's a bit of a difference: In explaining it to her, I did so socratically: "Ok, so imagine a balloon. If you heat the air inside the balloon, what happens?" "Um, it gets bigger, right?" "Yup, ..." (and now, knowing that she got that part, we could go on...) That's something you can absolutely imagine trying to program around an LLM, but it's not a native way of interacting with it. So ... I'd instead be a little more cautious here and say that ChatGPT potentially provides a really useful piece of what a human tutor offers, but it loses on the interactive exchange that helps much more rapidly zoom in on the source of confusion and correct it. Assuming that it's right. I think that for a particularly sophisticated consumer, it can be more valuable, but it requires knowing what you don't know, in some sense: The ability to already isolate what you're confused by. Once you know the question to ask, ChatGPT can often provide it -- again, assuming that some quirk of its training or your phrasing doesn't cause it to generate an answer that's wrong in some way.
  - CamperBob2 2y ago
    I wonder if you are getting very different results with Copilot relative to what I've seen lately from ChatGPT4o. It did a pretty good job with the successive "Explain Galois theory like I am 5/16/133" prompts that I tried earlier. Seems likely that the chat system prompt is different from Copilot's, but who knows how. I will say that I have asked it to solve gas-law problems before, back when the base ChatGPT4 model was in use, and it didn't do so well there. It came up with the correct logic but I found it hard to get it to use consistent units, so the numbers kept coming out wrong. In general I find it super useful at improving my conceptual understanding of things -- it's at least as good as a good human tutor, at levels up to and including fairly-advanced signal processing topics -- but the details are less reliable. Feeding the same question to Gemini is a good way to double-check the numbers in my experience. If they agree, the numbers are probably right, while if they don't, then it's likely they're both wrong.
    - silasdavis 2y ago
      Can you ask it to explain galois theory like he's 21?
      - CamperBob2 2y ago
        ---------------- "Let’s imagine Évariste Galois survived that fateful duel and continued his work. What would he have done next? If Galois had lived past 21, he would have likely expanded and deepened his revolutionary ideas. By that time, he had already laid down the foundation of what we now call Galois theory, but there were still many directions he could have explored further. Generalization of Galois Theory: Galois had just scratched the surface of the powerful connection between field theory and group theory. His next logical step might have been to generalize his ideas to more complex situations, such as understanding the symmetries of solutions to equations in multiple variables or extending his work to non-polynomial equations. Exploration of Solvable Groups: Galois introduced the concept of solvable groups to determine whether a polynomial equation could be solved by radicals (using addition, multiplication, and roots). If he had lived longer, he would likely have delved deeper into classifying different types of groups and exploring their properties. This could have led to a more detailed theory of group structures, potentially anticipating parts of what we now know as modern group theory. Linking Galois Theory with Other Areas: Galois might have explored connections between his theory and other areas of mathematics, such as geometry, number theory, or even early versions of what we now call topology. For instance, he could have investigated how the symmetries of geometric shapes relate to the symmetries of algebraic equations. Developing Further Applications: Beyond theory, Galois could have sought applications of his work in more practical areas. Perhaps he would have applied his ideas to solving problems in physics, mechanics, or other emerging fields, where symmetry plays a crucial role. Inspiring the Next Generation: If Galois had lived, he might have become a prominent teacher and mentor, influencing a new generation of mathematicians. His radical ideas would have challenged and inspired others to push the boundaries of mathematics even further. Galois could have spearheaded a new movement in algebra, similar to how Einstein revolutionized physics. Refining His Work: Galois' original manuscripts were hastily written under dire circumstances. Given more time, he would likely have polished and clarified his ideas, making them more accessible to other mathematicians of the time. This might have sped up the acceptance and expansion of his ideas in the mathematical community." ---------------- I'd give it a C for effort, I guess. I don't know enough about the topic to ask it for more specific predictions. At the end of the day it can only work with what it was trained with, so any genuine insights it comes up with will be due more to accident than inspiration.
- jacobolus 2y ago
  ChatGPT is an unbelievably bad tutor if what you want a tutorial about is even a little bit obscure (e.g. the answer you want isn't already included in Wikipedia). It just confidently states vaguely plausible sounding made up nonsense, and then when you ask it if it was mistaken it shamelessly makes up different total nonsense, and if you ask it for sources it makes up non-existent sources, and then when you look whatever it was up for yourself you have to spend 2x as long as you originally would have chasing down wrong paths and trying to understand exactly which parts ChatGPT was wrong about (most of them). And that's assuming you are a very savvy and media literate inquirer with plenty of domain expertise. In cases where the answer you want was already easily findable, ChatGPT still is wrong about a lot of it, and you could have more easily gotten a (mostly) correct answer by looking at standard sources, or if you want to be more careful tracking down their actually existing cited sources or doing a skim search through the academic literature. If you ask it something in a topic you are not already an expert about, or if you are e.g. an ordinary high school or college student, you are almost certainly coming away from the conversation with serious misconceptions.
  - CamperBob2 2y ago
    ChatGPT is an unbelievably bad tutor if what you want a tutorial about is even a little bit obscure That has absolutely not been my experience at all. It's brought me up to speed in areas from ML to advanced DSP that I'd been struggling with for a long time. How long has it been since you used it, and what did you ask it?
    - jacobolus 2y ago
      I have tried asking it all sorts of questions about specific obscure word etymologies and translations, obscure people's biographies (ancient and modern), historical events, organizations, academic citations, mathematical definitions and theorems, physical experiments, old machines, native plants, chemical reactions, diseases, engineering methods, ..., and it almost invariably flubs every question I throw at it, sometimes subtly and sometimes quite dramatically, often making up abject nonsense out of whole cloth. As a result I don't bother too much; I've found it to waste more time than it saves. To be fair, the kinds of questions I would want a tool like this to answer are usually ones I would have to spend some time and effort hunting to answer properly, and I'm pretty fast and effective at finding information. I haven't tried asking too much about questions that I could trivially answer some other way. If what you want to know can be found in any intro undergrad textbook or standard dictionary (or Wikipedia), it's plausible that it would be better able to parrot back more or less the correct thing. But again, I haven't done much of this, preferring to just get hold of the relevant dictionary or textbook and read it directly. I'll give you an example. I just now asked chatgpt.com what Lexell's theorem is and it says this: > Lexell's theorem is a result in geometry related to spherical triangles. Named after the mathematician Michel Léonard Jean Leclerc, known as Lexell, it states: ¶ In a spherical triangle, if the sum of the angles is greater than π radians (or 180 degrees), then the spherical excess (the amount by which the sum of the angles exceeds π) is equal to the area of the spherical triangle on a unit sphere. ¶ In simpler terms, for a spherical triangle, the difference between the sum of its angles and π radians (180 degrees) gives the area of the triangle when the sphere is of unit radius. This theorem is fundamental in spherical geometry and helps relate angular measurements directly to areas on a sphere. This gets the basic topic right ("is a result in geometry related to spherical triangles", involves area or spherical excess) but everything else about the answer, starting with the mathematician's identity, is completely wrong. If I tell it that this is incorrect, it repeats a random assortment of other statements, none of which is actually the theorem I am asking about. E.g. > [...] In a spherical triangle, if you have a spherical triangle with vertices A, B, and C, and the sides of the triangle are a, b, and c (measured in radians), then: ¶ cos⁡(a)cos⁡(b) + sin⁡(a)sin⁡(b)cos⁡(C) = cos⁡(c). [...] or > [...] In a spherical polyhedron, the sum of the angles at each vertex is equal to 2π radians minus the sum of the interior angles of the faces meeting at that vertex. [...] If you want to know what Lexell's theorem actually is, you can read the Wikipedia article I wrote last year: https://en.wikipedia.org/wiki/Lexell%27s_theorem https://en.wikipedia.org/wiki/Lexell%27s_theorem > every spherical triangle with the same surface area on a fixed base has its apex on a small circle, called Lexell's circle or Lexell's locus, passing through each of the two points antipodal to the two base vertices. The problem ChatGPT has is that it's not able to just say something true but incomplete such as "I'm not sure what Lexell's theorem is or who Lexell was, but I know the theorem has something to do with spherical trigonometry; maybe it could be found in the more comprehensive books about the subject such as Todhunter & Leathem 1901 or Casey 1889". Instead it just authoritatively spouts one bit of nonsense after another. (Every topic I have ever tried asking it about in detail is more or less the same.) The incorrect statements range from subtly wrong (e.g. two different things with similar names got conflated and some of the properties of the more common one were incorrectly applied to the other) to complete nonsense (jumbles of technical jargon strung together that are more or less gibberish). It's clear if you read carefully about any technical topic that it doesn't actually understand what it is saying, and is just combining bits of vaguely related material. Answers to technical questions are almost never entirely technically accurate unless you ask a very standard question about a very basic topic. Anyone using it for any purpose should (a) be already pretty media literate with some domain expertise, and (b) be willing to carefully verify every part of every statement.

by acchow 2y ago

> required having a tutor (me) on hand to help make the context make sense. I wonder how good ChatGPT would be as that tutor. You can ask it to “explain like I’m 12”

CamperBob2 2y ago

It's almost unbelievably good as a tutor. Still, you need to check everything it tells you. Treat verification as part of the lesson.

dgacmu 2y ago
So, I'm a professor, and I have, um, really strong opinions about this. :-) Perhaps too strong and long for the current forum. But I'll see if I can be brief. ChatGPT is really, really good at providing solid answers of varying levels of detail and complexity to hyper-common questions such as those used in problem sets. This is one part of the skill set of a tutor, and it's a valuable one. When I interview TAs for my classes, however, I actually put a lot more emphasis on a different skill: The ability to get into a student's head and understand where their conceptual difficulty or misunderstanding is. This is a very different skill, and it's one that ChatGPT isn't as good at, because we've gone from "maximum likelihood answers from questions that are in the middle of the distribution" into a wide range of possible sources of confusion, which the student may lack the words to explain in a precise way. In the case of my kid, the PV=nRT question manifested as "I don't get it!" (with more exclamation points). Asking ChatGPT (well, copilot, since I have institutional access to that, but it uses ChatGPT) to help understand the problem: It digressed and introduced Boyle's Law, threw in a new symbol "I" (ok, the 12 year old) had never seen for "proportional to", and ... in some sense just added to the cognitive overload. The human approach was to ask a question: Have you ever seen this equation before? (No) Oh! Well, let's talk a little about gases.. Now, responding to ChatGPT and asking "No, that didn't help. Please ELI5 instead?" actually produced a much better answer: An analogy using a balloon. Which, amusingly, is exactly how I explained the behavior of gases to her. But even here, there's a bit of a difference: In explaining it to her, I did so socratically: "Ok, so imagine a balloon. If you heat the air inside the balloon, what happens?" "Um, it gets bigger, right?" "Yup, ..." (and now, knowing that she got that part, we could go on...) That's something you can absolutely imagine trying to program around an LLM, but it's not a native way of interacting with it. So ... I'd instead be a little more cautious here and say that ChatGPT potentially provides a really useful piece of what a human tutor offers, but it loses on the interactive exchange that helps much more rapidly zoom in on the source of confusion and correct it. Assuming that it's right. I think that for a particularly sophisticated consumer, it can be more valuable, but it requires knowing what you don't know, in some sense: The ability to already isolate what you're confused by. Once you know the question to ask, ChatGPT can often provide it -- again, assuming that some quirk of its training or your phrasing doesn't cause it to generate an answer that's wrong in some way.
- CamperBob2 2y ago
  I wonder if you are getting very different results with Copilot relative to what I've seen lately from ChatGPT4o. It did a pretty good job with the successive "Explain Galois theory like I am 5/16/133" prompts that I tried earlier. Seems likely that the chat system prompt is different from Copilot's, but who knows how. I will say that I have asked it to solve gas-law problems before, back when the base ChatGPT4 model was in use, and it didn't do so well there. It came up with the correct logic but I found it hard to get it to use consistent units, so the numbers kept coming out wrong. In general I find it super useful at improving my conceptual understanding of things -- it's at least as good as a good human tutor, at levels up to and including fairly-advanced signal processing topics -- but the details are less reliable. Feeding the same question to Gemini is a good way to double-check the numbers in my experience. If they agree, the numbers are probably right, while if they don't, then it's likely they're both wrong.
  - silasdavis 2y ago
    Can you ask it to explain galois theory like he's 21?
    - CamperBob2 2y ago
      ---------------- "Let’s imagine Évariste Galois survived that fateful duel and continued his work. What would he have done next? If Galois had lived past 21, he would have likely expanded and deepened his revolutionary ideas. By that time, he had already laid down the foundation of what we now call Galois theory, but there were still many directions he could have explored further. Generalization of Galois Theory: Galois had just scratched the surface of the powerful connection between field theory and group theory. His next logical step might have been to generalize his ideas to more complex situations, such as understanding the symmetries of solutions to equations in multiple variables or extending his work to non-polynomial equations. Exploration of Solvable Groups: Galois introduced the concept of solvable groups to determine whether a polynomial equation could be solved by radicals (using addition, multiplication, and roots). If he had lived longer, he would likely have delved deeper into classifying different types of groups and exploring their properties. This could have led to a more detailed theory of group structures, potentially anticipating parts of what we now know as modern group theory. Linking Galois Theory with Other Areas: Galois might have explored connections between his theory and other areas of mathematics, such as geometry, number theory, or even early versions of what we now call topology. For instance, he could have investigated how the symmetries of geometric shapes relate to the symmetries of algebraic equations. Developing Further Applications: Beyond theory, Galois could have sought applications of his work in more practical areas. Perhaps he would have applied his ideas to solving problems in physics, mechanics, or other emerging fields, where symmetry plays a crucial role. Inspiring the Next Generation: If Galois had lived, he might have become a prominent teacher and mentor, influencing a new generation of mathematicians. His radical ideas would have challenged and inspired others to push the boundaries of mathematics even further. Galois could have spearheaded a new movement in algebra, similar to how Einstein revolutionized physics. Refining His Work: Galois' original manuscripts were hastily written under dire circumstances. Given more time, he would likely have polished and clarified his ideas, making them more accessible to other mathematicians of the time. This might have sped up the acceptance and expansion of his ideas in the mathematical community." ---------------- I'd give it a C for effort, I guess. I don't know enough about the topic to ask it for more specific predictions. At the end of the day it can only work with what it was trained with, so any genuine insights it comes up with will be due more to accident than inspiration.
jacobolus 2y ago
ChatGPT is an unbelievably bad tutor if what you want a tutorial about is even a little bit obscure (e.g. the answer you want isn't already included in Wikipedia). It just confidently states vaguely plausible sounding made up nonsense, and then when you ask it if it was mistaken it shamelessly makes up different total nonsense, and if you ask it for sources it makes up non-existent sources, and then when you look whatever it was up for yourself you have to spend 2x as long as you originally would have chasing down wrong paths and trying to understand exactly which parts ChatGPT was wrong about (most of them). And that's assuming you are a very savvy and media literate inquirer with plenty of domain expertise. In cases where the answer you want was already easily findable, ChatGPT still is wrong about a lot of it, and you could have more easily gotten a (mostly) correct answer by looking at standard sources, or if you want to be more careful tracking down their actually existing cited sources or doing a skim search through the academic literature. If you ask it something in a topic you are not already an expert about, or if you are e.g. an ordinary high school or college student, you are almost certainly coming away from the conversation with serious misconceptions.
- CamperBob2 2y ago
  ChatGPT is an unbelievably bad tutor if what you want a tutorial about is even a little bit obscure That has absolutely not been my experience at all. It's brought me up to speed in areas from ML to advanced DSP that I'd been struggling with for a long time. How long has it been since you used it, and what did you ask it?
  - jacobolus 2y ago
    I have tried asking it all sorts of questions about specific obscure word etymologies and translations, obscure people's biographies (ancient and modern), historical events, organizations, academic citations, mathematical definitions and theorems, physical experiments, old machines, native plants, chemical reactions, diseases, engineering methods, ..., and it almost invariably flubs every question I throw at it, sometimes subtly and sometimes quite dramatically, often making up abject nonsense out of whole cloth. As a result I don't bother too much; I've found it to waste more time than it saves. To be fair, the kinds of questions I would want a tool like this to answer are usually ones I would have to spend some time and effort hunting to answer properly, and I'm pretty fast and effective at finding information. I haven't tried asking too much about questions that I could trivially answer some other way. If what you want to know can be found in any intro undergrad textbook or standard dictionary (or Wikipedia), it's plausible that it would be better able to parrot back more or less the correct thing. But again, I haven't done much of this, preferring to just get hold of the relevant dictionary or textbook and read it directly. I'll give you an example. I just now asked chatgpt.com what Lexell's theorem is and it says this: > Lexell's theorem is a result in geometry related to spherical triangles. Named after the mathematician Michel Léonard Jean Leclerc, known as Lexell, it states: ¶ In a spherical triangle, if the sum of the angles is greater than π radians (or 180 degrees), then the spherical excess (the amount by which the sum of the angles exceeds π) is equal to the area of the spherical triangle on a unit sphere. ¶ In simpler terms, for a spherical triangle, the difference between the sum of its angles and π radians (180 degrees) gives the area of the triangle when the sphere is of unit radius. This theorem is fundamental in spherical geometry and helps relate angular measurements directly to areas on a sphere. This gets the basic topic right ("is a result in geometry related to spherical triangles", involves area or spherical excess) but everything else about the answer, starting with the mathematician's identity, is completely wrong. If I tell it that this is incorrect, it repeats a random assortment of other statements, none of which is actually the theorem I am asking about. E.g. > [...] In a spherical triangle, if you have a spherical triangle with vertices A, B, and C, and the sides of the triangle are a, b, and c (measured in radians), then: ¶ cos⁡(a)cos⁡(b) + sin⁡(a)sin⁡(b)cos⁡(C) = cos⁡(c). [...] or > [...] In a spherical polyhedron, the sum of the angles at each vertex is equal to 2π radians minus the sum of the interior angles of the faces meeting at that vertex. [...] If you want to know what Lexell's theorem actually is, you can read the Wikipedia article I wrote last year: https://en.wikipedia.org/wiki/Lexell%27s_theorem https://en.wikipedia.org/wiki/Lexell%27s_theorem > every spherical triangle with the same surface area on a fixed base has its apex on a small circle, called Lexell's circle or Lexell's locus, passing through each of the two points antipodal to the two base vertices. The problem ChatGPT has is that it's not able to just say something true but incomplete such as "I'm not sure what Lexell's theorem is or who Lexell was, but I know the theorem has something to do with spherical trigonometry; maybe it could be found in the more comprehensive books about the subject such as Todhunter & Leathem 1901 or Casey 1889". Instead it just authoritatively spouts one bit of nonsense after another. (Every topic I have ever tried asking it about in detail is more or less the same.) The incorrect statements range from subtly wrong (e.g. two different things with similar names got conflated and some of the properties of the more common one were incorrectly applied to the other) to complete nonsense (jumbles of technical jargon strung together that are more or less gibberish). It's clear if you read carefully about any technical topic that it doesn't actually understand what it is saying, and is just combining bits of vaguely related material. Answers to technical questions are almost never entirely technically accurate unless you ask a very standard question about a very basic topic. Anyone using it for any purpose should (a) be already pretty media literate with some domain expertise, and (b) be willing to carefully verify every part of every statement.