5 ms·

We obviously work in different ways, but how it comes up with the answer doesn't really matter. > So what is really the utility of "testing" it? Eventually th

We obviously work in different ways, but how it comes up with the answer doesn't really matter. > So what is really the utility of "testing" it? Eventually these models will be integrated into everyday products so we need to learn how trustworthy they are. These tests are written in natural language, the same way everybody interacts with them.

beepbooptheory 4y ago
It absolutely matters if you're going to say it passing a human test speaks to anything at all! If humans had a hardcoded multiplication table in their brain, it would be silly to "test" them on their multiplication ability, it would be pointless. Test's are useful precisely to gauge human retention/understanding, but they are even then a crude tool as I think anyone whose taken a test would agree. It borders on absurd to take it seriously for a model. Its like applauding the calculator for doing math. If you really do think this stuff is important, show it on its own terms. Make tests suited for the model with rigorous expectations tuned for a computer, not a human. Then you can start showing some impressive things. I just urge you to see the basic flaw in your conceptions here. I don't want to rob you of your enthusiasm, just better direct it.
- FeepingCreature 4y ago
  If you wanted to hire people to multiply numbers, it would be very worthwhile to test the functioning of the hardcoded multiplication table, or whatever other implementation they had. Nobody is saying that GPT-3 is smart. But it is surprisingly capable.
  - beepbooptheory 4y ago
    Right, and again, if there was some virtue to passing a test in itself, I would be on board. But a test for a human is meant to show something about the human that isnt just their test taking abilities. But with GPT we can only ever say that it has good human test taking abilities. Again, I know people are excited, and they have reason to be, but you gotta adjust your mindset here a little or you will end up in a weird B.F. Skinner logic hole.
- tiborsaas 4y ago
  In elementary school they hardcode the multiplication table and they do test for them, at least it was my experience. For some weird reason we were taught to remember 8x125 is 1000. It came handy maybe 3 times in the past 35 years, but in those cases people looked at me like a wizard :) > Test's are useful precisely to gauge human retention/understanding Indeed, and these models are also compressing information they were trained on. It makes sense to test them how well can we access and query that knowledge. I'm not saying at all, that passing tests make them human or anything like us, but it speaks a lot about the quality of the model. It's not the ultimate quality check of course, but it's an important one, because you can assign a score to the results of the test. > Its like applauding the calculator for doing math. Show that calculator to Charles Babbage, he would be in awe :) This is probably the leap why most people are freaking out, it feels a significant step towards intelligent machines regardless of the system under the hood. I actually tested ChatGPT, (like a few million other people). I'm not saying it was done with scientific rigour of course. It passed a few flawlessly and failed some other. I believe I know what I'm interacting with, I've read about how GPT works and I'm planning to learn more. Don't worry about my enthusiasm :)

beepbooptheory 4y ago

It absolutely matters if you're going to say it passing a human test speaks to anything at all! If humans had a hardcoded multiplication table in their brain, it would be silly to "test" them on their multiplication ability, it would be pointless. Test's are useful precisely to gauge human retention/understanding, but they are even then a crude tool as I think anyone whose taken a test would agree. It borders on absurd to take it seriously for a model. Its like applauding the calculator for doing math. If you really do think this stuff is important, show it on its own terms. Make tests suited for the model with rigorous expectations tuned for a computer, not a human. Then you can start showing some impressive things. I just urge you to see the basic flaw in your conceptions here. I don't want to rob you of your enthusiasm, just better direct it.

FeepingCreature 4y ago
If you wanted to hire people to multiply numbers, it would be very worthwhile to test the functioning of the hardcoded multiplication table, or whatever other implementation they had. Nobody is saying that GPT-3 is smart. But it is surprisingly capable.
- beepbooptheory 4y ago
  Right, and again, if there was some virtue to passing a test in itself, I would be on board. But a test for a human is meant to show something about the human that isnt just their test taking abilities. But with GPT we can only ever say that it has good human test taking abilities. Again, I know people are excited, and they have reason to be, but you gotta adjust your mindset here a little or you will end up in a weird B.F. Skinner logic hole.
tiborsaas 4y ago
In elementary school they hardcode the multiplication table and they do test for them, at least it was my experience. For some weird reason we were taught to remember 8x125 is 1000. It came handy maybe 3 times in the past 35 years, but in those cases people looked at me like a wizard :) > Test's are useful precisely to gauge human retention/understanding Indeed, and these models are also compressing information they were trained on. It makes sense to test them how well can we access and query that knowledge. I'm not saying at all, that passing tests make them human or anything like us, but it speaks a lot about the quality of the model. It's not the ultimate quality check of course, but it's an important one, because you can assign a score to the results of the test. > Its like applauding the calculator for doing math. Show that calculator to Charles Babbage, he would be in awe :) This is probably the leap why most people are freaking out, it feels a significant step towards intelligent machines regardless of the system under the hood. I actually tested ChatGPT, (like a few million other people). I'm not saying it was done with scientific rigour of course. It passed a few flawlessly and failed some other. I believe I know what I'm interacting with, I've read about how GPT works and I'm planning to learn more. Don't worry about my enthusiasm :)