3 ms·

I wish I had less morals to make this myself, but how long untill GPT-3 (or 4) is used to just produce material to pass the whole university process? It could p

I wish I had less morals to make this myself, but how long untill GPT-3 (or 4) is used to just produce material to pass the whole university process? It could probably solve exam questions.

samhw 5y ago
GPT-3 (and almost certainly also GPT-4) is not going to be capable of solving exam questions. I think you've got a radically exaggerated view of what it's capable of. It's a tool for generating roughly coherent text given a prompt.
- emteycz 5y ago
  I think you've got a radically under-exaggerated view of what it's capable of. It's fully capable of answering 98% of exam questions. It's even capable of producing working code from description, and much more.
  - samhw 5y ago
    > It's fully capable of answering 98% of exam questions. What does this even mean? Is there some dataset of all exam questions? And to what standard is it capable of answering them? Do you have a source to substantiate any of this?
    - emteycz 5y ago
      I simply tried. You can too. It gives fully correct answers deemed correct by the textbooks I took them from - and not word by word so it's not simple matching. It failed on a fusion physics scripta question - but I'm pretty sure almost nobody in the world knows that answer as it's very cutting edge stuff.
      - tomerv 5y ago
        What sort of questions? Care to give a concrete example? GPT-3 is certainly capable of regurgitating facts, even to the point of rewording them, and connecting a small number of facts together. But it's not capable of making logical deductions in any consistent manner.
      - probably_wrong 5y ago
        I tried it. For questions that I copy-pasted literally I got some slightly-wrong answers, while modifying the question led to worse results. Here are two examples. Everything after "A:" was generated by GPT3 Neo. Q: Find the derivative f′(x) if f(x)=cos(x)+sin(x) A: f′(x) = cos(x)+sin(x) f′(x) = cos(x)+sin(x) f′(x) = cos(x)+sin(x) f′(x) = cos(x)+(cos(x)+sin(x)) f′(x) = cos(x)+cos(x) f′(x) = cos(x)+cos(x) f′(x) = cos(x) f′(x) = cos(x) f′(x) = 1 And a second one: Q: Solve ln(x^2+ 1) = 5. A: x^2+1 = e^5. Q: Solve ln(x^3+ 1) = 5. A: x^3+1 = e^5. Q: Solve ln(x^4+ 1) = 5. A: x^4+1 = e^5. Q: Solve ln(x^5+ 1) = 5. A: x^5+1 = e^5. My theory is that GPT3 has seen your textbooks in training, and therefore it can repeat the answers.
        emteycz 5y ago
        I agree it's bad at math. I don't believe it knew my exam questions because I translated them from Czech printed books. I tried computer science and literature questions - it got 100% on the literature section (5 questions) and 4/5 on the CS section. I'll try again later and post the log too, but I'm on the go now.
        samhw 5y ago
        Thanks - I missed these replies, but this is an interesting result. I'll grant you that the results in all the comments are certainly far better than I would have expected.
  - mkl 5y ago
    > It's fully capable of answering 98% of exam questions. Citation? I haven't heard of it being that good at such a general task, and DDG & Google don't seem to return any hits for that statistic.

I wish I had less morals to make this myself, but how long untill GPT-3 (or 4) is used to just produce material to pass the whole university process? It could probably solve exam questions.

samhw 5y ago

GPT-3 (and almost certainly also GPT-4) is not going to be capable of solving exam questions. I think you've got a radically exaggerated view of what it's capable of. It's a tool for generating roughly coherent text given a prompt.

emteycz 5y ago
I think you've got a radically under-exaggerated view of what it's capable of. It's fully capable of answering 98% of exam questions. It's even capable of producing working code from description, and much more.
- samhw 5y ago
  > It's fully capable of answering 98% of exam questions. What does this even mean? Is there some dataset of all exam questions? And to what standard is it capable of answering them? Do you have a source to substantiate any of this?
  - emteycz 5y ago
    I simply tried. You can too. It gives fully correct answers deemed correct by the textbooks I took them from - and not word by word so it's not simple matching. It failed on a fusion physics scripta question - but I'm pretty sure almost nobody in the world knows that answer as it's very cutting edge stuff.
    - tomerv 5y ago
      What sort of questions? Care to give a concrete example? GPT-3 is certainly capable of regurgitating facts, even to the point of rewording them, and connecting a small number of facts together. But it's not capable of making logical deductions in any consistent manner.
    - probably_wrong 5y ago
      I tried it. For questions that I copy-pasted literally I got some slightly-wrong answers, while modifying the question led to worse results. Here are two examples. Everything after "A:" was generated by GPT3 Neo. Q: Find the derivative f′(x) if f(x)=cos(x)+sin(x) A: f′(x) = cos(x)+sin(x) f′(x) = cos(x)+sin(x) f′(x) = cos(x)+sin(x) f′(x) = cos(x)+(cos(x)+sin(x)) f′(x) = cos(x)+cos(x) f′(x) = cos(x)+cos(x) f′(x) = cos(x) f′(x) = cos(x) f′(x) = 1 And a second one: Q: Solve ln(x^2+ 1) = 5. A: x^2+1 = e^5. Q: Solve ln(x^3+ 1) = 5. A: x^3+1 = e^5. Q: Solve ln(x^4+ 1) = 5. A: x^4+1 = e^5. Q: Solve ln(x^5+ 1) = 5. A: x^5+1 = e^5. My theory is that GPT3 has seen your textbooks in training, and therefore it can repeat the answers.
      - emteycz 5y ago
        I agree it's bad at math. I don't believe it knew my exam questions because I translated them from Czech printed books. I tried computer science and literature questions - it got 100% on the literature section (5 questions) and 4/5 on the CS section. I'll try again later and post the log too, but I'm on the go now.
        samhw 5y ago
        Thanks - I missed these replies, but this is an interesting result. I'll grant you that the results in all the comments are certainly far better than I would have expected.
- mkl 5y ago
  > It's fully capable of answering 98% of exam questions. Citation? I haven't heard of it being that good at such a general task, and DDG & Google don't seem to return any hits for that statistic.