5 ms·

Oh how quickly the goalposts shift... If the test was done correctly and there was no data leakage, this absolutely does not compare to someone equipped with th

Oh how quickly the goalposts shift... If the test was done correctly and there was no data leakage, this absolutely does not compare to someone equipped with the answer key.

salamo 4y ago
There's even a specific term for shifting AI goalposts: https://en.wikipedia.org/wiki/AI_effect https://en.wikipedia.org/wiki/AI_effect
- chamwislothe2nd 4y ago
  https://en.m.wikipedia.org/wiki/Moving_the_goalposts https://en.m.wikipedia.org/wiki/Moving_the_goalposts reviewing the 'see also' section of the goalposts wikipage is like looking at a syllabus for PR. People are terrible.
- bamboozled 4y ago
  I also think there is a novel new effect happening which is that if one question anything about AI. For example: I tired to use Dall-E for something useful and it sucked. You get labelled...Anti-AI. Being critical about certain things isn't allowed at the moment. Kind of like the 2023 replacement for Anti-vaxxer if you don't want to take medicine every 3 months. My honest opinion, is that while a lot of this technology is cool and novel, it's not perfect, it has uses and then it doesn't. What it does have is a lot of good marketing and PR behind it and so you're up against a lot of that too. One example I like is the Dall-e homepage shoes the curated "best" images generated, which might be showing only 1 in 1000 or more of the best images. I was excited when I first tried it based on the front page. But now it's drawing people with feet for hands for me and it's super super fun and entertaining to play with, I don't feel it works quite as advertised most of the time. Disclaimer: Not anti-ai and I took my Covid19 vaccines (in the beginning, no boosters)
ipaddr 4y ago
It compares to someone with a textbook and/or access to the internet.
- zone411 4y ago
  ...and the ability to synthetize both into a response to a question it has never encountered before
reichardt 4y ago
Do you know that the answer key wasn't part of the train set of ChatGPT?
- mshake2 4y ago
  >Argument from ignorance, also known as appeal to ignorance (in which ignorance represents "a lack of contrary evidence"), is a fallacy in informal logic. It asserts that a proposition is true because it has not yet been proven false or a proposition is false because it has not yet been proven true.[0] Why should we assume that the training set includes the answer key in the absence of evidence? 0. https://en.wikipedia.org/wiki/Argument_from_ignorance https://en.wikipedia.org/wiki/Argument_from_ignorance
  - 8note 4y ago
    We know that the training set is large and non public, and that answer keys are generally available and ML likes to output exact answers when it's been trained on the questions. It's significantly more likely that a model outputting the right answer was trained in that answer than not
- zone411 4y ago
  The author said that he comes up with new questions every year.
  - reichardt 4y ago
    Still questions that I would assume are semantically similar to the questions you can find in exam prep material all over the internet. My point is that exams are a crutch we use to determine how well a person studied a subject. A crutch we use because we seem to lack better measuring devices. It's very much possible to ace an exam while at the same time being horrible at actually applying/working on a subject. I'd argue therefore that measuring how well LLMs perform on exams designed for humans is simply a more complicated Turing test, with all its shortcomings.
bamboozled 4y ago
Your comment isn't original either, it's almost always the same response if anyone questions many technologies. If you think about it critically, maybe we will find there isn't actually much use for ChatGPT outside of it being a pretty cool party trick. Even if it gets better. I think it pains people to think about it this way but if we have bullshit jobs, and now have bullshit bots to do bullshit jobs, what's really changed? Likely we'll just end up with more bullshit. I'm sorry but I think it wouldn't hurt for you to also be objective yourself when evaluating the system your defending. When you think about it this way, if ChatGPT has the answer to all answerable questions loaded into it's memory, is any of this really that impressive? Initially yes, in the long term maybe so. I would argue it speeds up finding the answer to many questions which I can find online, but I pretty much had access to any answer I needed already. My feeling is we think these machines are impressive, because we think we're impressive, and ChatGPT is doing something that looks like what we do in a new and novel way. I also get the vibe that we almost "have" to be impressed with it because as a species, we've invest a lot into these endeavors and I guess the thought of it being not useful it just as scary as it being useful and replacing everyone's work and crashing the economy of whatever. The psychology of it all is fascinating. I guess we all thought steam trains were amazing and unfathomable at one point in time, is this really much different time will tell :)
8note 4y ago
It does compare to inputting a math question into a calculator though. Do calculators mean that we shouldn't teach multiplication anymore? That we shouldn't have tests for multiplication? The calculator will always get the right answer. The bigger question imo is what humans are doing with these new tools. We no longer need the human computers of old now that we have calculators, and almost certainly we can innovate over automated MBAs now

Oh how quickly the goalposts shift... If the test was done correctly and there was no data leakage, this absolutely does not compare to someone equipped with the answer key.

salamo 4y ago

There's even a specific term for shifting AI goalposts: https://en.wikipedia.org/wiki/AI_effect https://en.wikipedia.org/wiki/AI_effect

chamwislothe2nd 4y ago
https://en.m.wikipedia.org/wiki/Moving_the_goalposts https://en.m.wikipedia.org/wiki/Moving_the_goalposts reviewing the 'see also' section of the goalposts wikipage is like looking at a syllabus for PR. People are terrible.
bamboozled 4y ago
I also think there is a novel new effect happening which is that if one question anything about AI. For example: I tired to use Dall-E for something useful and it sucked. You get labelled...Anti-AI. Being critical about certain things isn't allowed at the moment. Kind of like the 2023 replacement for Anti-vaxxer if you don't want to take medicine every 3 months. My honest opinion, is that while a lot of this technology is cool and novel, it's not perfect, it has uses and then it doesn't. What it does have is a lot of good marketing and PR behind it and so you're up against a lot of that too. One example I like is the Dall-e homepage shoes the curated "best" images generated, which might be showing only 1 in 1000 or more of the best images. I was excited when I first tried it based on the front page. But now it's drawing people with feet for hands for me and it's super super fun and entertaining to play with, I don't feel it works quite as advertised most of the time. Disclaimer: Not anti-ai and I took my Covid19 vaccines (in the beginning, no boosters)

ipaddr 4y ago

It compares to someone with a textbook and/or access to the internet.

zone411 4y ago
...and the ability to synthetize both into a response to a question it has never encountered before

reichardt 4y ago

Do you know that the answer key wasn't part of the train set of ChatGPT?

mshake2 4y ago
>Argument from ignorance, also known as appeal to ignorance (in which ignorance represents "a lack of contrary evidence"), is a fallacy in informal logic. It asserts that a proposition is true because it has not yet been proven false or a proposition is false because it has not yet been proven true.[0] Why should we assume that the training set includes the answer key in the absence of evidence? 0. https://en.wikipedia.org/wiki/Argument_from_ignorance https://en.wikipedia.org/wiki/Argument_from_ignorance
- 8note 4y ago
  We know that the training set is large and non public, and that answer keys are generally available and ML likes to output exact answers when it's been trained on the questions. It's significantly more likely that a model outputting the right answer was trained in that answer than not
zone411 4y ago
The author said that he comes up with new questions every year.
- reichardt 4y ago
  Still questions that I would assume are semantically similar to the questions you can find in exam prep material all over the internet. My point is that exams are a crutch we use to determine how well a person studied a subject. A crutch we use because we seem to lack better measuring devices. It's very much possible to ace an exam while at the same time being horrible at actually applying/working on a subject. I'd argue therefore that measuring how well LLMs perform on exams designed for humans is simply a more complicated Turing test, with all its shortcomings.

bamboozled 4y ago

Your comment isn't original either, it's almost always the same response if anyone questions many technologies. If you think about it critically, maybe we will find there isn't actually much use for ChatGPT outside of it being a pretty cool party trick. Even if it gets better. I think it pains people to think about it this way but if we have bullshit jobs, and now have bullshit bots to do bullshit jobs, what's really changed? Likely we'll just end up with more bullshit. I'm sorry but I think it wouldn't hurt for you to also be objective yourself when evaluating the system your defending. When you think about it this way, if ChatGPT has the answer to all answerable questions loaded into it's memory, is any of this really that impressive? Initially yes, in the long term maybe so. I would argue it speeds up finding the answer to many questions which I can find online, but I pretty much had access to any answer I needed already. My feeling is we think these machines are impressive, because we think we're impressive, and ChatGPT is doing something that looks like what we do in a new and novel way. I also get the vibe that we almost "have" to be impressed with it because as a species, we've invest a lot into these endeavors and I guess the thought of it being not useful it just as scary as it being useful and replacing everyone's work and crashing the economy of whatever. The psychology of it all is fascinating. I guess we all thought steam trains were amazing and unfathomable at one point in time, is this really much different time will tell :)

8note 4y ago

It does compare to inputting a math question into a calculator though. Do calculators mean that we shouldn't teach multiplication anymore? That we shouldn't have tests for multiplication? The calculator will always get the right answer. The bigger question imo is what humans are doing with these new tools. We no longer need the human computers of old now that we have calculators, and almost certainly we can innovate over automated MBAs now