4 ms·

LLMs still do not match humans on Winograd Schema Challenge but the top ones already get pretty close. The current top model, Vega v2, is already at 98.6 (comp

LLMs still do not match humans on Winograd Schema Challenge but the top ones already get pretty close. The current top model, Vega v2, is already at 98.6 (compared to human’s 100.0). https://super.gluebenchmark.com/leaderboard https://super.gluebenchmark.com/leaderboard Several other LLMs score higher than 95.0 as well. I suspect some upcoming multimodal LFMs will be very close to human level since their training data would include video, audio, and possibly even physical interaction input. Moreover, top LLMs already surpass the human baseline on SuperGLUE, a suite of many language tasks. This is my experience when using GPT-4 as well. It’s like an extremely well-read intern who follows instructions much better than an average person on the street. It is just often a little drunk or sleepy at work ¯\_(ツ)_/¯, ……for now.

sublinear 3y ago
All these responses are missing the point. Like the Turing test, Winograd is not a very good test and more of a philosophical question than a metric for engineering. Sure there's nothing wrong with a model that at some point in time passes 100% and is even more consistent than humans, but it's not possible to make an exhaustive list of what humans can disambiguate that these models can't. It's a moving target. We're just a cultural shift, scientific discovery, or political regime change away from the models needing to be updated all over again. For some applications these lagging inaccuracies are not acceptable. For yet still more applications, it's not possible to allow any amount of time to train the models ahead of seeing novel ambiguity. There's a lot of work left for AI research and the threat of authoritarian-induced cultural stagnation just to serve the AI gods and the convenience of governance is absolutely terrifying. i.e. You can't speak that way because it confuses the AI, you can't look or act that way because it confuses the AI, etc. AI in its current form totally sucks and is unfit for much beyond low risk tasks.
- nopinsight 3y ago
  I was addressing your statements regarding the capabilities of neural AI: > Anyway, all known AI techniques struggle with common sense reasoning. The majority of ambiguities in natural language are distinct and uncorrelated. Combining known techniques is not going to solve the problem. > Statistical techniques underperform compared to symbolic rules no matter how much hardware you throw at the problem. The best empirical machine learning techniques to measure a system's capabilities are good enough for many practical purposes. AI's commonsense and language understanding differs from that of humans but it's not necessarily inferior to the average human's understanding, even in its current state. AI's understanding is shallower but more interconnected to more diverse domains of knowledge, which is very useful for practical purposes. We're not going to change our language to avoid confusing AI. If anything, it can already communicate, using our natural language, better than an average human in some practical contexts. I do agree though that AI research in language understanding is far from completed.

sublinear 3y ago

All these responses are missing the point. Like the Turing test, Winograd is not a very good test and more of a philosophical question than a metric for engineering. Sure there's nothing wrong with a model that at some point in time passes 100% and is even more consistent than humans, but it's not possible to make an exhaustive list of what humans can disambiguate that these models can't. It's a moving target. We're just a cultural shift, scientific discovery, or political regime change away from the models needing to be updated all over again. For some applications these lagging inaccuracies are not acceptable. For yet still more applications, it's not possible to allow any amount of time to train the models ahead of seeing novel ambiguity. There's a lot of work left for AI research and the threat of authoritarian-induced cultural stagnation just to serve the AI gods and the convenience of governance is absolutely terrifying. i.e. You can't speak that way because it confuses the AI, you can't look or act that way because it confuses the AI, etc. AI in its current form totally sucks and is unfit for much beyond low risk tasks.

nopinsight 3y ago
I was addressing your statements regarding the capabilities of neural AI: > Anyway, all known AI techniques struggle with common sense reasoning. The majority of ambiguities in natural language are distinct and uncorrelated. Combining known techniques is not going to solve the problem. > Statistical techniques underperform compared to symbolic rules no matter how much hardware you throw at the problem. The best empirical machine learning techniques to measure a system's capabilities are good enough for many practical purposes. AI's commonsense and language understanding differs from that of humans but it's not necessarily inferior to the average human's understanding, even in its current state. AI's understanding is shallower but more interconnected to more diverse domains of knowledge, which is very useful for practical purposes. We're not going to change our language to avoid confusing AI. If anything, it can already communicate, using our natural language, better than an average human in some practical contexts. I do agree though that AI research in language understanding is far from completed.