6 ms·

Going meta a bit: comments so far on this post show diametrically opposing understandings of the paper, which demonstrates just how varied the interpretation of

Going meta a bit: comments so far on this post show diametrically opposing understandings of the paper, which demonstrates just how varied the interpretation of complex text can be. We hold AI to a pretty high standard of correctness, as we should, but humans are not that reliable on matters of fact, let alone on rigor of reasoning.

sigmoid10 2y ago
This is extremely common in these discussions. Most humans are not that good at reasoning themselves and fall for the same kind of fallacies over and over because of the way they were brought up (their training data so to speak). And yet they somehow think they can argue why or why not LLMs should be able to do the same. If anything, the current limits of these morels show the limits of human cognition which is spread throughout the internet - because this is literally what they learned from. I believe once we achieve a more independent learning (like we've seen glimpses of in the MuZero paper) these models will blow human intelligence out of the water.
- pclmulqdq 2y ago
  It's because we can put responsibility on humans to be correct but we can't on computers. Humans given the appropriate incentives are very good at their jobs and there is a path for compensation if they screw up. Computers have neither of these things.
  - sigmoid10 2y ago
    Humans already put a lot of trust in computers not because they can take responsibility but because traditional software can be made very predictable or at least compliant. There are whole industries built around software standards to ensure that. The problem is we don't yet know enough about identifying and patching problems in these models. Once we get something equivalent to MISRA for LLMs to achieve the same level of compliance, there is very little that could still hold them back.
    - pclmulqdq 2y ago
      Yes. Traditional software has an unbroken chain of hard responsibility back to a human. Once you introduce non-determinism, things get weird.
      - eru 2y ago
        Non-determinism is a well-understood tool, and does not diminish responsibility. Grep works just fine, despite implementing non-deterministic finite state machines. Monte Carlo simulations are behind nuclear weapons (where they were invented), weather forecasts, financial trading, etc. Las Vegas algorithms like randomised quicksort also diminish no one's responsibility. In principle, you can run training and inference on neural networks completely deterministically. But I don't think that makes any difference to responsibility. (To make them deterministic, you obviously have to use pseudo-random number generators with fixed seeds, but less obviously you also have to make sure that when you merge the results of parallel runs, the results 'merge' deterministically. Deterministic parallelism is an extremely interesting field of study! Or, since we are only talking about principles, not what's practical, you could just run everything in series.) The problem with LLMs is that they are complicated and their actions are hard for humans to predict or reason through. Complexity is the bane of responsibility: if you have a complicated enough system (and a complicated enough management structure involved in producing that system), that's where responsibility goes to die, unless you specifically work to establish it. In this case, employing LLMs is no worse than employing humans. If upper management gives bad instructions and incentives for lower level employees, we tend to pin the responsibility on upper management.
  - ndm000 2y ago
    I think this is a key argument in how powerful AI can become. We may be able to create incredibly intelligent systems, but at the end of the day you can’t send a computer to jail. That inherently limits the power that will be given over to AI. If an AI accidentally kills a person, the worst that could be done to it is that it is turned off, whereas the owners of the AI would be held liable.
- patcon 2y ago
  > Most humans are not that good at reasoning themselves and fall for the same kind of fallacies over and over because of the way they were brought up Disagree that it's easy to pin on "how they were brought up". It seems very likely that we may learn that the flaws are part of what makes our intelligence "work" and be adaptive to changing environments. It may be favourable in terms of cultural evolution for parents to indoctrinate flawed logic, not unlike how replication errors are part of how evolution can and must work. In other words: I'm not sure these "failures" of the models are actual failures (in the sense of being non-adaptive and important to the evolutionary processes of intelligence), and further, it is perhaps us humans that are "failing" by over-indexing on "reason" as explanation for how we arrived here and continue to persist in time ;)
  - eru 2y ago
    > Disagree that it's easy to pin on "how they were brought up". Indeed. That might play a role, but another less politically charged aspect to look at is just: how much effort is the human currently putting in? Humans are often on autopilot, perhaps even most of the time. Autopilot means taking lazy intellectual shortcuts. And to echo your argument: in familiar environments those shortcuts are often a good idea! If you just do whatever worked last time you were in a similar situation, or whatever your peers are doing, chances are you'll have an easier time than reasoning everything out from scratch. Especially in any situations involving other humans cooperating with you, predictability itself is an asset.
    - tsumnia 2y ago
      > And to echo your argument: in familiar environments those shortcuts are often a good idea! Only to continue to reaffirm the original post, this was some of the basis for my dissertation. Lower-level practice, or exposure to tons of interactive worked examples, allowed students to train the "mental muscle memory" for coding syntax to learn the more general CS concept (like loops instead of for(int i = 0...). The shortcut in this case is learning what the syntax for a loop looks like so that it can BECOME a shortcut. Once its automatic, then it can be compartmentalized as "loop" instead of getting anxious over where the semicolons go.
- eru 2y ago
  > Most humans are not that good at reasoning themselves [...] I'd say most humans most of the time. Individual humans can do a lot better (or worse) depending on how much effort they put in, and whether they slept well, had their morning coffee, etc. > If anything, the current limits of these morels show the limits of human cognition which is spread throughout the internet - because this is literally what they learned from. I wouldn't go quite so far. Especially because some tasks require smarts, even though there's no smarts in the training data. The classic example is perhaps programming: the Python interpreter is not intelligent by any stretch of the imagination, but an LLM (or a human) needs smarts to predict what's going to do, especially if you are trying to get it to do something specific. That example might skirt to close to the MuZero paper that you already mentioned as an exception / extension. So let's go with a purer example: even the least smart human is a complicated system with a lot of hidden state, parts of that state shine through when that human produces text. Predicting the next token of text just from the previous text is a lot harder and requires a lot more smarts than if you had access to the internal state directly. It's sort-of like an 'inverse problem'. https://en.wikipedia.org/wiki/Inverse_problem https://en.wikipedia.org/wiki/Inverse_problem
- InDubioProRubio 2y ago
  Human society, when the individual reaches the limits of their "reasoning", usually produce growths to circumvent these limitations to produce and use artifacts that lurk beyond their limitations. A illiterate can still use Netflix, etc. The ability to circumvent these limitations, is encoded in company procedures, architecture of hierarchies/gremiums within companies and states. Could AI be "upgraded" beyond human reasoning, by referencing these "meta-organisms" and their reasoning processes that can produce things that are larger then the sum of its parts? Could AI become smarter by rewarding this meta-reasoning and prompting for it? "Chat GPT for your next task, you are going to model a company reasoning process internally to produce a better outcome" This should also allow to circumvent human reasoning bugs - like tribal thinking (which is the reason why we have black and white thinking. You goto agree with the tribes-group-think, else there be civil war risking all members of the tribe. Which is why there always can only be ONE answer, one idea, one plan, one leader - and multiple simultaneous explorations at once as in capitalism cause deep unease)
cscurmudgeon 2y ago
> We hold AI to a pretty high standard of correctness, as we should, but humans are not that reliable on matters of fact, let alone on rigor of reasoning. I never understood this line of reasoning. 1. Humans can't run faster than 30 mph. 2. Therefore we can't complain if cars/trains/transport always go slower than 30 mph. These comparisons also hide that we are comparing best of AI (massive LLMs) with median/average human reasoning.
- XenophileJKO 2y ago
  The whole social dynamic of this conversation is amazing. How fast complacency happened. In 2010 if you told me I could get a model to respond approximately as intelligently as a low intelligence human, I would be amazed. As a matter of perspective,I am still amazed. At the same time I see such negative sentiment around the capabilities at their current limits. We are reaching an era of commodified intelligence, which will be disruptive and surprising. Even the current limited models change the economics dramatically.
  - eru 2y ago
    > Even the current limited models change the economics dramatically. Yes, though at the moment they hype is still a lot bigger than the impact. But I am fairly confident that even without any new technical ideas for the networks themselves, we will see a lot more economic impact over the next few years, as people work out how to use these new tools. (Of course, the networks will also evolve still.)
    - naasking 2y ago
      I think the impact is already understated. Every non-technical person I know that's still working has used ChatGPT for work at some point, and quite a few of them are using it regularly. And I'm nowhere near Silicon Valley or any serious tech hub.
      - eru 2y ago
        Yes, they are great for helping with writer's block or for replacing a Google search. But there's a lot of stuff they can't really do (in their current form), or can't do reliable, yet. > Every non-technical person I know that's still working has used ChatGPT for work at some point, and quite a few of them are using it regularly. And I'm nowhere near Silicon Valley or any serious tech hub. Yes, that makes me optimistic for their future, too.
fsndz 2y ago
I find it bizarre that people often criticize LLMs’ capabilities by stating they don’t exhibit human-level performance because they sometimes fail, when occasionally failing is, in fact, a quintessentially human trait. Moreover, how do we reconcile the fact that LLMs perform far better than most humans on all the standardized tests they’ve passed? https://medium.com/@fsndzomga/there-will-be-no-agi-d9be9af4428d https://medium.com/@fsndzomga/there-will-be-no-agi-d9be9af44...
deleted 2y ago
[deleted]

sigmoid10 2y ago

This is extremely common in these discussions. Most humans are not that good at reasoning themselves and fall for the same kind of fallacies over and over because of the way they were brought up (their training data so to speak). And yet they somehow think they can argue why or why not LLMs should be able to do the same. If anything, the current limits of these morels show the limits of human cognition which is spread throughout the internet - because this is literally what they learned from. I believe once we achieve a more independent learning (like we've seen glimpses of in the MuZero paper) these models will blow human intelligence out of the water.

pclmulqdq 2y ago
It's because we can put responsibility on humans to be correct but we can't on computers. Humans given the appropriate incentives are very good at their jobs and there is a path for compensation if they screw up. Computers have neither of these things.
- sigmoid10 2y ago
  Humans already put a lot of trust in computers not because they can take responsibility but because traditional software can be made very predictable or at least compliant. There are whole industries built around software standards to ensure that. The problem is we don't yet know enough about identifying and patching problems in these models. Once we get something equivalent to MISRA for LLMs to achieve the same level of compliance, there is very little that could still hold them back.
  - pclmulqdq 2y ago
    Yes. Traditional software has an unbroken chain of hard responsibility back to a human. Once you introduce non-determinism, things get weird.
    - eru 2y ago
      Non-determinism is a well-understood tool, and does not diminish responsibility. Grep works just fine, despite implementing non-deterministic finite state machines. Monte Carlo simulations are behind nuclear weapons (where they were invented), weather forecasts, financial trading, etc. Las Vegas algorithms like randomised quicksort also diminish no one's responsibility. In principle, you can run training and inference on neural networks completely deterministically. But I don't think that makes any difference to responsibility. (To make them deterministic, you obviously have to use pseudo-random number generators with fixed seeds, but less obviously you also have to make sure that when you merge the results of parallel runs, the results 'merge' deterministically. Deterministic parallelism is an extremely interesting field of study! Or, since we are only talking about principles, not what's practical, you could just run everything in series.) The problem with LLMs is that they are complicated and their actions are hard for humans to predict or reason through. Complexity is the bane of responsibility: if you have a complicated enough system (and a complicated enough management structure involved in producing that system), that's where responsibility goes to die, unless you specifically work to establish it. In this case, employing LLMs is no worse than employing humans. If upper management gives bad instructions and incentives for lower level employees, we tend to pin the responsibility on upper management.
- ndm000 2y ago
  I think this is a key argument in how powerful AI can become. We may be able to create incredibly intelligent systems, but at the end of the day you can’t send a computer to jail. That inherently limits the power that will be given over to AI. If an AI accidentally kills a person, the worst that could be done to it is that it is turned off, whereas the owners of the AI would be held liable.
patcon 2y ago
> Most humans are not that good at reasoning themselves and fall for the same kind of fallacies over and over because of the way they were brought up Disagree that it's easy to pin on "how they were brought up". It seems very likely that we may learn that the flaws are part of what makes our intelligence "work" and be adaptive to changing environments. It may be favourable in terms of cultural evolution for parents to indoctrinate flawed logic, not unlike how replication errors are part of how evolution can and must work. In other words: I'm not sure these "failures" of the models are actual failures (in the sense of being non-adaptive and important to the evolutionary processes of intelligence), and further, it is perhaps us humans that are "failing" by over-indexing on "reason" as explanation for how we arrived here and continue to persist in time ;)
- eru 2y ago
  > Disagree that it's easy to pin on "how they were brought up". Indeed. That might play a role, but another less politically charged aspect to look at is just: how much effort is the human currently putting in? Humans are often on autopilot, perhaps even most of the time. Autopilot means taking lazy intellectual shortcuts. And to echo your argument: in familiar environments those shortcuts are often a good idea! If you just do whatever worked last time you were in a similar situation, or whatever your peers are doing, chances are you'll have an easier time than reasoning everything out from scratch. Especially in any situations involving other humans cooperating with you, predictability itself is an asset.
  - tsumnia 2y ago
    > And to echo your argument: in familiar environments those shortcuts are often a good idea! Only to continue to reaffirm the original post, this was some of the basis for my dissertation. Lower-level practice, or exposure to tons of interactive worked examples, allowed students to train the "mental muscle memory" for coding syntax to learn the more general CS concept (like loops instead of for(int i = 0...). The shortcut in this case is learning what the syntax for a loop looks like so that it can BECOME a shortcut. Once its automatic, then it can be compartmentalized as "loop" instead of getting anxious over where the semicolons go.
eru 2y ago
> Most humans are not that good at reasoning themselves [...] I'd say most humans most of the time. Individual humans can do a lot better (or worse) depending on how much effort they put in, and whether they slept well, had their morning coffee, etc. > If anything, the current limits of these morels show the limits of human cognition which is spread throughout the internet - because this is literally what they learned from. I wouldn't go quite so far. Especially because some tasks require smarts, even though there's no smarts in the training data. The classic example is perhaps programming: the Python interpreter is not intelligent by any stretch of the imagination, but an LLM (or a human) needs smarts to predict what's going to do, especially if you are trying to get it to do something specific. That example might skirt to close to the MuZero paper that you already mentioned as an exception / extension. So let's go with a purer example: even the least smart human is a complicated system with a lot of hidden state, parts of that state shine through when that human produces text. Predicting the next token of text just from the previous text is a lot harder and requires a lot more smarts than if you had access to the internal state directly. It's sort-of like an 'inverse problem'. https://en.wikipedia.org/wiki/Inverse_problem https://en.wikipedia.org/wiki/Inverse_problem
InDubioProRubio 2y ago
Human society, when the individual reaches the limits of their "reasoning", usually produce growths to circumvent these limitations to produce and use artifacts that lurk beyond their limitations. A illiterate can still use Netflix, etc. The ability to circumvent these limitations, is encoded in company procedures, architecture of hierarchies/gremiums within companies and states. Could AI be "upgraded" beyond human reasoning, by referencing these "meta-organisms" and their reasoning processes that can produce things that are larger then the sum of its parts? Could AI become smarter by rewarding this meta-reasoning and prompting for it? "Chat GPT for your next task, you are going to model a company reasoning process internally to produce a better outcome" This should also allow to circumvent human reasoning bugs - like tribal thinking (which is the reason why we have black and white thinking. You goto agree with the tribes-group-think, else there be civil war risking all members of the tribe. Which is why there always can only be ONE answer, one idea, one plan, one leader - and multiple simultaneous explorations at once as in capitalism cause deep unease)

cscurmudgeon 2y ago

> We hold AI to a pretty high standard of correctness, as we should, but humans are not that reliable on matters of fact, let alone on rigor of reasoning. I never understood this line of reasoning. 1. Humans can't run faster than 30 mph. 2. Therefore we can't complain if cars/trains/transport always go slower than 30 mph. These comparisons also hide that we are comparing best of AI (massive LLMs) with median/average human reasoning.

XenophileJKO 2y ago
The whole social dynamic of this conversation is amazing. How fast complacency happened. In 2010 if you told me I could get a model to respond approximately as intelligently as a low intelligence human, I would be amazed. As a matter of perspective,I am still amazed. At the same time I see such negative sentiment around the capabilities at their current limits. We are reaching an era of commodified intelligence, which will be disruptive and surprising. Even the current limited models change the economics dramatically.
- eru 2y ago
  > Even the current limited models change the economics dramatically. Yes, though at the moment they hype is still a lot bigger than the impact. But I am fairly confident that even without any new technical ideas for the networks themselves, we will see a lot more economic impact over the next few years, as people work out how to use these new tools. (Of course, the networks will also evolve still.)
  - naasking 2y ago
    I think the impact is already understated. Every non-technical person I know that's still working has used ChatGPT for work at some point, and quite a few of them are using it regularly. And I'm nowhere near Silicon Valley or any serious tech hub.
    - eru 2y ago
      Yes, they are great for helping with writer's block or for replacing a Google search. But there's a lot of stuff they can't really do (in their current form), or can't do reliable, yet. > Every non-technical person I know that's still working has used ChatGPT for work at some point, and quite a few of them are using it regularly. And I'm nowhere near Silicon Valley or any serious tech hub. Yes, that makes me optimistic for their future, too.

fsndz 2y ago

I find it bizarre that people often criticize LLMs’ capabilities by stating they don’t exhibit human-level performance because they sometimes fail, when occasionally failing is, in fact, a quintessentially human trait. Moreover, how do we reconcile the fact that LLMs perform far better than most humans on all the standardized tests they’ve passed? https://medium.com/@fsndzomga/there-will-be-no-agi-d9be9af4428d https://medium.com/@fsndzomga/there-will-be-no-agi-d9be9af44...

deleted 2y ago

[deleted]