9 ms·

There is a lot in here, various paths to venture off, but the bottom line seems to be trust is important when running commands on a machine, and LLMs are not tr

by dpflan 3y ago

There is a lot in here, various paths to venture off, but the bottom line seems to be trust is important when running commands on a machine, and LLMs are not trustable. What else?

altruios 3y ago
what would be required for trusting an LLM? 1: 100% transparency. Open Source code, fully (and correctly) attributed training data. 2: A predictable model of what these models are actually encoding (so that hypothetical new models (or modifications) can be reasoned about).
- thesz 3y ago
  I think I have to know what chain of reasoning is behind this or that fact and/or deduction. i would like to be able to verify that. For example, the proof of the absence of solution in SAT should be accompanied with the easily verifiable chain of reasoning. This shows the absence of incorrect deductions and missing assignments. Another example is autovectorization in contemporary compilers, they can show you why parts of your loops are not eligible for vectoriztion. All LM's can do is to show me that these parts of those inputs are important for that output, but nothing else. Thus, they cannot be trusted even for minimally critical tasks.
  - pixl97 3y ago
    Even you can't do that yourself. You can only mad post ad-hoc justifications for the choices you made.
    - thesz 3y ago
      What cannot I do myself? Look at the output of gcc -ftree-vectorizer-verbose=2? Run the verifier after picosat? I did both.
    - thesz 3y ago
      To illustrate: https://arxiv.org/abs/2401.05566 https://arxiv.org/abs/2401.05566 One can teach LLM to write good code now and bad code in future (when everyone lower their guards). And no one can prove who and how made LLM to do that. Also, no one can prove formally the absence of such a plant.
- utopcell 3y ago
  Open sourcing would only guard against human programmers. The fact of the matter is, we don't _really_ know why LLMs work.
- falsaberN1 3y ago
  Transparency won't help a lot from a technical standpoint (seems more like a solution to a legal issue than a technical one). I can't trust LLMs because they just...recombine text by probabilities and aren't deterministic. I get incorrect information every time I ask them a thing, and it's incorrect in different ways every time. The only things they seem to get consistently correct are very widespread facts that are much faster and trivial to google. I still find it funny we managed to get image generation working so much better than text.
  - ToucanLoucan 3y ago
    So much of the odd almost cultish community around LLMs seems to just be people who really want to be at the ground floor of the Next Big Thing who are so wildly biased into this being that next big thing that they will spend all their time, all their energy, not just on other people but on themselves, convincing themselves over and over that their LLM girlfriend really does love them, that their LLM assistant is going to be the next DaVinci, that their generated art is so much better than anything else. More than anything it makes me sad. ANY amount of critical thinking would tell you all of this is not true, which isn't to say there's NO USE AT ALL for this technology, it certainly exists and has it's applications, and I also do more or less believe someday we'll create digital intelligence, but at the same time... ChatGPT is not that. DALL-E is not that. These systems are interesting and they have uses but they are not emergent intelligence, they don't know anything, they just assemble words from massive probability matrices and then the people who read those words ascribe meaning to them that is far, far beyond what originated them. In this way it's not so dissimilar from any garden variety religion, it's just religion for people who think they're too smart to fall into the trap of motivated reasoning and magical thinking.
    - RodgerTheGreat 3y ago
      I think it's under-appreciated how much LLMs harvest the natural, human tendency to generously ascribe meaning, subtext, and intent to text they read, glossing over flaws and small mistakes so long as the overall "thrust" seems reasonable enough. In a sense, LLMs have reinvented cold-reading from first-principles and created the cleverest Hans of them all.
- markrages 3y ago
  Even with these in place, the output of the LLM is not trustworthy. In the sense that it doesn't care what it true, only what is plausible.
- tjr 3y ago
  I would add, repeatable / reproducible results. Given the same input, you get the same output every time. If the input includes some sort of random seed for the express purpose of getting different output intentionally, then so be it, but I can't trust a program that I never know what it will do in response to what I tell it.
fzzzy 3y ago
Isn't the gist that you should be able to trust your teacher?

by dpflan 3y ago

There is a lot in here, various paths to venture off, but the bottom line seems to be trust is important when running commands on a machine, and LLMs are not trustable. What else?

altruios 3y ago

what would be required for trusting an LLM? 1: 100% transparency. Open Source code, fully (and correctly) attributed training data. 2: A predictable model of what these models are actually encoding (so that hypothetical new models (or modifications) can be reasoned about).

thesz 3y ago
I think I have to know what chain of reasoning is behind this or that fact and/or deduction. i would like to be able to verify that. For example, the proof of the absence of solution in SAT should be accompanied with the easily verifiable chain of reasoning. This shows the absence of incorrect deductions and missing assignments. Another example is autovectorization in contemporary compilers, they can show you why parts of your loops are not eligible for vectoriztion. All LM's can do is to show me that these parts of those inputs are important for that output, but nothing else. Thus, they cannot be trusted even for minimally critical tasks.
- pixl97 3y ago
  Even you can't do that yourself. You can only mad post ad-hoc justifications for the choices you made.
  - thesz 3y ago
    What cannot I do myself? Look at the output of gcc -ftree-vectorizer-verbose=2? Run the verifier after picosat? I did both.
  - thesz 3y ago
    To illustrate: https://arxiv.org/abs/2401.05566 https://arxiv.org/abs/2401.05566 One can teach LLM to write good code now and bad code in future (when everyone lower their guards). And no one can prove who and how made LLM to do that. Also, no one can prove formally the absence of such a plant.
utopcell 3y ago
Open sourcing would only guard against human programmers. The fact of the matter is, we don't _really_ know why LLMs work.
falsaberN1 3y ago
Transparency won't help a lot from a technical standpoint (seems more like a solution to a legal issue than a technical one). I can't trust LLMs because they just...recombine text by probabilities and aren't deterministic. I get incorrect information every time I ask them a thing, and it's incorrect in different ways every time. The only things they seem to get consistently correct are very widespread facts that are much faster and trivial to google. I still find it funny we managed to get image generation working so much better than text.
- ToucanLoucan 3y ago
  So much of the odd almost cultish community around LLMs seems to just be people who really want to be at the ground floor of the Next Big Thing who are so wildly biased into this being that next big thing that they will spend all their time, all their energy, not just on other people but on themselves, convincing themselves over and over that their LLM girlfriend really does love them, that their LLM assistant is going to be the next DaVinci, that their generated art is so much better than anything else. More than anything it makes me sad. ANY amount of critical thinking would tell you all of this is not true, which isn't to say there's NO USE AT ALL for this technology, it certainly exists and has it's applications, and I also do more or less believe someday we'll create digital intelligence, but at the same time... ChatGPT is not that. DALL-E is not that. These systems are interesting and they have uses but they are not emergent intelligence, they don't know anything, they just assemble words from massive probability matrices and then the people who read those words ascribe meaning to them that is far, far beyond what originated them. In this way it's not so dissimilar from any garden variety religion, it's just religion for people who think they're too smart to fall into the trap of motivated reasoning and magical thinking.
  - RodgerTheGreat 3y ago
    I think it's under-appreciated how much LLMs harvest the natural, human tendency to generously ascribe meaning, subtext, and intent to text they read, glossing over flaws and small mistakes so long as the overall "thrust" seems reasonable enough. In a sense, LLMs have reinvented cold-reading from first-principles and created the cleverest Hans of them all.
markrages 3y ago
Even with these in place, the output of the LLM is not trustworthy. In the sense that it doesn't care what it true, only what is plausible.
tjr 3y ago
I would add, repeatable / reproducible results. Given the same input, you get the same output every time. If the input includes some sort of random seed for the express purpose of getting different output intentionally, then so be it, but I can't trust a program that I never know what it will do in response to what I tell it.

fzzzy 3y ago

Isn't the gist that you should be able to trust your teacher?