6 ms·

I can walk and chew bubble gum at the same time: on one hand, yes, there's certainly a lot of Kool-Aid being drank by the AI folks. Even on HN, I constantly arg

by dvt 4y ago

I can walk and chew bubble gum at the same time: on one hand, yes, there's certainly a lot of Kool-Aid being drank by the AI folks. Even on HN, I constantly argue with people that genuinely think LLMs are some kind of magical black box that contain "knowledge" or "intelligence" or "meaning" when in reality, it's just a very fancy Markov chain. And on the other hand, I think that language interfaces are probably the next big leap in how we interact with our computers, but more to the point of the article: > To conclude: one must have different standards for developing systems than for testing, deploying, or using systems. In my opinion, you unfortunately will never (and, in fact could never) have reliable development and testing standards when designing purely stochastic systems like, e.g., large language models. Intuitively, the fact that these are stochastic systems is why we need things like hyper-parameters, fiddling with seeds, and prompt engineering.

charcircuit 4y ago
>contain "knowledge" or "intelligence" or "meaning" when in reality, it's just a very fancy Markov chain These are not mutually exclusive. If you have a Markov chain that 100% of the time outputs "A cat is an animal", then it has knowledge that a cat is an animal.
- version_five 4y ago
  $ yes "a cat is an animal" Does `yes` also have "knowledge"?
  - zacgarby 4y ago
    no, but `$ yes "a cat is an animal"` does (if you know how to interpret it)
    - maxbond 4y ago
      If it requires interpretation, than it is the "yes+human" system that has the knowledge. What really happened here is that a human wrote down, "a cat is an animal," and then another human read it, understood it, and believed it. And so the knowledge moved from one human to another. `yes` was only a conduit for that information to travel through.
      - charcircuit 4y ago
        >was only a conduit If something was a conduit for knowledge wouldn't it make sense that it would have at some point contained the knowledge? The knowledge is stored into it by one human and extracted by another human.
        maxbond 4y ago
        Absolutely you can store knowledge in text. You can store quite a bit of knowledge in a book for instance, but the book doesn't have any beliefs and doesn't know anything. Whether a sufficiently complex Markov chain or ANN can have beliefs, I don't know but I'm skeptical that these ANNs do in particular. It's ability to produce text containing true statements isn't sufficient evidence to conclude that it has beliefs, and it's easy to find cases where it contradicts itself (eg, if you play around with the wording you can find a prompt where it tells you that solving the trolley problem is a matter of harming the fewest people but proposes a solution that harms the most people). I take that as an indication it's primarily regurgitating text and rearranging the prompt rather than applying knowledge (which, to be clear, is useful for a number of tasks).
- tifik 4y ago
  I think you are operating with a different definition of "knowledge" that the parent does.
  - WastingMyTime89 4y ago
    The heart of the issue is always the same: is a perfect simulation actually the same thing as what it simulates. I would argue that yes especially when the definition of knowledge or intelligence is already so fuzzy but some people will probably always disagree.
  - archgoon 4y ago
    [dead]
  - stametseater 4y ago
    Knowledge is awareness of information. "Awareness" is a quagmire because a lot of people believe that 'true' awareness requires possessing a sort of soul which machines can't possess. I think the important part is information, the matter of 'awareness' can simply be ignored as a philosophical/religious disagreement which will never be resolved. What's important is: Does the system contain information? Can it reliably convey that information? In which ways can it manipulate that information?
    - maxbond 4y ago
      "Awareness of information" describes belief. Knowledge is justified, true belief (you can believe things you don't actually know/don't have justification for, and you can be made aware of information you don't believe). If you're dismissive of philosophy and then ask epistemological questions, you'll miss out on a lot of good pondering people have done on the subject, and end up reinventing some of it without encountering the criticism of those ideas.
      - stametseater 4y ago
        1. https://en.wiktionary.org/wiki/knowledge https://en.wiktionary.org/wiki/knowledge 2. "Belief" isn't any less of a quagmire than "awareness." Materialists and dualists will never agree on whether machines can have 'belief' or 'awareness', so discussions between the two will always be fruitless.
- dsr_ 4y ago
  It does not. It has a language output that you find pleasing.
cjbprime 4y ago
The Microsoft Research "Sparks of AGI" paper spends 154 pages describing behaviors of GPT-4 that are inconsistent with the understanding of it being a "fancy Markov chain": https://arxiv.org/abs/2303.12712 https://arxiv.org/abs/2303.12712 I expect that the reason people are constantly arguing with you is that your analysis does not explain some easily testable experiences, such as why GPT-4 has the ability to explain what some non-trivial and unique Python programs would output if they were run, despite GPT-4 not having access to a Python interpreter itself.
- dvt 4y ago
  > trivial and unique Python programs would output if they were run, despite GPT-4 not having access to a Python interpreter itself Trivially explained as "even a broken clock is right twice a day." I skimmed the paper, as it was linked here on HN iirc. First, it was published by Microsoft, a company that absolutely has a horse in this race (what were they supposed to say? "The AI bot our search engine uses is dumb?"). Second of all, I was very interested in their methodology, so I fully read the first section, which is woefully hand-wavy, a fact with which even the authors would agree: > We acknowledge that this approach is somewhat subjective and informal, and that it may not satisfy the rigorous standards of scientific evaluation. The paper, for instance, is amazed that GPT knows how to draw a unicorn in TikZ, but we already know it was trained on the Pile, which includes all Stack Exchange websites, which happens to include answers like this one[1]. So to make the argument that it's being creative, when the answer (or, more charitably, something extremely close to it) is literally in the training set, is just disingenuous. [1] https://tex.stackexchange.com/questions/312199/i-need-a-tex-donkey https://tex.stackexchange.com/questions/312199/i-need-a-tex-...
  - gaganyaan 4y ago
    You're one of the people that will be yelling to everyone else in a potential future "We're only seemingly oppressed! It's just a parlor trick that they've turned most of humanity into paperclips, sci-fi authors wrote about this already!" I don't think GPT-4 is magic, but unless the unicorn is literally an exact replica of something from it's training set, it clearly has "knowledge", and it's weird that you'd try to deny that.
sebzim4500 4y ago
>it's just a very fancy Markov chain Could you provide an argument for why an LLM is a fancy markov chain that does not apply equally well to a human?
- dvt 4y ago
  Well, for one, humans are obviously at least more than a fancy Markov chain because we have genetically hard-wired instincts, so we are, in some sense, "hard-coded" if you forgive my programming metaphor. Hard-coded to breed, multiply, care for our young, seek shelter, among many other things.
  - ummonk 4y ago
    Markov chains, like any algorithm, are hard-coded. And just as evolution hard-codes our genes, supervised learning (and in the future reinforcement learning) hard-codes LLMs and other AI models.
- CyberRabbi 4y ago
  Good point. While it seems obvious to me that LLMs can never be anything more than fancy Markov chains, in my experience it seems the majority of human "logic" does not operate much differently. Very rare to encounter someone who is able to think or speak critically. Most regurgitate canned responses based on keywords.
  - sebzim4500 4y ago
    I don't just mean the dumb ones. Einstein was just as much a markov chain as GPT-2 is. The term is utterly useless in this context.
- orwin 4y ago
  I'm gonna respond to you, because i think you like GPT4 and i do too (even if the only use i trust for now is "Resume me this **lot of text/research article** in less than 200 words", which is already great for a knowledge hoarder like me) You can think against yourself, a LLM have troubles doing so. Also, they fail spectacularly when asked to do real-life operation: "I have to buy two bagettes at one euros, then five chocolatine, croissants and raisin bread at 1.40, 1.20 and 1.60 respectively, how much should i take with me?" when in my head, i just know it'll be between 20 and 25 in seconds (and in fact it's 23, i took random numbers but they are quite easy to add).

by dvt 4y ago

charcircuit 4y ago

>contain "knowledge" or "intelligence" or "meaning" when in reality, it's just a very fancy Markov chain These are not mutually exclusive. If you have a Markov chain that 100% of the time outputs "A cat is an animal", then it has knowledge that a cat is an animal.

version_five 4y ago
$ yes "a cat is an animal" Does `yes` also have "knowledge"?
- zacgarby 4y ago
  no, but `$ yes "a cat is an animal"` does (if you know how to interpret it)
  - maxbond 4y ago
    If it requires interpretation, than it is the "yes+human" system that has the knowledge. What really happened here is that a human wrote down, "a cat is an animal," and then another human read it, understood it, and believed it. And so the knowledge moved from one human to another. `yes` was only a conduit for that information to travel through.
    - charcircuit 4y ago
      >was only a conduit If something was a conduit for knowledge wouldn't it make sense that it would have at some point contained the knowledge? The knowledge is stored into it by one human and extracted by another human.
      - maxbond 4y ago
        Absolutely you can store knowledge in text. You can store quite a bit of knowledge in a book for instance, but the book doesn't have any beliefs and doesn't know anything. Whether a sufficiently complex Markov chain or ANN can have beliefs, I don't know but I'm skeptical that these ANNs do in particular. It's ability to produce text containing true statements isn't sufficient evidence to conclude that it has beliefs, and it's easy to find cases where it contradicts itself (eg, if you play around with the wording you can find a prompt where it tells you that solving the trolley problem is a matter of harming the fewest people but proposes a solution that harms the most people). I take that as an indication it's primarily regurgitating text and rearranging the prompt rather than applying knowledge (which, to be clear, is useful for a number of tasks).
tifik 4y ago
I think you are operating with a different definition of "knowledge" that the parent does.
- WastingMyTime89 4y ago
  The heart of the issue is always the same: is a perfect simulation actually the same thing as what it simulates. I would argue that yes especially when the definition of knowledge or intelligence is already so fuzzy but some people will probably always disagree.
- archgoon 4y ago
  [dead]
- stametseater 4y ago
  Knowledge is awareness of information. "Awareness" is a quagmire because a lot of people believe that 'true' awareness requires possessing a sort of soul which machines can't possess. I think the important part is information, the matter of 'awareness' can simply be ignored as a philosophical/religious disagreement which will never be resolved. What's important is: Does the system contain information? Can it reliably convey that information? In which ways can it manipulate that information?
  - maxbond 4y ago
    "Awareness of information" describes belief. Knowledge is justified, true belief (you can believe things you don't actually know/don't have justification for, and you can be made aware of information you don't believe). If you're dismissive of philosophy and then ask epistemological questions, you'll miss out on a lot of good pondering people have done on the subject, and end up reinventing some of it without encountering the criticism of those ideas.
    - stametseater 4y ago
      1. https://en.wiktionary.org/wiki/knowledge https://en.wiktionary.org/wiki/knowledge 2. "Belief" isn't any less of a quagmire than "awareness." Materialists and dualists will never agree on whether machines can have 'belief' or 'awareness', so discussions between the two will always be fruitless.
dsr_ 4y ago
It does not. It has a language output that you find pleasing.

cjbprime 4y ago

The Microsoft Research "Sparks of AGI" paper spends 154 pages describing behaviors of GPT-4 that are inconsistent with the understanding of it being a "fancy Markov chain": https://arxiv.org/abs/2303.12712 https://arxiv.org/abs/2303.12712 I expect that the reason people are constantly arguing with you is that your analysis does not explain some easily testable experiences, such as why GPT-4 has the ability to explain what some non-trivial and unique Python programs would output if they were run, despite GPT-4 not having access to a Python interpreter itself.

dvt 4y ago
> trivial and unique Python programs would output if they were run, despite GPT-4 not having access to a Python interpreter itself Trivially explained as "even a broken clock is right twice a day." I skimmed the paper, as it was linked here on HN iirc. First, it was published by Microsoft, a company that absolutely has a horse in this race (what were they supposed to say? "The AI bot our search engine uses is dumb?"). Second of all, I was very interested in their methodology, so I fully read the first section, which is woefully hand-wavy, a fact with which even the authors would agree: > We acknowledge that this approach is somewhat subjective and informal, and that it may not satisfy the rigorous standards of scientific evaluation. The paper, for instance, is amazed that GPT knows how to draw a unicorn in TikZ, but we already know it was trained on the Pile, which includes all Stack Exchange websites, which happens to include answers like this one[1]. So to make the argument that it's being creative, when the answer (or, more charitably, something extremely close to it) is literally in the training set, is just disingenuous. [1] https://tex.stackexchange.com/questions/312199/i-need-a-tex-donkey https://tex.stackexchange.com/questions/312199/i-need-a-tex-...
- gaganyaan 4y ago
  You're one of the people that will be yelling to everyone else in a potential future "We're only seemingly oppressed! It's just a parlor trick that they've turned most of humanity into paperclips, sci-fi authors wrote about this already!" I don't think GPT-4 is magic, but unless the unicorn is literally an exact replica of something from it's training set, it clearly has "knowledge", and it's weird that you'd try to deny that.

sebzim4500 4y ago

>it's just a very fancy Markov chain Could you provide an argument for why an LLM is a fancy markov chain that does not apply equally well to a human?

dvt 4y ago
Well, for one, humans are obviously at least more than a fancy Markov chain because we have genetically hard-wired instincts, so we are, in some sense, "hard-coded" if you forgive my programming metaphor. Hard-coded to breed, multiply, care for our young, seek shelter, among many other things.
- ummonk 4y ago
  Markov chains, like any algorithm, are hard-coded. And just as evolution hard-codes our genes, supervised learning (and in the future reinforcement learning) hard-codes LLMs and other AI models.
CyberRabbi 4y ago
Good point. While it seems obvious to me that LLMs can never be anything more than fancy Markov chains, in my experience it seems the majority of human "logic" does not operate much differently. Very rare to encounter someone who is able to think or speak critically. Most regurgitate canned responses based on keywords.
- sebzim4500 4y ago
  I don't just mean the dumb ones. Einstein was just as much a markov chain as GPT-2 is. The term is utterly useless in this context.
orwin 4y ago
I'm gonna respond to you, because i think you like GPT4 and i do too (even if the only use i trust for now is "Resume me this **lot of text/research article** in less than 200 words", which is already great for a knowledge hoarder like me) You can think against yourself, a LLM have troubles doing so. Also, they fail spectacularly when asked to do real-life operation: "I have to buy two bagettes at one euros, then five chocolatine, croissants and raisin bread at 1.40, 1.20 and 1.60 respectively, how much should i take with me?" when in my head, i just know it'll be between 20 and 25 in seconds (and in fact it's 23, i took random numbers but they are quite easy to add).