5 ms·

Humans. After all, LLMs are designed to reason equal to or better than humans.

maweaver 3y ago
By "Humans", I assume you mean something like "adult humans, well-educated in the relevant fields". Otherwise, most of these responses look like they would easily beat most humans.
- DylanDmitri 3y ago
  I think most high-school educated adults, with the ability to make a couple web searches, would do fine on all these questions. It would take the humans minutes instead of seconds because they don't have the internet memorized. Me, Kubernetes Haikus, time taken 84 seconds: ---------- Kubernetes rules With its smooth orchestration You can reach web scale ---------- Kubernetes sucks Lost in endless YAML hell Why is it broken?
  - seabass-labrax 3y ago
    I think you're spot on here. Yes, if one's trying to compare human and GPT intelligence, then you have to define what counts as memorisation and what counts as reasoning. But what most people outside of academia are trying to do is work out whether a GPT can effectively replace a human in some time-consuming task, and to be able to do so without access to the internet is rarely an important factor.
sirk390 3y ago
Humans are pretty bad at these questions. Even with the simplest questions like "Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have?" I think that a lot of people will give an incorrect answer. And for questions like "Argue for and against the use of kubernetes in the style of a haiku", 99.99% will not be able to do it.
- earthboundkid 3y ago
  The thing with humans is they will say “I don’t remember how many syllables a haiku has” and “what the hell is kubernetes?” No LLM can reliably produce a haiku because their lexing process deprives them of reliable information about syllable counts. They should all say “I’m sorry, I can’t count syllables, but I’ll try my best anyway.” But the current models don’t do that because they were trained on texts by humans, who can do haiku, and not properly taught their own limits by reinforcement learning. It’s Dunning Kruger gone berserk.
  - pixl97 3y ago
    Eh, it's not D&K gone berserk, it's what happens when you attempt to compress reality down to a single dimension (text). If you're doing a haiku, you will likely subvocalize it to ensure you're saying it correctly. It will be interesting when we get multimodal AI that can speak and listen to itself to detect things like this.
    - earthboundkid 3y ago
      The problem isn’t just that everything is text. It’s that everything is a Fourier transform of text in such a way that it’s not actually possible for an LLM to learn to count syllables.
      - pixl97 3y ago
        Again, that is just using text only. Imagine you have a lot more computing resources in a multimodal LLM. It sees your request of count the syllables and realizes it can't do them from text alone (hell I can't and have to vocalize it). It then sends your request to a audio module and 'says' the sentence, then another listening module that understand syllables 'hears' the sentence. This is how it works in most humans, now if you do this every day you'll likely make some kind of mental shortcut to reduce the effort needed, but at the end of the day there is no unsolvable problem on the AI side.
- deleted 3y ago
  [deleted]
rvz 3y ago
> After all, LLMs are designed to reason equal to or better than humans. No. I doubt you would fully trust a LLM to replace high risk jobs such as lawyers, doctors or pilots such that when something goes wrong as it is used unattended, there is no-one held to account for it to transparently explain its own mistakes and errors. It is just nonsense to suggest that such systems are capable of ‘reasoning’ when it pretends to do so and repeats itself without understanding their own errors. Thus, LLMs and other black-box AIs cannot be trusted for those high risk situations over a consensus of human professionals.

By "Humans", I assume you mean something like "adult humans, well-educated in the relevant fields". Otherwise, most of these responses look like they would easily beat most humans.

DylanDmitri 3y ago
I think most high-school educated adults, with the ability to make a couple web searches, would do fine on all these questions. It would take the humans minutes instead of seconds because they don't have the internet memorized. Me, Kubernetes Haikus, time taken 84 seconds: ---------- Kubernetes rules With its smooth orchestration You can reach web scale ---------- Kubernetes sucks Lost in endless YAML hell Why is it broken?
- seabass-labrax 3y ago
  I think you're spot on here. Yes, if one's trying to compare human and GPT intelligence, then you have to define what counts as memorisation and what counts as reasoning. But what most people outside of academia are trying to do is work out whether a GPT can effectively replace a human in some time-consuming task, and to be able to do so without access to the internet is rarely an important factor.

sirk390 3y ago

Humans are pretty bad at these questions. Even with the simplest questions like "Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have?" I think that a lot of people will give an incorrect answer. And for questions like "Argue for and against the use of kubernetes in the style of a haiku", 99.99% will not be able to do it.

earthboundkid 3y ago
The thing with humans is they will say “I don’t remember how many syllables a haiku has” and “what the hell is kubernetes?” No LLM can reliably produce a haiku because their lexing process deprives them of reliable information about syllable counts. They should all say “I’m sorry, I can’t count syllables, but I’ll try my best anyway.” But the current models don’t do that because they were trained on texts by humans, who can do haiku, and not properly taught their own limits by reinforcement learning. It’s Dunning Kruger gone berserk.
- pixl97 3y ago
  Eh, it's not D&K gone berserk, it's what happens when you attempt to compress reality down to a single dimension (text). If you're doing a haiku, you will likely subvocalize it to ensure you're saying it correctly. It will be interesting when we get multimodal AI that can speak and listen to itself to detect things like this.
  - earthboundkid 3y ago
    The problem isn’t just that everything is text. It’s that everything is a Fourier transform of text in such a way that it’s not actually possible for an LLM to learn to count syllables.
    - pixl97 3y ago
      Again, that is just using text only. Imagine you have a lot more computing resources in a multimodal LLM. It sees your request of count the syllables and realizes it can't do them from text alone (hell I can't and have to vocalize it). It then sends your request to a audio module and 'says' the sentence, then another listening module that understand syllables 'hears' the sentence. This is how it works in most humans, now if you do this every day you'll likely make some kind of mental shortcut to reduce the effort needed, but at the end of the day there is no unsolvable problem on the AI side.
deleted 3y ago
[deleted]

rvz 3y ago

> After all, LLMs are designed to reason equal to or better than humans. No. I doubt you would fully trust a LLM to replace high risk jobs such as lawyers, doctors or pilots such that when something goes wrong as it is used unattended, there is no-one held to account for it to transparently explain its own mistakes and errors. It is just nonsense to suggest that such systems are capable of ‘reasoning’ when it pretends to do so and repeats itself without understanding their own errors. Thus, LLMs and other black-box AIs cannot be trusted for those high risk situations over a consensus of human professionals.