3 ms·

Nah not really, there’s just a lot of hyperbole going around right now. They are RNNs, the difference is they have a layer/layers being fed back instead of a s

Nah not really, there’s just a lot of hyperbole going around right now. They are RNNs, the difference is they have a layer/layers being fed back instead of a single neuron. That’s a fancy recurrent network. They are not machines that reason, they are approximators. It’s all just token matching based on data we’ve fed it. Further it didn’t happen over night but through successive improvements and at no point was the next improvement considered some infeasible thing. Seems to me maybe you’ve bought into the hype here, and are confusing that with reality.

famouswaffles 3y ago
>They are not machines that reason, they are approximators. And meaningless distinction of the year award goes to.. "It's not real [insert property]" is not an intelligent argument. By all means, divine the way to distinguish results of the two. After all, what kind of important distinction can't be distinguished by results? >Further it didn’t happen over night but through successive improvements The only difference between GPT-3 and GPT-2 was scale. They didn't even change the tokenizer until 4. There were no "successful improvements" to smoothen the massive gap in capabilities between the two. So to say that was expected just shows more lack of knowledge here.
- airgapstopgap 3y ago
  > The only difference between GPT-3 and GPT-2 was scale. They didn't even change the tokenizer until 4. There were no "successful improvements" to smoothen the massive gap in capabilities between the two. "Other than that, how was the play, Mrs. Lincoln?" GPT-2 is two orders of magnitude smaller. In terms of forebrain neuron count this is the difference between a human and an elephant shrew or budgerigar. It was absolutely expected by reasonable people that 2 OOMs of scaling will provide a qualitative jump.
  - famouswaffles 3y ago
    >GPT-2 is two orders of magnitude smaller. In terms of forebrain neuron count this is the difference between a human and an elephant shrew or budgerigar. It was absolutely expected by reasonable people that 2 OOMs of scaling will provide a qualitative jump. When GPT-3 was released, it was by far the largest artificial neural network ever trained. And I mean by far. Now there wasn't any big jump in hardware capabilities to spur this sort of gulf. It wasn't a case of "Oh now we can train a very large model" So Want to know why there was such a gap ? It's because most researchers assumed the models would overfit the data or display diminishing returns long before 175b. Brain neurons are not comparable to ann parameters.
- Reki 3y ago
  You would expect a machine that can reason to improve in its responses as a conversation progresses, instead of hallucinating.
chpatrick 3y ago
> They are not machines that reason, they are approximators. They approximate a function that performs reasoning...

famouswaffles 3y ago

>They are not machines that reason, they are approximators. And meaningless distinction of the year award goes to.. "It's not real [insert property]" is not an intelligent argument. By all means, divine the way to distinguish results of the two. After all, what kind of important distinction can't be distinguished by results? >Further it didn’t happen over night but through successive improvements The only difference between GPT-3 and GPT-2 was scale. They didn't even change the tokenizer until 4. There were no "successful improvements" to smoothen the massive gap in capabilities between the two. So to say that was expected just shows more lack of knowledge here.

airgapstopgap 3y ago
> The only difference between GPT-3 and GPT-2 was scale. They didn't even change the tokenizer until 4. There were no "successful improvements" to smoothen the massive gap in capabilities between the two. "Other than that, how was the play, Mrs. Lincoln?" GPT-2 is two orders of magnitude smaller. In terms of forebrain neuron count this is the difference between a human and an elephant shrew or budgerigar. It was absolutely expected by reasonable people that 2 OOMs of scaling will provide a qualitative jump.
- famouswaffles 3y ago
  >GPT-2 is two orders of magnitude smaller. In terms of forebrain neuron count this is the difference between a human and an elephant shrew or budgerigar. It was absolutely expected by reasonable people that 2 OOMs of scaling will provide a qualitative jump. When GPT-3 was released, it was by far the largest artificial neural network ever trained. And I mean by far. Now there wasn't any big jump in hardware capabilities to spur this sort of gulf. It wasn't a case of "Oh now we can train a very large model" So Want to know why there was such a gap ? It's because most researchers assumed the models would overfit the data or display diminishing returns long before 175b. Brain neurons are not comparable to ann parameters.
Reki 3y ago
You would expect a machine that can reason to improve in its responses as a conversation progresses, instead of hallucinating.

chpatrick 3y ago

> They are not machines that reason, they are approximators. They approximate a function that performs reasoning...