3 ms·

Perhaps your framework is not as state-of-the-art as you believe?

robbedpeter 5y ago
Why is it so difficult to conceive that what these models produce is novel, and capable of high level technical and semantic correctness? Transformer models aren't auto-complete engines. They're semantic graphs describing the probabilities of token positions within a sequence of a few thousand tokens. They go deep, with a nuanced and hierarchical representation of concepts. The representation is functional, conditionally altering the behavior of tokens within its sequence context. Because of the hierarchical nature of the internal rules, when the model "learns" things, it extracts meta-patterns from the training data, and can chain a seemingly arbitrary number of such rules together to produce an output. Auto-complete systems generally fail at the first level of abstraction. If transformers have sufficient vocabulary size and high quality data, they're capable of learning any degree of abstractions represented within the training corpus. They're not magic, or intelligent like human brains, but they're allowing programmers to model very complex systems and produce novel material. This ranges from constructed languages to the rules of programming to storytelling and poetry to protein sequencing and more. You should be in awe of gpt-3, or at the very least, far less dismissive. It represents a quantum leap in the power of software, and we're barely scratching the surface of what's possible. Recurrence and episodal memory, along with Moore's law and/or a substantial decrease in the size of the models, could be the thing that achieves human level agi that lives on consumer grade devices.
baby 5y ago
I’ll take the bait. What makes you say that?
- vermilingua 5y ago
  Because —and this serves as a reply to the sibling comment too— GPT-3 is just a relatively simple construct pumped full of data and compute. After training it is relatively static, and while output might appear novel in some cases I think it is far more likely that GPT-3 is not more imaginative than most people suspect, but that people are less imaginative than most people expect. I’m not knocking your work, it certainly may be the first (or near enough to) instance that techniques are applied in the way that they are; but in my understanding and opinion, the very fact that Copilot can output code that seems novel and groundbreaking suggests the opposite. Occam’s razor.

Why is it so difficult to conceive that what these models produce is novel, and capable of high level technical and semantic correctness? Transformer models aren't auto-complete engines. They're semantic graphs describing the probabilities of token positions within a sequence of a few thousand tokens. They go deep, with a nuanced and hierarchical representation of concepts. The representation is functional, conditionally altering the behavior of tokens within its sequence context. Because of the hierarchical nature of the internal rules, when the model "learns" things, it extracts meta-patterns from the training data, and can chain a seemingly arbitrary number of such rules together to produce an output. Auto-complete systems generally fail at the first level of abstraction. If transformers have sufficient vocabulary size and high quality data, they're capable of learning any degree of abstractions represented within the training corpus. They're not magic, or intelligent like human brains, but they're allowing programmers to model very complex systems and produce novel material. This ranges from constructed languages to the rules of programming to storytelling and poetry to protein sequencing and more. You should be in awe of gpt-3, or at the very least, far less dismissive. It represents a quantum leap in the power of software, and we're barely scratching the surface of what's possible. Recurrence and episodal memory, along with Moore's law and/or a substantial decrease in the size of the models, could be the thing that achieves human level agi that lives on consumer grade devices.

baby 5y ago

I’ll take the bait. What makes you say that?

vermilingua 5y ago
Because —and this serves as a reply to the sibling comment too— GPT-3 is just a relatively simple construct pumped full of data and compute. After training it is relatively static, and while output might appear novel in some cases I think it is far more likely that GPT-3 is not more imaginative than most people suspect, but that people are less imaginative than most people expect. I’m not knocking your work, it certainly may be the first (or near enough to) instance that techniques are applied in the way that they are; but in my understanding and opinion, the very fact that Copilot can output code that seems novel and groundbreaking suggests the opposite. Occam’s razor.