4 ms·

As someone working in the field I have a few thoughts: 1. The way I think about these huge models is no longer as something that makes predictions, but rather

As someone working in the field I have a few thoughts: 1. The way I think about these huge models is no longer as something that makes predictions, but rather as a kind of huge knowledge base (compressed training data as representations) with smart query capabilities, which come from the fact that the model is constrained to create syntactically correct output. This could be the next generation of search. There is a fine line between search and creating novel outputs, since most novelty is just a combination of old things with constraints on correct syntax. 2. The shown results are likely a result of massive cherry-picking in the typical "pitch deck demo" fashion. You'll see a lot more failures when trying to use this and you need to get the query "just right" for it to produce good output, which a skill in itself. Also let's not forget that the model must be fed fine-tuning examples, which also need to be "just right.". That's why I believe these kinds of models aren't that useful for fully automating things - for something to be used in production it should be 100% correct - but they are very useful as a search function and alternative to e.g. StackOverflow in this case. Querying the model will give you good enough results that you can use as a starting point for your own use case.

dang 6y ago
That's interesting. Are there examples of these techniques being used for searchy things?
- gas9S9zw3P9c 6y ago
  Well, I think it comes down to how you define search. People are amazed and how good GPT-3 is at generating text, but looking at it from a search lens, you could argue that rather than "generating" it searches for the most relevant pieces to the input, combines them, and then enforces correct grammatical rules on the output. A recent example I saw on Twitter was someone asking GPT-3 questions about human evolution, and the model answers with a bunch of book references and links - that's something you would have done in Google before. The fact that the Transformer attention mechanisms works based on matching queries and keys/values is probably why the model is so good at finding relevant information to your input - that's kind of what it was trained to do.
bvm 6y ago
Some relatively unstructured thoughts: I'm not sure I agree with 2 for this use case: writing the GDP table is a pain because table markup is (necessarily) verbose, and it's fiddly to copy and paste all of those fields individually. If I can get 95% of the way there with a couple of syntax errors that are picked up/fixed by my linter then I'm a happy customer. That said the way he generates a table is in no way idiomatic react, where you take an array of objects and map the data on to the table. In terms of code generation, is it actually any better than using an emmet expression? `table>tr*10>td` (I've probably mangled that). What's interesting with the table example is probably the combination of "knowledge" and knowing enough about a DSL to generate a coherent output. I agree re cherry picking, though the initial fine tuning set is extraordinarily small, just 2 items. Generating a much larger dataset shouldn't actually be that tricky, though perhaps getting agreement on how to actually describe a layout in plain English might be more of a challenge. Would a plethora of descriptions for the same layout be a bad thing? Lastly, I've been using TabNine (https://www.tabnine.com/ https://www.tabnine.com/) for the past few months, and I'm convinced that some kind of language model text generation is going to change how we write code, it's already changed it for me. If I signal my intent pretty clearly up front, I would say most of my coding is now tabbing and occasionally pressing the down arrow. Even writing comments becomes an exercise writing an initial three of four words and tabbing out the rest. Ditto tests.
ChrisRR 6y ago
Your first point is actually covered in the computerphile video on GPT-3. They discuss a paper that studies the ability of GPT-3 to do arithmetic as it's part of the learned text database Check a look https://www.youtube.com/watch?v=_8yVOC4ciXc https://www.youtube.com/watch?v=_8yVOC4ciXc

dang 6y ago

That's interesting. Are there examples of these techniques being used for searchy things?

gas9S9zw3P9c 6y ago
Well, I think it comes down to how you define search. People are amazed and how good GPT-3 is at generating text, but looking at it from a search lens, you could argue that rather than "generating" it searches for the most relevant pieces to the input, combines them, and then enforces correct grammatical rules on the output. A recent example I saw on Twitter was someone asking GPT-3 questions about human evolution, and the model answers with a bunch of book references and links - that's something you would have done in Google before. The fact that the Transformer attention mechanisms works based on matching queries and keys/values is probably why the model is so good at finding relevant information to your input - that's kind of what it was trained to do.

bvm 6y ago

Some relatively unstructured thoughts: I'm not sure I agree with 2 for this use case: writing the GDP table is a pain because table markup is (necessarily) verbose, and it's fiddly to copy and paste all of those fields individually. If I can get 95% of the way there with a couple of syntax errors that are picked up/fixed by my linter then I'm a happy customer. That said the way he generates a table is in no way idiomatic react, where you take an array of objects and map the data on to the table. In terms of code generation, is it actually any better than using an emmet expression? `table>tr*10>td` (I've probably mangled that). What's interesting with the table example is probably the combination of "knowledge" and knowing enough about a DSL to generate a coherent output. I agree re cherry picking, though the initial fine tuning set is extraordinarily small, just 2 items. Generating a much larger dataset shouldn't actually be that tricky, though perhaps getting agreement on how to actually describe a layout in plain English might be more of a challenge. Would a plethora of descriptions for the same layout be a bad thing? Lastly, I've been using TabNine (https://www.tabnine.com/ https://www.tabnine.com/) for the past few months, and I'm convinced that some kind of language model text generation is going to change how we write code, it's already changed it for me. If I signal my intent pretty clearly up front, I would say most of my coding is now tabbing and occasionally pressing the down arrow. Even writing comments becomes an exercise writing an initial three of four words and tabbing out the rest. Ditto tests.

ChrisRR 6y ago

Your first point is actually covered in the computerphile video on GPT-3. They discuss a paper that studies the ability of GPT-3 to do arithmetic as it's part of the learned text database Check a look https://www.youtube.com/watch?v=_8yVOC4ciXc https://www.youtube.com/watch?v=_8yVOC4ciXc