3 ms·

Thanks a lot. If I may ask one more question, would you happen to know if the authors or other researchers who are entertaining any theoretical work on the exp

by wadkar 6y ago

Thanks a lot. If I may ask one more question, would you happen to know if the authors or other researchers who are entertaining any theoretical work on the experimental design and training methodologies of GPT/BERT? As in why does it work? What is the significance of training via the "fill-in-the-blanks" method? Don't get me wrong - the work is great and the SOTAs are amazing, I would be just happy to have a chat to discuss and bounce some ideas what all this means and why do these methods seem to be working so well. Papers/articles/blog-posts are always a pleasure to read!

6gvONxR4sf7o 6y ago
I think it's just kind of understood, so I don't have any real references for you. Filling in "A dog has ___ feet" requires actual facts. Or compare these two: "The city councilmen refused the demonstrators a permit because they advocated violence. It wasn't the first time the _____ had advocated violence." "The city councilmen refused the demonstrators a permit because they feared violence. It wasn't the first time the _____ had feared violence." The syntax is identical. The words are identical, except that I swapped "advocated" out for "feared". When I swap it, the ____ changes from "demonstrators" to "councilmen." Think about what kinds of reasoning and experience and knowledge it takes you to resolve which group "they" refers to in this sentence. Most blanks might be simpler and just correspond to learning english, like when the blank is "the," but learning that is a feat too. Filling in the blanks that require broader knowledge requires somehow capturing that broader knowledge.