3 ms·
If you compare what BigBird is good at to what GPT-3 is good at, what are the relative strengths of each system?
by lacker 6y ago
If you compare what BigBird is good at to what GPT-3 is good at, what are the relative strengths of each system?
- phillypham 6y agoRight now, BigBird is encoder only so it doesn't generate text. Causal attention with the global memory is a bit weird, but we could probably do it. GPT-3 is only using a sequence length of 2048. In most of our paper, we use 4096, but we can go much larger 16k+. Of course, we don't nearly have as many parameters as GPT-3, so our generalization may not be as good. BigBird is just an attention mechanism and could actually be complementary to GPT-3.