4 ms·
The main advantage of Big Bird is its linear complexity in sequence length. If it were to be trained on the same corpus as GPT-3 what would be the advanta
by visarga 6y ago
The main advantage of Big Bird is its linear complexity in sequence length. If it were to be trained on the same corpus as GPT-3 what would be the advantages/disadvantages?
I am thinking maybe longer context window, faster training and less memory use, but what about performance, will it measure up?
- phillypham 6y agoThis is something we want to explore. BigBird just replaces the attention mechanism in BERT. We believe something like BigBird can be complementary to GPT-3. GPT-3 is still limited to 2048 tokens. We'd like to think that we could generate longer, more coherent stories by using more context.