This is something we want to explore. BigBird just replaces the attention mechanism in BERT. We believe something like BigBird can be complementary to GPT-3. G

This is something we want to explore. BigBird just replaces the attention mechanism in BERT. We believe something like BigBird can be complementary to GPT-3. GPT-3 is still limited to 2048 tokens. We'd like to think that we could generate longer, more coherent stories by using more context.