Your first suggestion seems like the way to go, I read a couple of papers this morning which seemed to confirm that it is a solid strategy. The first one claims

Your first suggestion seems like the way to go, I read a couple of papers this morning which seemed to confirm that it is a solid strategy. The first one claims that the Longformer architecture is tenable for truly long documents with some minor parameter optimization [0]. The Amazon/MSFT paper claims that just looking at the first part of the doc is superior to most other more complex solutions, like Longformer/Big Bird or hierarchical classification [1]. But here's what's really cool, they concluded that if you still use only 512 tokens, but you don't just look at the first ones, you look at basically an extractive summary of the most salient phrases in the document, it's more accurate. They call it BERT-TextRank, and it's using an extractive summarization algorithm from 2004. Found a tutorial which at least makes classification with RoBERTa pretty straightforward. I honestly didn't think this was possible before today but it's always nice to have your preconceptions challenged. [0] Revisiting Transformer-based Models for Long Document Classification (anonymous ACL submission, published yesterday) https://arxiv.org/abs/2204.06683 https://arxiv.org/abs/2204.06683 [1] Efficient classification of long documents using transformers (Amazon, MSFT, March 12, 2022) https://www.amazon.science/publications/efficient-classification-of-long-documents-using-transformers https://www.amazon.science/publications/efficient-classifica...