3 ms·

For the second part, I’d also train a fastText classifier. It’s cheap to do and can often get very usable results at 1/500th the computing power of a BERT/SBERT

by leobg 5y ago

For the second part, I’d also train a fastText classifier. It’s cheap to do and can often get very usable results at 1/500th the computing power of a BERT/SBERT model. Re: the length problem, some ideas: a) Just use the first part of each doc. For most humans, in most cases, that would be enough to do the classification. b) Mix head and tail of each doc (i.e. cut out as much of the middle as necessary). c) Split each doc into as many parts as needed, classify all of them, and then go with the majority vote. Good luck with the interview!

bpiche 5y ago
Your first suggestion seems like the way to go, I read a couple of papers this morning which seemed to confirm that it is a solid strategy. The first one claims that the Longformer architecture is tenable for truly long documents with some minor parameter optimization [0]. The Amazon/MSFT paper claims that just looking at the first part of the doc is superior to most other more complex solutions, like Longformer/Big Bird or hierarchical classification [1]. But here's what's really cool, they concluded that if you still use only 512 tokens, but you don't just look at the first ones, you look at basically an extractive summary of the most salient phrases in the document, it's more accurate. They call it BERT-TextRank, and it's using an extractive summarization algorithm from 2004. Found a tutorial which at least makes classification with RoBERTa pretty straightforward. I honestly didn't think this was possible before today but it's always nice to have your preconceptions challenged. [0] Revisiting Transformer-based Models for Long Document Classification (anonymous ACL submission, published yesterday) https://arxiv.org/abs/2204.06683 https://arxiv.org/abs/2204.06683 [1] Efficient classification of long documents using transformers (Amazon, MSFT, March 12, 2022) https://www.amazon.science/publications/efficient-classification-of-long-documents-using-transformers https://www.amazon.science/publications/efficient-classifica...

by leobg 5y ago

bpiche 5y ago

Your first suggestion seems like the way to go, I read a couple of papers this morning which seemed to confirm that it is a solid strategy. The first one claims that the Longformer architecture is tenable for truly long documents with some minor parameter optimization [0]. The Amazon/MSFT paper claims that just looking at the first part of the doc is superior to most other more complex solutions, like Longformer/Big Bird or hierarchical classification [1]. But here's what's really cool, they concluded that if you still use only 512 tokens, but you don't just look at the first ones, you look at basically an extractive summary of the most salient phrases in the document, it's more accurate. They call it BERT-TextRank, and it's using an extractive summarization algorithm from 2004. Found a tutorial which at least makes classification with RoBERTa pretty straightforward. I honestly didn't think this was possible before today but it's always nice to have your preconceptions challenged. [0] Revisiting Transformer-based Models for Long Document Classification (anonymous ACL submission, published yesterday) https://arxiv.org/abs/2204.06683 https://arxiv.org/abs/2204.06683 [1] Efficient classification of long documents using transformers (Amazon, MSFT, March 12, 2022) https://www.amazon.science/publications/efficient-classification-of-long-documents-using-transformers https://www.amazon.science/publications/efficient-classifica...