3 ms·
The limitation on sequence length in these architectures is important. For example if you ask someone about a book they just read they can probably give you a s
by iandanforth 6y ago
The limitation on sequence length in these architectures is important. For example if you ask someone about a book they just read they can probably give you a summary fairly easily. This is beyond Xformer models today. Note this is distinct from training on a book or series of books which networks handle fairly well.
More practically lets say I want to do an English language -> SQL translation task. If I feed a schema to GPT-3 along with the English language this actually works amazingly well. Unfortunately with only a couple thousand tokens to work with as input budget my schemas have to be short. I can't feed in a ton of schemas and meta data and have it work out complicated joins.
- petulla 6y agoThis is sort of right. Reformer has taken books as a single input and not just in split batches. There are pretrained book-trained models on HuggingFace now. I'm not aware of book-length summarization models yet, but that's not due to the input length issues of the previous generation of xformer models (maybe due to lack of an obvious training set?). So your SQL example should be a thing of the past before long.
- riku_iki 6y agoI think they didn't release any benchmark results for reformer, so yes, it can take whole book as input, but quality is unknown.