5 ms·

Longer inputs consume more memory, and if your input are shorter than the token length, they have to be padded. So asking a simple one sentence question to a m

Longer inputs consume more memory, and if your input are shorter than the token length, they have to be padded. So asking a simple one sentence question to a model that has a 81,920 token limit would be a collosal waste of resources.

visarga 4y ago
Padding + masking. So the transformer doesn't waste time on the padding.