4 ms·
Yes; and the amount of training examples that are really long (relative to the other training examples) becomes small really fast, so it's also a problem of the
by ofermend 3y ago
Yes; and the amount of training examples that are really long (relative to the other training examples) becomes small really fast, so it's also a problem of the long-tail in a sense.
- mike_hearn 3y agoIt feels like this is where training on code is going to go from important to critical. Most human texts won't require you to look back 100,000 tokens to understand what it means, but if you dump 1000 source files one after the other than it will.