4 ms·
The largest Python model (codegen-16B-mono) should be pretty competitive! The biggest thing that it lacks is the ability to make use of context after the cursor
by moyix 4y ago
The largest Python model (codegen-16B-mono) should be pretty competitive! The biggest thing that it lacks is the ability to make use of context after the cursor. Unfortunately, that capability will require retraining the models:
https://arxiv.org/abs/2207.14255 https://arxiv.org/abs/2207.14255
- lee101 4y agoInteresting, lots of models are already trained on MLM (masked language modelling), so you can iteratively add mask tokens in the middle but you need some rules around termination such as probability/length of mask tokens... I'd be interested in how to convert models without retaining, or minimal retaining, I'd have thought it would just work with iterative mask tokens at least for models trained with MLM