4 ms·
Fine-tuning the model to force it to understand the insane tokenization its forced to use for its vocabulary and perform accurate addition sounds great until yo
by Der_Einzige 3y ago
Fine-tuning the model to force it to understand the insane tokenization its forced to use for its vocabulary and perform accurate addition sounds great until you start trying very large numbers or messing with whatever decoding settings you're using (and let me guess, it was beam search).
Or, you can simply ask it to use a tool, like a calculator for you. This is more reliable than fine-tuning a current technique BPE tokenized model ever will be.
- GaggiX 3y agoI trained my models from scratch, the tokenization is extremely simple and designed just for this task, decoder-only models can learn to do sums of arbitrarily large numbers if the model is large enough (they are computationally bounded). The decoding method used was greedy or simple sampling.