3 ms·
How do you determine “ right around the time the model starts to go off the rails”?
by aantix 3y ago
How do you determine “ right around the time the model starts to go off the rails”?
- valine 3y agoAs part of the training loop I will periodically ask the model a question. At first the model will respond normally, and then get progressively more repetitive until it starts repeating tokens from the training data. The point in time where the model starts repeating itself aligns with the sudden change in the visualization.