3 ms·
If the learning rate was scaled based on its own confidence output, then ideally the system should learn to have a low confidence in everything early in trainin
by josephg 4y ago
If the learning rate was scaled based on its own confidence output, then ideally the system should learn to have a low confidence in everything early in training, and only raise its confidence late in training as it gets increasingly confident in its own outputs.
- naijaboiler 4y agoLLM would have to fundamentally change. Current transformer architecture can not but hallucinate. It is inherent. More training won't solve it. A complete rethinking of the approach is the only way to