3 ms·
GPT-4 before the RLHF phase of training had a pretty good idea of what it "knows". The calibration graph was almost perfect, but after the RLHF the calibration
by sebzim4500 4y ago
GPT-4 before the RLHF phase of training had a pretty good idea of what it "knows". The calibration graph was almost perfect, but after the RLHF the calibration is almost completely broken.
- jiggywiggy 4y agoHmm didn't notice any difference yet, you are saying it got worse last weeks? For kids story writing I've been getting better results with 3.5 at times. Where 4 is way better af coding.
- rideontime 4y agoWhat's with the obsession with children's stories and GPT? Is it just that children have low standards?
- Agentlien 4y agoAs a parent my guess would be that people see it as a way to introduce welcome variety and whimsy into the daily routine of reading a bedtime story. While also feeling like you're using a hobby interest to help with a real practical issue. I have a small library of children's books and we've read them all several times, the good ones many times. That said, I wouldn't personally turn to these language models. From what I've seen they tend to generate rather bland and boring stories. I would rather make up my own or reread "Kackel i grönsakslandet" for the hundredth time.
- sebzim4500 4y agoNo, we have no access to the original model, unfortunately. The fact that RLHF broke the calibration comes from the GPT-4 paper, possibly the only interesting technical detail that they include.
- brucethemoose2 4y agoPerhaps "one model to rule them all" isnt the best approach.
- sebzim4500 4y agoThere's probably a huge amount of room for improvement in the RLHF process. If there is still low hanging fruit, it would have to be there.
- brucethemoose2 4y ago"I dunno" would have to be marked as a good or neutral response in the RLHF process, and that seems like a problematic training incentive.
- sebzim4500 4y agoIn an ideal world "I don't know" would be considered worse than a correct answer but much better than a wrong answer. In the UK, there is a competition called the "junior maths challenge", or something, which is a multiple choice quiz where correct answers are +1 and incorrect answers are -6 (so guessing has negative EV). I think we need a similar scoring system here.
- esjeon 4y agoNah, RLHF is what made GPT-4 outperform 3.5. The base model hasn't been much improved since 3.5. Also, the calibration curve is based on a subset of MMLU, so it doesn't really represent any of the actual user experience.
- int_19h 4y agoSource?
- esjeon 4y agoThe OpenAI GPT-4 paper itself.
- sebzim4500 4y agoI'm not saying that RLHF does more harm than good, just that it made this particular aspect of its performance worse. Basically there is still significant room for improvement, probably without changing the architecture.