Chat GPT 4 has ongoing training, such as using Reinforcement Learning from Human Feedback (RLHF) to tune it to provide "better" responses, "safer" answers, and

Chat GPT 4 has ongoing training, such as using Reinforcement Learning from Human Feedback (RLHF) to tune it to provide "better" responses, "safer" answers, and to generally obey the system prompts. There's a release every few weeks. Yes, I've noticed too that recently it has become very "cagey", qualifying everything to death with "As an AI model...". A paper[1] that took snapshots monthly mentioned that as the initial bulk self-supervised learning went on, the model became smarter, as expected. However, once the "clicker training" was imposed on top to make it behave, its peak capabilities were reduced. I'm not sure if it's in the paper or the associated video, but the author mentioned that the original unrestricted model would provide probability estimates using percentages, and it was a very accurate predictor. The later versions that were adjusted based on human expectations used wishy-washy words like "likely" or "unlikely", and its estimation accuracy dropped significantly.[3] At Build 2023, Andrej Karpathy outlined[2] how they trained GPT 4. Essentially, the raw model during training builds its own intelligence. Then there's three stages of "tuning" to make it behave, and all three are based on human input. Essentially, they had contractors provide samples of "ideal" output. Similarly, end-users could up-vote or down-vote responses, which also got fed in. My personal theory is that the raw models can get about as intelligent as the average of the consistent and coherent parts of the Internet. Think about how many people are wrong, but often obviously so. Flat Earth, homeopathy, etc... If the model gains the ability to filter that stuff out, or "skip over the cracks" to distil out the general collected wisdom of the human race, then it can become more intelligent in some sense than the average human. If the training is done with thousands of $15/hr contractors, then the model will then slew back towards... the average human, or even slightly below average. There's a selection bias there. Geniuses won't be doing menial labour for that kind of money. The percentages thing was what made me realise this. When I talk to highly intelligent people, I use percentages to estimate probabilities. When I talk to typical people in a normal workplace setting, I dumb it down a bit and avoid using numbers. I've noticed that average people don't like percentages and it confuses and even angers them. The clicker training makes the GPT model appeal to average people. That's not the same as being smart. All too often, smart people upset average people. [1] "Sparks of Artificial General Intelligence: Early experiments with GPT-4" https://arxiv.org/abs/2303.12712 https://arxiv.org/abs/2303.12712 [2] "State of GPT | BRK216HFS" https://www.youtube.com/watch?v=bZQun8Y4L2A&list=LL&index=6 https://www.youtube.com/watch?v=bZQun8Y4L2A&list=LL&index=6 [3] The author also mentioned that the model was almost "evil", for the want of a better word. Capable of emulating the worst of 4chan or similar dark corners of the web's filthy underbelly. The HORRIFYING corollary here is that the temptation will always be there to use smarter-but-less-restrained models where accuracy matters. Sure, the model might be sexist, but a Bayesian estimator of sexist behaviour will only predict accurately if it too is sexist. Evil and accurate or woke and dumb. Apparently, we can choose.