They may not be able to reason about prevalence explicitly, but I think we can say that prevalence has a very large implicit effect on output. You'll be hard p

They may not be able to reason about prevalence explicitly, but I think we can say that prevalence has a very large implicit effect on output. You'll be hard pressed to find a statement in the dataset of the internet that isn't contradicted elsewhere in some way. This includes basic facts like living on a spherical planet. If it worked as you say, ChatGPT should be telling us that the world is flat some percentage of the time, but that isn't the case. It "knows" that one claim is true and the other false. Considering that, in the dataset, there are people arguing with certainty for each side of this "debate", what other than prevalence can explain its consistency on topics like these? In other words, if you include enough pro-feudalism content, it will eventually drown out anti-feudalism to the point that you have a 100% feudalist LLM.