You can't blame this on the math or transformers though. This is the thing about statistic, - the datasets have a tendency inherited from the human perception a

You can't blame this on the math or transformers though. This is the thing about statistic, - the datasets have a tendency inherited from the human perception and biases. You can actually spend some time and money to analyze the LION 5B dataset to find out how homogeneous it is. People create photos and tag them, however small the bias in initial data is, it will be exposed by the NN, because it is more probable.