For GPT-3, they use the data everyone is using. Their cleaning is unique, but so is everyone's. They don't do anything special, but put in some more effort than

For GPT-3, they use the data everyone is using. Their cleaning is unique, but so is everyone's. They don't do anything special, but put in some more effort than most though. As far as I can tell, it is a combination of smart algorithms, good engineering, and the hardware to make it happen. And scientists that had the right hunch for which direction to push in.