What may be relevant in this context is the observation that models ultimately converge on their dataset: > trained on the same dataset for long enough, pretty

What may be relevant in this context is the observation that models ultimately converge on their dataset: > trained on the same dataset for long enough, pretty much every model with enough weights and training time converges to the same point. (…) > This is a surprising observation! It implies that model behavior is not determined by architecture, hyperparameters, or optimizer choices. It’s determined by your dataset, nothing else. Everything else is a means to an end in efficiently delivery compute to approximating that dataset. > Then, when you refer to “Lambda”, “ChatGPT”, “Bard”, or “Claude” then, it’s not the model weights that you are referring to. It’s the dataset. https://nonint.com/2023/06/10/the-it-in-ai-models-is-the-dataset/ https://nonint.com/2023/06/10/the-it-in-ai-models-is-the-dat... – Is scraping good enough? Can we do without curated datasets? (This would be the most significant encyclopedic effort in human history – and we are by no means ready for this.) Edit: It may be worth remembering the Mundaneum, https://en.wikipedia.org/wiki/Mundaneum https://en.wikipedia.org/wiki/Mundaneum