that was the first stab at it; but it's pretty clear that you try to get good data, especially when you see how much the open source chinese models are doing wi

that was the first stab at it; but it's pretty clear that you try to get good data, especially when you see how much the open source chinese models are doing with far less compute and parameter sizes. These are the open source datasets: https://huggingface.co/datasets/HuggingFaceFW/fineweb https://huggingface.co/datasets/HuggingFaceFW/fineweb I don't think they do that anymore, but at the beginning, yes, they threw all kind of bullshit in there to get a proper gradient.