I read the paper "Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling" that was published last week and started building a tool for pe

I read the paper "Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling" that was published last week and started building a tool for people to collectively generate synthetic training data. The tool still needs a trust mechanism and a coherent incremental publishing strategy to be able to operate in a public fashion. Right now, running one node using my RTX 3060 it would take 1.2 years to do one split of the C4 dataset. https://arxiv.org/abs/2401.16380 https://arxiv.org/abs/2401.16380 https://www.emergentmind.com/papers/2401.16380 https://www.emergentmind.com/papers/2401.16380 https://github.com/gardner/gsd https://github.com/gardner/gsd